Difference between revisions of "Scalability testing"

From Archivematica
Jump to navigation Jump to search
 
(64 intermediate revisions by 8 users not shown)
Line 1: Line 1:
[[Main Page]] > [[Development roadmap]] > Scalability testing
+
[[Main Page]] > Scalability testing
  
= Test File Sets =
+
= Objectives =
[http://archivematica.org/downloads/docZips/ Test Documents]
 
  
= Test design =
+
'''1. set up a dedicated testing environment'''
 +
<br>
 +
The testing environment will start with 5 virtual machines set up in a hosted environment, where hardware resources can be scaled up and down between tests.  It is expected that the test environment will be ready to use by January 15th. 
 +
[[Test Environment Documentation]]
  
Maximums to test for:
 
*Max number of SIPS - 10
 
*Max number of files in SIP - 10,000
 
*Max size of individual file - 30 GiB
 
*Max size of SIP - 100 GiB
 
  
Baseline amounts:
+
'''2. develop a new initial repeatable test suite''' 
* number of SIPS - 1
+
<br>Initial tests will focus on two main areas - file io and documenting individual micro-service performance.  Data will be collected from external monitoring tools as well as from internal instrumentation. 
* number of files in SIP - 10
 
* size of individual file - 1 MiB
 
* size of SIP - 100 MiB
 
  
{| class="wikitable"
+
External monitoring will be done with two open source packages, munin and collectd.  This will provide data at the operating system level.
|-
+
Internal instrumentation already exists within the Archivematica source code, where each step in the process has a start time and end time recorded in the local database. This instrumentation will be extended and refined during the buildout of the test suite. The data collected will be used to identify which specific micro-services, and which steps within those micro-services are taking the longest time to complete.
! Test
 
! No. of SIPs
 
! No. of files in SIP
 
! Max size of individual file
 
! Max size of SIP
 
|-
 
| 1. Baseline Test
 
| 1
 
| 10
 
| 1 MiB
 
| 100 MiB
 
|-
 
| 2. No. of SIPs
 
| '''10'''
 
| 10
 
| 1 MiB
 
| 100 MiB
 
|-
 
| 3. No. of files
 
| 1
 
| '''10,000'''
 
| 1 MiB
 
| 100 MiB
 
|-
 
| 4. Max file size
 
| 1
 
| 10
 
| '''30 GiB'''
 
| 100 MiB
 
|-
 
| 5. Max SIP size
 
| 1
 
| 10
 
| 1 MiB
 
| '''100 GiB'''
 
|-
 
| ...
 
|
 
|
 
|
 
|
 
|}
 
  
*Other tests: combination of maximums
+
'''3. document a full matrix of test parameters'''
 +
<br>Archivematica workflow can vary considerably depending on use case.  Artefactual will document all testing efforts on this wiki, building out a matrix of test cases.  For example, we expect that adding additional storage subsystem capacity will allow for linear growth in scalability (add more disks, it should all go faster).  This will be one of the first 'columns' in our test matrix, repeating tests with the same workload, changing the capacity (maximum io's per second) of the storage subsystem between tests. 
  
= Test results =
+
Initial tests will focus on the 4 primary stages in the Archivematica workflow - Transfer, Ingest, creation of SIP, creation of AIP.  There are additional steps required, both before Transfer, and after creation of AIP, however these steps do not necessarily involve the use of Archivematica code.  For example, moving digital objects to a shared folder that Archivematica can access is a prerequisite of the Transfer stage, and can take a considerable amount of time.  We will document best practices for how to complete that work after initial scalability testing is complete.
  
== Transfer ==
+
'''4. repeat test suite at customer sites'''
 +
<br>The two initial customer sites have been identified by Archivematica and tests will be repeated at both customer sites.
  
{| border="1" cellpadding="10" cellspacing="0" width=90%
+
= Test Structure =
|-
 
! Test date
 
!System setup
 
!No. transfers
 
!No. files
 
!Total file size
 
!Largest file size
 
!AIP size
 
!Total time
 
!Comments
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|}
 
</br>
 
  
== Ingest ==
+
Scalability testing is done using a scripted workload, where all decision points, that are normally left to the Archivist to make using the Dashboard, are instead automated through the use of a configuration file.  This allows for repeatable test cases.  Example test scripts will be posted here over the coming weeks.
  
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
|-
 
! Test date
 
!System setup
 
!No. SIPS
 
!No. files
 
!Total file size
 
!Largest file size
 
!Pass/fail
 
!Total time
 
!Comments
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|}
 
</br>
 
  
= CVA tests =
+
= Test File Sets =
 
+
[http://archivematica.org/downloads/docZips/ Test Documents]
System setup:
 
 
 
*Bare-metal install, 1 processor
 
*2 cores
 
*4GB ram 9 GB swap
 
*xubuntu
 
 
 
Note: excludes store AIP and upload DIP micro-services except where noted
 
 
 
 
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
|-
 
!Test date
 
!No. transfers/SIPs
 
!No. files
 
!Total file size
 
!Largest file size
 
!AIP size
 
!Total time
 
!Comments
 
|-
 
|2011/11/10
 
|1/1
 
|1,000
 
|12.1 GB
 
|60 MB
 
|
 
|
 
|
 
*Failed at prepareAIP due to max Bag size: <strike>Issue 785</strike>
 
*Failed at uploadDIP due to max post size limit in ica-atom (8M).
 
|-
 
|2011/11/10
 
|1/1
 
|1
 
|2.7 GB
 
|2.7 GB
 
|
 
|
 
|Failed at prepareAIP due to max Bag size: <strike>Issue 785</strike>
 
|-
 
|2011/11/18
 
|1/1
 
|1,000
 
|12.1 GB
 
|60 MB
 
|7.2 GB
 
|4 hrs 30 mins
 
|Access normalization only
 
|-
 
|2011/12/02
 
|2/2
 
|1,998
 
|13 GB
 
|21 MB
 
|
 
|
 
|Access normalization only
 
|-
 
|2011/12/11
 
|1/1
 
|1,000
 
|6.51 GB
 
|21 MB
 
|3.5 GB
 
|
 
|Access normalization only
 
|-
 
|2011/12/11
 
|2/2
 
|1,996
 
|13.8 GB
 
|27 MB
 
|7.2 GB
 
|
 
|Access normalization only
 
|-
 
|2011/12/13
 
|3/3
 
|2,974
 
|18.6 GB
 
|20 MB
 
|10.3 GB
 
|3 hrs 19 mins
 
|Access normalization only
 
|-
 
|2011/12/14
 
|4/4
 
|3,993
 
|24.6 GB
 
|22 MB
 
|13.2 GB
 
|3 hrs 16 mins
 
|Access normalization only
 
|-
 
|2011/12/15
 
|4/4
 
|3,982
 
|43 GB
 
|12 MB
 
|15 GB
 
|3 hrs 30 mins
 
|Access normalization only
 
|-
 
|2011/12/15
 
|6/6
 
|5,113
 
|34.1 GB
 
|38 MB
 
|19.8 GB
 
|4 hrs 2 mins
 
|Access normalization only
 
|-
 
|2012/01/04
 
|6/6
 
|5,845
 
|42.4 GB
 
|33 MB
 
|24 GB
 
|3 hrs 52 mins
 
|Access normalization only
 
|-
 
|2012/01/05
 
|3/3
 
|2,957
 
|20.9 GB
 
|45 MB
 
|13.6 GB
 
|4 hrs
 
|Access normalization only
 
|-
 
|2012/01/05
 
|6/6
 
|'''5,947'''
 
|33 GB
 
|52 MB
 
|19.2 GB
 
|4 hrs 47 mins
 
|Access normalization only
 
|-
 
|2012/01/12
 
|6/6
 
|4,847
 
|38.5 GB
 
|58 MB
 
|23.2 GB
 
|4 hrs 43 mins
 
|Access normalization only
 
|-
 
|2012/01/13
 
|6/6
 
|5,912
 
|'''101.6 GB'''
 
|175 MB
 
|'''63.8 GB'''
 
|8 hrs 53 mins
 
|Access normalization only
 
|-
 
|2012/01/17
 
|1/1
 
|1
 
|1.4 GB
 
|1.4 GB
 
|0.6 GB
 
|25 mins
 
|Access normalization only
 
|-
 
|2012/01/17
 
|5/5
 
|23
 
|19.7 GB
 
|2.1 GB
 
|19 GB
 
|4 hrs 1 min
 
|Access normalization only
 
|-
 
|2012/01/18
 
|2/2
 
|2
 
|3.8 GB
 
|2.1 GB
 
|3.7 GB
 
|1 hr 11 mins
 
|Access normalization only
 
|-
 
|2012/01/20
 
|6/6
 
|14
 
|6.1 GB
 
|1.3 GB
 
|5.9 GB
 
|48 mins
 
|Access normalization only
 
|-
 
|2012/02/07
 
|5/5
 
|5
 
|56.7 GB
 
|'''25.4 GB'''
 
|55.5 GB
 
|4 hrs 51 mins
 
|No normalization
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|
 
|-
 
|}
 
</br>
 
 
 
= Old test findings =
 
  
Findings when running a 2,000 and 10,000 object sip through (May 2011):
+
= Archived Scalability Test Results =
* the dashboard would lose contact with the MCP intermittently this maybe fixed with more cores in a processor
+
Historical test results are available [[Archived Scalability Test Results]]
* transcoder extract packages was producing ALOT of log, causing the logs to rotate every few seconds.  Is a less verbose log possible, with option for debug mode?
 
* Sanitize (/usr/lib/sanitizeNames.py) names hung For a very long time, MCPServer and MCPClient seemed fine however. My 2,000 object sip contained ~1,000 objects with spaces in the name, the 10,000 object sip contained ~9,000 objects with spaces in the name.  On both the 2,000 object sip and the 10,000 object SIP the processing failed to complete.
 

Latest revision as of 19:08, 16 October 2013

Main Page > Scalability testing

Objectives[edit]

1. set up a dedicated testing environment
The testing environment will start with 5 virtual machines set up in a hosted environment, where hardware resources can be scaled up and down between tests. It is expected that the test environment will be ready to use by January 15th. Test Environment Documentation


2. develop a new initial repeatable test suite
Initial tests will focus on two main areas - file io and documenting individual micro-service performance. Data will be collected from external monitoring tools as well as from internal instrumentation.

External monitoring will be done with two open source packages, munin and collectd. This will provide data at the operating system level. Internal instrumentation already exists within the Archivematica source code, where each step in the process has a start time and end time recorded in the local database. This instrumentation will be extended and refined during the buildout of the test suite. The data collected will be used to identify which specific micro-services, and which steps within those micro-services are taking the longest time to complete.

3. document a full matrix of test parameters
Archivematica workflow can vary considerably depending on use case. Artefactual will document all testing efforts on this wiki, building out a matrix of test cases. For example, we expect that adding additional storage subsystem capacity will allow for linear growth in scalability (add more disks, it should all go faster). This will be one of the first 'columns' in our test matrix, repeating tests with the same workload, changing the capacity (maximum io's per second) of the storage subsystem between tests.

Initial tests will focus on the 4 primary stages in the Archivematica workflow - Transfer, Ingest, creation of SIP, creation of AIP. There are additional steps required, both before Transfer, and after creation of AIP, however these steps do not necessarily involve the use of Archivematica code. For example, moving digital objects to a shared folder that Archivematica can access is a prerequisite of the Transfer stage, and can take a considerable amount of time. We will document best practices for how to complete that work after initial scalability testing is complete.

4. repeat test suite at customer sites
The two initial customer sites have been identified by Archivematica and tests will be repeated at both customer sites.

Test Structure[edit]

Scalability testing is done using a scripted workload, where all decision points, that are normally left to the Archivist to make using the Dashboard, are instead automated through the use of a configuration file. This allows for repeatable test cases. Example test scripts will be posted here over the coming weeks.


Test File Sets[edit]

Test Documents

Archived Scalability Test Results[edit]

Historical test results are available Archived Scalability Test Results