Scalability testing

Main Page > Development roadmap > Scalability testing

Test File Sets

Test Documents

Test design

Maximums to test for:

• Max number of SIPS - 10
• Max number of files in SIP - 10,000
• Max size of individual file - 30 GiB
• Max size of SIP - 100 GiB

Baseline amounts:

• number of SIPS - 1
• number of files in SIP - 10
• size of individual file - 1 MiB
• size of SIP - 100 MiB

Test	No. of SIPs	No. of files in SIP	Max size of individual file	Max size of SIP
1. Baseline Test	1	10	1 MiB	100 MiB
2. No. of SIPs	10	10	1 MiB	100 MiB
3. No. of files	1	10,000	1 MiB	100 MiB
4. Max file size	1	10	30 GiB	100 MiB
5. Max SIP size	1	10	1 MiB	100 GiB
...

• Other tests: combination of maximums

Test results

Transfer

Test date	System setup	No. transfers	No. files	Total file size	Largest file size	Pass/fail	Total time	Comments

Ingest

Test date	System setup	No. SIPS	No. files	Total file size	Largest file size	Pass/fail	Total time	Comments

CVA tests

System setup:

• 4 systems, bare-metal install.
• 2 cores
• 4GB ram 9 GB swap
• xubuntu

Test date	No. transfers	No. SIPs	No. files	Total file size	Largest file size	Pass/fail	Total time	Comments
11/10/11	1	1	1,000	12.1 GB	60 MB	Fail		Failed at prepareAIP due to max Bag size: Issue 785 Failed at uploadDIP due to max post size limit in ica-atom (8M).
11/10/11	1	1	1	2.7 GB	2.7 GB	Fail		Failed at prepareAIP due to max Bag size: Issue 785
11/18/11	1	1	1,000	12.1 GB	60 MB	Pass	4.5 hours	Did not normalize to preservation copies because all files were uncompressed TIFF Compress AIP took 2.5 hours: AIP was compressed to 7.1 GB. Did not upload DIP
12/02/11	2	2	1,998	13 GB		Pass		Did not normalize to preservation copies because all files were uncompressed TIFF Compress AIP took hours: AIP was compressed to GB. Did not upload DIP

Old test findings

Findings when running a 2,000 and 10,000 object sip through (May 2011):

• the dashboard would lose contact with the MCP intermittently this maybe fixed with more cores in a processor
• transcoder extract packages was producing ALOT of log, causing the logs to rotate every few seconds. Is a less verbose log possible, with option for debug mode?
• Sanitize (/usr/lib/sanitizeNames.py) names hung For a very long time, MCPServer and MCPClient seemed fine however. My 2,000 object sip contained ~1,000 objects with spaces in the name, the 10,000 object sip contained ~9,000 objects with spaces in the name. On both the 2,000 object sip and the 10,000 object SIP the processing failed to complete.