Scalability testing

Main Page > Development roadmap > Scalability testing

Test File Sets

Test Documents

Test design

Maximums to test for:

• Max number of SIPS - 10
• Max number of files in SIP - 10,000
• Max size of individual file - 30 GiB
• Max size of SIP - 100 GiB

Baseline amounts:

• number of SIPS - 1
• number of files in SIP - 10
• size of individual file - 1 MiB
• size of SIP - 100 MiB

Test	No. of SIPs	No. of files in SIP	Max size of individual file	Max size of SIP
1. Baseline Test	1	10	1 MiB	100 MiB
2. No. of SIPs	10	10	1 MiB	100 MiB
3. No. of files	1	10,000	1 MiB	100 MiB
4. Max file size	1	10	30 GiB	100 MiB
5. Max SIP size	1	10	1 MiB	100 GiB
...

• Other tests: combination of maximums

Test results

Transfer

Test date	System setup	No. transfers	No. files	Total file size	Largest file size	AIP size	Total time	Comments

Ingest

Test date	System setup	No. SIPS	No. files	Total file size	Largest file size	Pass/fail	Total time	Comments

CVA tests

System setup:

• 4 systems, bare-metal install.
• 2 cores
• 4GB ram 9 GB swap
• xubuntu

Test date	No. transfers/SIPs	No. files	Total file size	Largest file size	AIP size	Total time	Comments
2011/11/10	1/1	1,000	12.1 GB	60 MB			Failed at prepareAIP due to max Bag size: Issue 785 Failed at uploadDIP due to max post size limit in ica-atom (8M).
2011/11/10	1/1	1	2.7 GB	2.7 GB			Failed at prepareAIP due to max Bag size: Issue 785
2011/11/18	1/1	1,000	12.1 GB	60 MB	7.2 GB	4.5 hours	Did not normalize to preservation copies because all files were uncompressed TIFF Compress AIP took 2.5 hours: AIP was compressed to 7.1 GB. Did not upload DIP
2011/12/02	2/2	1,998	13 GB	21 MB			Did not normalize to preservation copies because all files were uncompressed TIFF Did not upload DIP
2011/12/11	1/1	1,000	6.51 GB	21 MB	3.5 GB		Did not normalize to preservation copies because all files were uncompressed TIFF Did not upload DIP
2011/12/11	2/2	1,996	13.8 GB	27 MB	7.2 GB	Did not normalize to preservation copies because all files were uncompressed TIFF
2011/12/13	3/3	2,974	18.6 GB	20 MB	10.3 GB	3 hrs 19 mins	Did not normalize to preservation copies because all files were uncompressed TIFF

Old test findings

Findings when running a 2,000 and 10,000 object sip through (May 2011):

• the dashboard would lose contact with the MCP intermittently this maybe fixed with more cores in a processor
• transcoder extract packages was producing ALOT of log, causing the logs to rotate every few seconds. Is a less verbose log possible, with option for debug mode?
• Sanitize (/usr/lib/sanitizeNames.py) names hung For a very long time, MCPServer and MCPClient seemed fine however. My 2,000 object sip contained ~1,000 objects with spaces in the name, the 10,000 object sip contained ~9,000 objects with spaces in the name. On both the 2,000 object sip and the 10,000 object SIP the processing failed to complete.