Difference between revisions of "Scalability testing"
Jump to navigation
Jump to search
| Line 211: | Line 211: | ||
|7.2 GB | |7.2 GB | ||
|4 hrs 30 mins | |4 hrs 30 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/02 | |2011/12/02 | ||
| Line 220: | Line 220: | ||
| | | | ||
| | | | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/11 | |2011/12/11 | ||
| Line 229: | Line 229: | ||
|3.5 GB | |3.5 GB | ||
| | | | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/11 | |2011/12/11 | ||
| Line 238: | Line 238: | ||
|7.2 GB | |7.2 GB | ||
| | | | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/13 | |2011/12/13 | ||
| Line 247: | Line 247: | ||
|10.3 GB | |10.3 GB | ||
|3 hrs 19 mins | |3 hrs 19 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/14 | |2011/12/14 | ||
| Line 256: | Line 256: | ||
|13.2 GB | |13.2 GB | ||
|3 hrs 16 mins | |3 hrs 16 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/15 | |2011/12/15 | ||
| Line 265: | Line 265: | ||
|15 GB | |15 GB | ||
|3 hrs 30 mins | |3 hrs 30 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2011/12/15 | |2011/12/15 | ||
| Line 274: | Line 274: | ||
|19.8 GB | |19.8 GB | ||
|4 hrs 2 mins | |4 hrs 2 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
|2012/01/04 | |2012/01/04 | ||
| Line 283: | Line 283: | ||
|24 GB | |24 GB | ||
|3 hrs 52 mins | |3 hrs 52 mins | ||
| − | | | + | |Access normalization only |
|- | |- | ||
| − | | | + | |2012/01/05 |
| − | | | + | |3/3 |
| − | | | + | |2,957 |
| − | | | + | |20.9 GB |
| − | | | + | |45 MB |
| − | | | + | |13.6 GB |
| − | | | + | |4 hrs |
| − | | | + | |Access normalization only |
|- | |- | ||
| | | | ||
Revision as of 16:34, 13 February 2012
Main Page > Development roadmap > Scalability testing
Test File Sets
Test design
Maximums to test for:
- Max number of SIPS - 10
- Max number of files in SIP - 10,000
- Max size of individual file - 30 GiB
- Max size of SIP - 100 GiB
Baseline amounts:
- number of SIPS - 1
- number of files in SIP - 10
- size of individual file - 1 MiB
- size of SIP - 100 MiB
| Test | No. of SIPs | No. of files in SIP | Max size of individual file | Max size of SIP |
|---|---|---|---|---|
| 1. Baseline Test | 1 | 10 | 1 MiB | 100 MiB |
| 2. No. of SIPs | 10 | 10 | 1 MiB | 100 MiB |
| 3. No. of files | 1 | 10,000 | 1 MiB | 100 MiB |
| 4. Max file size | 1 | 10 | 30 GiB | 100 MiB |
| 5. Max SIP size | 1 | 10 | 1 MiB | 100 GiB |
| ... |
- Other tests: combination of maximums
Test results
Transfer
| Test date | System setup | No. transfers | No. files | Total file size | Largest file size | AIP size | Total time | Comments |
|---|---|---|---|---|---|---|---|---|
Ingest
| Test date | System setup | No. SIPS | No. files | Total file size | Largest file size | Pass/fail | Total time | Comments |
|---|---|---|---|---|---|---|---|---|
CVA tests
System setup:
- Bare-metal install, 1 processor
- 2 cores
- 4GB ram 9 GB swap
- xubuntu
Note: excludes store AIP and upload DIP micro-services except where noted
| Test date | No. transfers/SIPs | No. files | Total file size | Largest file size | AIP size | Total time | Comments |
|---|---|---|---|---|---|---|---|
| 2011/11/10 | 1/1 | 1,000 | 12.1 GB | 60 MB |
| ||
| 2011/11/10 | 1/1 | 1 | 2.7 GB | 2.7 GB | Failed at prepareAIP due to max Bag size: Issue 785 | ||
| 2011/11/18 | 1/1 | 1,000 | 12.1 GB | 60 MB | 7.2 GB | 4 hrs 30 mins | Access normalization only |
| 2011/12/02 | 2/2 | 1,998 | 13 GB | 21 MB | Access normalization only | ||
| 2011/12/11 | 1/1 | 1,000 | 6.51 GB | 21 MB | 3.5 GB | Access normalization only | |
| 2011/12/11 | 2/2 | 1,996 | 13.8 GB | 27 MB | 7.2 GB | Access normalization only | |
| 2011/12/13 | 3/3 | 2,974 | 18.6 GB | 20 MB | 10.3 GB | 3 hrs 19 mins | Access normalization only |
| 2011/12/14 | 4/4 | 3,993 | 24.6 GB | 22 MB | 13.2 GB | 3 hrs 16 mins | Access normalization only |
| 2011/12/15 | 4/4 | 3,982 | 43 GB | 12 MB | 15 GB | 3 hrs 30 mins | Access normalization only |
| 2011/12/15 | 6/6 | 5,113 | 34.1 GB | 38 MB | 19.8 GB | 4 hrs 2 mins | Access normalization only |
| 2012/01/04 | 6/6 | 5,845 | 42.4 GB | 33 MB | 24 GB | 3 hrs 52 mins | Access normalization only |
| 2012/01/05 | 3/3 | 2,957 | 20.9 GB | 45 MB | 13.6 GB | 4 hrs | Access normalization only |
Old test findings
Findings when running a 2,000 and 10,000 object sip through (May 2011):
- the dashboard would lose contact with the MCP intermittently this maybe fixed with more cores in a processor
- transcoder extract packages was producing ALOT of log, causing the logs to rotate every few seconds. Is a less verbose log possible, with option for debug mode?
- Sanitize (/usr/lib/sanitizeNames.py) names hung For a very long time, MCPServer and MCPClient seemed fine however. My 2,000 object sip contained ~1,000 objects with spaces in the name, the 10,000 object sip contained ~9,000 objects with spaces in the name. On both the 2,000 object sip and the 10,000 object SIP the processing failed to complete.