Difference between revisions of "Optimization"
| Line 20: | Line 20: | ||
== Single Large File per Transfer == | == Single Large File per Transfer == | ||
| − | + | === Materials === | |
| − | |||
| − | |||
The user has large files, such as videos that are from 10GB to several hundred GB per file in size. Each Transfer is made up of one single file. | The user has large files, such as videos that are from 10GB to several hundred GB per file in size. Each Transfer is made up of one single file. | ||
| Line 28: | Line 26: | ||
Assuming separate file systems for Transfer Source Location, Processing Location and AIP Storage Location. These could be mounted by NFS, or CIFS, the main point being that there is, somewhere, separate hardware providing each file system. | Assuming separate file systems for Transfer Source Location, Processing Location and AIP Storage Location. These could be mounted by NFS, or CIFS, the main point being that there is, somewhere, separate hardware providing each file system. | ||
| + | |||
| + | The Archivematica server has 21 cpu cores, and 192GB of ram. The file systems are served from a high capacity Isilon system. | ||
=== Example Performance === | === Example Performance === | ||
| Line 36: | Line 36: | ||
AIP Storage Location is configured in Space A. | AIP Storage Location is configured in Space A. | ||
| − | + | Based on timing sample transfers, there are two main bottlenecks in this scenario: | |
| + | |||
| + | # checksumming | ||
| + | # moving/copying files | ||
Revision as of 15:51, 18 March 2015
Main Page > Development > Development documentation > Optimization
Introduction
Archivematica is a complex application, with many moving parts, that can be used in a wide variety of configurations, to process a wide variety of workflows. The topic of optimization therefore has many aspects, and many possible approaches. This page will attempt to document some of the use cases, and different techniques to improve performance.
Definition
One measure of performance in an Archivematica installation is throughput - the raw number of files, or gigabytes of files, that can be processed from initial transfer to safe storage as an AIP, per unit of time. Depending on the composition of the original materials, the configuration of Archivematica and its different components, and the hardware resources available, throughput, as measured in GB processed per hour, can vary dramatically. It is difficult to accomodate all possible permutations, so in this discussion we will start with specific use cases, and examine options for improving performance in those specific examples.
Measuring Performance
There are a number of ways to measure performance. Counting the number of files that get stored in AIP's each day is one, although there can be large delays, at times when Archivematica is waiting on user input, for example, and the system is idle. Rough counts of gb in storage per week/month are still useful, as they measure actual results.
Use Cases
Single Large File per Transfer
Materials
The user has large files, such as videos that are from 10GB to several hundred GB per file in size. Each Transfer is made up of one single file.
Hardware Configuration
Assuming separate file systems for Transfer Source Location, Processing Location and AIP Storage Location. These could be mounted by NFS, or CIFS, the main point being that there is, somewhere, separate hardware providing each file system.
The Archivematica server has 21 cpu cores, and 192GB of ram. The file systems are served from a high capacity Isilon system.
Example Performance
starting with a sample transfer consisting of a single .mov file, 110GB in size. Transfer Source location is configured in the Storage Service in Space A - corresponds to a CIFS mount. Processing Location is configured in Space B, - corresponds to an NFS mount. AIP Storage Location is configured in Space A.
Based on timing sample transfers, there are two main bottlenecks in this scenario:
- checksumming
- moving/copying files