Difference between revisions of "Archivematica 0.8 Micro-services"

From Archivematica
Jump to navigation Jump to search
Line 1: Line 1:
 
[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 0.8 Micro-services
 
[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 0.8 Micro-services
−
 
−
Page in progress.
 
  
 
This page describes the key micro-services that are undertaken during transfer and ingest.
 
This page describes the key micro-services that are undertaken during transfer and ingest.
Line 42: Line 40:
 
|Extract packages
 
|Extract packages
 
|Extracts objects from any zipped files or other packages.
 
|Extracts objects from any zipped files or other packages.
 +
|-
 +
|Scan for viruses
 +
|Uses [http://www.clamav.net/lang/en/ ClamAV], parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in ''/sharedDirectoryStructure/failed/''.
 
|-
 
|-
 
|Sanitize object's file and directory names
 
|Sanitize object's file and directory names
Line 48: Line 49:
 
|Sanitize transfer name
 
|Sanitize transfer name
 
|Same as above except does it for the transfer folder name.
 
|Same as above except does it for the transfer folder name.
−
|-
 
−
|Scan for viruses
 
−
|Uses [http://www.clamav.net/lang/en/ ClamAV], parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in ''/sharedDirectoryStructure/watchedDirectories/failed/''.
 
 
|-
 
|-
 
|Characterize and extract metadata
 
|Characterize and extract metadata
Line 59: Line 57:
 
|-
 
|-
 
|}
 
|}
 +
</br>
  
 
=Ingest tab=
 
=Ingest tab=
Line 68: Line 67:
 
!style="width:70%"|'''Description'''
 
!style="width:70%"|'''Description'''
 
|-
 
|-
 +
|Verify SIP compliance
 +
|Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
 +
|-
 +
|Rename SIP directory with SIP UUID
 +
|Adds a unique universal identifier to the SIP folder name.
 +
|-
 +
|Include default SIP processingMCP.xml
 +
|
 +
|-
 +
|Remove thumbs.db files
 +
|
 +
|-
 +
|Sanitize object's file and directory names
 +
|
 +
|-
 +
|Sanitize SIP name
 +
|
 +
|-
 +
|Check for Service directory
 +
|
 +
|-
 +
|Check for Access directory
 +
|
 +
|-
 +
|Normalize
 +
|
 +
|-
 +
|Create DIP directory
 +
|
 +
|-
 +
|Normalize access
 +
|
 +
|-
 +
|Normalize preservation
 +
|
 +
|-
 +
|Approve normalization
 +
|
 +
|-
 +
|Check for submission documentation
 
|
 
|
 +
|-
 +
|Move submission documentation to objects directory
 
|
 
|
 
|-
 
|-
 +
|Assign file UUIDs to submission documentation
 +
|
 +
|-
 +
|Assign checksums and filesizes to submission documentation
 +
|
 +
|-
 +
|Extract packages in submission documentation
 +
|
 +
|-
 +
|Characterize and extract metadata on submission documentation
 +
|
 +
|-
 +
|Normalize submission documentation to preservation format
 +
|
 +
|-
 +
|Verify checksums generated on ingest
 +
|
 +
|-
 +
|Remove empty directories
 +
|
 +
|-
 +
|Generate METS.xml document
 +
|
 +
|-
 +
|Copy transfers metadata and logs
 +
|
 +
|-
 +
|Copy METS to DIP directory
 +
|
 +
|-
 +
|Generate DIP
 +
|
 +
|-
 +
|Upload DIP
 +
|
 +
|-
 +
|Prepare AIP
 +
|
 +
|-
 +
|Compress AIP
 +
|
 +
|-
 +
|Store AIP
 +
|
 +
|-
 +
|}

Revision as of 18:18, 14 February 2012

Main Page > Documentation > Technical Architecture > Micro-services > Archivematica 0.8 Micro-services

This page describes the key micro-services that are undertaken during transfer and ingest.

Transfer tab

Micro-service Description
Approve transfer Once the transfer has all its digital objects and has been formatted for processing, the user selects "Transfer complete" from the Actions drop-down menu.
Verify transfer compliance Verifies that the transfer conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename with transfer UUID Adds a unique universal identifier to the transfer folder name.
Include default Transfer processingMCP.xml Adds defaultTransferProcessing.xml file from /sharedDirectoryStructure/sharedMicroServiceTasksConfigs/ to the transfer directory. This xml file can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
Workflow decision - create transfer backup The user can choose to create a complete backup of the transfer in case transfer or ingest are interrupted or fail. The transfer backup is placed in /sharedDirectoryStructure/transferBackups/ and will automatically be deleted once the AIP has been moved into storage.
Assign file UUIDs to objects Assigns a unique universal identifier to each file in the /objects/ directory.
Assign checksums and file sizes to objects Assigns a sha-256 checksum to each file in the /objects/ directory and calculates file sizes.
Verify metadata directory checksums Checks any checksum files that were placed in the /metadata/ folder of the SIP prior to ingest. Note that the filenames need to be named based on their algorithm: checksum.sha1, checksum.sha256, checksum.md5.
Generate METS.xml document Generates a basic mets file with a fileSec and structMap to record the presence of all objects in the /objects/ directory and their location in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. The mets file is automatically added to any SIP generated from the transfer.
Extract packages Extracts objects from any zipped files or other packages.
Scan for viruses Uses ClamAV, parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in /sharedDirectoryStructure/failed/.
Sanitize object's file and directory names Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
Sanitize transfer name Same as above except does it for the transfer folder name.
Characterize and extract metadata Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Adds output to the PREMIS metadata.
Create SIP(s) The user chooses among a number of SIP creation options. See the user manual for details.


Ingest tab

Micro-service Description
Verify SIP compliance Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename SIP directory with SIP UUID Adds a unique universal identifier to the SIP folder name.
Include default SIP processingMCP.xml
Remove thumbs.db files
Sanitize object's file and directory names
Sanitize SIP name
Check for Service directory
Check for Access directory
Normalize
Create DIP directory
Normalize access
Normalize preservation
Approve normalization
Check for submission documentation
Move submission documentation to objects directory
Assign file UUIDs to submission documentation
Assign checksums and filesizes to submission documentation
Extract packages in submission documentation
Characterize and extract metadata on submission documentation
Normalize submission documentation to preservation format
Verify checksums generated on ingest
Remove empty directories
Generate METS.xml document
Copy transfers metadata and logs
Copy METS to DIP directory
Generate DIP
Upload DIP
Prepare AIP
Compress AIP
Store AIP
Retrieved from "https://wiki.archivematica.org/index.php?title=Archivematica_0.8_Micro-services&oldid=5428"