Archivematica 0.9 Micro-services

Main Page > Documentation > Technical Architecture > Micro-services > Archivematica 0.9 Micro-services

This page describes the key micro-services that are undertaken during transfer and ingest.

Transfer

Micro-service	Description
Approve transfer	Once the transfer has all its digital objects and has been formatted for processing, the user selects "Transfer complete" from the Actions drop-down menu.
Verify transfer compliance	Verifies that the transfer conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename with transfer UUID	Adds a unique universal identifier to the transfer folder name.
Include default Transfer processingMCP.xml	Adds defaultTransferProcessing.xml file from /sharedDirectoryStructure/sharedMicroServiceTasksConfigs/ to the transfer directory. This xml file can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
Workflow decision - create transfer backup	The user can choose to create a complete backup of the transfer in case transfer or ingest are interrupted or fail. The transfer backup is placed in /sharedDirectoryStructure/transferBackups/ and will automatically be deleted once the AIP has been moved into storage.
Assign file UUIDs to objects	Assigns a unique universal identifier to each file in the /objects/ directory.
Assign checksums and file sizes to objects	Assigns a sha-256 checksum to each file in the /objects/ directory and calculates file sizes.
Verify metadata directory checksums	Checks any checksum files that were placed in the /metadata/ folder of the SIP prior to ingest. Note that the filenames need to be named based on their algorithm: checksum.sha1, checksum.sha256, checksum.md5.
Generate METS.xml document	Generates a basic mets file with a fileSec and structMap to record the presence of all objects in the /objects/ directory and their location in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. The mets file is automatically added to any SIP generated from the transfer.
Extract packages	Extracts objects from any zipped files or other packages.
Scan for viruses	Uses ClamAV, parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in /sharedDirectoryStructure/failed/.
Sanitize object's file and directory names	Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
Sanitize transfer name	Same as above except does it for the transfer folder name.
Characterize and extract metadata	Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Adds output to the PREMIS metadata.
Create SIP(s)	The user chooses among a number of SIP creation options. See the user manual for details.

Ingest

Micro-service	Description
Verify SIP compliance	Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename SIP directory with SIP UUID	Adds a unique universal identifier to the SIP folder name.
Include default SIP processingMCP.xml	Adds defaultSIPProcessing.xml file from /sharedDirectoryStructure/sharedMicroServiceTasksConfigs/ to the transfer directory. This xml file canbe used to configure SIP workflow options.
Remove thumbs.db files	Removes any Thumbs.db files. May be expanded to others in future releases.
Sanitize object's file and directory names	If user created new folder and/or file names during SIP creation, any prhobited characters are removed from these names and replaced with dashes.
Sanitize SIP name	Same as above except that it's done for the SIP folder name.
Check for Service directory	For digitization output workflows, checks to see if the SIP contains any service copies of master files. See the user manual for details.
Check for Access directory	For digitization output workflows, checks to see if the SIP contains any access copies of master files. See the user manual for details.
Normalize	The user can choose to create preservation and/or access copies of the ingested files based on rules in the transcoder database. These rules can be seen under the Preservation planning tab in the Archivematica dashboard.
Normalize access	Creates access copies to be added to the DIP.
Normalize preservation	Creates preservation copies to be included in the AIP along with the original files.
Approve normalization	If desired, the user can verify the quality of normalized files in /sharedDirectoryStructure/watchedDirectories/approveNormalization/.
Check for submission documentation	Checks for files in /metadata/submissionDocumentation/.
Move submission documentation to objects directory	Moves the /submissionDocumentation/ directory from /metadata/ to /objects/ for ingest processing.
Assign file UUIDs to submission documentation	Assigns a unique universal identifier to each file in the /objects/submissionDocumentation/ directory.
Assign checksums and filesizes to submission documentation	Assigns a sha-256 checksum to each file in the /objects/submissionDocumentation directory and calculates file sizes.
Extract packages in submission documentation	Extracts objects from any zipped files or other packages in the /objects/ directory.
Characterize and extract metadata on submission documentation	Identifies and validates formats and extracts object metadata for files in the /objects/submissionDocumentation directory using the File Information Tool Set (FITS). Adds output to the PREMIS metadata.
Normalize submission documentation to preservation format	Creates preservation copies of all files in the /objects/submissionDocumentation directory to be included in the AIP along with the original files.
Verify checksums generated on ingest	Verifies checksums that were generated during transfer processing to ensure that the files have not been corrupted during transfer or ingest.
Remove empty directories	Removes any empty directories from the SIP.
Generate METS.xml document	Generates a METS file with PREMIS metadata. For more information on the METS file, see METS.
Copy transfers metadata and logs	Copies all submission documentation included in the original transfer to the SIP. Copies all logs generated during transfer processing to the SIP.
Copy METS to DIP directory	Creates a copy of the METS file in the DIP directory.
Generate DIP	Moves the DIP to the DIP upload directory.
Upload DIP	The user uploads the DIP to a selected description in the access system. See the user manual for details.
Prepare AIP	Packages the SIP into an AIP using BagIt
Compress AIP	Losslessly compresses the AIP for storage using p7zip.
Store AIP	Moves the AIP to a specified directory. In the demo version of Archivematica the directory is /sharedDirectoryStructure/www/AIPsStore/. In other environments it can be a remote network mounted directory. The directory structure of the AIP store contains UUID quad directories and an index.html file listing the AIPs in storage. The index.html file is displayed in the dashboard in the Archival storage tab. Once the AIP has been stored, a copy of the AIP is extracted from storage to a local temp directory, and is validated with the various BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.

0.9

Micro-service	Description
Normalize thumbnail normalization failed - copying
Normalize Access normalization failed - copying
Normalize Normalize thumbnail
Normalize Normalize access - exclude
Rename with transfer UUID Set file permissions
Rename with transfer UUID Move to processing directory
Rename with transfer UUID Set transfer type: Maildir
Rename with transfer UUID Attempt restructure for compliance
Extract packages Extract attachments
Approve transfer Approve bagit transfer
Approve transfer Extract zipped bag transfer
Approve transfer Set file permissions
Approve transfer Move to processing directory
Approve transfer Rename with transfer UUID
Approve transfer Assign file UUIDs to objects
Approve transfer Assign checksums and file sizes to objects
Approve transfer Verify bag, and restructure for compliance
Approve transfer Include default Transfer processingMCP.xml
Approve transfer Approve DSpace transfer
Verify transfer compliance Set transfer type: DSpace
Include default Transfer processingMCP.xml Failed compliance. See output in dashboard. Transfer moved back to activeTransfers.
Include default Transfer processingMCP.xml Rename with transfer UUID
Include default Transfer processingMCP.xml Create transfer backup (sharedDirectory/transferBackups)
Verify transfer checksum Verify metadata directory checksums
Generate METS.xml Generate METS.xml document
Quarantine Designate to process as a DSpace transfer
Quarantine Designate to process as a DSpace transfer when unquarantined
Quarantine Move to quarantined
Verify transfer checksums Verify checksums in fileSec of DSpace METS files
Clean up names Identify DSpace text files
Clean up names Identify DSpace license files
Clean up names Identify DSpace mets files
Characterize and extract metadata Characterize and extract metadata on objects
Approve transfer Approve transfer
Verify transfer compliance Set file permissions
Verify transfer compliance Move to processing directory
Verify transfer compliance Set transfer type: Standard
Verify transfer compliance Remove hidden files and directories
Verify transfer compliance Attempt restructure for compliance
Verify transfer compliance Verify transfer compliance
Failed transfer compliance Designate to process as a standard transfer
Failed transfer compliance Failed compliance.
Failed transfer compliance Find branch to continue processing
Failed transfer compliance Attempt restructure for compliance?
Failed transfer compliance Move to processing directory
Failed transfer compliance Attempt restructure for compliance
Failed transfer compliance Move transfer back to activeTransfers directory.
Rename with transfer UUID Rename with transfer UUID
Include default Transfer processingMCP.xml Include default Transfer processingMCP.xml
Assign file UUIDs and checksums Designate to process as a standard transfer
Assign file UUIDs and checksums Move to workFlowDecisions-createTransferBackup directory
Assign file UUIDs and checksums Find type to process as
Assign file UUIDs and checksums Workflow decision - create transfer backup
Assign file UUIDs and checksums Create transfer backup (sharedDirectory/transferBackups)
Assign file UUIDs and checksums Move to processing directory
Assign file UUIDs and checksums Set file permissions
Assign file UUIDs and checksums Assign file UUIDs to objects
Assign file UUIDs and checksums Assign checksums and file sizes to objects
Verify transfer checksums Verify metadata directory checksums
Generate METS.xml document Generate METS.xml document
Quarantine Designate to process as a standard transfer
Quarantine Move to workFlowDecisions-quarantineSIP directory
Quarantine Find type to process as
Quarantine Workflow decision - send transfer to quarantine
Quarantine Set quarantine permissions on transfer
Quarantine Create quarantine PREMIS events
Quarantine Designate to process as a standard transfer when unquarantined
Quarantine Move to quarantine
Quarantine Find type to remove from quarantine as
Quarantine Remove from quarantine
Quarantine Set unquarantined file permissions on Transfer
Quarantine Move to processing directory
Quarantine Create unquarantine PREMIS events
Extract packages Move to processing directory
Extract packages Extract packages
Scan for viruses Scan for viruses
Clean up names Sanitize object's file and directory names
Clean up names Sanitize Transfer name
Characterize and extract metadata Characterize and extract metadata
Characterize and extract metadata Identify Files ByExtension
Characterize and extract metadata Load labels from metadata/file_labels.csv
Complete transfer Set file permissions
Complete transfer Move to completedTransfers directory
Complete transfer Index transfer contents
Create SIP from Transfer Check transfer directory for objects
Create SIP from Transfer Create SIP(s)
Create SIP from Transfer Move to processing directory
Create SIP from Transfer Create SIP from transfer objects
Create SIP from Transfer Move to completedTransfers directory
Reject transfer Move to the rejected directory
Failed transfer Move to the failed directory
Approve SIP creation Approve SIP Creation
Verify SIP compliance Set file permissions
Verify SIP compliance Move to processing directory
Verify SIP compliance Verify SIP compliance
Verify transfer compliance Verify mets_structmap.xml compliance
Failed compliance Failed compliance. See output in dashboard. SIP moved back to SIPsUnderConstruction
Rename SIP directory with SIP UUID Rename SIP directory with SIP UUID
Include default SIP processingMCP.xml Include default SIP processingMCP.xml
Remove cache files Remove cache files
Clean up names Set file permissions
Clean up names Sanitize SIP name
Normalize Check for Service directory
Normalize Check for Access directory
Normalize Grant normalization options for pre-existing DIP
Normalize Grant normalization options for no pre-existing DIP
Normalize Move to workFlowDecisions-createDip directory
Normalize Find options to normalize as
Normalize Normalize
Process submission documentation Remove files without linking information (failed normalization artifacts etc.)
Process submission documentation Normalize submission documentation to thumbnail format
Normalize Verify checksums generated on ingest
Normalize Copy transfers metadata and logs
Normalize Generate METS.xml document
Normalize Normalize service files for access
Normalize Normalize access
Normalize Create DIP directory
Normalize Normalize for preservation and access
Normalize Normalize preservation
Normalize Remove files without linking information (failed normalization artifacts etc.)
Normalize Move to approve normalization directory
Normalize Approve normalization
Normalize Move to processing directory
Normalize Set file permissions
Normalize Create thumbnails directory
Normalize Normalize thumbnails
Process submission documentation Move to processing directory
Process submission documentation Set file permissions
Process submission documentation Copy transfer submission documentation
Process submission documentation Check for submission documentation
Process submission documentation Move submission documentation into objects directory
Process submission documentation Assign file UUIDs to submission documentation
Process submission documentation Assign checksums and file sizes to submissionDocumentation
Process submission documentation Extract packages in submission documentation
Process submission documentation Sanitize file and directory names in submission documentation
Process submission documentation Scan for viruses in submission documentation
Process submission documentation Characterize and extract metadata on submission documentation
Process submission documentation Identify Files ByExtension
Process submission documentation Normalize submission documentation to preservation format
Prepare AIP Remove files without linking information (failed normalization artifacts etc.)
Prepare AIP Verify checksums generated on ingest
Prepare AIP Copy transfers metadata and logs
Prepare AIP Generate METS.xml document
Prepare DIP Copy thumbnails to DIP directory
Prepare DIP Copy METS to DIP directory
Prepare DIP Set file permissions
Prepare DIP Generate DIP
Prepare AIP Index AIP contents
Prepare AIP Prepare AIP
Prepare AIP Select compression algorithm
Prepare AIP Select compression level
Prepare AIP Compress AIP
Prepare AIP Set bag file permissions
Prepare AIP Removed bagged files
Store AIP Move to the store AIP approval directory
Store AIP Store AIP
Store AIP Store AIP location
Store AIP Move to processing directory
Store AIP Store the AIP
Store AIP Remove the processing directory
Upload DIP Select target CONTENTdm server
Upload DIP Get list of collections on server
Upload DIP Select destination collection
Upload DIP Select upload type (Project Client or direct upload)
Upload DIP Restructure DIP for CONTENTdm upload
Upload DIP Upload DIP to contentDM
Upload DIP Upload DIP
Upload DIP Move to the uploadedDIPs directory
Reject DIP Move to the rejected directory
Reject SIP Move to the rejected directory
Failed SIP Move to the failed directory
Normalize Find preservation links to run.
Normalize Find access links to run.
Normalize Find thumbnail links to run.
Process submission documentation Find thumbnail links to run.
Process submission documentation Find preservation links to run. |