Archivematica 0.9 Micro-services

From Archivematica
Revision as of 16:42, 22 August 2012 by Joseph (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Main Page > Documentation > Technical Architecture > Micro-services > Archivematica 0.9 Micro-services

This page describes the key micro-services that are undertaken during transfer and ingest.

Transfer

Micro-service Description
Approve transfer Once the transfer has all its digital objects and has been formatted for processing, the user selects "Transfer complete" from the Actions drop-down menu.
Verify transfer compliance Verifies that the transfer conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename with transfer UUID Adds a unique universal identifier to the transfer folder name.
Include default Transfer processingMCP.xml Adds defaultTransferProcessing.xml file from /sharedDirectoryStructure/sharedMicroServiceTasksConfigs/ to the transfer directory. This xml file can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
Workflow decision - create transfer backup The user can choose to create a complete backup of the transfer in case transfer or ingest are interrupted or fail. The transfer backup is placed in /sharedDirectoryStructure/transferBackups/ and will automatically be deleted once the AIP has been moved into storage.
Assign file UUIDs to objects Assigns a unique universal identifier to each file in the /objects/ directory.
Assign checksums and file sizes to objects Assigns a sha-256 checksum to each file in the /objects/ directory and calculates file sizes.
Verify metadata directory checksums Checks any checksum files that were placed in the /metadata/ folder of the SIP prior to ingest. Note that the filenames need to be named based on their algorithm: checksum.sha1, checksum.sha256, checksum.md5.
Generate METS.xml document Generates a basic mets file with a fileSec and structMap to record the presence of all objects in the /objects/ directory and their location in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. The mets file is automatically added to any SIP generated from the transfer.
Extract packages Extracts objects from any zipped files or other packages.
Scan for viruses Uses ClamAV, parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in /sharedDirectoryStructure/failed/.
Sanitize object's file and directory names Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
Sanitize transfer name Same as above except does it for the transfer folder name.
Characterize and extract metadata Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Adds output to the PREMIS metadata.
Create SIP(s) The user chooses among a number of SIP creation options. See the user manual for details.


Ingest

Micro-service Description
Verify SIP compliance Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename SIP directory with SIP UUID Adds a unique universal identifier to the SIP folder name.
Include default SIP processingMCP.xml Adds defaultSIPProcessing.xml file from /sharedDirectoryStructure/sharedMicroServiceTasksConfigs/ to the transfer directory. This xml file canbe used to configure SIP workflow options.
Remove thumbs.db files Removes any Thumbs.db files. May be expanded to others in future releases.
Sanitize object's file and directory names If user created new folder and/or file names during SIP creation, any prhobited characters are removed from these names and replaced with dashes.
Sanitize SIP name Same as above except that it's done for the SIP folder name.
Check for Service directory For digitization output workflows, checks to see if the SIP contains any service copies of master files. See the user manual for details.
Check for Access directory For digitization output workflows, checks to see if the SIP contains any access copies of master files. See the user manual for details.
Normalize The user can choose to create preservation and/or access copies of the ingested files based on rules in the transcoder database. These rules can be seen under the Preservation planning tab in the Archivematica dashboard.
Normalize access Creates access copies to be added to the DIP.
Normalize preservation Creates preservation copies to be included in the AIP along with the original files.
Approve normalization If desired, the user can verify the quality of normalized files in /sharedDirectoryStructure/watchedDirectories/approveNormalization/.
Check for submission documentation Checks for files in /metadata/submissionDocumentation/.
Move submission documentation to objects directory Moves the /submissionDocumentation/ directory from /metadata/ to /objects/ for ingest processing.
Assign file UUIDs to submission documentation Assigns a unique universal identifier to each file in the /objects/submissionDocumentation/ directory.
Assign checksums and filesizes to submission documentation Assigns a sha-256 checksum to each file in the /objects/submissionDocumentation directory and calculates file sizes.
Extract packages in submission documentation Extracts objects from any zipped files or other packages in the /objects/ directory.
Characterize and extract metadata on submission documentation Identifies and validates formats and extracts object metadata for files in the /objects/submissionDocumentation directory using the File Information Tool Set (FITS). Adds output to the PREMIS metadata.
Normalize submission documentation to preservation format Creates preservation copies of all files in the /objects/submissionDocumentation directory to be included in the AIP along with the original files.
Verify checksums generated on ingest Verifies checksums that were generated during transfer processing to ensure that the files have not been corrupted during transfer or ingest.
Remove empty directories Removes any empty directories from the SIP.
Generate METS.xml document Generates a METS file with PREMIS metadata. For more information on the METS file, see METS.
Copy transfers metadata and logs Copies all submission documentation included in the original transfer to the SIP. Copies all logs generated during transfer processing to the SIP.
Copy METS to DIP directory Creates a copy of the METS file in the DIP directory.
Generate DIP Moves the DIP to the DIP upload directory.
Upload DIP The user uploads the DIP to a selected description in the access system. See the user manual for details.
Prepare AIP Packages the SIP into an AIP using BagIt
Compress AIP Losslessly compresses the AIP for storage using p7zip.
Store AIP Moves the AIP to a specified directory. In the demo version of Archivematica the directory is /sharedDirectoryStructure/www/AIPsStore/. In other environments it can be a remote network mounted directory. The directory structure of the AIP store contains UUID quad directories and an index.html file listing the AIPs in storage. The index.html file is displayed in the dashboard in the Archival storage tab.

Once the AIP has been stored, a copy of the AIP is extracted from storage to a local temp directory, and is validated with the various BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.

0.9

Micro-service Description
Normalize

thumbnail normalization failed - copying

Normalize

Access normalization failed - copying

Normalize

Normalize thumbnail

Normalize

Normalize access - exclude

Rename with transfer UUID

Set file permissions

Rename with transfer UUID

Move to processing directory

Rename with transfer UUID

Set transfer type: Maildir

Rename with transfer UUID

Attempt restructure for compliance

Extract packages

Extract attachments

Approve transfer

Approve bagit transfer

Approve transfer

Extract zipped bag transfer

Approve transfer

Set file permissions

Approve transfer

Move to processing directory

Approve transfer

Rename with transfer UUID

Approve transfer

Assign file UUIDs to objects

Approve transfer

Assign checksums and file sizes to objects

Approve transfer

Verify bag, and restructure for compliance

Approve transfer

Include default Transfer processingMCP.xml

Approve transfer

Approve DSpace transfer

Verify transfer compliance

Set transfer type: DSpace

Include default Transfer processingMCP.xml

Failed compliance. See output in dashboard. Transfer moved back to activeTransfers.

Include default Transfer processingMCP.xml

Rename with transfer UUID

Include default Transfer processingMCP.xml

Create transfer backup (sharedDirectory/transferBackups)

Verify transfer checksum

Verify metadata directory checksums

Generate METS.xml

Generate METS.xml document

Quarantine

Designate to process as a DSpace transfer

Quarantine

Designate to process as a DSpace transfer when unquarantined

Quarantine

Move to quarantined

Verify transfer checksums

Verify checksums in fileSec of DSpace METS files

Clean up names

Identify DSpace text files

Clean up names

Identify DSpace license files

Clean up names

Identify DSpace mets files

Characterize and extract metadata

Characterize and extract metadata on objects

Approve transfer

Approve transfer

Verify transfer compliance

Set file permissions

Verify transfer compliance

Move to processing directory

Verify transfer compliance

Set transfer type: Standard

Verify transfer compliance

Remove hidden files and directories

Verify transfer compliance

Attempt restructure for compliance

Verify transfer compliance

Verify transfer compliance

Failed transfer compliance

Designate to process as a standard transfer

Failed transfer compliance

Failed compliance.

Failed transfer compliance

Find branch to continue processing

Failed transfer compliance

Attempt restructure for compliance?

Failed transfer compliance

Move to processing directory

Failed transfer compliance

Attempt restructure for compliance

Failed transfer compliance

Move transfer back to activeTransfers directory.

Rename with transfer UUID

Rename with transfer UUID

Include default Transfer processingMCP.xml

Include default Transfer processingMCP.xml

Assign file UUIDs and checksums

Designate to process as a standard transfer

Assign file UUIDs and checksums

Move to workFlowDecisions-createTransferBackup directory

Assign file UUIDs and checksums

Find type to process as

Assign file UUIDs and checksums

Workflow decision - create transfer backup

Assign file UUIDs and checksums

Create transfer backup (sharedDirectory/transferBackups)

Assign file UUIDs and checksums

Move to processing directory

Assign file UUIDs and checksums

Set file permissions

Assign file UUIDs and checksums

Assign file UUIDs to objects

Assign file UUIDs and checksums

Assign checksums and file sizes to objects

Verify transfer checksums

Verify metadata directory checksums

Generate METS.xml document

Generate METS.xml document

Quarantine

Designate to process as a standard transfer

Quarantine

Move to workFlowDecisions-quarantineSIP directory

Quarantine

Find type to process as

Quarantine

Workflow decision - send transfer to quarantine

Quarantine

Set quarantine permissions on transfer

Quarantine

Create quarantine PREMIS events

Quarantine

Designate to process as a standard transfer when unquarantined

Quarantine

Move to quarantine

Quarantine

Find type to remove from quarantine as

Quarantine

Remove from quarantine

Quarantine

Set unquarantined file permissions on Transfer

Quarantine

Move to processing directory

Quarantine

Create unquarantine PREMIS events

Extract packages

Move to processing directory

Extract packages

Extract packages

Scan for viruses

Scan for viruses

Clean up names

Sanitize object's file and directory names

Clean up names

Sanitize Transfer name

Characterize and extract metadata

Characterize and extract metadata

Characterize and extract metadata

Identify Files ByExtension

Characterize and extract metadata

Load labels from metadata/file_labels.csv

Complete transfer

Set file permissions

Complete transfer

Move to completedTransfers directory

Complete transfer

Index transfer contents

Create SIP from Transfer

Check transfer directory for objects

Create SIP from Transfer

Create SIP(s)

Create SIP from Transfer

Move to processing directory

Create SIP from Transfer

Create SIP from transfer objects

Create SIP from Transfer

Move to completedTransfers directory

Reject transfer

Move to the rejected directory

Failed transfer

Move to the failed directory

Approve SIP creation

Approve SIP Creation

Verify SIP compliance

Set file permissions

Verify SIP compliance

Move to processing directory

Verify SIP compliance

Verify SIP compliance

Verify transfer compliance

Verify mets_structmap.xml compliance

Failed compliance

Failed compliance. See output in dashboard. SIP moved back to SIPsUnderConstruction

Rename SIP directory with SIP UUID

Rename SIP directory with SIP UUID

Include default SIP processingMCP.xml

Include default SIP processingMCP.xml

Remove cache files

Remove cache files

Clean up names

Set file permissions

Clean up names

Sanitize SIP name

Normalize

Check for Service directory

Normalize

Check for Access directory

Normalize

Grant normalization options for pre-existing DIP

Normalize

Grant normalization options for no pre-existing DIP

Normalize

Move to workFlowDecisions-createDip directory

Normalize

Find options to normalize as

Normalize

Normalize

Process submission documentation

Remove files without linking information (failed normalization artifacts etc.)

Process submission documentation

Normalize submission documentation to thumbnail format

Normalize

Verify checksums generated on ingest

Normalize

Copy transfers metadata and logs

Normalize

Generate METS.xml document

Normalize

Normalize service files for access

Normalize

Normalize access

Normalize

Create DIP directory

Normalize

Normalize for preservation and access

Normalize

Normalize preservation

Normalize

Remove files without linking information (failed normalization artifacts etc.)

Normalize

Move to approve normalization directory

Normalize

Approve normalization

Normalize

Move to processing directory

Normalize

Set file permissions

Normalize

Create thumbnails directory

Normalize

Normalize thumbnails

Process submission documentation

Move to processing directory

Process submission documentation

Set file permissions

Process submission documentation

Copy transfer submission documentation

Process submission documentation

Check for submission documentation

Process submission documentation

Move submission documentation into objects directory

Process submission documentation

Assign file UUIDs to submission documentation

Process submission documentation

Assign checksums and file sizes to submissionDocumentation

Process submission documentation

Extract packages in submission documentation

Process submission documentation

Sanitize file and directory names in submission documentation

Process submission documentation

Scan for viruses in submission documentation

Process submission documentation

Characterize and extract metadata on submission documentation

Process submission documentation

Identify Files ByExtension

Process submission documentation

Normalize submission documentation to preservation format

Prepare AIP

Remove files without linking information (failed normalization artifacts etc.)

Prepare AIP

Verify checksums generated on ingest

Prepare AIP

Copy transfers metadata and logs

Prepare AIP

Generate METS.xml document

Prepare DIP

Copy thumbnails to DIP directory

Prepare DIP

Copy METS to DIP directory

Prepare DIP

Set file permissions

Prepare DIP

Generate DIP

Prepare AIP

Index AIP contents

Prepare AIP

Prepare AIP

Prepare AIP

Select compression algorithm

Prepare AIP

Select compression level

Prepare AIP

Compress AIP

Prepare AIP

Set bag file permissions

Prepare AIP

Removed bagged files

Store AIP

Move to the store AIP approval directory

Store AIP

Store AIP

Store AIP

Store AIP location

Store AIP

Move to processing directory

Store AIP

Store the AIP

Store AIP

Remove the processing directory

Upload DIP

Select target CONTENTdm server

Upload DIP

Get list of collections on server

Upload DIP

Select destination collection

Upload DIP

Select upload type (Project Client or direct upload)

Upload DIP

Restructure DIP for CONTENTdm upload

Upload DIP

Upload DIP to contentDM

Upload DIP

Upload DIP

Upload DIP

Move to the uploadedDIPs directory

Reject DIP

Move to the rejected directory

Reject SIP

Move to the rejected directory

Failed SIP

Move to the failed directory

Normalize

Find preservation links to run.

Normalize

Find access links to run.

Normalize

Find thumbnail links to run.

Process submission documentation

Find thumbnail links to run.

Process submission documentation

Find preservation links to run. |

Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox