Difference between revisions of "Micro-services"

From Archivematica
Jump to navigation Jump to search
 
(31 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > Micro-services
 
[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > Micro-services
  
 +
<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>
  
 
[[File:Micro-service.png|320px|thumb|right]]  
 
[[File:Micro-service.png|320px|thumb|right]]  
  
−
The Archivematica [http://www.cdlib.org/services/uc3/curation/ micro-services] are granular system tasks which operate on a conceptual entity that is equivalent to an OAIS information package: Submission Information Package (SIP), Archival Information Package (AIP), Dissemination Information Package (DIP). The physical structure of an information package will include files, checksums, logs, XML metadata, etc..
+
Archivematica implements a [http://www.cdlib.org/services/uc3/curation/ micro-service] approach to digital preservation. The Archivematica micro-services are granular system tasks which operate on a conceptual entity that is equivalent to an OAIS information package: Submission Information Package (SIP), Archival Information Package (AIP), Dissemination Information Package (DIP). The physical structure of an information package will include files, checksums, logs, submission documentation, XML metadata, etc..  
  
−
These information packages are moved from one service to the next using the well-established [http://en.wikipedia.org/wiki/Pipeline_%28Unix%29 Unix pipeline] design pattern. Each micro-service is defined in a simple XML configuration file and associated with a watched directory. When an information package is moved to that directory it triggers the micro-service.  
+
These information packages are processed using a series of micro-services. Micro-services are provided by a combination of Archivematica Python scripts and one or more of the free, open-source [[External tools|software tools]] bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is processed accordingly by the next micro-service. There are a variety of mechanisms used to connect the various micro-services together into complex, custom workflows. Resulting in a complete ingest to access system.
  
−
Each service is provided by a combination of Archivematica Python scrips and one or more of the free, open-source [[External tools|software tools]] bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is moved accordingly to a success or error directory. Each success or error directory is the watched directory for a subsequent micro-service. This allows for the chaining of directories into complex, custom workflows. Archivematica implements a default ingest to access workflow that is [[Requirements|compliant with the ISO-OAIS]] functional model.  
+
Archivematica implements a [[Micro-services#Archivematica_Micro-services|default ingest to access workflow]] that is [[Requirements|compliant with the ISO-OAIS]] functional model. Micro-services can be distributed to processing clusters for highly scalable configurations.  
  
−
==Archivematica 0.7.1 Micro-services==
+
==Current Micro-services in Archivematica==
  
 +
A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).
 +
 +
</br>
 +
 +
[[Image:Micro-services1.png|600px|center|thumb|Archivematica dashboard showing a micro-service and its jobs]]
 +
 +
</br>
 +
 +
The table below shows micro-services and jobs in the current release of Archivematica ('''Updated July 2015 for release 1.4.1''').
 +
 +
</br>
 +
===Transfer===
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
|-
 
|-
Line 18: Line 31:
 
!style="width:70%"|'''Description'''
 
!style="width:70%"|'''Description'''
 
|-
 
|-
−
|Create SIP backup
+
|-
−
|Creates a backup of the SIP. By default these are stored in /sharedDirectoryStructure/SIPbackups/. The backups are automatically removed at the end of SIP processing, when the AIP has been moved to archival storage.
+
|  <div class="toccolours mw-collapsible mw-collapsed">'''Approve Transfer'''<div class="mw-collapsible-content">
 +
</div>
 +
</div>
 +
| This is the approval step that moves the transfer into the Archivematica processing pipeline.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content">
 +
<br/>Set file permissions
 +
<br/>Move to processing directory
 +
<br/>Set transfer type: (Standard, Zipped bag, Unzipped bag, DSpace, Maildir)
 +
<br/>Remove hidden files and directories
 +
<br/>Remove unneeded files
 +
<br/>Attempt restructure for compliance
 +
<br/>Verify transfer compliance
 +
<br/>Verify mets_structmap.xml compliance
 +
</div>
 +
</div>
 +
| Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Rename with transfer UUID'''<div class="mw-collapsible-content">
 +
<br/>Rename with transfer UUID
 +
</div>
 +
</div>
 +
| Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Include default Transfer processingMCP.xml'''<div class="mw-collapsible-content">
 +
<br/>Include default Transfer processingMCP.xml
 +
</div>
 +
</div>
 +
| Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Assign file UUIDs and checksums'''<div class="mw-collapsible-content">
 +
<br/>Set file permissions
 +
<br/>Assign file UUIDs to objects
 +
<br/>Assign checksums and file sizes to objects
 +
</div>
 +
</div>
 +
| Assigns a unique universal identifier and sha-256 checksum to each file in the ''/objects/'' directory and sets file permission to allow for continued processing.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer checksums'''<div class="mw-collapsible-content">
 +
<br/>Verify metadata directory checksums
 +
</div>
 +
</div>
 +
| Checks any checksum files that were placed in the ''/metadata/'' folder of the transfer prior to moving the transfer into Archivematica.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Generate METS.xml document'''<div class="mw-collapsible-content">
 +
<br/>Generate METS.xml document
 +
</div>
 +
</div>
 +
| Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the ''/objects/'' directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.
 +
|-
 +
| <div class="toccolours mw-collapsible mw-collapsed">'''Reformat metadata files'''<div class="mw-collapsible-content">
 +
<br/>Process transfer JSON metadata
 +
</div>
 +
</div>
 +
| Transforms JSON metadata included with transfer to metadata.csv
 
|-
 
|-
−
|Verify SIP compliance
+
| <div class="toccolours mw-collapsible mw-collapsed">'''Quarantine'''<div class="mw-collapsible-content">
−
|Verify that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/logs/fileMeta/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
+
<br/>Workflow decision - send transfer to quarantine
 +
<br/>Move to quarantine
 +
<br/>Remove from quarantine
 +
</div>
 +
</div>
 +
| Quarantines the transfer for a set duration, to allow virus definitions to update, before virus scan.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content">
 +
<br/>Scan for viruses
 +
</div>
 +
</div>
 +
| Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in ''/sharedDirectoryStructure/failed/'' and all processing on the transfer is stopped.
 
|-
 
|-
−
|Assign file UUIDs and checksums
+
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate transfer structure report'''<div class="mw-collapsible-content">
−
|Assigns file UUIDs and generates checksums for each file in the /objects/ directory. This step also creates the PREMIS files located in the /logs/fileMeta/ directory. The files in this directory are named based on the fileUUID of the file they represent.
+
<br/>Generate transfer structure report
 +
<br/>Move to generate transfer tree
 +
</div>
 +
</div>
 +
| Uses Tree to create transfer structure report (original order report) in txt format
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content">
 +
<br/>Sanitize object's file and directory names
 +
<br/>Sanitize Transfer name
 +
</div>
 +
</div>
 +
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Identify file format'''<div class="mw-collapsible-content">
 +
<br/>Move to select file ID tool
 +
<br/>Select file format identification command
 +
<br/>Determine which files to identify
 +
<br/> Identify file format
 +
</div>
 +
</div>
 +
| Identifies formats of the objects in the transfer using either FIDO or file extension based on user choice. Format types are managed in the Format Policy Registry. This micro-service can be skipped and done in Ingest instead.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Extract packages'''<div class="mw-collapsible-content">
 +
<br/>Move to processing directory
 +
<br/>Move to extract packages
 +
<br/>Extract packages? (preconfigured choice)
 +
<br/>Delete package after extraction? (preconfigured choice)
 +
<br/>Extract contents from compressed archives
 +
</div>
 +
</div>
 +
| Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers. User can preconfigure choices about whether to extract and whether to keep one or both of extracted object contents and/or the package itself.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Update METS.xml document'''<div class="mw-collapsible-content">
 +
<br/>Add processed structMap to METS.xml document
 +
</div>
 +
</div>
 +
| Creates a structMap in the transfer METS to reflect structure of transfer
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content">
 +
<br/>Characterize and extract metadata
 +
<br/>Load labels from metadata/file_labels.csv
 +
</div>
 +
</div>
 +
| Characterizes formats and extracts object metadata using File Information Tool Set (FITS), FFprobe, Exiftool and/or Mediainfo
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Validation'''<div class="mw-collapsible-content">
 +
<br/>Validate formats
 +
</div>
 +
</div>
 +
| Validates formats using JHOVE.  
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Examine Contents'''<div class="mw-collapsible-content">
 +
<br/>Examine contents
 +
</div>
 +
</div>
 +
| Runs Bulk Extractor on transfer contents and creates reports
 
|-
 
|-
−
|Verify metadata directory checksums
+
| <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content">
−
|Checks any checksum files that were placed in the /metadata/ folder of the SIP prior to ingest. Note that the filenames need to be named based on their algorithm: ''checksum.sha1'', ''checksum.sha256'', ''checksum.md5''.
+
<br/>Index transfer contents
 +
<br/>Create transfer metadata XML
 +
<br/>Move to SIP creation directory for completed transfers
 +
</div>
 +
</div>
 +
| Indexes transfer contents, creates transfer METS XML, then marks the transfer as complete.  
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content">
 +
<br/>Check transfer directory for objects
 +
<br/>Updating transfer file index
 +
<br/>Create placement in backlog PREMIS events
 +
<br/>Move transfer to backlog
 +
<br/>Check transfer directory for objects
 +
<br/>Load options to create SIP
 +
</div>
 +
</div>
 +
| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest) if user chooses to Create single SIP and continue processing. User can also choose to Send transfer to backlog at this time, which creates an associated PREMIS event.
 
|-
 
|-
−
|Remove thumbs.db files
+
|}
−
|Removes any [http://en.wikipedia.org/wiki/Windows_thumbnail_cache Thumbs.db] files. May be expanded to others in future releases.
+
</br>
 +
 
 +
===Ingest===
 +
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
|-
 
|-
−
|Create Dublin Core template
+
|- style="background-color:#cccccc;"
−
|If the ingested SIP does not already contain one, a Dublin Core xml template is added to the ''/metadata/'' folder in the SIP. The user can fill in fields as desired. These values are uploaded to the access system as part of the DIP created by Archivematica.
+
!style="width:30%"|'''Micro-service'''
−
|-
+
!style="width:70%"|'''Description'''
−
|Set file permissions
 
−
|Changes file permissions on the SIP to allow the user to modify the SIP contents.
 
−
|-
 
−
|Appraise SIP for submission
 
−
|Manual approval step. Review the SIP to confirm that it conforms to any submission agreements and remove files and folders if desired. Do not move or rename files or folders as this will cause them to be excluded from the AIP.
 
−
|-
 
−
|Scan for removed files post appraise SIP for submission
 
−
|Checks to see if any files were deleted and creates a list of them at ''/logs/removedFilesAppraiseSIPForSubmission.log''.
 
−
|-
 
−
|Place in quarantine
 
−
|Places SIP in quarantine for a pre-set period of time. The purpose of this is to allow time for new viruses to be identified, and antivirus groups to update their virus definitions. Note: for demonstration purposes, the quarantine period is set to a minute.
 
−
|-
 
−
|Remove from quarantine
 
−
|Archivematica uses a cron job to periodically check for SIPs that have met the configured quarantine time. Keeping in mind the purpose of the quarantine period, if you know the virus definitions are up to date for any virus possibly contained in the SIP (eg. The SIP source is a cd from 4 years ago) then you can remove it from quarantine manually.
 
−
|-
 
−
|Extract packages
 
−
|Extracts objects from any zipped files or other packages.
 
−
|-
 
−
|Sanitize file and directory names
 
−
|Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
 
−
|-
 
−
|Scan for viruses
 
−
|Uses [http://www.clamav.net/lang/en/ ClamAV], parses the output and creates a PREMIS event. If a virus is found, the SIP is automatically placed in ''/sharedPath/watchedDirectories/failed/''.
 
−
|-
 
−
|Characterize and extract metadata
 
−
|Identifies and validates formats and extracts object metadata using the [http://code.google.com/p/fits/ File Information Tool Set (FITS)]. Adds output to the PREMIS files.
 
−
|-
 
−
|Set file permissions
 
−
|Changes file permissions on the SIP to allow the user to modify the SIP contents.
 
−
|-
 
−
|Appraise SIP for preservation
 
−
|Manual approval step. If desired, appraise SIP contents for preservation and delete any unwanted files and folders. Do not move or rename files or folders as this will cause them to be excluded from the AIP. Note: in future releases, appraisal decisions at this point will be assisted by a summary of technical information about the files (format, validation status, characteristics such as compression etc.).
 
−
|-
 
−
|Scan for removed files post appraise SIP for preservation
 
−
|Checks to see if any files were deleted and creates a list of them at ''/logs/removedFilesAppraiseSIPForPreservation.log''.
 
−
|-
 
−
|Create DIP directory
 
−
|Creates a directory for access copies of the ingested files.
 
−
|-
 
−
|Normalize
 
−
|Creates preservation copies and access copies of the ingested files based on rules in the transcoder database. These rules can be seen under the '''Preservation planning''' tab in the Archivematica dashboard. Adds access copies to the DIP directory.
 
−
|-
 
−
|Set file permissions
 
−
|Changes file permissions on the SIP to allow the user to modify the SIP contents.
 
−
|-
 
−
|Approve normalization
 
−
|Manual approval step. This micro-service allows for manual normalization. See [[0.7.1_How-To#Manual_Normalization]]. Note that the user cannot delete normalized files: see issue 678.
 
−
|-
 
−
|Check for submission documentation
 
−
|Checks for files in ''metadata/submissionDocumentation/''; if folder is empty, creates a log file indicating that no submission documentation was included in the SIP.
 
−
|-
 
−
|Move Submission Documentation into objects directory
 
−
|Moves the ''/submissionDocumentation/'' directory from ''/metadata/'' to ''/objects/'' for ingest processing.
 
−
|-
 
−
|Assign file UUIDs and checksums to submission documentation
 
−
|See Assign file UUIDs and checksums, above.
 
−
|-
 
−
|Extract packages in submission documentation
 
−
|See Extract packages, above.
 
−
|-
 
−
|Sanitize file and directory names in submission documentation
 
−
|See Sanitize file and directory names, above.
 
−
|-
 
−
|Scan for viruses in submission documentation
 
−
|See Scan for viruses, above.
 
 
|-
 
|-
−
|Characterize and extract metadata in submission documentation
+
|-
−
|See Characterize and extract metadata, above.
+
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve SIP creation'''<div class="mw-collapsible-content">
−
|-
+
<br/>Approve SIP creation
−
|Normalize submission documentation
+
<br/> Create removal from backlog PREMIS events
−
|See Normalize, above. No access copies are made since the submission documentation is not included in the DIP.
+
</div>
−
|-
+
</div>
−
|Remove files without PREMIS
+
| Applies to SIPs that have been created from backlog. Approves creation of SIP, and creates PREMIS event for removal from backlog.
−
|Removes any files in the ''/objects/'' directory that don't have PREMIS entries. This is done because some normalizations that fail will still leave behind artifacts/files (for example, 0 byte files) that don't belong in the AIP.
+
|-  
−
|-
+
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content">
−
|Verify PREMIS checksums
+
<br/>Verify mets_structmap.xml compliance
−
|Verifies the checksums assigned at ingest to ensure that the files have not been modified while being processed by Archivematica.
+
</div>
−
|-
+
</div>
−
|Compile METS
+
| Verifies the METS from the transfer.
−
|Creates a ''METS.xml'' file using the PREMIS files in ''/logs/fileMeta/''.
 
−
|-
 
−
|Add Dublin Core to METS
 
−
|Adds the ''/metadata/dublincore.xml'' file to the ''METS.xml'' file.
 
−
|-
 
−
|Copy METS to DIP directory
 
−
|Creates a copy of the ''METS.xml'' file in the DIP directory.
 
−
|-
 
−
|Generate DIP
 
−
|Moves the DIP directory out of the SIP and into another Archivematica watched directory.
 
−
|-
 
−
|Set file permissions
 
−
|Changes file permissions on the DIP to allow the user to modify the DIP contents.
 
−
|-
 
−
|Prepare AIP
 
−
|Packages the SIP into an AIP using [http://sourceforge.net/projects/loc-xferutils/ BagIt]
 
−
|-
 
−
|Upload DIP
 
−
|Uploads the DIP to the access system ([http://www.ica-atom.org ICA-AtoM]). Removes the UUID from the filename as the upload title, but original filename still contains this UUID so it can be traced back to the file in the AIP.
 
−
|-
 
−
|Store AIP
 
−
|Moves the AIP to a specified directory. In the demo version of Archivematica the directory is ''/sharedDirectoryStructure/www/AIPsStore/''. In other environments it can be a remote network mounted directory. The directory structure of the AIP store contains UUID quad directories and an ''index.html'' file listing the AIPs in storage. The ''index.html'' file is displayed in the dashboard in the '''Archival storage''' tab.
 
−
*Once the AIP has been stored, a copy of the AIP is extracted from storage to a local temp directory, and is validated with the various BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.
 
 
|-
 
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Verify SIP compliance'''<div class="mw-collapsible-content">
 +
<br/>Set file permissions
 +
<br/>Move to processing directory
 +
<br/>Verify SIP compliance
 +
</div>
 +
</div>
 +
| Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content">
 +
<br/>Rename SIP directory with SIP UUID
 +
<br/>Check if SIP is from Maildir Transfer
 +
</div>
 +
</div>
 +
| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name and checks if SIP is from Maildir transfer type to determine workflow.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content">
 +
<br/>Include default SIP processingMCP.xml
 +
</div>
 +
</div>
 +
| Copies the processing configuration file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content">
 +
<br/>Remove cache files
 +
</div>
 +
</div>
 +
| Removes any thumbs.db files.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content">
 +
<br/>Load Dublin Core metadata from disk
 +
<br/>Sanitize SIP name
 +
<br/>Set file permissions
 +
</div>
 +
</div>
 +
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content">
 +
<br/>Identify manually normalized files
 +
<br/>Check for Service directory
 +
<br/>Check for Access directory
 +
<br/>Set remove preservation and access normalized files to renormalize link.
 +
<br/>Grant normalization options for no pre-existing DIP
 +
<br/>Move to workFlowDecisions-createDip directory
 +
<br/>Find options to normalize as
 +
<br/>Set resume link after tool selected
 +
<br/>Move to select file ID tool
 +
<br/>Select pre-normalize file format identification commant
 +
<br/>Identify file format
 +
<br/>Resume after normalization file identification tool selected
 +
<br/>Normalize
 +
<br/>Move to processing directory
 +
<br/>Create DIP directory
 +
<br/>Create thumbnails directory
 +
<br/>Normalize thumbnails
 +
<br/>Normalize access
 +
<br/>Normalize preservation
 +
<br/>Set file permissions
 +
<br/>Remove files without linking information (failed normalization artifacts etc.)
 +
<br/>Move to approve normalization directory
 +
<br/>Approve normalization
 +
<br/>Load post approve normalization link
 +
<br/>Set resume link after handling any manually normalized files
 +
<br/>Move to processing directory
 +
<br/>Set file permissions
 +
<br/>Load finished with manual normalized link
 +
</div>
 +
</div>
 +
| Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Process manually normalized files'''<div class="mw-collapsible-content">
 +
<br/>Check for manual normalized files
 +
</div>
 +
</div>
 +
| Processed files manually normalized during processing
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Add final metadata'''<div class="mw-collapsible-content">
 +
<br/> Move to metadata reminder
 +
<br/>Reminder: add metadata if desired
 +
<br/>Set file permissions
 +
</div>
 +
</div>
 +
| Provides the operator a reminder to add metadata through the UI if desired.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Transcribe SIP contents'''<div class="mw-collapsible-content">
 +
<br/>Transcribe SIP contents
 +
<br/>Transcribe
 +
</div>
 +
</div>
 +
| Runs Tesseract OCR tool on any JPG or TIFF image files in SIP.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content">
 +
<br/>Copy transfer submission documentation
 +
<br/>Check for submission documentation
 +
<br/>Move submission documentation into objects directory
 +
<br/>Assign file UUIDs to submission documentation
 +
<br/>Assign checksums and file sizes to submissionDocumentation
 +
<br/>Sanitize file and directory names in submission documentation
 +
<br/>Scan for viruses in submission documentation
 +
<br/>Select file format identification command
 +
<br/>Identify file format
 +
<br/>Characterize and extract metadata on submission documentation
 +
<br/>Remove files without linking information (failed normalization artifacts etc.)
 +
</div>
 +
</div>
 +
| Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Process metadata directory'''<div class="mw-collapsible-content">
 +
<br/>Copy transfers metadata and logs
 +
<br/>Process JSON metadata
 +
<br/>Move metadata to objects directory
 +
<br/>Assign file UUIDs to metadata
 +
<br/>Assign checksums and file sizes to metadata
 +
<br/>Sanitize file and directory names in metadata
 +
<br/>Scan for viruses in metadata
 +
<br/>Identify file format of metadata files
 +
<br/>Characterize and extract metadata on metadata files
 +
<br/>Remove empty manual normalization directories
 +
</div>
 +
</div>
 +
| Processes metadata.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Verify checksums'''<div class="mw-collapsible-content">
 +
<br/>Verify checksums generated on ingest
 +
</div>
 +
</div>
 +
| Verifies checksums generated on ingest
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Generate AIP METS'''<div class="mw-collapsible-content">
 +
<br/>Generate METS.xml document 
 +
</div>
 +
</div>
 +
| Generates Archivematica AIP METS.xml document
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content">
 +
<br/>Copy thumbnails to DIP directory
 +
<br/>Copy METS to DIP directory
 +
<br/>Copy preconfigured choice XML to DIP directory
 +
<br/>Generate DIP
 +
<br/>Copy OCR data to DIP directory
 +
<br/>Set file permissions
 +
</div>
 +
</div>
 +
| Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content">
 +
<br/>Check if DIP should be generated
 +
<br/>Prepare AIP
 +
<br/>Move to compressionAIPDecisions directory
 +
<br/>Select compression algorithm
 +
<br/>Select compression level
 +
<br/>Compress AIP
 +
<br/>Copy submission documentation
 +
<br/>Create AIP pointer file
 +
<br/>Set bag file permissions
 +
<br/>Check if AIP is a file or directory
 +
<br/>Removed bagged files
 +
</div>
 +
</div>
 +
| Creates an AIP in Bagit format. Creates the AIP pointer file. Indexes the AIP, then losslessly compresses it.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP'''<div class="mw-collapsible-content">
 +
<br/>Upload DIP
 +
</div>
 +
</div>
 +
| Allows the user to choose to upload the DIP AtoM, CONTENTdm, Archivist's Toolkit. Also allows user to store DIP or reject DIP.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP  to AtoM'''<div class="mw-collapsible-content">
 +
<br/>Upload DIP 
 +
<br/>Move to the uploadedDIPs directory
 +
</div>
 +
</div>
 +
| The user uploads the DIP to a selected description in AtoM.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP  to CONTENTdm'''<div class="mw-collapsible-content">
 +
<br/>Upload DIP
 +
<br/>Restructure DIP for CONTENTdm upload
 +
<br/>Move to the uploadedDIPs directory
 +
</div>
 +
</div>
 +
| The user creates a DIP suitable for upload to CONTENTdm using Project Client.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP  to Archivists' Toolkit'''<div class="mw-collapsible-content">
 +
<br/>
 +
</div>
 +
</div>
 +
| The user uploads the DIP metadata to Archivists' Toolkit
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Store DIP'''<div class="mw-collapsible-content">
 +
<br/>Upload DIP
 +
<br/>Retrieve DIP Storage Locations
 +
<br/>Store DIP location
 +
<br/>Store DIP
 +
<br/>Move to the uploadedDIPs directory
 +
</div>
 +
</div>
 +
| The user stores the DIP in a location pre-configured in the Storage Service.
 +
|-
 +
|  <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content">
 +
<br/>Move to the store AIP approval directory
 +
<br/>Store AIP   
 +
<br/>Retrieve AIP Storage Locations
 +
<br/>Store AIP location
 +
<br/>Move to processing directory
 +
<br/>Verify AIP
 +
<br/>Store the AIP
 +
<br/>Index AIP
 +
<br/>Clean up after storing AIP
 +
<br/>Remove processing directory
 +
</div>
 +
</div>
 +
| Moves the AIP to ''/sharedDirectoryStructure/www/AIPsStore/'' or another specified directory. Before the AIP has been stored, a copy of it is extracted to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.
 +
|-
 
|}
 
|}
 +
 +
=Historical Micro-services in Archivematica Releases=
 +
 +
*[[Archivematica 1.1 Micro-services]]
 +
*[[Archivematica 1.0 Micro-services]]
 +
*[[Archivematica 0.10 Micro-services]]
 +
*[[Archivematica 0.9 Micro-services]]
 +
*[[Archivematica 0.8 Micro-services]]
 +
*[[Archivematica 0.7.1 Micro-services]]

Latest revision as of 16:36, 11 February 2020

Main Page > Documentation > Technical Architecture > Micro-services

This page is no longer being maintained and may contain inaccurate information. Please see the Archivematica documentation for up-to-date information.

Micro-service.png

Archivematica implements a micro-service approach to digital preservation. The Archivematica micro-services are granular system tasks which operate on a conceptual entity that is equivalent to an OAIS information package: Submission Information Package (SIP), Archival Information Package (AIP), Dissemination Information Package (DIP). The physical structure of an information package will include files, checksums, logs, submission documentation, XML metadata, etc..

These information packages are processed using a series of micro-services. Micro-services are provided by a combination of Archivematica Python scripts and one or more of the free, open-source software tools bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is processed accordingly by the next micro-service. There are a variety of mechanisms used to connect the various micro-services together into complex, custom workflows. Resulting in a complete ingest to access system.

Archivematica implements a default ingest to access workflow that is compliant with the ISO-OAIS functional model. Micro-services can be distributed to processing clusters for highly scalable configurations.

Current Micro-services in Archivematica[edit]

A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).


Archivematica dashboard showing a micro-service and its jobs


The table below shows micro-services and jobs in the current release of Archivematica (Updated July 2015 for release 1.4.1).


Transfer[edit]

Micro-service Description
Approve Transfer
 This is the approval step that moves the transfer into the Archivematica processing pipeline.
Verify transfer compliance


Set file permissions
Move to processing directory
Set transfer type: (Standard, Zipped bag, Unzipped bag, DSpace, Maildir)
Remove hidden files and directories
Remove unneeded files
Attempt restructure for compliance
Verify transfer compliance
Verify mets_structmap.xml compliance

Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename with transfer UUID


Rename with transfer UUID

Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.
Include default Transfer processingMCP.xml


Include default Transfer processingMCP.xml

Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
Assign file UUIDs and checksums


Set file permissions
Assign file UUIDs to objects
Assign checksums and file sizes to objects

Assigns a unique universal identifier and sha-256 checksum to each file in the /objects/ directory and sets file permission to allow for continued processing.
Verify transfer checksums


Verify metadata directory checksums

Checks any checksum files that were placed in the /metadata/ folder of the transfer prior to moving the transfer into Archivematica.
Generate METS.xml document


Generate METS.xml document

Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the /objects/ directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.
Reformat metadata files


Process transfer JSON metadata

Transforms JSON metadata included with transfer to metadata.csv
Quarantine


Workflow decision - send transfer to quarantine
Move to quarantine
Remove from quarantine

Quarantines the transfer for a set duration, to allow virus definitions to update, before virus scan.
Scan for viruses


Scan for viruses

Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in /sharedDirectoryStructure/failed/ and all processing on the transfer is stopped.
Generate transfer structure report


Generate transfer structure report
Move to generate transfer tree

Uses Tree to create transfer structure report (original order report) in txt format
Clean up names


Sanitize object's file and directory names
Sanitize Transfer name

Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
Identify file format


Move to select file ID tool
Select file format identification command
Determine which files to identify
Identify file format

Identifies formats of the objects in the transfer using either FIDO or file extension based on user choice. Format types are managed in the Format Policy Registry. This micro-service can be skipped and done in Ingest instead.
Extract packages


Move to processing directory
Move to extract packages
Extract packages? (preconfigured choice)
Delete package after extraction? (preconfigured choice)
Extract contents from compressed archives

Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers. User can preconfigure choices about whether to extract and whether to keep one or both of extracted object contents and/or the package itself.
Update METS.xml document


Add processed structMap to METS.xml document

Creates a structMap in the transfer METS to reflect structure of transfer
Characterize and extract metadata


Characterize and extract metadata
Load labels from metadata/file_labels.csv

Characterizes formats and extracts object metadata using File Information Tool Set (FITS), FFprobe, Exiftool and/or Mediainfo
Validation


Validate formats

Validates formats using JHOVE.
Examine Contents


Examine contents

Runs Bulk Extractor on transfer contents and creates reports
Complete transfer


Index transfer contents
Create transfer metadata XML
Move to SIP creation directory for completed transfers

Indexes transfer contents, creates transfer METS XML, then marks the transfer as complete.
Create SIP from Transfer


Check transfer directory for objects
Updating transfer file index
Create placement in backlog PREMIS events
Move transfer to backlog
Check transfer directory for objects
Load options to create SIP

This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest) if user chooses to Create single SIP and continue processing. User can also choose to Send transfer to backlog at this time, which creates an associated PREMIS event.


Ingest[edit]

Micro-service Description
Approve SIP creation


Approve SIP creation
Create removal from backlog PREMIS events

Applies to SIPs that have been created from backlog. Approves creation of SIP, and creates PREMIS event for removal from backlog.
Verify transfer compliance


Verify mets_structmap.xml compliance

Verifies the METS from the transfer.
Verify SIP compliance


Set file permissions
Move to processing directory
Verify SIP compliance

Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: /logs/, /metadata/, /metadata/submissionDocumentation/, /objects/.
Rename SIP directory with SIP UUID


Rename SIP directory with SIP UUID
Check if SIP is from Maildir Transfer

Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name and checks if SIP is from Maildir transfer type to determine workflow.
Include default SIP processingMCP.xml


Include default SIP processingMCP.xml

Copies the processing configuration file added to the transfer in Include default Transfer processingMCP.xml, above, to the SIP.
Remove cache files


Remove cache files

Removes any thumbs.db files.
Clean up names


Load Dublin Core metadata from disk
Sanitize SIP name
Set file permissions

Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
Normalize


Identify manually normalized files
Check for Service directory
Check for Access directory
Set remove preservation and access normalized files to renormalize link.
Grant normalization options for no pre-existing DIP
Move to workFlowDecisions-createDip directory
Find options to normalize as
Set resume link after tool selected
Move to select file ID tool
Select pre-normalize file format identification commant
Identify file format
Resume after normalization file identification tool selected
Normalize
Move to processing directory
Create DIP directory
Create thumbnails directory
Normalize thumbnails
Normalize access
Normalize preservation
Set file permissions
Remove files without linking information (failed normalization artifacts etc.)
Move to approve normalization directory
Approve normalization
Load post approve normalization link
Set resume link after handling any manually normalized files
Move to processing directory
Set file permissions
Load finished with manual normalized link

Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.
Process manually normalized files


Check for manual normalized files

Processed files manually normalized during processing
Add final metadata


Move to metadata reminder
Reminder: add metadata if desired
Set file permissions

Provides the operator a reminder to add metadata through the UI if desired.
Transcribe SIP contents


Transcribe SIP contents
Transcribe

Runs Tesseract OCR tool on any JPG or TIFF image files in SIP.
Process submission documentation


Copy transfer submission documentation
Check for submission documentation
Move submission documentation into objects directory
Assign file UUIDs to submission documentation
Assign checksums and file sizes to submissionDocumentation
Sanitize file and directory names in submission documentation
Scan for viruses in submission documentation
Select file format identification command
Identify file format
Characterize and extract metadata on submission documentation
Remove files without linking information (failed normalization artifacts etc.)

Processes any submission documentation included in the SIP and adds it to the /objects/ directory.
Process metadata directory


Copy transfers metadata and logs
Process JSON metadata
Move metadata to objects directory
Assign file UUIDs to metadata
Assign checksums and file sizes to metadata
Sanitize file and directory names in metadata
Scan for viruses in metadata
Identify file format of metadata files
Characterize and extract metadata on metadata files
Remove empty manual normalization directories

Processes metadata.
Verify checksums


Verify checksums generated on ingest

Verifies checksums generated on ingest
Generate AIP METS


Generate METS.xml document

Generates Archivematica AIP METS.xml document
Prepare DIP


Copy thumbnails to DIP directory
Copy METS to DIP directory
Copy preconfigured choice XML to DIP directory
Generate DIP
Copy OCR data to DIP directory
Set file permissions

Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.
Prepare AIP


Check if DIP should be generated
Prepare AIP
Move to compressionAIPDecisions directory
Select compression algorithm
Select compression level
Compress AIP
Copy submission documentation
Create AIP pointer file
Set bag file permissions
Check if AIP is a file or directory
Removed bagged files

Creates an AIP in Bagit format. Creates the AIP pointer file. Indexes the AIP, then losslessly compresses it.
Upload DIP


Upload DIP

Allows the user to choose to upload the DIP AtoM, CONTENTdm, Archivist's Toolkit. Also allows user to store DIP or reject DIP.
Upload DIP to AtoM


Upload DIP
Move to the uploadedDIPs directory

The user uploads the DIP to a selected description in AtoM.
Upload DIP to CONTENTdm


Upload DIP
Restructure DIP for CONTENTdm upload
Move to the uploadedDIPs directory

The user creates a DIP suitable for upload to CONTENTdm using Project Client.
Upload DIP to Archivists' Toolkit


 The user uploads the DIP metadata to Archivists' Toolkit
Store DIP


Upload DIP
Retrieve DIP Storage Locations
Store DIP location
Store DIP
Move to the uploadedDIPs directory

The user stores the DIP in a location pre-configured in the Storage Service.
Store AIP


Move to the store AIP approval directory
Store AIP
Retrieve AIP Storage Locations
Store AIP location
Move to processing directory
Verify AIP
Store the AIP
Index AIP
Clean up after storing AIP
Remove processing directory

Moves the AIP to /sharedDirectoryStructure/www/AIPsStore/ or another specified directory. Before the AIP has been stored, a copy of it is extracted to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.

Historical Micro-services in Archivematica Releases[edit]

Retrieved from "https://wiki.archivematica.org/index.php?title=Micro-services&oldid=13402"