Difference between revisions of "Micro-services"

From Archivematica
Jump to navigation Jump to search
Line 8: Line 8:
 
These information packages are moved from one service to the next using the well-established [http://en.wikipedia.org/wiki/Pipeline_%28Unix%29 Unix pipeline] design pattern. Each micro-service is defined in a simple XML configuration file and associated with a watched directory. When an information package is moved to that directory it triggers the micro-service.  
 
These information packages are moved from one service to the next using the well-established [http://en.wikipedia.org/wiki/Pipeline_%28Unix%29 Unix pipeline] design pattern. Each micro-service is defined in a simple XML configuration file and associated with a watched directory. When an information package is moved to that directory it triggers the micro-service.  
  
−
Each service is provided by a combination of Archivematica Python scrips and one or more of the free, open-source [[External tools|software tools]] bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is moved accordingly to a success or error directory. Each success or error directory is the watched directory for a subsequent micro-service. This allows for the chaining of directories into complex, custom workflows. Archivematica implements a default ingest to access workflow that is [[Requirements|compliant with the ISO-OAIS]] functional model.  
+
Each service is provided by a combination of Archivematica Python scrips and one or more of the free, open-source [[External tools|software tools]] bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is moved accordingly to a success or error directory. Each success or error directory is the watched directory for a subsequent micro-service. This allows for the chaining of directories into complex, custom workflows. Archivematica implements a default ingest to access workflow that is [[Requirements|compliant with the ISO-OAIS]] functional model.
−
 
 
−
==Archivematica Micro-services==
 
−
 
 
−
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
−
|-
 
−
|- style="background-color:#cccccc;"
 
−
!style="width:30%"|'''Micro-service'''
 
−
!style="width:70%"|'''Description'''
 
−
|-
 
−
|backupSIP
 
−
|Create a backup of the entire SIP as soon as it is ingested.
 
−
|-
 
−
|verifySIPcompliance
 
−
|Verify that the SIP conforms to the folder structure required for processing in Archivematica.
 
−
|-
 
−
|assignIdentifier
 
−
|Each file in the SIP is assigned a universal unique identifier and a sha-1 checksum for future integrity checks.
 
−
|-
 
−
|verifyChecksums
 
−
|If the ingested SIP already contains a checksum file, this micro-service will check it to confirm that none of the files were deleted or altered upon transfer to Archivematica.
 
−
|-
 
−
|createDublinCore
 
−
|If the ingested SIP does not already contain one, a Dublin Core xml template is added to the metadata folder in the SIP. The user can fill in fields as desired. These values are uploaded to the access system as part of the DIP created by Archivematica.
 
−
|-
 
−
|appraiseForSubmission
 
−
|The user may review the SIP to confirm that it complies with any submission agreements. The user can delete unwanted files at this point; a log of the deleted files will be added to the information package.
 
−
|-
 
−
|quarantine
 
−
|The SIP is placed in quarantine for a pre-set period of time. The user can move the SIP out of quarantine before the pre-set time has expired, if desired.
 
−
|-
 
−
|extractPackages
 
−
|Files are extracted from any .zip, .tar or other file package formats; each extracted file is assigned a universal unique identifier and a sha-1 checksum.
 
−
|-
 
−
|sanitizeNames
 
−
|Prohibited characters which may cause processing errors on known operating systems (e.g. spaces or ampersands) are removed from file and directory names and replaced with underscores.
 
−
|-
 
−
|virusScan
 
−
|ClamAV scans all files in the SIP. In the event that a virus or other malware is found, the SIP is placed in a folder called SIPerrors and all processing on the SIP is stopped.
 
−
|-
 
−
|validateFormatsAndExtractMetadata
 
−
|File formats are identified and the files validated against external format specifications. Technical metadata is extracted from the file.
 
−
|-
 
−
|appraiseForPreservation
 
−
|The user may appraise the contents of the SIP and delete unwanted files; a log of the deleted files is added to the information package.
 
−
|-
 
−
|normalize
 
−
|Normalize SIP files into a preservation format copy and an access format copy for each file according to its [[Media type preservation plans|media type preservation plan]]. These are packaged along with the original file in the AIP.
 
−
|-
 
−
|compilePreservationMetadata
 
−
|Compile a METS file with a complete set of PREMIS metadata for each ingested file. Technical metadata is placed in the PREMIS bjectCharacteristicsExtension element.
 
−
|-
 
−
|createAIPchecksum
 
−
|Generate a sha-1 checksum for all AIP contents.
 
−
|-
 
−
|prepareAIP
 
−
|Package AIP using the Library of Congress Bagit specification.
 
−
|-
 
−
|storeAIP
 
−
|The user may review the AIP and approve it for archival storage. The AIP is moved into the AIPsStore folder which is synced to the storage system.
 
−
|-
 
−
|generateDIP
 
−
|The access copies that were created during the "transcode" micro-service are placed in a DIP folder and the METS file is added to the DIP.
 
−
|-
 
−
|uploadDIP
 
−
|The user may review the DIP and remove any access copies that cannot be sent to the public access system due to copyright, security or other issues. The user then approves the DIP for upload and the DIP is uploaded into the public access system (in Archivematica, the default access system is the open-source archival description tool ICA-AtoM). A backup copy of the DIP, including files that were deleted, is sent to the DIPbackups folder.
 
−
|-
 
−
|}
 

Revision as of 18:06, 8 August 2011

Main Page > Documentation > Technical Architecture > Micro-services


Micro-service.png

The Archivematica micro-services are granular system tasks which operate on a conceptual entity that is equivalent to an OAIS information package: Submission Information Package (SIP), Archival Information Package (AIP), Dissemination Information Package (DIP). The physical structure of an information package will include files, checksums, logs, XML metadata, etc..

These information packages are moved from one service to the next using the well-established Unix pipeline design pattern. Each micro-service is defined in a simple XML configuration file and associated with a watched directory. When an information package is moved to that directory it triggers the micro-service.

Each service is provided by a combination of Archivematica Python scrips and one or more of the free, open-source software tools bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is moved accordingly to a success or error directory. Each success or error directory is the watched directory for a subsequent micro-service. This allows for the chaining of directories into complex, custom workflows. Archivematica implements a default ingest to access workflow that is compliant with the ISO-OAIS functional model.

Retrieved from "https://wiki.archivematica.org/index.php?title=Micro-services&oldid=4292"