Archivematica 1.0 Micro-services - Revision history

Ablewer at 20:13, 11 December 2019

2019-12-11T20:13:58Z

Courtney at 19:53, 6 February 2014

2014-02-06T19:53:09Z

Courtney at 18:42, 6 February 2014

2014-02-06T18:42:58Z

Courtney: /* Ingest */

2014-02-06T18:40:53Z

Ingest

Courtney: /* Ingest */

2014-02-06T18:34:06Z

Ingest

Courtney: /* Ingest */

2014-02-06T18:13:46Z

Ingest

Courtney at 18:07, 6 February 2014

2014-02-06T18:07:08Z

Courtney: Created page with "Main Page > Documentation > Technical Architecture > Micro-services > Archivematica 1.0 Micro-services A micro-service may consist of a number of discrete tas..."

2014-02-06T17:52:44Z

Created page with "Main Page > Documentation > Technical Architecture > Micro-services > Archivematica 1.0 Micro-services A micro-service may consist of a number of discrete tas..."

New page

[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 1.0 Micro-services

A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 1.o dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).

</br>

[[Image:NormPresAccess-10.png|600px|center|thumb|Arhivematica dashboard showing a micro-service and its jobs]]

</br>

The table below shows micro-services and jobs in Archivematica 1.0. Note that this is is only a list of micro-services; detailed user instructions are available in the [[User manual 1.0|user manual]].
</br>
=Transfer=
{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:30%"|'''Micro-service'''
!style="width:70%"|'''Description'''
|-
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve Transfer'''<div class="mw-collapsible-content">
Set file permissions
</div>
</div>
| This is the approval step that moves the transfer into the Archivematica processing pipeline.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Move to processing directory
<br/>Set transfer type: (Standard, Zipped bag, Unzipped bag, DSpace, Maildir)
<br/>Remove hidden files and directories
<br/>Remove unneeded files
<br/>Attempt restructure for compliance
<br/>Verify transfer compliance
<br/>Verify mets_structmap.xml compliance
</div>
</div>
| Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename with transfer UUID'''<div class="mw-collapsible-content">
<br/>Rename with transfer UUID
</div>
</div>
| Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default Transfer processingMCP.xml'''<div class="mw-collapsible-content">
<br/>Include default Transfer processingMCP.xml
</div>
</div>
| Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Assign file UUIDs and checksums'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Assign file UUIDs to objects
<br/>Assign checksums and file sizes to objects
</div>
</div>
| Assigns a unique universal identifier and sha-256 checksum to each file in the ''/objects/'' directory and sets file permission to allow for continued processing.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer checksums'''<div class="mw-collapsible-content">
<br/>Verify metadata directory checksums
</div>
</div>
| Checks any checksum files that were placed in the ''/metadata/'' folder of the transfer prior to moving the transfer into Archivematica.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate METS.xml document'''<div class="mw-collapsible-content">
<br/>Generate METS.xml document
</div>
</div>
| Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the ''/objects/'' directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Quarantine'''<div class="mw-collapsible-content">
<br/>Workflow decision - send transfer to quarantine
<br/>Move to quarantine
<br/>Remove from quarantine
</div>
</div>
| Quarantine's the transfer for a set duration, to allow virus definitions to update, before virus scan.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Extract packages'''<div class="mw-collapsible-content">
<br/>Extract packages
<br/>Extract attachments
</div>
</div>
| Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content">
<br/>Scan for viruses
</div>
</div>
| Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in ''/sharedDirectoryStructure/failed/'' and all processing on the transfer is stopped.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content">
<br/>Sanitize object's file and directory names
<br/>Sanitize Transfer name
</div>
</div>
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content">
<br/>Characterize and extract metadata
<br/>Identify Files ByExtension
<br/>Load labels from metadata/file_labels.csv
</div>
</div>
| Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Also identifies file extensions, which are used for selecting normalization paths.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content">
<br/>Index transfer contents
<br/>Set file permissions
<br/>Move to completedTransfers directory
</div>
</div>
| Indexes transfer contents, then marks the transfer as complete.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content">
<br/>Check transfer directory for objects
<br/>Create SIP(s)
<br/>Create SIP from transfer objects
<br/>Send transfer to backlog
</div>
</div>
| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest).
|-
|}
</br>

=Ingest=
{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:30%"|'''Micro-service'''
!style="width:70%"|'''Description'''
|-
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify SIP compliance'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Move to processing directory
<br/>Verify SIP compliance
</div>
</div>
| Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content">
<br/>Rename SIP directory with SIP UUID
</div>
</div>
| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content">
<br/>Include default SIP processingMCP.xml
</div>
</div>
| Copies the processing config file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content">
<br/>Remove cache files
</div>
</div>
| Removes any Thumbs.db files.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content">
<br/>Identify manually normalized files
<br/>Check for Service directory
<br/>Check for Access directory
<br/>Find options to normalize as
<br/>Create DIP directory
<br/>Normalize
<br/>Normalize access
<br/>Normalize preservation
<br/>Remove files without linking information (failed normalization artifacts etc.)
<br/>Create thumbnails directory
<br/>Normalize thumbnails
<br/>Verify checksums generated on ingest
<br/>Approve normalization
<br/>Check for manual normalized files
<br/>Move access files to DIP
<br/>Assign UUIDs to manually normalized preservation files
<br/>Assign checksums to manually normalized preservation files
<br/>Run FITS on manually normalized preservation files
<br/>Relate manually normalized preservation files to the original files
</div>
</div>
| Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content">
<br/>Check for submission documentation
<br/>Copy transfer submission documentation
<br/>Move to processing directory
<br/>Set file permissions
<br/>Assign file UUIDs to submission documentation
<br/>Assign checksums and file sizes to submissionDocumentation
<br/>Extract packages in submission documentation
<br/>Sanitize file and directory names in submission documentation
<br/>Scan for viruses in submission documentation
<br/>Characterize and extract metadata on submission documentation
<br/>Identify Files ByExtension
<br/>Normalize submission documentation to preservation format
</div>
</div>
| Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content">
<br/>Copy thumbnails to DIP directory
<br/>Copy METS to DIP directory
<br/>Set file permissions
<br/>Generate DIP
</div>
</div>
| Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP'''<div class="mw-collapsible-content">
<br/>Upload DIP
</div>
</div>
| Allows the user to choose to upload the DIP to either ICA-AtoM or CONTENTdm.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to ICA-AtoM'''<div class="mw-collapsible-content">
<br/>Upload DIP
<br/>Move to the uploadedDIPs directory
</div>
</div>
| The user uploads the DIP to a selected description in ICA-AtoM.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to CONTENTdm'''<div class="mw-collapsible-content">
<br/>Restructure DIP for CONTENTdm upload
<br/>Select upload type (Project Client or direct upload)
<br/>Select target CONTENTdm server
<br/>Get list of collections on server
<br/>Select destination collection
<br/>Upload DIP to contentDM
<br/>Move to the uploadedDIPs directory
</div>
</div>
| The user uploads the DIP to a selected description in CONTENTdm.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content">
<br/>Copy transfers metadata and logs
<br/>Remove files without linking information (failed normalization artifacts etc.)
<br/>Verify checksums generated on ingest
<br/>Generate METS.xml document
<br/>Index AIP contents
<br/>Prepare AIP
<br/>Select compression algorithm
<br/>Select compression level
<br/>Compress AIP
<br/>Set bag file permissions
<br/>Removed bagged files
</div>
</div>
| Creates an AIP in Bagit format. Indexes the AIP, then losslessly compresses it.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content">
<br/>Move to the store AIP approval directory
<br/>Store AIP
<br/>Store AIP location
<br/>Store the AIP
</div>
</div>
| Moves the AIP to ''/sharedDirectoryStructure/www/AIPsStore/'' or another specified directory. Once the AIP has been stored, a copy of it is extracted from storage to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.
|-
|}

← Older revision		Revision as of 20:13, 11 December 2019
Line 1:		Line 1:
	[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 1.0 Micro-services		[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 1.0 Micro-services
		+
		+	<blockquote style="background-color:orange;">
		+	'''Note: The documentation on this page is out of date. Please see the [https://www.archivematica.org/docs official documentation page] for the latest.'''
		+	</blockquote>

	A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 1.0 dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).		A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 1.0 dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).

@@ Line 5: / Line 5: @@
 </br>
-[[Image:NormPresAccess-10.png|600px|center|thumb|Arhivematica dashboard showing a micro-service and its jobs]]
+[[Image:Micro-services1.png|600px|center|thumb|Archivematica dashboard showing a micro-service and its jobs]]
 </br>

@@ Line 1: / Line 1: @@
 [[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 1.0 Micro-services
-A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 1.o dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).
+A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 1.0 dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).
 </br>

@@ Line 280: / Line 280: @@
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content">
 <br/>Copy transfers metadata and logs
-<br/>Remove files without linking information (failed normalization artifacts etc.)
+<br/>Remove empty manual normalization directories
 <br/>Generate METS.xml document
 <br/>Index AIP contents
 <br/>Prepare AIP
 <br/>Select compression algorithm
 <br/>Select compression level
 <br/>Compress AIP
 <br/>Set bag file permissions
 <br/>Removed bagged files
 </div>
 </div>
-| Creates an AIP in Bagit format. Indexes the AIP, then losslessly compresses it.
+| Creates an AIP in Bagit format. Creates the AIP pointer file. Indexes the AIP, then losslessly compresses it.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content">
 <br/>Move to the store AIP approval directory
 <br/>Store AIP
 <br/>Store AIP location
 <br/>Store the AIP
 </div>
 </div>

@@ Line 184: / Line 184: @@
 </div>
 </div>
+| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content">
@@ Line 190: / Line 190: @@
 <br/>Check for Service directory
 <br/>Check for Access directory
 <br/>Find options to normalize as
 <br/>Create DIP directory
-<br/>Normalize
+<br/>Create thumbnails directory
 <br/>Normalize access
 <br/>Normalize preservation
 <br/>Remove files without linking information (failed normalization artifacts etc.)
-<br/>Create thumbnails directory
+<br/>Move to approve normalization directory
 <br/>Approve normalization
-<br/>Check for manual normalized files
+<br/>Load post approve normalization link
-<br/>Move access files to DIP
+<br/>Set resume link after handling any manually normalized files
-<br/>Assign UUIDs to manually normalized preservation files
+<br/>Move to processing directory
-<br/>Assign checksums to manually normalized preservation files
+<br/>Set file permissions
-<br/>Run FITS on manually normalized preservation files
+<br/>Check for manually normalized files
-<br/>Relate manually normalized preservation files to the original files
+<br/>Load finished with manual normalized link
 </div>
 </div>
@@ Line 211: / Line 221: @@
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content">
 <br/>Check for submission documentation
-<br/>Copy transfer submission documentation
+<br/>Move submission documentation into objects directory
 <br/>Assign file UUIDs to submission documentation
 <br/>Assign checksums and file sizes to submissionDocumentation
 <br/>Sanitize file and directory names in submission documentation
 <br/>Scan for viruses in submission documentation
 <br/>Characterize and extract metadata on submission documentation
 </div>
 </div>
 | Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content">

@@ Line 153: / Line 153: @@
 </div>
 | Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content">
 <br/>Rename SIP directory with SIP UUID
 </div>
 </div>
-| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name.
+| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name and checks if SIP is from Maildir transfer type to determine workflow.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content">
@@ Line 164: / Line 171: @@
 </div>
 </div>
-| Copies the processing config file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.
+| Copies the processing configuration file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content">
@@ Line 170: / Line 177: @@
 </div>
 </div>
-| Removes any Thumbs.db files.
+| Removes any thumbs.db files.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content">

@@ Line 77: / Line 77: @@
 </div>
 | Quarantine's the transfer for a set duration, to allow virus definitions to update, before virus scan.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content">
@@ Line 98: / Line 91: @@
 | Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content">
 <br/>Characterize and extract metadata
 <br/>Load labels from metadata/file_labels.csv
 </div>
 </div>
-| Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Also identifies file extensions, which are used for selecting normalization paths.
+| Characterizes and validates formats and extracts object metadata using the File Information Tool Set (FITS).
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content">
 <br/>Index transfer contents
-<br/>Set file permissions
+<br/>Move to SIP creation directory for completed transfers
 </div>
 </div>
 | Indexes transfer contents, then marks the transfer as complete.
 |-
 |  <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content">
 <br/>Check transfer directory for objects
 <br/>Create SIP(s)
 <br/>Create SIP from transfer objects
 <br/>Send transfer to backlog
 </div>
 </div>
-| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest).
+| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest) if user chooses to Create single SIP and continue processing. User can also choose to Send transfer to backlog at this time.
 |-
 |}