Archivematica - User contributions [en]

MCPClient

2018-05-11T17:16:37Z

Jdunham: /* Config File */

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCPClient

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

Archivematica has one or more MCPClient instances to perform the actual work. They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.

== Client scripts ==
Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.

New scripts are defined in <code>[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/archivematicaClientModules src/MCPClient/lib/archivematicaClientModules]</code>, which is what is registered with Gearman on MCPClient startup.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: archivematicaClientModules lists both 'supportedCommandSpecial' and 'supportedCommands'. This distinction may have once been based on scripts that relied on external services, but serves no purpose now and should be removed.
|}

The name is what the [[MCPServer#StandardTasksConfigs | StandardTasksConfig]] table will refer to them as, and the value is the script that will be run. Some are defined as shell builtins (eg copy_v0.0 is cp). Most are paths to a script in the clientScripts directory, using the <code>%clientScriptsDirectory%</code> replacement variable. The name of the client script is usually the same as the name in archivematicaClientModules, but for very old scripts may have ‘archivematica’ at the beginning (eg createMETS_v2.0 = archivematicaCreateMETS2.py) or be named more pythonically (eg parseExternalMETS = parse_external_mets.py). Entries are added alphabetically.

The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen. Newer scripts may not have the version defined.

The list of client scripts is sorted roughly in order of appearance during processing

=== moveTransfer_v0.0 ===

* '''Purpose''': Move a Transfer & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveTransfer.py archivematicaMoveTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole Transfer and updates the database with the new location relative to the shared directory.

=== assignFileUUIDs_v0.0 ===

* '''Purpose''': Starts tracking files new to Archivematica
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaAssignFileUUID.py archivematicaAssignFileUUID.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': ingestion/reingestion, possibly registration
* '''FPR?''': No

This creates an entry in the Files table with the file's UUID, current & original paths and file group. It also creates an 'ingestion' Event and an 'registration' Event if an accession ID was specified. Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.

In ingest, is used on manually normalized files which may have been newly added, metadata and submission documentation.

On reingest, it parses the METS file instead of generating the file UUID, path & group. The Event type is 'reingestion'.

=== updateSizeAndChecksum_v0.0 ===

* '''Purpose''': Set file's size & checksum
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaUpdateSizeAndChecksum.py archivematicaUpdateSizeAndChecksum.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': message digest calculation
* '''FPR?''': No

Updates the entry in the Files table with a size and checksum. IT also generates a 'message digest calculation' Event.

On reingest, it parses the METS file instead of generating the checksums & sizes. It also re-adds Derivation & Format links.

Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].

=== createMETS_v0.0 ===

* '''Purpose''': Generate the transfer METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS.py archivematicaCreateMETS.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Creates the Transfer METS file. This will contain all the information generated on the transfer during processing, and is especially useful for backlogged transfers.

Not to be confused with [[#createMETS_v2.0]] for the AIP METS.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The Transfer METS file & related backlog functionality needs to be expanded. See [[Transfer_backlog_requirements#Proposed_improvements]] for details.
|}

=== createEvent_v0.0 ===

* '''Purpose''': Create an Event outside of other scripts
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/createEvent.py createEvent.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': as parameter: quarantine, unquarantine, placement in backlog, removal from backlog
* '''FPR?''': No

Used to generate events for when a file is placed in or removed from quarantine or backlog. These do not have scripts associated with them, so the event creation is handled here.

=== archivematicaClamscan_v0.0 ===

* '''Purpose''': Check for viruses in incoming files
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaClamscan.py archivematicaClamscan.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': virus check
* '''FPR?''': No

Runs clamscan on the file and generates a 'virus scan' event. If a scan has been run, it is not run again on the same file.

This is run on incoming files, files after extraction, metadata files and submission documentation. It is not run on normalized files.

=== sanitizeObjectNames_v0.0 ===

* '''Purpose''': Strip problematic characters from filenames
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeObjectNames.py sanitizeObjectNames.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': name cleanup
* '''FPR?''': No
* '''Tests''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_sanitize.py test_sanitize.py]

Sanitize object names by replacing anything that isn't an ASCII letter, number, hyphen, underscore, period or parenthesis. This runs on both files and directories, though only files have Events generated. An Event is generated for a file even if only a parent directory is sanitized.

This used code from [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeNames.py sanitizeNames.py]

=== identifyFileFormat_v0.0 ===

* '''Purpose''': Identify a file's format
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/identifyFileFormat.py identifyFileFormat.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': format identification
* '''FPR?''': IDCommand & IDRule

One of the most important scripts in Archivematica. Since the file format is used to determine many later actions (extraction, characterization, normalization etc), if this fails many important command later will also fail. This is the only script that uses the FPR that doesn't use the file format as a key for looking up what command to run. Instead, an IDCommand is selected and the output is matched to an IDRule to find the FormatVersion.

There is a short circuit handling of PRONOM ID (PUID) outputs. Since many FormatVersions have PUIDs, and both FIDO & Siegfried output PUIDs, this script looks for a FormatVersion with a given PUID. This reduces the number of IDRules that have to be created.

This also populates the legacy but still required FilesIDs table.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Only one identification tool can be run at a time currently. It would be better to allow a cascading of tools. E.g. if a file is identified as a video to subsequently run a tool specialized in identifying different types of video. Similarly, if the default tool failed, we could run a backup tool for a second opinion.
|}

=== extractContents_v0.0 ===

* '''Purpose''': Extract files from packages
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/extractContents.py extractContents.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': unpacking, registration, deletion, message digest calculation
* '''FPR?''': extract

Extracts files from a package (e.g. zip, tar, 7z etc) and optionally deletes the package. Generates Events for the new files.

=== characterizeFile_v0.0 ===

* '''Purpose''': Collects characterization information on a file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/characterizeFile.py characterizeFile.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': No
* '''FPR?''': characterization, default_characterization

Collects characterization commands for the provided file, then either
# Inserts the tool's XML output into the database, or
# Prints the tool's stdout, for tools which do not output XML

If a tool has no defined characterization commands, then the default will be run instead (currently FITS). Can run multiple characterization commands and log the output of all of them.

=== validateFile_v1.0 ===

* '''Purpose''': Validate a file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/validateFile.py validateFile.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': validation
* '''FPR?''': validation, default_validation

Validates files are correct, where correctness is defined by the file format.

=== examineContents_v0.0 ===

* '''Purpose''': Run bulk extractor for a detailed analysis of contents
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/examineContents.py examineContents.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': No
* '''FPR?''': No

Runs bulk extractor and stores the outputs in the logs directory.

=== elasticSearchIndex_v0.0 ===

* '''Purpose''': Index the Transfer METS into ElasticSearch when sending files to backlog
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/elasticSearchIndexProcessTransfer.py elasticSearchIndexProcessTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

The data in ElasticSearch is used by the Backlog tab, SIP Arrangement and the Appraisal tab when dealing with files from backlog. Note that this is not run if the transfer is not sent to backlog (since AM 1.5).

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The client config 'disableElasticsearchIndexing' can disable indexing, but this should be removed, since searching for files in backlog is required functionality.
|}

=== moveSIP_v0.0 ===

* '''Purpose''': Move a SIP & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveSIP.py archivematicaMoveSIP.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole SIP and updates the database with the new location relative to the shared directory.

=== normalize_v1.0 ===

* '''Purpose''': Generate a preservation or access derivative
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/normalize.py normalize.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': normalization
* '''FPR?''': Rules: access, preservation, thumbnail, default_access, default_thumbnail. Commands: normalization, verification

One of the most important scripts in Archivematica. This generates the preservation and access derivatives, as well as thumbnails. If manual normalization happens, this instead finds the manually normalized files and links them as derivatives.

The same script is used for generating preservation, access or thumbnail derivatives, controlled by the parameter passed in. In general, normalization is only performed on original files, but may be done on service files. This reads the FPR for the specific command, and may fall back to a default access or thumbnail command.

The output is used to generate the normalization report about normalization attempted and success/failure rates.

For historical reasons, much of the functionality is also implemented in [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/transcoder.py transcoder.py]. It runs an event detail & verification command associated with the normalization command.

On reingest, a 'deletion' Event is created for replaced derivatives.

=== transcribeFile_v0.0 ===

* '''Purpose''': Generate OCR of files.
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaTranscribeFile.py archivematicaTranscribeFile.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': transcription
* '''FPR?''': transcription

Optionally generates an OCR file for original files based on FPR entries for transcription. If the original file has no transcription rules, runs on the derivative. The new file is a derivation of the original, has a group of 'text/ocr' and is updated with a UUID, checksum, size etc.

=== createMETS_v2.0 ===

* '''Purpose''': Generate the AIP METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py archivematicaCreateMETS2.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No
* '''Tests''':
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_create_aip_mets.py test_create_aip_mets.py]
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_reingest_mets.py test_reingest_mets.py]

Perhaps the most important script in Archivematica: it creates the AIP METS which contains all the archival metadata generated by previous client scripts.

This script imports from several other files for additional functionality:
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSMetadataCSV.py archivematicaCreateMETSMetadataCSV]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRights.py archivematicaCreateMETSRights]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRightsDspaceMDRef.py archivematicaCreateMETSRightsDspaceMDRef]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSTrim.py archivematicaCreateMETSTrim]

On reingest, it short-circuits and runs [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSReingest.py archivematicaCreateMETSReingest] to update the METS file instead.

Not to be confused with [[#createMETS_v0.0]] for the transfer METS.

=== storeAIP_v0.0 ===

* '''Purpose''': Send the completed AIP to the storage service
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/storeAIP.py storeAIP.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Sends the currently processing AIP to the storage service. The Location is selected from the list of AIP Storage Locations associated with the Pipeline in previous tasks.

=== ===

* '''Purpose''':
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/]
* '''Used in''':
* '''Task type''':
* '''Event?''':
* '''FPR?''':
* '''Tests''':

== Config File ==

Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.

Variables in the MCPClient section:

{|
|-
! Variable !! Description !! Default value
|-
| MCPArchivematicaServer || URL of the MCP gearman server. Must match the [[MCPServer#Config File | server config file]]. || localhost:4730
|-
| sharedDirectoryMounted || Directory structure owned by Archivematica and shared between the MCPServer & MCPClient. Must match the [[MCPServer#Config File | server config file]]. || /var/archivematica/sharedDirectory/
|-
| archivematicaClientModules || Path to the list of jobs to register with Gearman || /usr/lib/archivematica/MCPClient/archivematicaClientModules
|-
| clientScriptsDirectory || Path to the directory where client scripts are installed. Used when parsing archivematicaClientModules || /usr/lib/archivematica/MCPClient/clientScripts/
|- style="background: #ffeecc;"
| LoadSupportedCommandsSpecial || Whether or not to register the SupportedCommandsSpecial section of archivematicaClientModules. This should be removed. || True
|-
| numberOfTasks || Number of MCPClient workers to created. 0 detects the number of cores and uses that. || 0
|-
| elasticsearchServer || URL of the ElasticSearch server. || localhost:9200
|- style="background: #ffeecc;"
| disableElasticsearchIndexing || If true, do not index AIPs or Transfers in backlog. This should be removed, since ElasticSearch indexing is required || False
|-
| temp_dir || Path to the temporary usage directory. Should be in the shared directory || /var/archivematica/sharedDirectory/tmp
|- style="background: #ffeecc;"
| kioskMode || Dashboard setting that disables editing users. This should be removed, or at least moved to dashboard settings || False
|-
| removableFiles || List of filenames that are not archivally significant and can be removed. || Thumbs.db, Icon, Icon\r, .DS_Store
|-
| django_settings_module || Name of the Django settings module, so the client scripts can access the database via the Django ORM. || settings.common
|-
|}

[[Category:Development documentation]]

Storage Service API

2018-02-27T23:45:38Z

Jdunham: /* Delete package request */

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: IP or hostname of the pipeline. If not provided and <code>create_default_locations</code> is set, will try to populate from the IP of the request.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:
$ curl -X POST -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" -H"Content-Type: application/json" -d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", "description": "Test pipeline", "create_default_locations": true, "api_username": "demo", "api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' 'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Is there no way to create Spaces in the API?
|}

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Is there no way to create Locations in the API?
|}

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the package path.
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Dashboard

2018-02-22T18:37:38Z

Jdunham: /* Sections */

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Dashboard

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

The dashboard manages a pipeline’s behaviour. It provides a view into the status of units, allows workflow decisions to be made, handles configuration, allows arrangement & description, and customization of the [[Format policy registry requirements | FPR]]. Code for the dashboard is found in <code>src/dashboard/src/</code>

Previously the dashboard also handled starting transfers from disk & storing files, but much of that functionality has been moved to the storage service.

== Sections ==

* <code>components/access</code>: Internal API to talk to access systems (ArchivesSpace, Archivist’s Toolkit). Primarily supports the appraisal tab, and uses [[https://github.com/artefactual-labs/agentarchives agentarchives]]
* <code>components/accounts</code>: Views related to account management (create, edit, delete, list)
* <code>components/administration</code>: Views & models related to the Administration tab. Contains the processing config form, DIP upload settings, etc
* <code>components/api</code>: [[ Archivematica_API | External API]] endpoints.
* <code>components/appraisal</code>: Appraisal tab view.
* <code>components/archival_storage</code>: Views related to the Archival Storage tab. Contains ElasticSearch queries, creating AICs, deleting AIPs and starting reingest.
* <code>components/backlog</code>: Views related to the Backlog tab. Contains ElasticSearch queries, deleting & downloading backlogged transfers.
* <code>components/file</code>: Internal API to get file information from ElasticSearch & Storage Service. Primarily supports the appraisal tab.
* <code>components/filesystem_ajax</code>: Internal API to support SIP Arrangement and moving files to and from the pipeline. Much of its previous functionality has been superseded by the storage service
* <code>components/ingest</code>: Views and internal API related to the Ingest tab.
* <code>components/mcp</code>: Internal API for status updates from [[MCPServer]]
* <code>components/rights</code>: Views and forms for editing unit rights.
* <code>components/transfer</code>: Views and internal API related to the Transfer tab.
* <code>components/unit</code>: Views and internal API common between Transfers and SIPs
{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This is newer than <code>components/transfer</code> and <code>components/ingest</code>. Functionality that is the same between the two should be moved here as appropriate
|}
* <code>external</code>: External dependencies, often git submodules. '''Note: as of Archivematica 1.7, there is no external/ directory here. The base64-helpers subdirectory is now required by the transfer-browser and appraisal-tab and is downloaded during the NPM build phase.'''
* <code>fpr</code>: Views related to the Preservation Planning tab. This in the FPR-admin submodule from externals, and contains all [[Format policy registry requirements | FPR]] modification views. '''Note: as of Archivematica 1.7, there is no fpr/ directory here. The archivematica-fpr-admin module is now a dependency of the dashboard; see dashboard/src/requirements/base.txt.'''
* <code>installer</code>: Views related to installation and setting up Archivematica
* <code>main</code>: Models, views and internal APIs related to core dashboard functionality. Contains all model definitions & migrations, as well as app-wide configuration. Also includes Task & Job display, and views related to the Access tab.
* <code>media</code>: All CSS, JS & images for the dashboard
* <code>middleware</code>: Django middleware definitions.
* <code>requirements</code>: Python requirements files
* <code>settings</code>: Django settings modules for development or production configuration
* <code>templates</code>: All templates for the dashboard. Structure broadly mirrors that of the components directory.

== Old documentation ==

===Technical Requirements===
The Dashboard is a web-based tool that is developed using Python-based [http://djangoproject.com Django] MVC framework.

===Functional Requirements===

*provide a web-based, multi-user interface that will report on the status of system events and make it simpler to control and trigger specific micro-services.
*provide a user-friendly interface to add/edit metadata
*coordinate the read and write operations of the AIP to file storage and the syncing of metadata updates between the AIPs and the access system.
*process Consumer AIP requests
*provide statistical information about Archivematica operation
*provide preservation planning information

===User interface===

Release 0.7-alpha (Feb 18, 2011)

[[File:Dashboard-0.7.png|680px]]

Django interface found in Archivematica 0.6.2 (dev tree, 29 Nov 2010)

[[File:Archivematica-dashboard-0.6.2-dev-29Nov.png|480px]]

Early mockup (March 2010)

[[File:ArchivematicaDashboardScreencap05Mar2010.png|480px]]

===Real-time interaction===

<div style="background-color: #fffccc; border: 1px solid #ddd; padding: 8px;">
Our preliminary design will be based in '''periodic refresh''', trying to minimize the risks of more sophisticated solutions before [http://archivematica.org/wiki/index.php?title=Development_roadmap#Release_0.7-alpha Release 0.7-alpha] is launched. In future releases, we will do more research on this topic trying to achieve the best user experience while we keep an eye on performance.</div>

The Ajax web application model came to made the Web UI experience dynamic and asynchronous, as a replacement of the classic page-by-page web application model (see [http://www.adaptivepath.com/images/publications/essays/ajax-fig2.png graph]). However, Ajax applications don't offer a duplex communication where both client and server can send messages at any time. A new model of web applications frequently called [http://infrequently.org/2006/03/comet-low-latency-data-for-the-browser/ Comet] appeared providing bi-directional communications using persistent long-lasting HTTP connections between the server and the client (see [http://infrequently.org/wp-content/Comet.png graph]). Comet is similar to Ajax in that it's asynchronous, but '''applications following the Comet model can communicate state changes on the server with almost negligible latency, which makes it suitable for monitoring or multi-user collaboration applications'''.

There exists different methods of implementing a Comet streaming transport (browser transport), but all of them are based in existing browser features: iframe HTML element, XMLHttpRequest or script tags. Between these methods, I think that we have two candidates:

* '''XMLHttpRequest long polling''': firstly, the browser creates an asynchronous XMLHttpRequest with a long time-out. When we receive a response, the server closes the connection and we launch another XHR request immediately afterward, waiting for a new event.
* '''Script tag long polling''': the browser creates script HTML elements dinamically and setting their source ("src" attribute) to the location of the server, which then send back JavaScript code. Each time the script requests is completed, the browser opens a new one, just like XHR long polling design does. This method bypass the [http://code.google.com/p/browsersec/wiki/Part2#Same-origin_policy same-origin policy security mechanism] implemented in modern browsers.

====Alternatives====

There exists other alternatives that we should consider:

* '''WebSockets''': this technology is part of HTML5 and provides full-duplex communications channels over a single TCP socket between the browser and the server. The [http://dev.w3.org/html5/websockets/ WebSocket API] is being standardized by the W3C and the [http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol WebSocket protocol] is being standardized by the IETF ([https://datatracker.ietf.org/wg/hybi/charter/ HyBi working group]). Chrome 4, Safari 5, Firefox 4 (not yet in FF3) and Opera 11 support WebSockets. However, the last two ones have disabled this protocol by default. HTML5 Labs at Microsoft interoperability group recently launched a [http://blogs.msdn.com/b/interoperability/archive/2010/12/21/introducing-the-websockets-prototype.aspx prototype] compatible with IE8 and IE9 based in Silverlight. WebSockets is a promissing technology but unfortunately in the development phase yet.
** There are some solutions which provide an API that looks like WebSocket API, and fallback to other techniques if WebSocket is not available. A good example is Socket.IO, which supports different transports: WebSocket, Adobe® Flash® Socket, Ajax long polling, etc... however, the server module was designed for Node.JS. Several implementations have been started for other languages / frameworks that are compatible with the Socket.IO client.
** Other products like CometD, Lightstreamer and others provide a higher-level API using pubsub (see mod_pubsub) or some other messaging protocol, and use WebSocket or whatever other transport is available that is the fastest and safest option.
* '''Server-Sent Events''': another [http://dev.w3.org/html5/eventsource/ draft API] included in HTML5 designed for scenarios where data does not need to be sent from the client, just need updates from the server (server push only). This technology, only supported by some browsers like Chrome or Opera, could be considered as a formal and efficient alternative to Comet, but based in the same method: HTTP long-held requests. The big difference with WebSockets is, therefore, that it does not to implement a new protocol (it is based in HTTP) and it is not really full-duplex, although it could be simulated with parallel XHR requests.
* '''Periodic refresh''' (simple polling): to keep users informated about changes occurring on the server we can make the browser generates requests periodically, at fixed intervals, to gain new information: for example, one call every five seconds. This is a valid approximation where the server push data if data latency is not a critical for users. A callback function would be responsible for updating the DOM according to the server's latest report and the browser script can do some monitoring and dynamically adjust the period of refreshes to minimize the workload (e.g.: to cease it when the system detects the user is no longer active, see this [http://ajaxpatterns.org/Heartbeat article]).
** [https://github.com/brettstimmerman/lace Lace]: Old open source chat based in simple polling (periodic refresh).

====Server design and scalability====

When a web application creates bi-directional connections between the browser and the server, new server software is often required in order to scale well. Take into account that traditional web-based solutions would break down very quickly due to memory consumption and the excess overhead of framework for each HTTP (and possibly long-held) request made.

More research on this must be done if we decide to take advantage of Comet or WebSockets technologies. These are some initial notes:

* [http://httpd.apache.org/docs/2.2/mod/event.html Apache MPM event]: this experimental module included in Apache 2.2 has the potential to bring Twisted-esque funcionality within the Apache pipeline. It can save significant overhead in creating TCP connections, however, Apache traditionally keeps an entire child process/thread waiting for data from the client, which brings its own disadvantages. To solve this problem, this MPM uses a dedicated thread to handle both the listening sockets, and all sockets that are in a Keep Alive state.
* [http://www.tornadoweb.org/ Tornado]: an open source version of the scalable, non-blocking web server and tools that power FriendFeed. It is ideal for real-time web services. It is not just a web server, it could be considered a real-time web framework. [http://lincolnloop.com/blog/2009/sep/15/using-django-inside-tornado-web-server/ It can serves] Django applications.
* Twisted, eventlet, gevent, Tornado, Node.JS, greenlet, celery

Some recipes:

* [http://www.clemesha.org/blog/realtime-web-apps-python-django-orbited-twisted Django + Orbited + Twisted]: and an example, a realtime webapp: [https://github.com/clemesha/hotdot Hotdot].
* [http://media.eflorenzano.com/dropbox/UsingDjangoInNonStandardWays.pdf Using Django in Non-Standard Ways]
* [http://prg10001.blogspot.com/2009/09/simpler-long-polling-with-django-and.html Simple long polling with Django and gevent]
* [http://ajaxian.com/archives/django-and-comet Django and Comet using Orbited]
* [http://blog.gevent.org/2010/02/27/why-gevent/ Comparing gevent to eventlet]
* [http://stackoverflow.com/questions/1824418/a-clean-lightweight-alternative-to-pythons-twisted A clean, lightweight alternative to Python's twisted?]
* [http://stackoverflow.com/questions/4363899/making-moves-w-websockets-and-python-django-twisted Making moves w/ websockets and python/django(/twisted?)]
* [http://www.saltycrane.com/blog/2010/05/quick-notes-trying-twisted-websocket-branch-example/ Quick notes on trying the Twisted websocket branch example]

===Debug mode===

By default, the dashboard runs in "production" mode. To diagnose application errors it is usually useful to run in debug mode. Debug mode will display error messages. If you want to enable it, please follow these instructions:

# Go to [http://code.google.com/p/archivematica/source/browse/trunk#trunk%2Fsrc%2Fdashboard dashboard sources directory]
# Open settings.py file with your preferred text editor
# Find the following line <pre>Debug = False</pre>
# Update the False flag to True <pre>Debug = True</pre>
# Save the file
# Restart Apache <pre>sudo /etc/init.d/apache2 restart</pre>

[[Category:Development documentation]]

Storage Service API

2017-03-21T20:34:14Z

Jdunham: /* Create new pipeline */

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: IP or hostname of the pipeline. If not provided and <code>create_default_locations</code> is set, will try to populate from the IP of the request.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:
$ curl -X POST -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" -H"Content-Type: application/json" -d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", "description": "Test pipeline", "create_default_locations": true, "api_username": "demo", "api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' 'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Is there no way to create Spaces in the API?
|}

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Is there no way to create Locations in the API?
|}

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: URI of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the package path.
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Storage Service

2017-03-21T20:33:38Z

Jdunham: /* AIP Recovery */

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Storage Service

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

The Archivematica Storage Service is a standalone web application that handles moving files to Archivematica for processing, from Archivematica into long term storage, and keeps track of their location for later retrieval.

There are 2 main configuration levels in the Storage Service: Spaces and Locations.
* [[#Space | Space]]: '''where''' the files are stored. This is the protocol used to fetch and store the files in a storage system. Examples: Local filesystem, Duracloud. Spaces contain Locations.
* [[#Location | Location]]: '''why''' the files are there. This is what purpose Archivematica is using them for. Examples: Transfer Source, AIP Storage. Locations are inside Spaces.

[[Storage API | Initial design requirements]] and [[Storage Service API | API documentation]] also exist.

== Space ==

A storage Space contains all the information necessary to connect to the physical storage. It is '''where''' the files are stored. Protocol-specific information, like an NFS export path and hostname, or the username of a system accessible only via SSH, is stored here. All locations must be contained in a space.

Because Spaces deal with many different protocols and transportation needs, there are many different types of them. Each different Space type defines its own Django model (class), which has an associated Space instance.

For path-based spaces, the Space is the immediate parent of the Location folders. For example, if you had transfer source locations at <code>/home/artefactual/archivematica-sampledata-2013-10-10-09-17-20</code> and <code>/home/artefactual/maildir_transfers</code>, the Space’s path could be <code>/home/artefactual/</code>

All protocols require a staging path. This is a temporary location on the Storage Service server that is used when moving files. The storage service moves files by first copying them to the destination Space's staging directory, and then to the actual destination space. This reduces complexity, because each Space only needs to know how to get files between the locally-accessible staging directory & its own protocol, not between all other protocols.

There are 4 core methods that a Space should implement
* <code>browse</code>
** Allows seeing the files available or stored here.
* <code>move_to_storage_service</code>
** Moves files from the remote storage to the staging path
* <code>move_from_storage_service</code>
** Moves files from the staging path to the remote storage.
* <code>delete_path</code>
** Deletes files at a path.

Additionally, several other functions enable custom behaviour

* <code>post_move_to_storage_service</code>
** Allows post-processing after fetching files. E.g. putting a split package back together
* <code>post_move_from_storage_service</code>
** Allows post-processing after storing files. E.g. Notifying the storage system of the new files
* <code>update_package_status</code>
** Allow the status of a package in this system to be checked and updated. E.g. ensure replication has completed
* <code>check_package_fixity</code>
** Allow a space's fixity check to be called, instead of downloading and running bagit manually

Spaces can also be configured with a maximum quota to allow in that space, and track how much data has been stored in them through the storage service.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Quota and storage tracking is a stub feature, and does not properly track how much has been stored, especially with uncompressed AIPs and deleting AIPs.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, the Spaces are distinct models with a OnetoOneField back to Space. This was done because of warnings against concrete inheritance in models [https://jacobian.org/writing/concrete-inheritance/] [http://stackoverflow.com/questions/23466577/should-i-avoid-multi-table-concrete-inheritance-in-django-by-any-means]. However, in the Storage Service we never want to access a child space without also accessing its parent, so that concern is probably not founded. A better future design would use concrete multi-table inheritance for the different types of Spaces.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: When originally written, the only Spaces conceived of were path-based (local filesystem, NFS, etc), so the Space/Location information reflected that. However, most new Spaces are object based, or otherwise don’t use the Space.path & Location.relative_path, and shouldn’t use os.path.join to join the two. A better future design would move all path-related features out of Space into LocalFilesystem etc and remove the implicit os.path.join with Location.relative_path
|}

Spaces are sorted alphabetically in the docs.

=== Arkivum ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': ARKIVUM

This uses Arkivum's A-Stor. A-Stor exposes a CIFS share, which is mounted on the Storage Service and treated like a local filesystem. After files are copied to the share, a release request is sent to A-Stor to start its internal processing (copying the files to multiple datapools). While the mount is exposed as a CIFS share, it is only imitating the behaviour. The files may not exist, and the package info must be checked before accessing files to ensure they are actually present and avoid long waits.

=== Dataverse ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]]
* '''Database code''': DV

Dataverse is a prototype Space that requires support in Archivematica that is [https://github.com/artefactual/archivematica/pull/347 not yet merged]. It uses the search interface to 'browse' datasets by querying with the provided path. It uses the returned JSON to fetch all files associated with that dataset, and the returned JSON is stored as <code>dataset.json</code>

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| When fetching datasets to start a transfer with, it assumes that the identifier is a digit, and the Location & Space paths contain no digits. This is a likely source of bugs.
|}

=== Duracloud ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]]
* '''Database code''': DC

A Duracloud Space corresponds with a Space in Archivematica, so to support multiple Duracloud Spaces, multiple storage service spaces must be created. The Location path is used as a prefix to the path. Duracloud is used in hosted Archivematica.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: The Location path should track which Space in Duracloud to upload to, instead of Duracloud.duraspace. This would also remove the unnecessary prefixes in paths when uploading. Care would have to be taken to migrate existing Duracloud configurations correctly. An optional path prefix could be useful. This would benefit from having Space.path removed first (see above improvement notes).
|}

=== DSpace ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': DSPACE

DSpace uses the SWORD2 API & DSpace REST API to upload the AIP. Before uploading, the AIP is split into two packages: one containing the objects, and one containing everything else (metadata, logs, bagit structure). It also uploads Dublin Core information to DSpace if available. For the Dublin Core upload to work, some configuration changes in DSpace are required.

To make DSpace fit with the path-based structure of Spaces, the <code>Space.path</code> is unused, and the service document URL is stored in the Service Document IRI field. The <code>Location.relative_path</code> is overloaded to represent the collection.

* Example service document IRI: http://dspace.institution.org:8080/swordv2/servicedocument
* Example location relative path: http://dspace.institution.org:8080/swordv2/collection/123456789/2

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: DSpace does not support fetching files from DSpace, so downloading the AIP, fixity check, and AIP reingest do not work. This should be implemented.
|}

=== FEDORA via SWORD2 ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#FEDORA Deposit|FEDORA Deposit]]
* '''Database code''': FEDORA

This offers a SWORD2 server API to allow another system (developed for [https://wiki.duraspace.org/display/ISLANDORA/Archidora Archidora], but could be others) to deposit content into Archivematica and trigger a Transfer. This is contrasted with the other Spaces, which require Archivematica to initiate contact. Examples and documentation at [[Sword API]].

=== Local Filesystem ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]], [[#Storage Service Internal | Storage Service Internal]]
* '''Database code''': FS

Local Filesystem spaces handle storage that is available locally on the machine running the storage service. This can be a hard drive, or a mounted remote filesystem. This is the default configured space.

=== LOCKSS-o-matic ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': LOM

This support storing AIPs in a LOCKSS network via LOCKSS-O-Matic, which uses SWORD to communicate between the Storage Service and a Private LOCKSS Network (PLN). The Space.path is used as a staging location when making files available for harvesting.

=== NFS ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]], [[#Storage Service Internal | Storage Service Internal]]
* '''Database code''': NFS

NFS is a stub space. It was intended to support auto mounting NFS shares, but is not significantly different from [[#Local Filesystem | Local Filesystem]]. Currently, NFS handling should be done outside of Archivematica.

=== Pipeline Local Filesystem ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]]
* '''Database code''': PIPE_FS

Pipeline Local Filesystems refer to the storage that is local to the Archivematica pipeline, but remote to the storage service. For this Space to work properly, passwordless SSH must be set up between the Storage Service host and the Archivematica host. This is the easiest way to support having the Pipeline and Storage Service on different machines.

=== Swift ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]]
* '''Database code''': SWIFT

This stores in OpenStack's Swift using the [https://pypi.python.org/pypi/python-swiftclient swiftclient] library.

== Locations ==

A storage Location is contained in a Space, and knows its purpose in the Archivematica system. This is '''why''' the files are there. A Location allows Archivematica to query for only storage that has been marked for a particular purpose.

Each Location should be associated with at least one pipeline. A pipeline can have multiple instances of any location, except for Backlog and Currently Processing locations which should only be one of. If you want the same directory on disk to have multiple purposes, multiple Locations with different purposes can be created.

Not all Spaces support all Location purposes. For example, several Spaces only allow AIP storage, because they are only suitable for long term storage and do not provide temporary storage (eg Transfer backlog) or easy access to files (eg Transfer source). When creating a new Location, only allowed purposes are selectable in the menu.

Locations are sorted by order of appearance in processing in the docs.

=== Transfer Source ===

* '''Purpose''': Input into Archivematica
* '''Required''': Yes
* '''Multiples allowed''': Yes
* '''Database code''': TS

Trasfer source locations are where Transfers can be started from and where metadata files can be added to a unit from. Transfer source locations display in Archivematica’s Transfer tab. Any folder in a transfer source can be selected to become a Transfer. The default value is <code>/home</code> in a Local Filesystem.

=== Currently Processing ===

* '''Purpose''': For Archivematica's internal processing
* '''Required''': Yes
* '''Multiples allowed''': No
* '''Database code''': CP

During processing, Archivematica uses the currently processing location associated with that pipeline. Exactly one currently processing location should be associated with a given pipeline. The default value is <code>/var/archivematica/sharedDirectory</code> in a Local Filesystem. This is required for Archivematica to run.

=== Transfer Backlog ===

* '''Purpose''': Store Transfers in backlog
* '''Required''': No (Yes if using Backlog)
* '''Multiples allowed''': No
* '''Database code''': BL

Transfer backlog stores transfers until such a time that the user continues processing them. The default value is <code>/var/archivematica/sharedDirectory/www/AIPsStore/transferBacklog</code> in a Local Filesystem. This is required to store and retrieve transfers in backlog.

=== AIP Storage ===

* '''Purpose''': Store AIPs for long term storage
* '''Required''': Yes
* '''Multiples allowed''': Yes
* '''Database code''': AS

AIP storage locations are where the completed AIPs are put for long-term storage. The default value is <code>/var/archivematica/sharedDirectory/www/AIPsStore</code> in a Local Filesystem. This is required to store and retrieve AIPs.

=== DIP Storage ===

* '''Purpose''': Store DIPs before uploading to access systems
* '''Required''': No
* '''Multiples allowed''': Yes
* '''Database code''': DS

DIP storage is used for storing DIPs until such a time that they can be uploaded to an access system. The default value is <code>/var/archivematica/sharedDirectory/www/DIPsStore</code> in a Local Filesystem. This is required to store and retrieve DIPs. This is not required to upload DIPs to access systems.

=== AIP Recovery ===

* '''Purpose''': Recover a corrupted AIP
* '''Required''': No
* '''Multiples allowed''': No
* '''Database code''': AR

AIP Recovery is where the AIP recovery feature looks for an AIP to recover. No more than one AIP recovery location should be associated with a given pipeline. The default value is <code>/var/archivematica/storage_service/recover</code> in a Local Filesystem. This is only required if AIP recovery is used.

Needs clarification: Is this for storing the corrupted AIP, or for storing a duplicate copy of the AIP from which it can be recovered, or for something else?

=== Storage Service Internal ===

* '''Purpose''': Internal staging area for the Storage Service
* '''Required''': Yes
* '''Multiples allowed''': No
* '''Database code''': SS
* '''Associated with a pipeline''': No

There should only be exactly one Storage Service Internal Processing location for each Storage Service installation. The default value is <code>/var/archivematica/storage_service</code> in a Local Filesystem. This is required for the Storage Service to run, and must be locally available to the storage service. It should not be associated with any pipelines.

=== FEDORA Deposit ===

* '''Purpose''': Store deposited transfers from Archidora before starting as a Transfer.
* '''Required''': No
* '''Multiples allowed''': Yes
* '''Database code''': SD

FEDORA Deposit is used with the Archidora plugin to ingest material from Islandora. This is only available to the FEDORA Space, and is only required for that space.

== Pipelines ==

Archivematica installations are tracked in the Storage Service as Pipelines. Locations are associated with Pipelines, which gives them access to that storage. If a Location is not associated with a Pipeline, it doesn't exist as far as that pipeline is concerned.

== Package ==

Packages are a file or directory (collection of files) that Archivematica knows about and can track. Most Packages are AIPs in long term storage, but they could also be Transfers in backlog or stored DIPs.

Most of the additional functionality in the storage service not directly related to moving files around is implemented on the Package class.

=== Status ===

Packages can have several different statuses, indicating where they are in terms of being stored.

* Upload Pending: Still on Archivematica
* Staged on Storage Service: In Storage Service staging directory
* Uploaded: In final storage location
* Verified: Verified to be in final storage location
* Failed: Error occurred - may or may not be at final location
* Delete requested: Delete requested, package state unchanged
* Deleted: Storage service tried to delete it, considers it gone
* Deposit Finalized: For SWORD API accepting deposits (unused for AIPs)

"Upload Pending" is set when storing an AIP, before it's been moved from the pipeline to the staging directory of the final destination.

"Staging" is set when storing an AIP after it's been moved to the staging directory of the destination Space.

"Uploaded" is set when storing an AIP after it has been moved to the final location, unless it's LOCKSS or Arkivum. Arkivum packages move to "Uploaded" once the replication status is "green", and LOCKSS once all server states are "agreement"

"Verified" exists but is never used.

"Failed" is the default state if none is set (eg if it doesn't get to Upload Pending), but is otherwise unused.

"Delete requested" is set when a request is deleted. Note that this replaces the previous state, so if the delete request is rejected the previous state is lost.

"Deleted" happens when a delete request is approved. The package entry still exists, but is assumed to be gone. A package can be in this state but still exist if the delete failed. The storage service doesn't enforce not interacting with deleted packages

"Deposit Finalized" is the state of a FEDORA transfer after the deposit is finalized and the transfer has been started.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Make "Uploaded" more consistent - state is always Uploaded after the AIP has been moved to the final location, and start using "Verified" for what Arkivum & LOCKSS currently use "Uploaded" for. Perhaps "Uploaded" should be renamed, since in the case of LOCKSS/Arkivum it may not have been actually uploaded yet. This is potentially confused though, because the 'completely stored' state (Uploaded vs Verified) is different depending on the Space.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Start using "Failed". Use cases might include Arkivum status going from Green to Red, or otherwise failing a fixity check.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Store the package state as set or a list. This would allow a package to be Uploaded and Delete requested, instead of losing that state. We could also mark something as both Uploaded and Verified, resolving how to handle Arkivum status. Failed could coexist with the last known good state.
|}

== API ==

See [[Storage Service API]]

== Troubleshooting ==

The Storage service keeps a log at /var/log/archivematica/storage-service.log and errors may also be logged to the nginx and uwsgi logs as well at: /var/log/uwsgi/app/storage.log

== See also ==

* [[Administrator manual 1.1#Storage service]]

[[Category:Development documentation]]

Storage Service

2017-03-21T20:32:16Z

Jdunham: /* Troubleshooting */

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Storage Service

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

The Archivematica Storage Service is a standalone web application that handles moving files to Archivematica for processing, from Archivematica into long term storage, and keeps track of their location for later retrieval.

There are 2 main configuration levels in the Storage Service: Spaces and Locations.
* [[#Space | Space]]: '''where''' the files are stored. This is the protocol used to fetch and store the files in a storage system. Examples: Local filesystem, Duracloud. Spaces contain Locations.
* [[#Location | Location]]: '''why''' the files are there. This is what purpose Archivematica is using them for. Examples: Transfer Source, AIP Storage. Locations are inside Spaces.

[[Storage API | Initial design requirements]] and [[Storage Service API | API documentation]] also exist.

== Space ==

A storage Space contains all the information necessary to connect to the physical storage. It is '''where''' the files are stored. Protocol-specific information, like an NFS export path and hostname, or the username of a system accessible only via SSH, is stored here. All locations must be contained in a space.

Because Spaces deal with many different protocols and transportation needs, there are many different types of them. Each different Space type defines its own Django model (class), which has an associated Space instance.

For path-based spaces, the Space is the immediate parent of the Location folders. For example, if you had transfer source locations at <code>/home/artefactual/archivematica-sampledata-2013-10-10-09-17-20</code> and <code>/home/artefactual/maildir_transfers</code>, the Space’s path could be <code>/home/artefactual/</code>

All protocols require a staging path. This is a temporary location on the Storage Service server that is used when moving files. The storage service moves files by first copying them to the destination Space's staging directory, and then to the actual destination space. This reduces complexity, because each Space only needs to know how to get files between the locally-accessible staging directory & its own protocol, not between all other protocols.

There are 4 core methods that a Space should implement
* <code>browse</code>
** Allows seeing the files available or stored here.
* <code>move_to_storage_service</code>
** Moves files from the remote storage to the staging path
* <code>move_from_storage_service</code>
** Moves files from the staging path to the remote storage.
* <code>delete_path</code>
** Deletes files at a path.

Additionally, several other functions enable custom behaviour

* <code>post_move_to_storage_service</code>
** Allows post-processing after fetching files. E.g. putting a split package back together
* <code>post_move_from_storage_service</code>
** Allows post-processing after storing files. E.g. Notifying the storage system of the new files
* <code>update_package_status</code>
** Allow the status of a package in this system to be checked and updated. E.g. ensure replication has completed
* <code>check_package_fixity</code>
** Allow a space's fixity check to be called, instead of downloading and running bagit manually

Spaces can also be configured with a maximum quota to allow in that space, and track how much data has been stored in them through the storage service.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Quota and storage tracking is a stub feature, and does not properly track how much has been stored, especially with uncompressed AIPs and deleting AIPs.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, the Spaces are distinct models with a OnetoOneField back to Space. This was done because of warnings against concrete inheritance in models [https://jacobian.org/writing/concrete-inheritance/] [http://stackoverflow.com/questions/23466577/should-i-avoid-multi-table-concrete-inheritance-in-django-by-any-means]. However, in the Storage Service we never want to access a child space without also accessing its parent, so that concern is probably not founded. A better future design would use concrete multi-table inheritance for the different types of Spaces.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: When originally written, the only Spaces conceived of were path-based (local filesystem, NFS, etc), so the Space/Location information reflected that. However, most new Spaces are object based, or otherwise don’t use the Space.path & Location.relative_path, and shouldn’t use os.path.join to join the two. A better future design would move all path-related features out of Space into LocalFilesystem etc and remove the implicit os.path.join with Location.relative_path
|}

Spaces are sorted alphabetically in the docs.

=== Arkivum ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': ARKIVUM

This uses Arkivum's A-Stor. A-Stor exposes a CIFS share, which is mounted on the Storage Service and treated like a local filesystem. After files are copied to the share, a release request is sent to A-Stor to start its internal processing (copying the files to multiple datapools). While the mount is exposed as a CIFS share, it is only imitating the behaviour. The files may not exist, and the package info must be checked before accessing files to ensure they are actually present and avoid long waits.

=== Dataverse ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]]
* '''Database code''': DV

Dataverse is a prototype Space that requires support in Archivematica that is [https://github.com/artefactual/archivematica/pull/347 not yet merged]. It uses the search interface to 'browse' datasets by querying with the provided path. It uses the returned JSON to fetch all files associated with that dataset, and the returned JSON is stored as <code>dataset.json</code>

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| When fetching datasets to start a transfer with, it assumes that the identifier is a digit, and the Location & Space paths contain no digits. This is a likely source of bugs.
|}

=== Duracloud ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]]
* '''Database code''': DC

A Duracloud Space corresponds with a Space in Archivematica, so to support multiple Duracloud Spaces, multiple storage service spaces must be created. The Location path is used as a prefix to the path. Duracloud is used in hosted Archivematica.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: The Location path should track which Space in Duracloud to upload to, instead of Duracloud.duraspace. This would also remove the unnecessary prefixes in paths when uploading. Care would have to be taken to migrate existing Duracloud configurations correctly. An optional path prefix could be useful. This would benefit from having Space.path removed first (see above improvement notes).
|}

=== DSpace ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': DSPACE

DSpace uses the SWORD2 API & DSpace REST API to upload the AIP. Before uploading, the AIP is split into two packages: one containing the objects, and one containing everything else (metadata, logs, bagit structure). It also uploads Dublin Core information to DSpace if available. For the Dublin Core upload to work, some configuration changes in DSpace are required.

To make DSpace fit with the path-based structure of Spaces, the <code>Space.path</code> is unused, and the service document URL is stored in the Service Document IRI field. The <code>Location.relative_path</code> is overloaded to represent the collection.

* Example service document IRI: http://dspace.institution.org:8080/swordv2/servicedocument
* Example location relative path: http://dspace.institution.org:8080/swordv2/collection/123456789/2

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: DSpace does not support fetching files from DSpace, so downloading the AIP, fixity check, and AIP reingest do not work. This should be implemented.
|}

=== FEDORA via SWORD2 ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#FEDORA Deposit|FEDORA Deposit]]
* '''Database code''': FEDORA

This offers a SWORD2 server API to allow another system (developed for [https://wiki.duraspace.org/display/ISLANDORA/Archidora Archidora], but could be others) to deposit content into Archivematica and trigger a Transfer. This is contrasted with the other Spaces, which require Archivematica to initiate contact. Examples and documentation at [[Sword API]].

=== Local Filesystem ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]], [[#Storage Service Internal | Storage Service Internal]]
* '''Database code''': FS

Local Filesystem spaces handle storage that is available locally on the machine running the storage service. This can be a hard drive, or a mounted remote filesystem. This is the default configured space.

=== LOCKSS-o-matic ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#AIP Storage | AIP Storage]]
* '''Database code''': LOM

This support storing AIPs in a LOCKSS network via LOCKSS-O-Matic, which uses SWORD to communicate between the Storage Service and a Private LOCKSS Network (PLN). The Space.path is used as a staging location when making files available for harvesting.

=== NFS ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]], [[#Storage Service Internal | Storage Service Internal]]
* '''Database code''': NFS

NFS is a stub space. It was intended to support auto mounting NFS shares, but is not significantly different from [[#Local Filesystem | Local Filesystem]]. Currently, NFS handling should be done outside of Archivematica.

=== Pipeline Local Filesystem ===

* '''Uses Space.path''': Yes
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Currently Processing | Currently Processing]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]], [[#AIP Recovery | AIP Recovery]]
* '''Database code''': PIPE_FS

Pipeline Local Filesystems refer to the storage that is local to the Archivematica pipeline, but remote to the storage service. For this Space to work properly, passwordless SSH must be set up between the Storage Service host and the Archivematica host. This is the easiest way to support having the Pipeline and Storage Service on different machines.

=== Swift ===

* '''Uses Space.path''': No
* '''Supported purposes''': [[#Transfer Source | Transfer Source]], [[#Transfer Backlog | Transfer Backlog]], [[#AIP Storage | AIP Storage ]], [[#DIP Storage | DIP Storage]]
* '''Database code''': SWIFT

This stores in OpenStack's Swift using the [https://pypi.python.org/pypi/python-swiftclient swiftclient] library.

== Locations ==

A storage Location is contained in a Space, and knows its purpose in the Archivematica system. This is '''why''' the files are there. A Location allows Archivematica to query for only storage that has been marked for a particular purpose.

Each Location should be associated with at least one pipeline. A pipeline can have multiple instances of any location, except for Backlog and Currently Processing locations which should only be one of. If you want the same directory on disk to have multiple purposes, multiple Locations with different purposes can be created.

Not all Spaces support all Location purposes. For example, several Spaces only allow AIP storage, because they are only suitable for long term storage and do not provide temporary storage (eg Transfer backlog) or easy access to files (eg Transfer source). When creating a new Location, only allowed purposes are selectable in the menu.

Locations are sorted by order of appearance in processing in the docs.

=== Transfer Source ===

* '''Purpose''': Input into Archivematica
* '''Required''': Yes
* '''Multiples allowed''': Yes
* '''Database code''': TS

Trasfer source locations are where Transfers can be started from and where metadata files can be added to a unit from. Transfer source locations display in Archivematica’s Transfer tab. Any folder in a transfer source can be selected to become a Transfer. The default value is <code>/home</code> in a Local Filesystem.

=== Currently Processing ===

* '''Purpose''': For Archivematica's internal processing
* '''Required''': Yes
* '''Multiples allowed''': No
* '''Database code''': CP

During processing, Archivematica uses the currently processing location associated with that pipeline. Exactly one currently processing location should be associated with a given pipeline. The default value is <code>/var/archivematica/sharedDirectory</code> in a Local Filesystem. This is required for Archivematica to run.

=== Transfer Backlog ===

* '''Purpose''': Store Transfers in backlog
* '''Required''': No (Yes if using Backlog)
* '''Multiples allowed''': No
* '''Database code''': BL

Transfer backlog stores transfers until such a time that the user continues processing them. The default value is <code>/var/archivematica/sharedDirectory/www/AIPsStore/transferBacklog</code> in a Local Filesystem. This is required to store and retrieve transfers in backlog.

=== AIP Storage ===

* '''Purpose''': Store AIPs for long term storage
* '''Required''': Yes
* '''Multiples allowed''': Yes
* '''Database code''': AS

AIP storage locations are where the completed AIPs are put for long-term storage. The default value is <code>/var/archivematica/sharedDirectory/www/AIPsStore</code> in a Local Filesystem. This is required to store and retrieve AIPs.

=== DIP Storage ===

* '''Purpose''': Store DIPs before uploading to access systems
* '''Required''': No
* '''Multiples allowed''': Yes
* '''Database code''': DS

DIP storage is used for storing DIPs until such a time that they can be uploaded to an access system. The default value is <code>/var/archivematica/sharedDirectory/www/DIPsStore</code> in a Local Filesystem. This is required to store and retrieve DIPs. This is not required to upload DIPs to access systems.

=== AIP Recovery ===

* '''Purpose''': Recover a corrupted AIP
* '''Required''': No
* '''Multiples allowed''': No
* '''Database code''': AR

AIP Recovery is where the AIP recovery feature looks for an AIP to recover. No more than one AIP recovery location should be associated with a given pipeline. The default value is <code>/var/archivematica/storage_service/recover</code> in a Local Filesystem. This is only required if AIP recovery is used.

Needs clarification: Is this to stored the corrupted AIP, or stores a duplicated copy of the AIP so it can be recovered from, or something else?

=== Storage Service Internal ===

* '''Purpose''': Internal staging area for the Storage Service
* '''Required''': Yes
* '''Multiples allowed''': No
* '''Database code''': SS
* '''Associated with a pipeline''': No

There should only be exactly one Storage Service Internal Processing location for each Storage Service installation. The default value is <code>/var/archivematica/storage_service</code> in a Local Filesystem. This is required for the Storage Service to run, and must be locally available to the storage service. It should not be associated with any pipelines.

=== FEDORA Deposit ===

* '''Purpose''': Store deposited transfers from Archidora before starting as a Transfer.
* '''Required''': No
* '''Multiples allowed''': Yes
* '''Database code''': SD

FEDORA Deposit is used with the Archidora plugin to ingest material from Islandora. This is only available to the FEDORA Space, and is only required for that space.

== Pipelines ==

Archivematica installations are tracked in the Storage Service as Pipelines. Locations are associated with Pipelines, which gives them access to that storage. If a Location is not associated with a Pipeline, it doesn't exist as far as that pipeline is concerned.

== Package ==

Packages are a file or directory (collection of files) that Archivematica knows about and can track. Most Packages are AIPs in long term storage, but they could also be Transfers in backlog or stored DIPs.

Most of the additional functionality in the storage service not directly related to moving files around is implemented on the Package class.

=== Status ===

Packages can have several different statuses, indicating where they are in terms of being stored.

* Upload Pending: Still on Archivematica
* Staged on Storage Service: In Storage Service staging directory
* Uploaded: In final storage location
* Verified: Verified to be in final storage location
* Failed: Error occurred - may or may not be at final location
* Delete requested: Delete requested, package state unchanged
* Deleted: Storage service tried to delete it, considers it gone
* Deposit Finalized: For SWORD API accepting deposits (unused for AIPs)

"Upload Pending" is set when storing an AIP, before it's been moved from the pipeline to the staging directory of the final destination.

"Staging" is set when storing an AIP after it's been moved to the staging directory of the destination Space.

"Uploaded" is set when storing an AIP after it has been moved to the final location, unless it's LOCKSS or Arkivum. Arkivum packages move to "Uploaded" once the replication status is "green", and LOCKSS once all server states are "agreement"

"Verified" exists but is never used.

"Failed" is the default state if none is set (eg if it doesn't get to Upload Pending), but is otherwise unused.

"Delete requested" is set when a request is deleted. Note that this replaces the previous state, so if the delete request is rejected the previous state is lost.

"Deleted" happens when a delete request is approved. The package entry still exists, but is assumed to be gone. A package can be in this state but still exist if the delete failed. The storage service doesn't enforce not interacting with deleted packages

"Deposit Finalized" is the state of a FEDORA transfer after the deposit is finalized and the transfer has been started.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Make "Uploaded" more consistent - state is always Uploaded after the AIP has been moved to the final location, and start using "Verified" for what Arkivum & LOCKSS currently use "Uploaded" for. Perhaps "Uploaded" should be renamed, since in the case of LOCKSS/Arkivum it may not have been actually uploaded yet. This is potentially confused though, because the 'completely stored' state (Uploaded vs Verified) is different depending on the Space.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Start using "Failed". Use cases might include Arkivum status going from Green to Red, or otherwise failing a fixity check.
|}

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Store the package state as set or a list. This would allow a package to be Uploaded and Delete requested, instead of losing that state. We could also mark something as both Uploaded and Verified, resolving how to handle Arkivum status. Failed could coexist with the last known good state.
|}

== API ==

See [[Storage Service API]]

== Troubleshooting ==

The Storage service keeps a log at /var/log/archivematica/storage-service.log and errors may also be logged to the nginx and uwsgi logs as well at: /var/log/uwsgi/app/storage.log

== See also ==

* [[Administrator manual 1.1#Storage service]]

[[Category:Development documentation]]