[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:
$ curl -X POST -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" -H"Content-Type: application/json" -d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", "description": "Test pipeline", "create_default_locations": true, "api_username": "demo", "api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' 'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Is there no way to create Spaces in the API?
|}

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the package path.
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Dataverse

2018-05-15T19:43:21Z

Joel-simpson: Added future considerations section

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse].

This page was originally created as part of an early Proof of Concept integration in 2017, which was only made available in a development branch of Archivematica. We have now started a phase 2 project to improve on that original integration work and merge it into a public release of Archivematica (exact release tbc). This work is being sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Current Status==

'''May 11, 2018'''
To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:

* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]

==Feature Files==
On this project we are using [http://docs.behat.org/en/v2.5/guides/1.gherkin.html Gherkin] feature files to define the desired behaviour of preserving a dataset from a Dataverse. Feature files are also known as Acceptance Tests, because they specify the behaviour that we will test at the end of the project.

The early drafts are documented in this google doc: [http://docs.behat.org/en/v2.5/guides/1.gherkin.html]
Once the draft has been reviewed we will publish it to our acceptance test repository in github.

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
This section is from the first phase project in 2017 and needs to be updated.

*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===
This section is from the first phase project in 2017 and needs to be updated.

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>

== Future Requirements & Considerations ==
This section includes working notes for future phases, as interesting opportunities or questions arise. At the end of the current phase we will be documenting the integration as well as future opportunities.

=== Notes from Feature File review meeting on May 1 2018 (2pm EST) ===

'''Choice & Versioning of Dataverse API:'''
The dataverse Search and Access APIs are not currently versioned.
The Native API is versioned: http://guides.dataverse.org/en/latest/api/native-api.html
There is an OAI-PMH interface (although it is not mentioned in the dataverse API guide). Amber said there were idiosyncrasies in the way dataverse implemented PMH, and wasn’t sure it would be a ‘safe’ option.
Amaz would like to see that we are either using a standard API (like OAI-PMH) or a versioned API.
Amaz thought wondered whether we could use PMH with the polling part of the solution; but given what Amber said, it doesn’t seem like a good way to go)
So as part of the project we need to see whether we could use the Native API (even if we don’t actually use it), or we need to raise it as an issue to discuss with the dataverse team.

'''Relationships between Datasets'''
Amber pointed out that they are not currently clear exactly what datasets should be preserved, and expects this will vary quite a bit by institution.
We discussed the question of whether all datasets in a dataverse would be preserved (not currently known), which brought up the question of how to relate datasets.
We talked about AICs as one possible solution. But agreed that it’s a new feature and needs to be thought through… there could be other solutions than AIC.

'''Improving agent info in event history in METS'''
We pointed out that having an agent other than Archivematica in the METS is a new feature
Discussed the fact that we could make this even more specific by adding more agents. For instance, differentiating between the researcher who uploaded files from the research data manager who published the dataset.

'''Notes from Dataverse Testing:'''

Should a preserved dataset include an equivalent of fixity check on any UNFs created by Dataverse?
https://dataverse.scholarsportal.info/guides/en/4.8.6/developers/unf/index.html#unf
Universal Numerical Fingerprint (UNF) is a unique signature of the semantic content of a digital object. It is not simply a checksum of a binary data file. Instead, the UNF algorithm approximates and normalizes the data stored within. A cryptographic hash of that normalized (or canonicalized) representation is then computed.

Dataverse

2018-05-15T19:19:43Z

Joel-simpson: Added link to feature file

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse].

This page was originally created as part of an early Proof of Concept integration in 2017, which was only made available in a development branch of Archivematica. We have now started a phase 2 project to improve on that original integration work and merge it into a public release of Archivematica (exact release tbc). This work is being sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Current Status==

'''May 11, 2018'''
To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:

* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]

==Feature Files==
On this project we are using [http://docs.behat.org/en/v2.5/guides/1.gherkin.html Gherkin] feature files to define the desired behaviour of preserving a dataset from a Dataverse. Feature files are also known as Acceptance Tests, because they specify the behaviour that we will test at the end of the project.

The early drafts are documented in this google doc: [http://docs.behat.org/en/v2.5/guides/1.gherkin.html]
Once the draft has been reviewed we will publish it to our acceptance test repository in github.

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
This section is from the first phase project in 2017 and needs to be updated.

*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===
This section is from the first phase project in 2017 and needs to be updated.

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>