Archivematica - User contributions [en]

Archivematica API

2021-03-02T14:41:28Z

Rspencer: Add api prefix to jobs and tasks call

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>).
 
''As GET parameters, format is:'' 
<code>?username=<''username''>&api_key=<''api_key''></code> 

''As a header, format is:'' 
<code>ApiKey <username>:<api_key></code> 

'''NOTE:''' The 'ApiKey' prefix is a requirement of the Tastypie library in use. See the Tastypie docs for [https://django-tastypie.readthedocs.io/en/latest/authentication.html#apikeyauthentication ApiKeyAuthentication].'' 

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/api/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"
'''Note:''' for consumers of this endpoint, it is possible for Archivematica to return a <code>status</code> of <code>COMPLETE</code> without a <code>sip_uuid</code>. Consumers looking to use the UUID of the AIP that will be created following Ingest should therefore test for both a <code>status</code> of <code>COMPLETE</code> and the existence of <code>sip_uuid</code> that does not also equal <code>BACKLOG</code> to ensure that they retrieve it. This might mean an additional call to the status endpoint while this data becomes available.

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Return a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Unit ==

''Note: Currently, you can only query strings in English.''

=== List jobs ===
* '''URL''': <code>/api/v2beta/jobs/<unit UUID>/</code>
* '''Verb''': GET
* Return a list of jobs for the passed unit (transfer or ingest).
* '''Parameters''':
* '''Optional filters''':
*** <code>microservice</code>: Name of the microservice the jobs belong to
*** <code>link_uuid</code>: UUID of the job chain link
*** <code>name</code>: Name of the job
* '''Response''': JSON body
** '''List of dicts with keys''':
*** <code>uuid</code>: UUID of the job
*** <code>name</code>: Name of the job
*** <code>status</code>: One of USER_INPUT, PROCESSING, COMPLETE, FAILED or UNKNOWN
*** <code>microservice</code>: Microservice the job belongs to
*** <code>link_uuid</code>: UUID of the job chain link
*** <code>tasks</code>: List of dicts with information about the microservice's tasks:
*** <code>uuid</code>: UUID of the task
*** <code>exit_code</code>: Exit code of the task

=== Task ===
* '''URL''': <code>/api/v2beta/task/<task UUID>/</code>
* '''Verb''': GET
* Return information about a task.
* '''Response''': JSON body
** <code>uuid</code>: UUID of the task
** <code>exit_code</code>: Exit code of the task
** <code>file_uuid</code>: UUID of the file used for the task
** <code>file_name</code>: File used for the task,
** <code<time_created</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task was created
** <code>time_started</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task started
** <code>time_ended</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task finished
** <code>duration</code>: Task duration in seconds (integer). If the duration is less than a second, this will be a < 1 string

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>true</code> or <code>false</code> to set the transfer to auto-approve, default: <code>true</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID ('''Note:''' as the package endpoint allows the caller to interact with Archivematica asynchronously it doesn't guarantee a transfer has started. The caller must use the UUID in the response to verify it has begun or any errors that were encountered initiating one.)
'''Examples:'''
 
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
 
''Starting a transfer using an absolute path:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransferCSV' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransferCSV' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>

=== Validate ===

Available in Archivematica 1.10+. This endpoint can be used to validate CSVs against embedded sets of rules. Currently works with Avalon Media System Manifest files.

In the future, this endpoint can be extended to support validation for <code>metadata.csv</code> or <code>rights.csv</code>, or other institutionally-based rules.

* '''URL''': <code>/api/v2beta/validate/avalon</code>
* '''Content-Type''': text/csv; charset=utf-8
* '''Parameters''': <code><document></code>

Usage example: (assuming that <code>avalon.csv</code> exists)

<pre>
curl http://127.0.0.1:62080/api/v2beta/validate/avalon \
--data-binary "@avalon.csv" \
--header "Authorization: ApiKey test:test" \
--header "Content-Type: text/csv; charset=utf-8"
</pre>

Response examples:

200 OK

<pre>
{
"valid": true
}
</pre>

400 Bad Request (expect reason to include different validation errors)

<pre>
{
"valid": false,
"reason": "Administrative data must include reference name and author."
}
</pre>

404 Not Found

<pre>
{
"error": true,
"message": "Unknown validator. Accepted values: avalon"
}
</pre>

[[Category:Development documentation]]

Contribute code

2020-07-15T23:32:52Z

Rspencer: Update sphinx link

[[Main Page]] > [[Development]] > Contribute code

Expanded contribution guidelines can also be found in the Archivematica [https://github.com/artefactual/archivematica/blob/qa/1.x/CONTRIBUTING.md contributing document].

[[Category:Process documentation]]

==Patches==

If you find a bug in this project or would like to make an enhancement, please be encouraged to contribute code via a pull request. Archivematica code is available on github. We will accept git pull requests from individuals who have submitted a [[Contributor Agreement]].

For more information please visit:
* https://help.github.com/articles/creating-a-pull-request
* https://help.github.com/articles/using-pull-requests
* https://help.github.com/articles/fork-a-repo

==Commit access==
Anyone can contribute code patches to this project. Project collaborators and regular patch contributors will be given access to commit directly to the git code repository.

==Contributor's Agreement==
In order to accept any patches or code commits, contributors must first sign the [[Contributor Agreement|Archivematica Contributor's Agreement]].

==Standards==

=== Code Style Guide For Archivematica ===

Archivematica follows common Python coding standards, linked below. We encourage installing a Python [http://stackoverflow.com/questions/8503559/what-is-linting linter] to help with this. [https://pypi.python.org/pypi/flake8 flake8] is recommended because it wraps three common linters (pep8, pyflakes, mccabe).
* [https://www.python.org/dev/peps/pep-0008/ PEP8] for styling
** Exception is line length, which should be wrapped at 79 characters or left long for the IDE to wrap.
** Imports are sorted alphabetically within their grouping to reduce duplicate imports
* [https://www.python.org/dev/peps/pep-0257/ PEP257] for docstring structure
** Parameters and return values should be specified in [http://sphinx-doc.org/domains.html#info-field-lists Sphinx-style].
* See also [https://github.com/artefactual/archivematica/pull/218 this pull request] and associated discussion.

Example docstrings with Sphinx markup below. Other attributes that can be used in a docstring can be found [https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#python-signatures on the Sphinx website].

<pre>def get_unit_status(unit_uuid, unit_type):
"""
Get status for a SIP or Transfer.

Returns a dict with status info. Keys will always include 'status' and
'microservice', and may include 'sip_uuid'.

Status is one of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING.
Microservice is the name of the current microservice.
SIP UUID is populated only if the unit_type was unitTransfer and status is
COMPLETE. Otherwise, it is None.

:param str unit_uuid: UUID of the SIP or Transfer
:param str unit_type: unitSIP or unitTransfer
:return: Dict with status info.
"""</pre>

<pre>def each_child(path, file_group_identifier, base_path, base_path_name, sip_uuid):
"""
Iterates over entries in a filesystem, beginning at `path`.

At each entry in the filesystem, yields either a File model instance
(for files) or a string (for directories).

When iterating, makes two passes: first iterating over files, then
directories. Does not iterate over directories; consumers should
call this function again on directory strings to recurse.

:param string path: path to a directory on disk to recurse into.
:raises ValueError: if the specified path does not exist, or is not a directory.
"""
</pre>

=== File Structure ===
The file structure in Archivematica will comply with the Filesystem Hierarchy Standard (FHS).
 More information on this standard is available at:
 http://www.pathname.com/fhs/

Research data management

2020-04-28T14:36:50Z

Rspencer: Update to GH link

[[Main Page]] > [[Documentation]] > [[Requirements]] > Research data management

=About=

This page describes requirements for enhancements to Archivematica to better handle research data management. It is funded by [https://www.jisc.ac.uk/ Jisc], through University of York and University of Hull.

'''See also'''

*[[Dataset preservation]]
*[[Dataverse]]
*[http://digital-archiving.blogspot.ca Digital Archiving blog] written by archivists at University of York
*[https://github.com/archivematica/archivematica-case-studies/blob/master/resources/2016-02-01-JiscDataSpringFinalReportphase2.pdf Filling the Digital Preservation Gap] written by archivists at the Universities of York and Hull



=Automated DIP generation=

The tasks related to this phase of development are:

* change workflow so that the ‘upload DIP’ choice can be preconfigured.
* update AIP reingest workflow to allow uncompressed AIPs to be reingested.
* enhance the callback functionality in the Storage Service, to notify third party apps when a DIP is ready to be used.

These developments are necessary for research data management to meet use cases when research data is stored without the expectation that it will be re-used, but then subsequently a need for re-use arises and/or is approved by the creator of the data.



==Workflow==

[[File:York_DIP_gen_v1.png|820px]]

*Access request is initiated through a staff alert to RDMonitor
*Request for DIP sent to Storage Service
**If DIP exists already in storage, return DIP
**If DIP does not exist, send wait response and initiate AIP re-ingest in Archivematica pipeline
*When DIP stored by Archivematica,return DIP message goes to RDMonitor
*If DIP creation/storage fails, failure message sent to RDMonitor.



=METS parsing=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in this package?
|
|integer
|-
|How many original/preservation/metadata files are in this package?
|mets:fileGrp USE="original" / mets:fileGrp USE="preservation" / mets:fileGrp USE="metadata"
|integer
|-
|What is the total volume of files in this package?
|premis:size
|integer and unit, eg 17 GB
|-
|Is there a DIP for this package?
|Storage Service
|Location of DIP
|-
|How many files with PRONOM puid X are in this package?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in this package?
|premis:formatName
|integer
|-
|How many files with PRONOM puid/format name X have been normalized?
|premis:formatRegistryKey / premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid/format name X have not been normalized?
|premis:formatRegistryKey / premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|Does the package include descriptive metadata?
|mets:dmdSec
|Yes or No
|-
|Does the package include rights metadata?
|mets:rightsMD
|Yes or No
|-
|How many files are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with PRONOM puid/format name X are invalid/not well-formed?
|premis:formatRegistryKey / premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|What is the directory structure of the files in this package?
|mets:structMap
|?
|-
|What is the size of file X?
|premis:size
|integer and unit, eg 2.2 GB
|-
|What is the checksum format for file X?
|premis:messageDigestAlgorithm
|md5, sha256
|-
|What is the checksum for file X?
|premis:messageDigest
|d32d41f7481afc1ab48779e2608v08d93b5d05cd217a4372e6a93957767ae651
|-
|Has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|Yes or no
|-
|To what format has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:formatName for preservation copy
|TIFF
|-
|When was file X normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:eventType:creation and premis:eventDateTime for preservation copy
|date
|-
|-
|}



=Generic search REST API=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in archival storage?
|Storage Service
|integer
|-
|How many AIPs are in archival storage?
|Storage Service
|integer
|-
|How many files are in a specified AIP?
|Storage Service
|integer
|-
|How many files with PRONOM puid X are in archival storage?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in archival storage?
|premis:formatName
|integer
|-
|How many [video, image, plain text etc.] files are in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? Alternatively, FPR groups?
|integer
|-
|What is the total volume of files with PRONOM puid X in archival storage?
|premis:formatRegistryKey; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of files with format name X in archival storage?
|premis:formatName; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of [video, image, plain text etc.] files in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? FPR groups?; premis:size
|integer and unit, eg 452 GB
|-
|How many files with PRONOM puid X have been normalized?
|premis:formatRegistryKey; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid X have not been normalized?
|premis:formatRegistryKey; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have been normalized?
|premis:premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have not been normalized?
|premis:premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have not been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files were ingested between date X and date Y?
|<premis:eventType>ingestion</premis:eventType>; premis:eventDateTime
|integer
|-
|How many files with PRONOM puid X are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with format name X are invalid/not well-formed?
|premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many AIPs have corresponding DIPs?
|Storage Service
|integer
|-
|How many AIPs do not have corresponding DIPs?
|Storage Service
|integer
|-
|}



=Multiple checksum algorithms=

=Enhance PRONOM integration=

Priorities for this phase are (in order of priority):

1. Provide report of non-identified files in a SIP or AIP, with access to the file identification tool output

2. Provide direct access to the PRONOM submission form from within Archivematica

Alternatives being discussed:

- Post tool output and optionally a sample file to the FPR, making that available publically and to PRONOM

3. Allow a user to manually assign pronom IDs to non-identified files; record manual selection in the AIP METS file

=Automation tools documentation=

Archivematica has features that enable quite a bit of automation and interaction from 3rd party applications, but most of that functionality is hidden and not well documented. The goal of this part of the work on improving Research data management workflows is about adding documentation, in written form and as screencasts/videos, aimed at developers, and other technical users of Archivematica. This will make it easier for developers working on other applications to integrate Archivematica into their organizations digital preservation workflows.

Potential topics

* introduction to the Archivematica
* setting up an Archivematica development
* automation tools introduction
* implementing an automated workflow in
* managing an Archivematica installation
* configuration and troubleshooting
* api documentation

Research data management

2020-04-28T14:00:07Z

Rspencer: Update refs

[[Main Page]] > [[Documentation]] > [[Requirements]] > Research data management

=About=

This page describes requirements for enhancements to Archivematica to better handle research data management. It is funded by [https://www.jisc.ac.uk/ Jisc], through University of York and University of Hull.

'''See also'''

*[[Dataset preservation]]
*[[Dataverse]]
*[http://digital-archiving.blogspot.ca Digital Archiving blog] written by archivists at University of York
*[https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/3702676/JiscDataSpringFinalReportphase2.pdf Filling the Digital Preservation Gap] written by archivists at the Universities of York and Hull



=Automated DIP generation=

The tasks related to this phase of development are:

* change workflow so that the ‘upload DIP’ choice can be preconfigured.
* update AIP reingest workflow to allow uncompressed AIPs to be reingested.
* enhance the callback functionality in the Storage Service, to notify third party apps when a DIP is ready to be used.

These developments are necessary for research data management to meet use cases when research data is stored without the expectation that it will be re-used, but then subsequently a need for re-use arises and/or is approved by the creator of the data.



==Workflow==

[[File:York_DIP_gen_v1.png|820px]]

*Access request is initiated through a staff alert to RDMonitor
*Request for DIP sent to Storage Service
**If DIP exists already in storage, return DIP
**If DIP does not exist, send wait response and initiate AIP re-ingest in Archivematica pipeline
*When DIP stored by Archivematica,return DIP message goes to RDMonitor
*If DIP creation/storage fails, failure message sent to RDMonitor.



=METS parsing=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in this package?
|
|integer
|-
|How many original/preservation/metadata files are in this package?
|mets:fileGrp USE="original" / mets:fileGrp USE="preservation" / mets:fileGrp USE="metadata"
|integer
|-
|What is the total volume of files in this package?
|premis:size
|integer and unit, eg 17 GB
|-
|Is there a DIP for this package?
|Storage Service
|Location of DIP
|-
|How many files with PRONOM puid X are in this package?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in this package?
|premis:formatName
|integer
|-
|How many files with PRONOM puid/format name X have been normalized?
|premis:formatRegistryKey / premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid/format name X have not been normalized?
|premis:formatRegistryKey / premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|Does the package include descriptive metadata?
|mets:dmdSec
|Yes or No
|-
|Does the package include rights metadata?
|mets:rightsMD
|Yes or No
|-
|How many files are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with PRONOM puid/format name X are invalid/not well-formed?
|premis:formatRegistryKey / premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|What is the directory structure of the files in this package?
|mets:structMap
|?
|-
|What is the size of file X?
|premis:size
|integer and unit, eg 2.2 GB
|-
|What is the checksum format for file X?
|premis:messageDigestAlgorithm
|md5, sha256
|-
|What is the checksum for file X?
|premis:messageDigest
|d32d41f7481afc1ab48779e2608v08d93b5d05cd217a4372e6a93957767ae651
|-
|Has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|Yes or no
|-
|To what format has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:formatName for preservation copy
|TIFF
|-
|When was file X normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:eventType:creation and premis:eventDateTime for preservation copy
|date
|-
|-
|}



=Generic search REST API=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in archival storage?
|Storage Service
|integer
|-
|How many AIPs are in archival storage?
|Storage Service
|integer
|-
|How many files are in a specified AIP?
|Storage Service
|integer
|-
|How many files with PRONOM puid X are in archival storage?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in archival storage?
|premis:formatName
|integer
|-
|How many [video, image, plain text etc.] files are in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? Alternatively, FPR groups?
|integer
|-
|What is the total volume of files with PRONOM puid X in archival storage?
|premis:formatRegistryKey; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of files with format name X in archival storage?
|premis:formatName; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of [video, image, plain text etc.] files in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? FPR groups?; premis:size
|integer and unit, eg 452 GB
|-
|How many files with PRONOM puid X have been normalized?
|premis:formatRegistryKey; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid X have not been normalized?
|premis:formatRegistryKey; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have been normalized?
|premis:premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have not been normalized?
|premis:premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have not been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files were ingested between date X and date Y?
|<premis:eventType>ingestion</premis:eventType>; premis:eventDateTime
|integer
|-
|How many files with PRONOM puid X are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with format name X are invalid/not well-formed?
|premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many AIPs have corresponding DIPs?
|Storage Service
|integer
|-
|How many AIPs do not have corresponding DIPs?
|Storage Service
|integer
|-
|}



=Multiple checksum algorithms=

=Enhance PRONOM integration=

Priorities for this phase are (in order of priority):

1. Provide report of non-identified files in a SIP or AIP, with access to the file identification tool output

2. Provide direct access to the PRONOM submission form from within Archivematica

Alternatives being discussed:

- Post tool output and optionally a sample file to the FPR, making that available publically and to PRONOM

3. Allow a user to manually assign pronom IDs to non-identified files; record manual selection in the AIP METS file

=Automation tools documentation=

Archivematica has features that enable quite a bit of automation and interaction from 3rd party applications, but most of that functionality is hidden and not well documented. The goal of this part of the work on improving Research data management workflows is about adding documentation, in written form and as screencasts/videos, aimed at developers, and other technical users of Archivematica. This will make it easier for developers working on other applications to integrate Archivematica into their organizations digital preservation workflows.

Potential topics

* introduction to the Archivematica
* setting up an Archivematica development
* automation tools introduction
* implementing an automated workflow in
* managing an Archivematica installation
* configuration and troubleshooting
* api documentation

Research data management

2020-04-28T13:56:29Z

Rspencer: Added final report

[[Main Page]] > [[Documentation]] > [[Requirements]] > Research data management

=About=

This page describes requirements for enhancements to Archivematica to better handle research data management. It is funded by [https://www.jisc.ac.uk/ Jisc], through University of York and University of Hull.

'''See also'''

*[[Dataset preservation]]
*[[Dataverse]]
*[http://digital-archiving.blogspot.ca Digital Archiving blog] written by archivists at University of York
*[https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/3702676/JiscDataSpringFinalReportphase2.pdf Filling the Digital Preservation Gap] written by archivists at University of York



=Automated DIP generation=

The tasks related to this phase of development are:

* change workflow so that the ‘upload DIP’ choice can be preconfigured.
* update AIP reingest workflow to allow uncompressed AIPs to be reingested.
* enhance the callback functionality in the Storage Service, to notify third party apps when a DIP is ready to be used.

These developments are necessary for research data management to meet use cases when research data is stored without the expectation that it will be re-used, but then subsequently a need for re-use arises and/or is approved by the creator of the data.



==Workflow==

[[File:York_DIP_gen_v1.png|820px]]

*Access request is initiated through a staff alert to RDMonitor
*Request for DIP sent to Storage Service
**If DIP exists already in storage, return DIP
**If DIP does not exist, send wait response and initiate AIP re-ingest in Archivematica pipeline
*When DIP stored by Archivematica,return DIP message goes to RDMonitor
*If DIP creation/storage fails, failure message sent to RDMonitor.



=METS parsing=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in this package?
|
|integer
|-
|How many original/preservation/metadata files are in this package?
|mets:fileGrp USE="original" / mets:fileGrp USE="preservation" / mets:fileGrp USE="metadata"
|integer
|-
|What is the total volume of files in this package?
|premis:size
|integer and unit, eg 17 GB
|-
|Is there a DIP for this package?
|Storage Service
|Location of DIP
|-
|How many files with PRONOM puid X are in this package?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in this package?
|premis:formatName
|integer
|-
|How many files with PRONOM puid/format name X have been normalized?
|premis:formatRegistryKey / premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid/format name X have not been normalized?
|premis:formatRegistryKey / premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|Does the package include descriptive metadata?
|mets:dmdSec
|Yes or No
|-
|Does the package include rights metadata?
|mets:rightsMD
|Yes or No
|-
|How many files are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with PRONOM puid/format name X are invalid/not well-formed?
|premis:formatRegistryKey / premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|What is the directory structure of the files in this package?
|mets:structMap
|?
|-
|What is the size of file X?
|premis:size
|integer and unit, eg 2.2 GB
|-
|What is the checksum format for file X?
|premis:messageDigestAlgorithm
|md5, sha256
|-
|What is the checksum for file X?
|premis:messageDigest
|d32d41f7481afc1ab48779e2608v08d93b5d05cd217a4372e6a93957767ae651
|-
|Has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|Yes or no
|-
|To what format has file X been normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:formatName for preservation copy
|TIFF
|-
|When was file X normalized for preservation?
|matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"; premis:eventType:creation and premis:eventDateTime for preservation copy
|date
|-
|-
|}



=Generic search REST API=

==METS questions==

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:25%"|'''Question'''
!style="width:40%"|'''Information source'''
!style="width:25%%"|'''Sample result'''
|-
|How many files are in archival storage?
|Storage Service
|integer
|-
|How many AIPs are in archival storage?
|Storage Service
|integer
|-
|How many files are in a specified AIP?
|Storage Service
|integer
|-
|How many files with PRONOM puid X are in archival storage?
|premis:formatRegistryKey
|integer
|-
|How many files with format name X are in archival storage?
|premis:formatName
|integer
|-
|How many [video, image, plain text etc.] files are in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? Alternatively, FPR groups?
|integer
|-
|What is the total volume of files with PRONOM puid X in archival storage?
|premis:formatRegistryKey; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of files with format name X in archival storage?
|premis:formatName; premis:size
|integer and unit, eg 452 GB
|-
|What is the total volume of [video, image, plain text etc.] files in archival storage?
|fits:mimetype, File:MIMEType, other mimetype sources? FPR groups?; premis:size
|integer and unit, eg 452 GB
|-
|How many files with PRONOM puid X have been normalized?
|premis:formatRegistryKey; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with PRONOM puid X have not been normalized?
|premis:formatRegistryKey; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have been normalized?
|premis:premis:formatName; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files with format name X have not been normalized?
|premis:premis:formatName; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''with''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many [video, image, plain text etc.] files have not been normalized?
|fits:mimetype, File:MIMEType, other mimetype sources?; files '''without''' matching GROUPID attributes in mets:fileGrp USE="original" and mets:fileGrp USE="preservation"
|integer
|-
|How many files were ingested between date X and date Y?
|<premis:eventType>ingestion</premis:eventType>; premis:eventDateTime
|integer
|-
|How many files with PRONOM puid X are invalid/not well-formed?
|premis:formatRegistryKey; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many files with format name X are invalid/not well-formed?
|premis:formatName; <premis:eventType>validation</premis:eventType>; <premis:eventOutcome>fail</premis:eventOutcome>
|integer
|-
|How many AIPs have corresponding DIPs?
|Storage Service
|integer
|-
|How many AIPs do not have corresponding DIPs?
|Storage Service
|integer
|-
|}



=Multiple checksum algorithms=

=Enhance PRONOM integration=

Priorities for this phase are (in order of priority):

1. Provide report of non-identified files in a SIP or AIP, with access to the file identification tool output

2. Provide direct access to the PRONOM submission form from within Archivematica

Alternatives being discussed:

- Post tool output and optionally a sample file to the FPR, making that available publically and to PRONOM

3. Allow a user to manually assign pronom IDs to non-identified files; record manual selection in the AIP METS file

=Automation tools documentation=

Archivematica has features that enable quite a bit of automation and interaction from 3rd party applications, but most of that functionality is hidden and not well documented. The goal of this part of the work on improving Research data management workflows is about adding documentation, in written form and as screencasts/videos, aimed at developers, and other technical users of Archivematica. This will make it easier for developers working on other applications to integrate Archivematica into their organizations digital preservation workflows.

Potential topics

* introduction to the Archivematica
* setting up an Archivematica development
* automation tools introduction
* implementing an automated workflow in
* managing an Archivematica installation
* configuration and troubleshooting
* api documentation

External tools

2019-12-13T02:16:22Z

Rspencer: Adding __NOINDEX__

__NOINDEX__
[[Main Page]] > [[Documentation]] > External tools

<blockquote style="background-color:orange;">
'''Note: The documentation on this page is out of date. Please see the [https://www.archivematica.org/docs official documentation page] for the latest.'''
</blockquote>

=External software tools used in Archivematica=

==Current release==

Archivematica integrates a suite of free and open-source tools that allows users to process digital objects from ingest to access based on the ISO-OAIS functional model. In addition to the core software which is released under AGPL v3 license, the following tools are bundled with Archivematica:

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''Tool'''
!style="width:40%"|'''Description'''
!style="width:30%"|'''License'''
|-
|-
|[http://www.accesstomemory.org AtoM]
|Web-based archival description and access tool.
|GNU General Public License (GPL)
|-
|[https://github.com/LibraryOfCongress/bagit-java/ BagIt]
|Standard and script to package digital objects and metadata for archival storage
|BSD License
|-
|[https://github.com/simsong/bulk_extractor bulk_extractor]
|Disk image and file contents analysis tool
|Public domain
|-
|[http://www.clamav.net/ Clam AV (anti-virus)]
|Anti-virus toolkit for UNIX
|GNU General Public License (GPL)
|-
|[https://www.djangoproject.com/ Django]
|Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
|BSD License
|-
|[http://www.elasticsearch.org/ ElasticSearch]
|Indexing and search
|Apache License 2.0
|-
|[http://www.sno.phy.queensu.ca/~phil/exiftool/index.html ExifTool]
|Multimedia metadata extraction
|GNU General Public License and Artistic License
|-
|[http://ffmpeg.org/ FFmpeg]
|Converts a wide variety of audio and video formats
|GNU Lesser General Public License (LGPL)
|-
|[https://github.com/openplanets/fido fido]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[https://projects.iq.harvard.edu/fits File Information Tool Set (FITS)]
|File format identification and validation software integration tool
|GNU Lesser General Public License (LGPL)
|-
|[http://gearman.org/ Gearman]
|Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work.
|BSD License
|-
|[http://www.imagemagick.org/script/index.php Imagemagick]
|Converts a wide variety of bitmap images
|GPL compatible [http://www.imagemagick.org/script/license.php Imagemagick license]
|-
|[http://www.inkscape.org/ Inkscape]
|Converts vector images to Scalable Vector Graphic (SVG) format
|GNU General Public License (GPL) version 2
|-
|[http://jhove.sourceforge.net/ JHOVE]
|File format validation tool
|GNU Lesser General Public License (LGPL)
|-
|[http://md5deep.sourceforge.net/ md5deep]
|Checksum generation and verification scripts
|GNU General Public License (GPL)
|-
|[https://mediaarea.net/MediaConch MediaConch]
|File policy conformance checker tool
|BSD (2-clause)
|-
|[https://mediaarea.net/en/MediaInfo MediaInfo]
|Multimedia metadata extraction
|BSD (2-clause), Zlib
|-
|[https://pkgs.org/download/nfs-common NFS-common]
|Network File System Access - allows access to files on network storage devices.
|GNU General Public License (GPL)
|-
|[http://p7zip.sourceforge.net/ p7zip]
|7-Zip is a file archiver with a high compression ratio. (LZMA)
|GNU General Public License (GPL)
|-
|[https://lxml.de/ Python-lxml]
|Python binding for libxml2 and libxslt
|GNU General Public License (GPL)
|-
|[https://github.com/richardlehane/siegfried Seigfried]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[http://www.sleuthkit.org/ Sleuthkit]
|Disk image management and extraction toolkit
|Common Public License / IBM Public License
|-
|[http://code.google.com/p/tesseract-ocr/ Tesseract]
|Optical Character Recognition -read image files and convert to text
|Apache Licence Vesion 2
|-
|[https://ubuntu.com/ Ubuntu Linux]
|Interface with computing hardware. Ubuntu Linux server edition.
|GNU General Public License (GPL)
|-
|[http://www.ossp.org/pkg/lib/uuid/ UUID]
|command line interface (CLI) for the generation of DCE 1.1, ISO/IEC 11578:1996 and IETF RFC-4122 compliant Universally Unique Identifier (UUID).
|GNU General Public License (GPL)
|-
|[http://unarchiver.c3.cx/commandline unar]
|The Unarchiver is an archive unpacker program
|GNU General Public License (GPL)
|-
|[http://manpages.ubuntu.com/manpages/hardy/man1/zip.1.html Zip]
|Utility called by Bagit to create AIP package
|Info-Zip license: "Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely"
|-
|}



==Past releases==

'''[[Release 1.6.0]]''' (March 2017)

'''[[Release 1.5]]''' (June 2016)

'''[[Release 1.4.1]]''' (June 2015)

'''[[Release 1.4]]''' (May 2015)

'''[[Release 1.3.1]]''' (February 2015)

'''[[Release 1.0]]''' (September 2014)

'''[[Release 0.10-beta]]''' (April 2013)

'''[[Release 0.9-beta]]''' (Aug 2012)

'''[[Release 0.8-alpha]]''' (Jan 2012)

'''[[Release 0.7.1-alpha]]''' (June 2011)

'''[[Release 0.7-alpha]]''' (Feb 2011)

'''[[Release 0.6.2-alpha]]''' (November 2010)

'''[[Release 0.6-alpha]]''' (May 2010)

'''[[Release 0.5-alpha]]''' (November 2009)

'''[[Release 0.4-alpha]]''' (October 2009)

'''[[Release 0.3.5-alpha]]''' (September 2009)

'''[[Release 0.3-alpha]]''' (June 2009)

'''[[Release 0.2-alpha]]''' (May 2009)

'''[[Release 0.1-alpha]]''' (Feb 2009)

External tools

2019-12-13T01:59:01Z

Rspencer: Remove __NOINDEX__

[[Main Page]] > [[Documentation]] > External tools

<blockquote style="background-color:orange;">
'''Note: The documentation on this page is out of date. Please see the [https://www.archivematica.org/docs official documentation page] for the latest.'''
</blockquote>

=External software tools used in Archivematica=

==Current release==

Archivematica integrates a suite of free and open-source tools that allows users to process digital objects from ingest to access based on the ISO-OAIS functional model. In addition to the core software which is released under AGPL v3 license, the following tools are bundled with Archivematica:

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''Tool'''
!style="width:40%"|'''Description'''
!style="width:30%"|'''License'''
|-
|-
|[http://www.accesstomemory.org AtoM]
|Web-based archival description and access tool.
|GNU General Public License (GPL)
|-
|[https://github.com/LibraryOfCongress/bagit-java/ BagIt]
|Standard and script to package digital objects and metadata for archival storage
|BSD License
|-
|[https://github.com/simsong/bulk_extractor bulk_extractor]
|Disk image and file contents analysis tool
|Public domain
|-
|[http://www.clamav.net/ Clam AV (anti-virus)]
|Anti-virus toolkit for UNIX
|GNU General Public License (GPL)
|-
|[https://www.djangoproject.com/ Django]
|Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
|BSD License
|-
|[http://www.elasticsearch.org/ ElasticSearch]
|Indexing and search
|Apache License 2.0
|-
|[http://www.sno.phy.queensu.ca/~phil/exiftool/index.html ExifTool]
|Multimedia metadata extraction
|GNU General Public License and Artistic License
|-
|[http://ffmpeg.org/ FFmpeg]
|Converts a wide variety of audio and video formats
|GNU Lesser General Public License (LGPL)
|-
|[https://github.com/openplanets/fido fido]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[https://projects.iq.harvard.edu/fits File Information Tool Set (FITS)]
|File format identification and validation software integration tool
|GNU Lesser General Public License (LGPL)
|-
|[http://gearman.org/ Gearman]
|Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work.
|BSD License
|-
|[http://www.imagemagick.org/script/index.php Imagemagick]
|Converts a wide variety of bitmap images
|GPL compatible [http://www.imagemagick.org/script/license.php Imagemagick license]
|-
|[http://www.inkscape.org/ Inkscape]
|Converts vector images to Scalable Vector Graphic (SVG) format
|GNU General Public License (GPL) version 2
|-
|[http://jhove.sourceforge.net/ JHOVE]
|File format validation tool
|GNU Lesser General Public License (LGPL)
|-
|[http://md5deep.sourceforge.net/ md5deep]
|Checksum generation and verification scripts
|GNU General Public License (GPL)
|-
|[https://mediaarea.net/MediaConch MediaConch]
|File policy conformance checker tool
|BSD (2-clause)
|-
|[https://mediaarea.net/en/MediaInfo MediaInfo]
|Multimedia metadata extraction
|BSD (2-clause), Zlib
|-
|[https://pkgs.org/download/nfs-common NFS-common]
|Network File System Access - allows access to files on network storage devices.
|GNU General Public License (GPL)
|-
|[http://p7zip.sourceforge.net/ p7zip]
|7-Zip is a file archiver with a high compression ratio. (LZMA)
|GNU General Public License (GPL)
|-
|[https://lxml.de/ Python-lxml]
|Python binding for libxml2 and libxslt
|GNU General Public License (GPL)
|-
|[https://github.com/richardlehane/siegfried Seigfried]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[http://www.sleuthkit.org/ Sleuthkit]
|Disk image management and extraction toolkit
|Common Public License / IBM Public License
|-
|[http://code.google.com/p/tesseract-ocr/ Tesseract]
|Optical Character Recognition -read image files and convert to text
|Apache Licence Vesion 2
|-
|[https://ubuntu.com/ Ubuntu Linux]
|Interface with computing hardware. Ubuntu Linux server edition.
|GNU General Public License (GPL)
|-
|[http://www.ossp.org/pkg/lib/uuid/ UUID]
|command line interface (CLI) for the generation of DCE 1.1, ISO/IEC 11578:1996 and IETF RFC-4122 compliant Universally Unique Identifier (UUID).
|GNU General Public License (GPL)
|-
|[http://unarchiver.c3.cx/commandline unar]
|The Unarchiver is an archive unpacker program
|GNU General Public License (GPL)
|-
|[http://manpages.ubuntu.com/manpages/hardy/man1/zip.1.html Zip]
|Utility called by Bagit to create AIP package
|Info-Zip license: "Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely"
|-
|}



==Past releases==

'''[[Release 1.6.0]]''' (March 2017)

'''[[Release 1.5]]''' (June 2016)

'''[[Release 1.4.1]]''' (June 2015)

'''[[Release 1.4]]''' (May 2015)

'''[[Release 1.3.1]]''' (February 2015)

'''[[Release 1.0]]''' (September 2014)

'''[[Release 0.10-beta]]''' (April 2013)

'''[[Release 0.9-beta]]''' (Aug 2012)

'''[[Release 0.8-alpha]]''' (Jan 2012)

'''[[Release 0.7.1-alpha]]''' (June 2011)

'''[[Release 0.7-alpha]]''' (Feb 2011)

'''[[Release 0.6.2-alpha]]''' (November 2010)

'''[[Release 0.6-alpha]]''' (May 2010)

'''[[Release 0.5-alpha]]''' (November 2009)

'''[[Release 0.4-alpha]]''' (October 2009)

'''[[Release 0.3.5-alpha]]''' (September 2009)

'''[[Release 0.3-alpha]]''' (June 2009)

'''[[Release 0.2-alpha]]''' (May 2009)

'''[[Release 0.1-alpha]]''' (Feb 2009)

External tools

2019-12-12T18:59:17Z

Rspencer: Adding __NOINDEX__

External tools

2019-12-12T18:58:39Z

Rspencer: Testing removing __NOINDEX__

External tools

2019-12-12T18:58:07Z

Rspencer: Adding __NOINDEX__

__NOINDEX__

[[Main Page]] > [[Documentation]] > External tools

<blockquote style="background-color:orange;">
'''Note: The documentation on this page is out of date. Please see the [https://www.archivematica.org/docs official documentation page] for the latest.'''
</blockquote>

=External software tools used in Archivematica=

==Current release==

Archivematica integrates a suite of free and open-source tools that allows users to process digital objects from ingest to access based on the ISO-OAIS functional model. In addition to the core software which is released under AGPL v3 license, the following tools are bundled with Archivematica:

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''Tool'''
!style="width:40%"|'''Description'''
!style="width:30%"|'''License'''
|-
|-
|[http://www.accesstomemory.org AtoM]
|Web-based archival description and access tool.
|GNU General Public License (GPL)
|-
|[https://github.com/LibraryOfCongress/bagit-java/ BagIt]
|Standard and script to package digital objects and metadata for archival storage
|BSD License
|-
|[https://github.com/simsong/bulk_extractor bulk_extractor]
|Disk image and file contents analysis tool
|Public domain
|-
|[http://www.clamav.net/ Clam AV (anti-virus)]
|Anti-virus toolkit for UNIX
|GNU General Public License (GPL)
|-
|[https://www.djangoproject.com/ Django]
|Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
|BSD License
|-
|[http://www.elasticsearch.org/ ElasticSearch]
|Indexing and search
|Apache License 2.0
|-
|[http://www.sno.phy.queensu.ca/~phil/exiftool/index.html ExifTool]
|Multimedia metadata extraction
|GNU General Public License and Artistic License
|-
|[http://ffmpeg.org/ FFmpeg]
|Converts a wide variety of audio and video formats
|GNU Lesser General Public License (LGPL)
|-
|[https://github.com/openplanets/fido fido]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[https://projects.iq.harvard.edu/fits File Information Tool Set (FITS)]
|File format identification and validation software integration tool
|GNU Lesser General Public License (LGPL)
|-
|[http://gearman.org/ Gearman]
|Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work.
|BSD License
|-
|[http://www.imagemagick.org/script/index.php Imagemagick]
|Converts a wide variety of bitmap images
|GPL compatible [http://www.imagemagick.org/script/license.php Imagemagick license]
|-
|[http://www.inkscape.org/ Inkscape]
|Converts vector images to Scalable Vector Graphic (SVG) format
|GNU General Public License (GPL) version 2
|-
|[http://jhove.sourceforge.net/ JHOVE]
|File format validation tool
|GNU Lesser General Public License (LGPL)
|-
|[http://md5deep.sourceforge.net/ md5deep]
|Checksum generation and verification scripts
|GNU General Public License (GPL)
|-
|[https://mediaarea.net/MediaConch MediaConch]
|File policy conformance checker tool
|BSD (2-clause)
|-
|[https://mediaarea.net/en/MediaInfo MediaInfo]
|Multimedia metadata extraction
|BSD (2-clause), Zlib
|-
|[https://pkgs.org/download/nfs-common NFS-common]
|Network File System Access - allows access to files on network storage devices.
|GNU General Public License (GPL)
|-
|[http://p7zip.sourceforge.net/ p7zip]
|7-Zip is a file archiver with a high compression ratio. (LZMA)
|GNU General Public License (GPL)
|-
|[https://lxml.de/ Python-lxml]
|Python binding for libxml2 and libxslt
|GNU General Public License (GPL)
|-
|[https://github.com/richardlehane/siegfried Seigfried]
|File format identification tool
|Licensed under the Apache License, Version 2.0 (the "License")
|-
|[http://www.sleuthkit.org/ Sleuthkit]
|Disk image management and extraction toolkit
|Common Public License / IBM Public License
|-
|[http://code.google.com/p/tesseract-ocr/ Tesseract]
|Optical Character Recognition -read image files and convert to text
|Apache Licence Vesion 2
|-
|[https://ubuntu.com/ Ubuntu Linux]
|Interface with computing hardware. Ubuntu Linux server edition.
|GNU General Public License (GPL)
|-
|[http://www.ossp.org/pkg/lib/uuid/ UUID]
|command line interface (CLI) for the generation of DCE 1.1, ISO/IEC 11578:1996 and IETF RFC-4122 compliant Universally Unique Identifier (UUID).
|GNU General Public License (GPL)
|-
|[http://unarchiver.c3.cx/commandline unar]
|The Unarchiver is an archive unpacker program
|GNU General Public License (GPL)
|-
|[http://manpages.ubuntu.com/manpages/hardy/man1/zip.1.html Zip]
|Utility called by Bagit to create AIP package
|Info-Zip license: "Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely"
|-
|}



==Past releases==

'''[[Release 1.6.0]]''' (March 2017)

'''[[Release 1.5]]''' (June 2016)

'''[[Release 1.4.1]]''' (June 2015)

'''[[Release 1.4]]''' (May 2015)

'''[[Release 1.3.1]]''' (February 2015)

'''[[Release 1.0]]''' (September 2014)

'''[[Release 0.10-beta]]''' (April 2013)

'''[[Release 0.9-beta]]''' (Aug 2012)

'''[[Release 0.8-alpha]]''' (Jan 2012)

'''[[Release 0.7.1-alpha]]''' (June 2011)

'''[[Release 0.7-alpha]]''' (Feb 2011)

'''[[Release 0.6.2-alpha]]''' (November 2010)

'''[[Release 0.6-alpha]]''' (May 2010)

'''[[Release 0.5-alpha]]''' (November 2009)

'''[[Release 0.4-alpha]]''' (October 2009)

'''[[Release 0.3.5-alpha]]''' (September 2009)

'''[[Release 0.3-alpha]]''' (June 2009)

'''[[Release 0.2-alpha]]''' (May 2009)

'''[[Release 0.1-alpha]]''' (Feb 2009)

MediaWiki:Robots.txt

2019-12-12T18:48:25Z

Rspencer: Updating rendering

# <source lang="robots">
# Trialing an Archivematica Wiki robots.txt
# Edit via: https://wiki.archivematica.org/index.php?title=MediaWiki:Robots.txt&action=edit
# Test this works by looking up Google: site:https://wiki.archivematica.org/
Disallow: /Meeting_2015*
Disallow: /Meeting_2014*
Disallow: /Meeting_2013*
Disallow: /Meeting_2012*
Disallow: /Meeting_2011*

MediaWiki:Robots.txt

2019-12-12T18:46:53Z

Rspencer: Creating a robots.txt

# Trialing an Archivematica Wiki robots.txt
# Edit via: https://wiki.archivematica.org/index.php?title=MediaWiki:Robots.txt&action=edit
# Test this works by looking up Google: site:https://wiki.archivematica.org/
Disallow: /Meeting_2015*
Disallow: /Meeting_2014*
Disallow: /Meeting_2013*
Disallow: /Meeting_2012*
Disallow: /Meeting_2011*

Release Process

2019-08-04T21:43:33Z

Rspencer: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python2 -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* See this pull request submitted to update to PRONOM 90: https://github.com/openpreserve/fido/pull/110
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request.
# Release new version of FIDO. Make sure to update __init__.py AND the versions XML with a new version before creating a release.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55
</div>

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python testproject/manage.py dumpdata fpr > fpr-current.json</code>
# Generate a blank data migration
#* E.g. <code>python testproject/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python testproject/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v89.xml --output-filename pronom89.txt</code>
# Copy the output into the blank migration above the Migration class
# Add the below RunPython operation into the Migration class, in the operations list
#* <code>migrations.RunPython(data_migration),</code>
# <s>Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports</s>
#* <s>Remove the <code>apps.get_model</code> lines</s>
#* <s>Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code></s>
# Commit
# Deploy on testing pipeline or locally

(Analysts)
# Update the new entries
#* '''Ask to edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly'''
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analysts)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python testproject/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python testproject/manage.py get_fpr_changes fpr-current.json fpr-updated.json fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove the direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge
# Make new FPR release. Don't forget to update fpr/__init__.py! Then tag the release (e.g. `v.1.7.9`) and push the latest tag.
# Update AM's dashboard base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/requirements/base.txt]

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via gitolite (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin (for now; hopefully Jenkins for 1.8) builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Archivematica API

2019-04-07T16:58:12Z

Rspencer: /* Status */ add information about the status: COMPLETE

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"
'''Note:''' for consumers of this endpoint, it is possible for Archivematica to return a <code>status</code> of <code>COMPLETE</code> without a <code>sip_uuid</code>. Consumers looking to use the UUID of the AIP that will be created following Ingest should therefore test for both a <code>status</code> of <code>COMPLETE</code> and the existence of <code>sip_uuid</code> that does not also equal <code>BACKLOG</code> to ensure that they retrieve it. This might mean an additional call to the status endpoint while this data becomes available.

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID ('''Note:''' as the package endpoint allows the caller to interact with Archivematica asynchronously it doesn't guarantee a transfer has started. The caller must use the UUID in the response to verify it has begun or any errors that were encountered initiating one.)
'''Examples:'''
 
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
 
''Starting a transfer using an absolute path:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

AIP pointer file

2019-04-04T19:40:43Z

Rspencer: Added information about purpose

Pointer files have been used to help provide sustainable access to compressed AIPs since Archivematica 1.0. [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ [1]]. The documentation about them, below, is currently in progress.

== Purpose ==

Pointer files can help users ascertain information about their AIPs without needing to consult the Storage Service API or database.

As an example, pointer files include:

* Information about compression, currently used internally by Archivematica itself for Re-ingest or unpacking for delivery via the API
* Information about encryption so that AIPs can be recovered if encrypted.
* Information about where the AIP is stored relative to the storage service. NB. there are some variations, see below.

=== Describing the location of the AIP ===

Information about the location of an AIP is stored in a <pre>mets:filesec</pre> inside the pointer file:

<pre>
<mets:fileSec>
<mets:fileGrp USE="Archival Information Package">
<mets:file ID="rs_s3-cc53bf2e-0eb9-4d00-a752-8f07c2886fe9"
GROUPID="Group-cc53bf2e-0eb9-4d00-a752-8f07c2886fe9"
ADMID="amdSec_698926">
<mets:FLocat xlink:href="/var/archivematica/storage_service/aipstore/cc53/.../ex1-cc53bf2e-0eb9-4d00-a752-8f07c2886fe9.7z"
LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>
<mets:transformFile TRANSFORMTYPE="decompression" TRANSFORMORDER="1" TRANSFORMALGORITHM="bzip2"/>
</mets:file>
</mets:fileGrp>
</mets:fileSec>
</pre>

In some storage options that can be configured via the storage service, such as LOCKSS you will see information about the remote location of the AIP:

<pre>
<mets:fileSec>
<mets:fileGrp USE="Archival Information Package">
<mets:file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6">
<mets:FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<mets:transformFile TRANSFORMALGORITHM="bzip2" TRANSFORMORDER="1" TRANSFORMTYPE="decompression"/>
</mets:file>
</mets:fileGrp>
</mets:fileSec>
</pre>

'''Note:''' Not all storage options will show the information about the remote location. Either this information isn't as easy to replay to the user, or it is functionality that was beyond the scope of development of a storage adapter at the time. S3 is a current example where the user would see the path as it is shown above: <pre>/var/archivematica/storage_service/aipstore/cc53/.../ex1-cc53bf2e-0eb9-4d00-a752-8f07c2886fe9.7z</pre> and they would need to ascertain for themselves that the location was relative to the S3 location created for use as an package store online.

== Accessing a pointer file ==

Accessing pointer files inside Archivematica can be done by navigating the Dashboard to '''Archival storage''' finding a suitable AIP, e.g. at time of writing a compressed AIP, and under '''Pointer file''' clicking '''View'''. Archivematica will make a call to the Storage Service API to return the file for that AIP. More information can be found about this in the Archivematica documentation under [https://www.archivematica.org/en/docs/archivematica-1.9/user-manual/archival-storage/archival-storage/#storing-an-aip Storing an AIP].

* [https://wiki.archivematica.org/Storage_Service_API#Get_pointer_file Storage Service API reference to return a Pointer File]

== Developer notes ==

=== Location of the pointer file inside Storage Service ===

Pointer files are stored in the Storage Service directories, for example:

/var/archivematica/storage_service/d5c3/3360/7b92/4a2f/9002/32f5/a143/daaf/pointer.d5c33360-7b92-4a2f-9002-32f5a143daaf.xml

You can see where they are stored looking up the Storage Service database:

<source lang="sql">
select locations_location.uuid,
locations_location.relative_path,
locations_location.description,
locations_location.purpose
from locations_location
inner join locations_package
on locations_location.uuid =
locations_package.pointer_file_location_id
</source>

== Additional references ==

* [https://wiki.archivematica.org/LOCKSS_Integration LOCKSS integration]

== Sample Pointer Files ==

* [https://wiki.archivematica.org/Dataset_preservation#Sample_pointer.xml_file Sample XML Pointer File]
* [https://wiki.archivematica.org/LOCKSS_Integration#Sample_pointer_files LOCKSS Integration Pointer File]

== External Links ==

* [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ Justin Simpson on the structure and purpose of pointer files at Google Groups]
* [https://projects.artefactual.com/issues/5159 Feature #5159 The original tranche of work resulting in the implementation of pointer files]

Archivematica API

2019-03-24T23:10:46Z

Rspencer: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID ('''Note:''' as the package endpoint allows the caller to interact with Archivematica asynchronously it doesn't guarantee a transfer has started. The caller must use the UUID in the response to verify it has begun or any errors that were encountered initiating one.)
'''Examples:'''
 
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
 
''Starting a transfer using an absolute path:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Archivematica API

2019-02-26T21:08:56Z

Rspencer: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** <code>name</code>: Transfer name
** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
** <code>accession</code>: Accession ID
** <code>accession_system_id</code>: TBC
** <code>path</code>: Path relative, or absolute to a storage service transfer source
** <code>metadata_set_id</code>: TBC
** <code>auto_apptove</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
** <code>wait_until_complete</code>: TBC
* '''Response''': JSON
** <code>id</code>: Transfer UUID
'''Examples:'''
 
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
 
''Starting a transfer using an absolute path:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
 
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Archivematica API

2019-02-26T21:02:51Z

Rspencer: /* Package */

Archivematica API

2019-02-26T20:54:52Z

Rspencer: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** <code>name</code>: Transfer name
** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
** <code>accession</code>: Accession ID
** <code>accession_system_id</code>: _
** <code>path</code>: Path relative, or absolute to a storage service transfer source
** <code>metadata_set_id</code>: _
** <code>auto_apptove</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
** <code>wait_until_complete</code>: _
* '''Response''': JSON
** <code>id</code>: "Transfer UUID"
'''Examples:'''
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and other's from which a transfer can be initiated is that a storage service transfer location UUID isn't always required. In some cases that might still be ideal.
''Starting a transfer using an absolute path:''
<code>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://127.0.0.1:62080/api/v2beta/package"
</code>
''Starting a transfer using an relative path with a transfer source UUID:''
<code>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://127.0.0.1:62080/api/v2beta/package"
</code>

[[Category:Development documentation]]

Archivematica API

2019-02-26T19:56:01Z

Rspencer: Add header to add content underneath

Storage Service API

2019-02-05T13:27:28Z

Rspencer: /* Create new pipeline */ formatting correction

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:

$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the package path.
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Storage Service API

2019-02-05T13:26:33Z

Rspencer: /* Create new pipeline */ split command onto multiple lines for readability

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:
$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the package path.
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Merging

2019-01-18T08:38:30Z

Rspencer: /* Merge process */

[[Main Page]] > [[Development]] > Merging & Branches

== Branching guidelines ==

Guidelines on when to create branches for new development in Archivematica projects. Short answer: always create branches!

* All development work happens in branches - nothing is committed directly to master, qa/ or stable/ branches
* Development branches should follow the naming format: <code>dev/issue-####-short-description</code>
** <code>dev/</code> marks it as being a development branch, not for QA integration or a stable release
** <code>issue-####</code> is the Redmine ticket that the work is mostly related to. If the branch expands in scope to include multiple tickets later, that's fine, and don't worry about renaming branches
** <code>short-description</code> is a couple word description of the branch, to make it easier to remember what the topic of the branch is.
** Example: <code>dev/issue-8161-dashboard-i18n</code> or <code>dev/issue-8693-dataverse</code>
* Development branches should generally be create from the branch they will be merged into. In most cases, this is master, qa/1.x (Archivematica) or qa/0.x (Storage Service). If something is a patch for an old release, it may be more appropriate to branch from the old stable branch (e.g. stable/1.5.x)
* All branches should undergo code review before being merged into the target branch. If something was developed for an old release, it should be cherry-picked to the current development branch, and a comment with the commit hash should be left in the pull request for tracking.

== Code review & approval ==

All code changes merged into a stable or qa branch must go through code review. We use Github's pull request feature for code review. Every change being submitted to an Artefactual project should be submitted as a pull request to the appropriate repository.

Another core contributor (usually an Artefactual employee) will review the code for content, style and suitability in the project. A more extensive guideline to code review is [https://github.com/artefactual/archivematica/blob/qa/1.x/code_review.md also available]. After the reviewer is satisfied with the code, they will approve it on github. If at least one reviewer approves, and there are no disapprovals, the branch can be merged.

In general, the original contributor will [[#Merge process | merge the branch]]. For external contributions, the reviewer can merge or cherry-pick the commits. This will leave the Author correct in git, but update the committer.

== Merge process ==

We use a rebase-based workflow, where branches are brought up to date with the latest stable by rebasing them. (This is compared to a merge-based workflow, where branches are merged, and conflicts resolved in the merge commit). This shows the process for merging development branch <code>dev/test</code> into target branch <code>qa/1.x</code>. The target branch is usually qa/1.x.

The instructions are formatted: <code>(current branch) $ <command to run></code>

# Check your remote is set correctly
#* <code>$ git remote -v</code> should show <code>origin https://github.com/<organisation>/<repository-name></code>
#* Archivematica uses GitHub now as its primary host with a mirror in GitLab so it would be very rare that your <code>origin</code> would be something different.
# Ensure your copy of the target branch is up to date, in most cases this will be <code>qa/1.x</code>
#* <code>$ git checkout qa/1.x</code>
#* <code>(master) $ git pull --rebase </code>
# Ensure your development branch is based off the latest target branch
#* <code>$ git checkout dev/test</code>
#* <code>(dev/test) $ git rebase qa/1.x</code>
# Check that your changes still work after rebasing
# Check tests pass:
#* In the docker-compose environment use the <code>MakeFile</code> tests, e.g. <code>make test-mcp-client</code>
#* Push your rebased development branch back to GitHub and ensure the Travis CI checks pass before merging (see below).
# Push your newly-rebased branch to remote. Because this is a rebase, which changes history, you'll need --force. The rebased dev branch has to be pushed before the master branch is or github gets confused.
#* '''Never force push on a stable/, qa/ or master branch!'''
#* <code>(dev/test) $ git push -f</code>
# Fast-forward merge
#* <code>$ git checkout qa/1.x</code>
#* <code>(master) $ git merge dev/test</code>
# Push merge!
#* <code>(master) $ git push</code>
# Delete dev branch
#* Locally: <code>(master) $ git branch -d dev/test</code>
#* Remote: <code>(master) $ git push origin --delete dev/test</code>
# Update the related ticket, if needed
# You're done!

== Proposed changes ==

=== Forks ===

* '''Current status''': All development branches are created in the artefactual (or artefactual-labs) owned repository, and core contributors (usually Artefactual employees) commit directly to the official repository
* '''Proposed change''': Development happens in contributor's forks, and the official repo contains only stable & QA branches
* '''Pros''':
** Fewer branches in the official repository, so there is less clutter and noise for people watching activity
** Handles more contributors, since there won't be dozens or hundreds of branches of uncertain development status
** Contribution process is the same for external and internal contributors
** Contributors can handle their repository branches however they want - naming conventions, deleting old branches or not, etc
* '''Cons''':
** Potential confusion about which repository is the correct official one
** Potential to push changes to the wrong repo (e.g. push a development branch to the official repo instead of the contributor's fork)
** Developers have to keep 2 repositories up to date: upstream and their fork
** Harder for analysts to check out development code, since they would have to set their remote to the contributor's fork

=== Hybrid merge/rebase model ===

* '''Current status''': All branches are rebased onto their target branch before being (fast-forward) merged
* '''Proposed change''': Small or minor branches are rebased, larger feature branches are true merged
* '''Pros''':
** Commit history clearly shows 'feature', and which commits came in together as a unit
** Rebasing small branches preserves clean history for minor changes
** Better reflects nature of development - shows minor changes as happening inline, and major changes as happening in parallel
** Merge commits are clear places to look for integration issues
* '''Cons''':
** Confusion about when to merged vs rebase
** Developers have to learn two workflows

[[Category:Process documentation]]

Improvements

2018-11-20T17:56:26Z

Rspencer: Move self-describing AIP to completed

This page is intended to list possible new features, enhancements, and other improvements to any of the code used in an Archivematica based digital preservation system.

== In Progress ==
*[[Performance Improvements]]
**Make indexing configurable
**Make capture output configurable
**Make performance metrics accessible via the API
*[[Improvements/Reporting| Searching and Reporting]]
**Add basic functionality to allow a user to search for files across AIPs
**Performance Metrics Reporting API
**Standard Framework for all API's in Archivematica
* [[Improvements/Containerization|Containerization]]
** ability to deploy Archivematica into Containers
* [[Asynchronous Processing in Storage Service]]
** Add ability to do work in the background
* [[Improvements/Dspace_AIPs|DSpace AIPs]]
** Improve how AIPs stored in DSpace are handled
* [[Improvements/Dataverse|Dataverse Integration]]
** Dataverse to Archivematica integration

== Proposed ==
*[[Validation check micro-service]]
**Check for valid ArchivesSpace / DSpace identifiers
*[[DSpace REST API version check]]
**Ensure that Archivematica is configurable to a new version of the DSpace REST API
*[[AIP Import]]
**Import AIPs into Storage Service from filesystem
* [[Enhance DIP Dissemination Workflows]]
** More control over structure of DIPs and more methods for disseminating
* [[Improvements/Email|Email Preservation]]
** improve email preservation workflows
* [[Improvements/Islandora|Islandora integration]]
** current status of Islandora implementation and proposed improvements
* [[Improvements/devdocs| Developer Focused Documentation]]
** improve documentation for developers integrating with Archivematica
* [[Improvements/aippackage| AIP Packaging]]
** support alternate physical structures for an aip
* [[Improvements/warc| Web Archiving]]
** extract metadata and files from WARCs for processing
** integrate with the Archive-It service
* [[Improvements/Appraisal|Appraisal and Arrangement tab]]
** Possible enhancements to the Appraisal and Arrangement tab, introduced in v1.6
* [[Improvements/Python3|Python3]]
** changes required to run all of Archivematica using Python3 instead of Python2
** [https://docs.google.com/document/d/1r_8426OluGe8ORnqNU7MeP8N3l4MxMhdKDwOUM4zvcg/edit# Google document] with status
* [[Improvements/EPrintsIntegration|EPrints Integration]]
** suggestions for workflows to connect EPrints repositories to Archivematica
* [[Improvements/Disk_Image_Preservation|Disk Image Preservation]]
** improve disk image handling to better manage different file systems
* [https://docs.google.com/spreadsheets/d/1n567VURzQKDIcd_hXTnrRmmXlYg_t2II-dp1P-Nbhjc/edit#gid=0 Google document] with other proposed projects and status

== Completed ==

* [[Improvements/Migrations| Migrations]]
** Use Django's migrations instead of custom SQL-based migrations
* [[Improvements/rpm|CentOS/RedHat support]]
** ability to run Archivematica on an rpm based version of linux
* [[Improvements/AIPEncryption| AIP Encryption]]
** Encrypt AIP's when storing and decrypt when retrieving
* [[Improvements/aipreadme| Self Describing AIP]]
** add a README file to Archivematica AIP's to make them self-describing

[[Category:Development documentation]]

Community/Camps/Houston2018

2018-11-14T22:39:04Z

Rspencer: /* Thursday, November 15: Day Two Archivematica Camp */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates and Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica. From Artefactual we have:

'''Ashley Blewer''' works at Artefactual as our AV Preservation Specialist, primarily on the Archivematica project. Ashley also works on the audiovisual open source software projects QCTools and MediaConch (which is also used in Archivematica!). Ashley will be leveraging her archival and technical backgrounds in Stream 1 and talking about all things Archivematica!

'''Jesús García Crespo''' is a software developer for Artefactual Systems Inc.

'''Evelyn McLellan''' is President of Artefactual Systems and has been involved in the Archivematica project since its earliest days. An archivist by training with a strange obsession with preservation metadata, she'll be leading some of the sessions and generally helping out where needed.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archive UK, he has experience with the unique requirements of a government workflow. Ross also has a broad interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

Our two community member counsellors will be:

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 9:30pm
|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica users and their workflows'''
Topics covered include Checking-in; technical architecture, deployment and working open-source.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: Texas institutions'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''
Topics include Monitoring, logs, the API, and automation tools.

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Computer History Museum and New York Public Library'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Open conversation and Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Arrival and planning'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - Special Topics'''
Topics covered include handling complex media (audiovisual and disk images), ArchivesSpace integration, and PREMIS linked open data and the OWL ontology
|'''Module 6: Stream 2 - GitHub and new-features showcase'''
Topics covered include accessing Archivematica’s data, other GitHub highlights, features showcase.

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica.
Campers are invited to share their particular Archivematica implementations, real or imagined.
Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Lunch ===
On-campus options:

* Eric's Restaurant
* Einstein Bros. Bagels
* McAlister's Deli
* Student Center: Starbucks, McDonald's, Chick-fil-A
* Near Calhoun Lofts: Cafe 101, Pink's Pizza, Jimmy Johns

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Community/Camps/Houston2018

2018-11-14T00:10:23Z

Rspencer: /* Friday, November 16: Day Three Archivematica Camp */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates and Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica. From Artefactual we have:

'''Ashley Blewer''' works at Artefactual as our AV Preservation Specialist, primarily on the Archivematica project. Ashley also works on the audiovisual open source software projects QCTools and MediaConch (which is also used in Archivematica!). Ashley will be leveraging her archival and technical backgrounds in Stream 1 and talking about all things Archivematica!

'''Jesús García Crespo''' is a software developer for Artefactual Systems Inc.

'''Evelyn McLellan''' is President of Artefactual Systems and has been involved in the Archivematica project since its earliest days. An archivist by training with a strange obsession with preservation metadata, she'll be leading some of the sessions and generally helping out where needed.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archive UK, he has experience with the unique requirements of a government workflow. Ross also has a broad interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

Our two community member counsellors will be:

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 9:30pm
|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica workflows'''
Topics covered include Checking-in; technical architecture, deployment and working open-source.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: Texas institutions'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''
Topics include Monitoring, logs, the API, and automation tools.

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Computer History Museum and New York Public Library'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Open conversation and Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Arrival and planning'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - Special Topics'''
Topics covered include handling complex media (audiovisual and disk images), ArchivesSpace integration, and PREMIS linked open data and the OWL ontology
|'''Module 6: Stream 2 - GitHub and new-features showcase'''
Topics covered include accessing Archivematica’s data, other GitHub highlights, features showcase.

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica.
Campers are invited to share their particular Archivematica implementations, real or imagined.
Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Community/Camps/Houston2018

2018-11-14T00:08:47Z

Rspencer: /* Friday, November 16: Day Three Archivematica Camp */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates and Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica. From Artefactual we have:

'''Ashley Blewer''' works at Artefactual as our AV Preservation Specialist, primarily on the Archivematica project. Ashley also works on the audiovisual open source software projects QCTools and MediaConch (which is also used in Archivematica!). Ashley will be leveraging her archival and technical backgrounds in Stream 1 and talking about all things Archivematica!

'''Jesús García Crespo''' is a software developer for Artefactual Systems Inc.

'''Evelyn McLellan''' is President of Artefactual Systems and has been involved in the Archivematica project since its earliest days. An archivist by training with a strange obsession with preservation metadata, she'll be leading some of the sessions and generally helping out where needed.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archive UK, he has experience with the unique requirements of a government workflow. Ross also has a broad interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

Our two community member counsellors will be:

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 9:30pm
|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica workflows'''
Topics covered include Checking-in; technical architecture, deployment and working open-source.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: Texas institutions'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''
Topics include Monitoring, logs, the API, and automation tools.

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Computer History Museum and New York Public Library'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Open conversation and Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Arrival and planning'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - Special Topics'''
Topics covered include handling complex media (audiovisual and disk images), ArchivesSpace integration, and PREMIS linked open data and the OWL ontology
|'''Module 6: Stream 2 - GitHub and new-features showcase'''
Topics covered include other GitHub highlights, features showcase.

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica.
Campers are invited to share their particular Archivematica implementations, real or imagined.
Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Community/Camps/Houston2018

2018-11-14T00:07:34Z

Rspencer: /* Thursday, November 15: Day Two Archivematica Camp */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates and Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica. From Artefactual we have:

'''Ashley Blewer''' works at Artefactual as our AV Preservation Specialist, primarily on the Archivematica project. Ashley also works on the audiovisual open source software projects QCTools and MediaConch (which is also used in Archivematica!). Ashley will be leveraging her archival and technical backgrounds in Stream 1 and talking about all things Archivematica!

'''Jesús García Crespo''' is a software developer for Artefactual Systems Inc.

'''Evelyn McLellan''' is President of Artefactual Systems and has been involved in the Archivematica project since its earliest days. An archivist by training with a strange obsession with preservation metadata, she'll be leading some of the sessions and generally helping out where needed.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archive UK, he has experience with the unique requirements of a government workflow. Ross also has a broad interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

Our two community member counsellors will be:

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 9:30pm
|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica workflows'''
Topics covered include Checking-in; technical architecture, deployment and working open-source.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: Texas institutions'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''
Topics include Monitoring, logs, the API, and automation tools.

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Computer History Museum and New York Public Library'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Open conversation and Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Arrival and planning'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - Special Topics'''
Topics covered include handling complex media (audiovisual and disk images), ArchivesSpace integration, and PREMIS linked open data and the OWL ontology
|'''Module 6: Stream 2 - AIPs for developers/systems administrators'''
Topics covered include how the METS file is written from the database

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica.
Campers are invited to share their particular Archivematica implementations, real or imagined.
Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Community/Camps/Houston2018

2018-11-02T18:41:52Z

Rspencer: /* Camp Counsellors */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates & Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica. From Artefactual we have:

'''Ashley Blewer''' works at Artefactual as our AV Preservation Specialist, primarily on the Archivematica project.

'''Jesús García Crespo''' is a software developer for Artefactual Systems Inc.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archive UK, he has experience with the unique requirements of a government workflow. Ross also has a broad interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

Our two community member counsellors will be:

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule (tentative) ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica workflows'''
Topics covered include how to install automation tools, REST APIs, logs, maintenance, and upgrades.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: University of Houston'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Computer History Museum and New York Public Library'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''How this day is going to unfold'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - Special Topics'''
Topics covered include handling complex media (audiovisual and disk images), ArchivesSpace integration, and PREMIS linked open data and the OWL ontology
|'''Module 6: Stream 2 - AIPs for developers/systems administrators'''
Topics covered include how the METS file is written from the database

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica.
Campers are invited to share their particular Archivematica implementations, real or imagined.
Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Community/Camps/Houston2018

2018-10-30T20:57:55Z

Rspencer: /* Social events */

We're thrilled to be headed to Houston, Texas for Archivematica Camp in November 2018! Hope y'all can make it!

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp is intended to provide a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.
[[File:ArchivematicaCamp.jpg|500px|center|thumb]]
A big thanks to University of Houston Libraries and the Texas Digital Library for co-sponsoring this camp.

== Dates & Location ==

'''November 14-16, 2018 - [http://libraries.uh.edu/ University of Houston Libraries]'''.

== Registration ==

https://www.eventbrite.com/e/archivematica-camp-houston-2018-registration-48420292398

== Camp Counsellors ==

Archivematica Camp counsellors are a mix of Artefactual staff and community members with experience/expertise in using Archivematica.

Our two community member counsellors will be Andrew Berger and Nick Krabbenhoeft!

'''Andrew Berger'''

Andrew Berger is Senior Digital Archivist at the Computer History Museum. He is the primary archivist responsible for overseeing the preservation of the museum’s digital collections and manages the museum’s digital repository. In 2014-2015, he oversaw the museum’s testing and implementation of Archivematica and maintains day-to-day responsibility for ingest and for monitoring digital repository storage via periodic fixity checks.

'''Nick Krabbenhoeft'''

Nick Krabbenhoeft is the Digital Preservation Manager at New York Public Library where he is responsible for the long-term accessibility of digitized images, audio, and video material and born-digital video and archival collections. He researches digital collections surveys, systems integrations, and digital object validation. Nick teaches on digital preservation and curation as a Visiting Assistant Professor at the Pratt Institute.

== Schedule (tentative) ==

=== Wednesday November 14: Day One (optional) Archivematica for Beginners ===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Topic'''

|-
|style="white-space: nowrap;"|9:00am - 10:30pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 5:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|}

'''Note''': participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.


=== Thursday, November 15: Day Two Archivematica Camp===

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 11:00am
|'''Module 4: Stream 1 - Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Stream 2 - Supporting Archivematica workflows'''
Topics covered include how to install automation tools, REST APIs, logs, maintenance, and upgrades.

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: TBD'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 5: Stream 1 - Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Stream 2 - Understanding Archivematica's logs and performance evaluation'''

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: TBD'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| Agenda adjustment for Day 3 - what do you want to talk about tomorrow? Also, wrap-up.
|}


=== Friday, November 16: Day Three Archivematica Camp ===

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 1 TBA'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Room 2 TBA'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''How this day is going to unfold'''

|-
|9:30pm - 10:30pm
|'''Module 6: Stream 1 - AIPs for Archivists'''
Topics covered include how the AIP is structured, and a close look at the METS file
|'''Module 6: Stream 2 - AIPs for developers/systems administrators'''
Topics covered include how the METS file is written from the database

|-
|style="white-space: nowrap;"|10:30-11:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 7: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.

|-
|style="white-space: nowrap;"|11:30am - 12:30pm
|colspan="2"|'''Brown Bag Lunch and Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:30pm - 2:00pm
|colspan="2" style="white-space: nowrap; width: 90%;"|'''Module 8: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented with a checklist of what you need to actually run Archivematica. Campers are invited to share their particular Archivematica implementations, real or imagined. Questions and participation mandatory! Wrap up and safe travels!
|}

'''Note on agenda adjusting''': based on camper interest, the schedule may be adjusted to accommodate topics of interest that will be covered during the concurrent sessions.

== Social events ==

=== Wednesday, November 14: Group event ===

Folks meet at [https://8thwonder.com/ 8th Wonder Breweries] from 6pm. The address is: [https://www.google.de/maps/search/2202+Dallas+Street+%7C+HouTex+%7C+77003/@29.7488668,-95.3580668,17z/data=!3m1!4b1 ''2202 Dallas Street | HouTex | 77003''] about 2.5 miles away from the University of Houston campus.

Everyone is invited to find shared rides to the event and folks from Artefactual will be on hand to help everyone remaining on campus find transport too. We can't wait to see y'all there!

=== Thursday, November 15: Small group dinners ===

We've created a sign-up sheet to help folk find others to dine with on Thursday. There are a variety of restaurants on the list to cater for all. The restaurants are across the Houston area. If you know Houston well, or have an idea for a restaurant not listed, please consider becoming a group leader.

The sign-up sheet for group dinners can be found here: https://docs.google.com/document/d/1rvnKP6lM3aeuRulgOnlmjYg_k8gaaCgfZdfvkU7sTBM/edit?usp=sharing

== Local Arrangements ==
Sessions will be held in the [https://www.google.com/maps/place/M.D.+Anderson+Library/@29.7216972,-95.3444726,15z/data=!4m8!1m2!3m1!2sM.D.+Anderson+Library!3m4!1s0x0:0x2a3f0ce5297e9c3a!8m2!3d29.7210839!4d-95.3420484 M.D. Anderson Library] on the University of Houston main campus.

=== Hotels ===
We have reserved a block of rooms in the UH Hilton at a rate of $179/night. Rooms at this rate are available on a first come, first served basis and must be booked before Sunday, October 14, 2018. To make reservations on the room block, call 832-531-6300 or 800-774-1500 and follow the automated prompts for room reservations. The group name for the room block is Artefactual Systems (code ARTE).

Hotels are also available downtown. There are many hotels near the [https://www.google.com/maps/place/Theater+District/@29.7588822,-95.3667289,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3a02f3a873:0x2b891dc7cc50c30f!8m2!3d29.7607509!4d-95.3676917 Theater District], [https://www.google.com/maps/place/Central+Station+Rusk/@29.7586121,-95.3638858,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf3b333c80a7:0x6fa09da76725dace!8m2!3d29.7577507!4d-95.3627443 Central Station Rusk], and [https://www.google.com/maps/place/Convention+District/@29.7567679,-95.3626305,17z/data=!4m8!1m2!2m1!1spurple+line!3m4!1s0x8640bf23e6dc48e3:0xe8d2427611ae1e92!8m2!3d29.7547783!4d-95.3578097 Convention District] stops on the [https://www.ridemetro.org/Pages/PurpleLine.aspx METRORail purple line], which provides frequent service between downtown Houston and the UH campus. Purple line stops on the UH main campus include Robertson Stadium/UH/TSU and UH South/University Oaks.

A quick review of Airbnb search results shows many rentals in Midtown, Montrose, Museum District, and EaDo/Eastwood neighborhoods within a short commute (~5 miles) to campus.

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

Dataverse

2018-08-06T19:30:38Z

Rspencer: Link out to supported tabular formats

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse].

This page was originally created as part of an early Proof of Concept integration in 2017, which was only made available in a development branch of Archivematica. We have now started a phase 2 project to improve on that original integration work and merge it into a public release of Archivematica (exact release tbc). This work is being sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Current Status==

'''May 11, 2018'''
To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:

* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]

==Feature Files==
On this project we are using [http://docs.behat.org/en/v2.5/guides/1.gherkin.html Gherkin] feature files to define the desired behaviour of preserving a dataset from a Dataverse. Feature files are also known as Acceptance Tests, because they specify the behaviour that we will test at the end of the project.

The early drafts are documented in this google doc: [http://docs.behat.org/en/v2.5/guides/1.gherkin.html]
Once the draft has been reviewed we will publish it to our acceptance test repository in github.

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
This section is from the first phase project in 2017 and needs to be updated.

*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===
This section is from the first phase project in 2017 and needs to be updated.

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

A [http://guides.dataverse.org/en/latest/user/dataset-management.html?highlight=bundle bundle] is a zipped object, documented by Dataverse as containing all of the below files:

* As tab-delimited data (with the variable names in the first row);
* The original file uploaded by the user;
* Saved as R data (if the original file was not in R format);
* Variable Metadata (as a DDI Codebook XML file);
* Data File Citation (currently in either RIS or EndNote XML format);

Supported tabular formats are listed in the Dataverse [http://guides.dataverse.org/en/latest/user/tabulardataingest/supportedformats.html manual]

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>

== Future Requirements & Considerations ==
This section includes working notes for future phases, as interesting opportunities or questions arise. At the end of the current phase we will be documenting the integration as well as future opportunities.

=== Notes from Feature File review meeting on May 1 2018 (2pm EST) ===

'''Choice & Versioning of Dataverse API:'''
The dataverse Search and Access APIs are not currently versioned.
The Native API is versioned: http://guides.dataverse.org/en/latest/api/native-api.html
There is an OAI-PMH interface (although it is not mentioned in the dataverse API guide). Amber said there were idiosyncrasies in the way dataverse implemented PMH, and wasn’t sure it would be a ‘safe’ option.
Amaz would like to see that we are either using a standard API (like OAI-PMH) or a versioned API.
Amaz thought wondered whether we could use PMH with the polling part of the solution; but given what Amber said, it doesn’t seem like a good way to go)
So as part of the project we need to see whether we could use the Native API (even if we don’t actually use it), or we need to raise it as an issue to discuss with the dataverse team.

'''Relationships between Datasets'''
Amber pointed out that they are not currently clear exactly what datasets should be preserved, and expects this will vary quite a bit by institution.
We discussed the question of whether all datasets in a dataverse would be preserved (not currently known), which brought up the question of how to relate datasets.
We talked about AICs as one possible solution. But agreed that it’s a new feature and needs to be thought through… there could be other solutions than AIC.

'''Improving agent info in event history in METS'''
We pointed out that having an agent other than Archivematica in the METS is a new feature
Discussed the fact that we could make this even more specific by adding more agents. For instance, differentiating between the researcher who uploaded files from the research data manager who published the dataset.

'''Notes from Dataverse Testing:'''

Should a preserved dataset include an equivalent of fixity check on any UNFs created by Dataverse?
https://dataverse.scholarsportal.info/guides/en/4.8.6/developers/unf/index.html#unf
Universal Numerical Fingerprint (UNF) is a unique signature of the semantic content of a digital object. It is not simply a checksum of a binary data file. Instead, the UNF algorithm approximates and normalizes the data stored within. A cryptographic hash of that normalized (or canonicalized) representation is then computed.

Dataverse

2018-08-05T22:47:32Z

Rspencer: Adding contextual information about the contents of a Bundle

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page sets out the requirements and designs for integration with [http://dataverse.org Dataverse].

This page was originally created as part of an early Proof of Concept integration in 2017, which was only made available in a development branch of Archivematica. We have now started a phase 2 project to improve on that original integration work and merge it into a public release of Archivematica (exact release tbc). This work is being sponsored by [https://scholarsportal.info/ Scholars Portal], a service of the Ontario Council of University Libraries (OCUL).

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Current Status==

'''May 11, 2018'''
To see the current status of work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:

* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]

==Feature Files==
On this project we are using [http://docs.behat.org/en/v2.5/guides/1.gherkin.html Gherkin] feature files to define the desired behaviour of preserving a dataset from a Dataverse. Feature files are also known as Acceptance Tests, because they specify the behaviour that we will test at the end of the project.

The early drafts are documented in this google doc: [http://docs.behat.org/en/v2.5/guides/1.gherkin.html]
Once the draft has been reviewed we will publish it to our acceptance test repository in github.

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
This section is from the first phase project in 2017 and needs to be updated.

*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===
This section is from the first phase project in 2017 and needs to be updated.

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

A [http://guides.dataverse.org/en/latest/user/dataset-management.html?highlight=bundle bundle] is a zipped object, documented by Dataverse as containing all of the below files:

* As tab-delimited data (with the variable names in the first row);
* The original file uploaded by the user;
* Saved as R data (if the original file was not in R format);
* Variable Metadata (as a DDI Codebook XML file);
* Data File Citation (currently in either RIS or EndNote XML format);

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>

== Future Requirements & Considerations ==
This section includes working notes for future phases, as interesting opportunities or questions arise. At the end of the current phase we will be documenting the integration as well as future opportunities.

=== Notes from Feature File review meeting on May 1 2018 (2pm EST) ===

'''Choice & Versioning of Dataverse API:'''
The dataverse Search and Access APIs are not currently versioned.
The Native API is versioned: http://guides.dataverse.org/en/latest/api/native-api.html
There is an OAI-PMH interface (although it is not mentioned in the dataverse API guide). Amber said there were idiosyncrasies in the way dataverse implemented PMH, and wasn’t sure it would be a ‘safe’ option.
Amaz would like to see that we are either using a standard API (like OAI-PMH) or a versioned API.
Amaz thought wondered whether we could use PMH with the polling part of the solution; but given what Amber said, it doesn’t seem like a good way to go)
So as part of the project we need to see whether we could use the Native API (even if we don’t actually use it), or we need to raise it as an issue to discuss with the dataverse team.

'''Relationships between Datasets'''
Amber pointed out that they are not currently clear exactly what datasets should be preserved, and expects this will vary quite a bit by institution.
We discussed the question of whether all datasets in a dataverse would be preserved (not currently known), which brought up the question of how to relate datasets.
We talked about AICs as one possible solution. But agreed that it’s a new feature and needs to be thought through… there could be other solutions than AIC.

'''Improving agent info in event history in METS'''
We pointed out that having an agent other than Archivematica in the METS is a new feature
Discussed the fact that we could make this even more specific by adding more agents. For instance, differentiating between the researcher who uploaded files from the research data manager who published the dataset.

'''Notes from Dataverse Testing:'''

Should a preserved dataset include an equivalent of fixity check on any UNFs created by Dataverse?
https://dataverse.scholarsportal.info/guides/en/4.8.6/developers/unf/index.html#unf
Universal Numerical Fingerprint (UNF) is a unique signature of the semantic content of a digital object. It is not simply a checksum of a binary data file. Instead, the UNF algorithm approximates and normalizes the data stored within. A cryptographic hash of that normalized (or canonicalized) representation is then computed.

Elasticsearch Development

2018-06-12T15:14:04Z

Rspencer: Updated with current Elasticsearch API/Archivematica information

From Archivematica 0.9, AIP package information, such as METS data, is indexed
using [http://www.elasticsearch.org/ Elasticsearch (ES)]. This data can be
searched from the Archival Storage area of the dashboard or can be interfaced
with programmatically. For Elasticsearch administration information, such as
how to delete an Elasticsearch index, please reference the
[[Administrator_manual_1.2#Elasticsearch|administrator manual]].

'''NB.''' From Archivematica 1.7.0, users are given the option of whether to
index information using Elasticsearch, and so the information below might not
work. It will be dependant on how your Archivematica instance has been
configured.

=Programmatic Access to indexed AIP data=

To access indexed AIP data using a custom script or application, find an
Elasticsearch API (Application Programming Interface) library for the
programming language you are most comfortable with. In Archivematica we use
Python with the Elasticsearch supported
[https://github.com/elastic/elasticsearch-py library]. In our developer
documentation, we will demonstrate how to use this and Python to access AIP
data, but any programming language, such as PHP and
[https://github.com/ruflin/Elastica/ Elastica], should work.

A list of [https://www.elastic.co/guide/en/elasticsearch/client/index.html officially]
supported and
[https://www.elastic.co/guide/en/elasticsearch/client/community/current/index.html community]
supported libraries can be found on the Elasticsearch website.

==Connecting to Elasticsearch (Archivematica 1.3+)==

The following example will demonstrate access to the indexes using
Elasticsearch's own
[https://github.com/elastic/elasticsearch-py Python library].

===Importing the Elasticsearch API module===

The first step is to import the Elasticsearch module and connect to the
Elasticsearch server.

<pre>
from __future__ import print_function
import sys

from elasticsearch import Elasticsearch
from elasticsearch.exceptions import RequestError, NotFoundError

conn = Elasticsearch(['127.0.0.1:9200'])
</pre>

'''NB.''' The additional module imports are used in this example so that the
examples below can be copied-and-pasted as desired.

===Full text searching===

Once connected to Elasticsearch, you can perform searches. Below is the code
needed to do a wildcard ('*') search for all indexed AIP files. We retrieve the
first 20 items. Instead of providing a wildcard you could also supply keywords,
such as a specific AIP UUID.

<pre>
start_page = 1
items_per_page = 20

# ref: string query, https://git.io/vhgUw
wildcard_query = { "query": {
"query_string" : {
"query": "*",
},
}}

try:
results = conn.search(
body=wildcard_query,
index="aips",
doc_type="aip",
from_=start_page - 1,
size=items_per_page,
)
except RequestError:
print("Query error")
sys.exit()
except NotFoundError:
print("No results found")
sys.exit()
</pre>

There are a number of ways to construct Elasticsearch queries. The
Elasticsearch website provides useful reference material:
[https://www.elastic.co/guide/en/elasticsearch/reference/current/full-text-queries.html Elasticsearch Full Text Queries].

===Querying for specific data===

While the string query-type is good for broad searches, you may want to
narrow a search down to a specific field of data to reduce false positives.
Below is an example of searching documents, using a "term" query to match
criteria within specific data.

<pre>
start_page = 1
items_per_page = 20

# ref: term query, https://git.io/vhrI9
term_query = {
"query": {
"constant_score": {
"filter": {
"term": {
"mets.ns0:mets_dict_list.ns0:amdSec_dict_list.@ID":
"amdsec_8"
}
}
}
}
}

try:
results = conn.search(
body=term_query,
index="aips",
doc_type="aip",
from_=start_page - 1,
size=items_per_page,
)
except RequestError:
print("Query error")
sys.exit()
except NotFoundError:
print("No results found")
sys.exit()
</pre>

Note that the construction of the query is not straightforward. Fields and
values are stored in Elasticsearch in lowercase. Properties work in uppercase
and might not work in lowercase. You can analyze a query with a `curl`
statement along the lines of:

<pre>
curl -X GET "http://127.0.0.1:9200/_analyze?pretty=true" \
-H 'Content-Type: application/json' -d'
{
"field": "METS.ns0:mets_dict_list.ns0:amdSec_dict_list.@ID",
"text": "amdSec_8"
}'
</pre>

The result is an analysis of the query string we want to use and it can reveal
common mistakes in our intuition, for example, searching for `amdSec_8` as a
mixed case string.

===Displaying search results===

Now that you have performed a couple of searches, you can display some results.
The logic below cycles through each hit in a results set. For each AIP file,
the UUID and filepath of the AIP are printed to the console.

<pre>
res = results.get("hits")
if res is not None:
for r_ in res.items():
if r_[0] == "total":
print("Total results:", r_[1])
if r_[0] == "hits":
print("Results returned:", len(r_[1]))
for aip_index in r_[1]:
# aip_index will be the complete AIP record as a Python dict
if aip_index.get("_source"):
print(
"AIP ID: {}".format(
aip_index.get("_source").get("filePath")
)
)
print(
"Filepath: {}".format(
aip_index.get("_source").get("uuid")
)
)
</pre>

===Fetching specific documents===

The AIP index inside Archivematica is separated into two Elasticsearch document
types. The AIP as a whole, and its individual files.

* ''aip''
* ''aipfile''

It might be easier to retrieve information about a specific digital object if
you know its UUID and want to query the ```aipfile``` instead. The example
below shows how we might retrieve the format identification for the file with
the UUID: ''f7428196-a11b-4093-b311-d43d607d54ca''

<pre>
from __future__ import print_function
import sys

from elasticsearch import Elasticsearch
from elasticsearch.exceptions import RequestError, NotFoundError

conn = Elasticsearch(["http://127.0.0.1:9200"])

start_page = 1
items_per_page = 20

term_query = {
"query": {
"constant_score": {
"filter": {
"term": {"FILEUUID": "f7428196-a11b-4093-b311-d43d607d54ca"}
}
}
}
}

try:
results = conn.search(
body=term_query,
index="aips",
doc_type="aipfile",
from_=start_page - 1,
size=items_per_page,
)
except RequestError:
print("Query error")
sys.exit()
except NotFoundError:
print("No results found")
sys.exit()

res = results.get("hits")
if res is not None:
for res_ in res.get("hits"):
file_record = res_.get("_source")
if file_record:
try:
puid = file_record["METS"]["amdSec"] \
["ns0:amdSec_dict_list"][0] \
["ns0:techMD_dict_list"][0] \
["ns0:mdWrap_dict_list"][0] \
["ns0:xmlData_dict_list"][0] \
["ns1:object_dict_list"][0] \
["ns1:objectCharacteristics_dict_list"][0] \
["ns1:format_dict_list"][0] \
["ns1:formatRegistry_dict_list"][0] \
["ns1:formatRegistryKey"]
print("Format ID: {}".format(puid))
except KeyError:
print("Problem accessing index.")
sys.exit(1)
</pre>

===Archivematica Transfer Index===

Information about Transfers is also indexed in Archivematica. The information
is less-rich when compared to what is stored in the AIP and so it is not
covered in detail here. We can use the wildcard query from the AIP examples
above to begin to look at what is in the transfer index. The query would look
like as follows:

<pre>
from __future__ import print_function
import sys

from elasticsearch import Elasticsearch
from elasticsearch.exceptions import RequestError, NotFoundError

conn = Elasticsearch(["http://127.0.0.1:62002"])

start_page = 1
items_per_page = 20

# ref: string query, https://git.io/vhgUw
wildcard_query = { "query": {
"query_string" : {
"query": "*",
},
}}

try:
results = conn.search(
body=wildcard_query,
index="transfers",
doc_type="transfer",
from_=start_page - 1,
size=items_per_page,
)
except RequestError:
print("Query error")
sys.exit()
except NotFoundError:
print("No results found")
sys.exit()

res = results.get("hits")
if res is not None:
for res_ in res.get("hits"):
print(res_.get("_source"))
</pre>

===Further reading===

Elasticsearch provides API functions beyond
[https://www.elastic.co/guide/en/elasticsearch/reference/current/search.html searching].
Users who wish to make use of these capabilities in Python
can look at the Python [https://elasticsearch-py.readthedocs.io/en/master/ library documentation].

The complete [https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html Elasticsearch documentation]
reference is also available. Many of the commands are described with examples
that can be run using the [https://curl.haxx.se/ curl] command line tool.

==Connecting to Elasticsearch (Archivematica 0.9 up to Archivematica 1.2)==

Here we will run through an example of interfacing with older versions of
Archivematica using Elasticsearch with a Python script that leverages the pyes
library.

'''NB.''' Pyes use was [https://github.com/artefactual/archivematica/commit/b0ac6c642a2f070fc7a0f7c198a51c2d0509b7f7 removed]
in Archivematica 1.3. Though with some modification to the examples below it
should still be possible to adopt it to query the ES indexes.

===Importing the pyes module===

The first step, when using pyes, is to require the module. The following code
imports pyes functionality on a system on which Archivematica is installed.

<pre>
import sys
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")
from pyes import *
</pre>

Next you'll want to create a connection to Elasticsearch.

<pre>
conn = ES('127.0.0.1:9200')
</pre>

===Full text searching===

Once connected to Elasticsearch, you can perform searches. Below is the code
needed to do a "wildcard" search for all AIP files indexed by Elasticsearch and
retrieve the first 20 items. Instead of doing a "wildcard" search you could
also supply keywords, such as a certain AIP UUID.

<pre>
start_page = 1
items_per_page = 20

q = StringQuery('*')

try:
results = conn.search_raw(
query=q,
indices='aips',
type='aip',
start=start_page - 1,
size=items_per_page
)
except:
print 'Query error.'
</pre>

===Querying for specific data===

While the "StringQuery" query type is good for broad searches, you may want to
narrow a search down to a specific field of data to reduce false-positives.
Below is an example of searching documents, using "TermQuery", matching
criteria within specific data. As, by default, Elasticsearch stores term values
in lowercase the term value searched for must also be lowercase.

<pre>
import sys
sys.path.append("/usr/lib/archivematica/archivematicaCommon/externals")
import pyes

conn = pyes.ES('127.0.0.1:9200')

q = pyes.TermQuery("METS.amdSec.ns0:amdSec_list.@ID", "amdsec_8")

try:
results = conn.search_raw(query=q, indices='aips')
except:
print 'Query failed.'
</pre>

===Displaying search results===

Now that you've performed a couple of searches, you can display some results.
The below logic cycles through each hit in a results set, representing an AIP
file, and prints the UUID of the AIP the file belongs in, the Elasticsearch
document ID corresponding to the indexed file data, and the path of the file
within the AIP.

<pre>
if results:
document_ids = []
for item in results.hits.hits:
aip = item._source
print 'AIP ID: ' + aip['AIPUUID'] + ' / Document ID: ' + item._id
print 'Filepath: ' + aip['filePath']
print
document_ids.append(item._id)
</pre>

===Fetching specific documents===

If you want to get Elasticsearch data for a specific AIP file, you can use the
Elasticsearch document ID. The above code populates the
<code>document_ids</code> array and the below code uses this data, retrieving
individual documents and extracting a specific item of data from each document.

<pre>
for document_id in document_ids:
data = conn.get(index_name, type_name, document_id)

format = data['METS']['amdSec'] \
['ns0:amdSec_list'][0] \
['ns0:techMD_list'][0] \
['ns0:mdWrap_list'][0] \
['ns0:xmlData_list'][0] \
['ns1:object_list'][0] \
['ns1:objectCharacteristics_list'][0] \
['ns1:format_list'][0] \
['ns1:formatDesignation_list'][0] \
['ns1:formatName']

print 'Format for document ID ' + document_id + ' is ' + format
</pre>

===Augmenting documents===

To add additional data to an Elasticsearch document, you will need the document
ID. The following code shows an Elasticsearch query being used to find a
document and update it with additional data. Note that the name of the data
field being added, "__public", is prefixed with two underscores. This practice
prevents the accidental overwriting of system or Archivematica-specific data.
System data is prefixed with a single underscore.

<pre>
import sys
sys.path.append("/usr/lib/archivematica/archivematicaCommon/externals")
import pyes

conn = pyes.ES('127.0.0.1:9200')

q = pyes.TermQuery("METS.amdSec.ns0:amdSec_list.@ID", "amdsec_8")

results = conn.search_raw(query=q, indices='aips')

try:
if results:
for item in results.hits.hits:
print 'Updating ID: ' + item['_id']

document = item['_source']
document['__public'] = 'yes'
conn.index(document, 'aips', 'aip', item['_id'])
except:
print 'Query failed.'
</pre>

AIP re-ingest

2018-05-22T09:32:06Z

Rspencer: /* techMD */ correct superseded vs. current

[[Main Page]] > [[Requirements]] > AIP re-ingest

This page documents requirements for retrieving an Archivematica AIP from archival storage and re-ingesting it for further processing.

[[Category:Feature requirements]]

=Use cases=

*Updating DC metadata (version 1.5)
*Updating rights metadata (version 1.5)
*Adding new DC metadata files (version 1.5)
*Adding new submission documentation (unsponsored)
*Adding new digital objects (unsponsored)
*Deleting original digital objects (unsponsored)
*Redoing format identification/validation/characterization (version 1.6)
*Redoing metadata extraction (version 1.6)
*Redoing preservation normalization (version 1.6)
*Generating new DIP (version 1.5)
*Running OCR on re-ingest (version 1.6)
*Rerunning other micro-services (Examine Contents, Transfer Structure Report) (unsponsored)
*Extracting packages and then deleting the packages (unsponsored)
*Sending AIP to backlog and re-arranging/selecting from contents (unsponsored)

==Full re-ingest==

To be supported in version 1.6. Sends an AIP to the beginning of transfer to run all micro-services on AIP, and re-normalize for preservation and access if desired.

[[File:AIP_re-ingest_Preservation_and_metadata.png|800px|thumb|center]]

==Metadata re-ingest==

Supported in version 1.5. Sends an AIP to the beginning of ingest to allow user to update metadata.

[[File:AIP_re-ingest_workflows_-_Metadata_only.png|800px|thumb|center]]

==Partial re-ingest==

Supported in version 1.5. Sends an AIP to the beginning of ingest to allow user to update metadata and normalize for access.

[[File:AIP_re-ingest_objects_with_metadata.png|800px|thumb|center]]

=New micro-services=

==New micro-services for all workflows==

*Retrieve AIP from archival storage
*Place AIP in active transfers for processing
*Extract AIP contents and run BagIt checks
*Add approve AIP re-ingest micro-service
*Re-use existing file UUIDs
*Identify new metadata files
*Validate schemas in new metadata files
*Update METS file
*Replace AIP

=METS versioning=

*Versioning will be captured via METS file updates.
*METS file updates will be handled through <status>, <created> and <groupID> attributes in the various METS sections. See for example Element <dmdSec> at http://www.loc.gov/standards/mets/docs/mets.v1-9.html.
*This means that there will always only be one AIP METS file, but it will contain both superseded and current metadata and versioning information for all updates.

==dmdSec==

*The first dmdSec created will be marked as status="original", and updated/revised dmdSecs will be marked as status="updated". The timestamp (created="[date/time]") is also updated.

==amdSec==

===techMD===

*After re-ingest, the original techMD will contain the PREMIS:Object metadata and the status="superseded."
*The next techMD will be status="current" and contain the PREMIS:Object metadata generated on re-ingest, as well as all PREMIS events from each ingest and re-ingest. This shows the complete set of actions taken upon the objects since they came into the repository.
*The timestamp (created="[date/time]") is also updated.

===rightsMD===

*After re-ingest, the original rightsMD will have the status="superseded" and the revised rightsMD will have the status="current".
*The timestamp (created="[date/time]") is also updated.
*Altering either the rights statement or the act(s) will result in a new rightsMD.

==fileSec==

The fileSec does not indicate its status, but should reflect the files in the AIP currently. For example, if normalization is not performed on the first ingest, but is performed on reingest, the fileSec in the reingested AIP will include preservation derivatives.

Dataverse

2018-05-11T12:51:38Z

Rspencer: Adding a current status bullet point

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page tracks development of a proof of concept integration of Archivematica with [http://dataverse.org Dataverse]. Please note that this feature is experimental and Artefactual has not tested it in a production environment.

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Current Status==

'''May 11, 2018'''
Work has begun to see the OCUL Dataverse project migrated from a development branch to a stable branch of Archivematica. To see the current status of that work, and any outstanding issue, please see the Waffle Board or Board's linked to [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse below]:

* [https://waffle.io/artefactual/archivematica?label=OCUL:%20AM-Dataverse Waffle board for the Dataverse Feature]

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>

Dataverse

2018-04-18T10:07:40Z

Rspencer: Update on PREMIS derivative term status

[[Main Page]] > [[Documentation]] > [[Requirements]] > Dataverse

This page tracks development of a proof of concept integration of Archivematica with [http://dataverse.org Dataverse]. Please note that this feature is experimental and Artefactual has not tested it in a production environment.

[[Category:Feature requirements]]

===See also===

* [[Sword API]]
* [[Dataset preservation]]

==Overview==
This wiki captures requirements for ingesting studies (datasets) from Dataverse into Archivematica for long-term preservation.

==Installation==

April 24, 2017
This feature requires a development branch of Archivematica, which can be installed with the following steps:

1) install deploy-pub. https://github.com/artefactual/deploy-pub
2) use the archivematica-centos7 playbook in deploy-pub https://github.com/artefactual/deploy-pub/tree/master/playbooks/archivematica-centos7
3) create a hosts file that lists your target machine (see digital ocean example linked from playbook)
4) in requirements.yml change version of ansible-archivematica-src to "stable/1.6.x"
5) change singlenode.yml to point to the host you defined in your hosts file.
6) change the vars-singlenode.yml to include the following info:
#required for dataverse testing
archivematica_src_am_version: "dev/dataverse-poc"
archivematica_src_automationtools: "yes"
archivematica_src_automationtools_version: "dev/dataverse"

==Workflow==
*The proposed workflow consists of issuing API calls to Dataverse, receiving content (data files and metadata) for ingest into Archivematica, preparing Archivematica Archival Information Packages (AIPs) and placing them in archival storage, <strike> and updating the Dataverse study with the AIP UUIDs </strike> (this was determined to be out of scope).
*Analysis is based on Dataverse tests using [https://apitest.dataverse.org https://apitest.dataverse.org] and [https://demo.dataverse.org https://demo.dataverse.org], online documentation at http://guides.dataverse.org/en/latest/api/index.html and discussions with Dataverse developers and users.
*Proposed integration is for Archivematica 1.5 and higher and Dataverse 4.x.

===Workflow diagram===

[[File:Dataverse - Archivematica workflow_1.png|800px|thumb|center]]

===Workflow diagram notes===

[1] "Ingest script" refers to an [https://github.com/artefactual/automation-tools automation tool] designed to automate ingest into Archivematica for bulk processing. An existing automation tool would be modified to accomplish the tasks described in the workflow.

[2] A new or updated study is one that has been published, either for the first time or as a new version, since the last API call.

[3] The json file contains citation and other study-level metadata, an entity_id field that is used to identify the study in Dataverse, version information, a list of data files with their own entity_id values, and md5 checksums for each data file.

[4] If json file has content_type of tab separated values, Archivematica issues API call for multiple file ("bundled") content download. This returns a zipped package for tsv files containing the .tab file, the original uploaded file, several other derivative formats, a DDI XML file and file citations in Endnote and RIS formats.

[5] The METS file will consist of a dmdSec containing the DC elements extracted from the json file, and a fileSec and structMap indicating the relationships between the files in the transfer (eg. original uploaded data file, derivative files generated for tabular data, metadata/citation files). This will allow Archivematica to apply appropriate preservation micro-services to different filetypes and provide an accurate representation of the study in the AIP METS file (step 1.9).

[6] Archivematica ingests all content returned from Dataverse, including the json file, plus the METS file generated in step 1.6.

[7] Standard and pre-configured micro-services include: assign UUID, verify checksums, generate checksums, extract packages, scan for viruses, clean up filenames, identify formats, validate formats, extract metadata and normalize for preservation.

== Transfer METS file ==

When the ingest script retrieves content from Dataverse, it generates a METS file to allow Archivematica to understand the contents of the transfer and the relationships between its various data and metadata files.

=== Sample transfer METS file ===

Original Dataverse study retrieved through API call:

*dataset.json (a JSON file generated by Dataverse consisting of study-level metadata and information about data files)
*Study_info.pdf (a non-tabular data file)
*A zipped bundle consisting of the following:
**YVR_weather_data.sav (an SPSS SAV file uploaded by the researcher)
**YVR_weather_data.tab (a TAB file generated from the SPSS SAV file by Dataverse)
**YVR weather_data.RData (an R file generated from the SPSS SAV file by Dataverse)
**YVR_weather_data-ddi.xml, YVR_weather_datacitation-endnote.xml, and YVR_weather_datacitation-ris.ris (three metadata files generated for the TAB file by Dataverse)


Resulting transfer METS file

*The fileSec in the METS file consists of three file groups, USE="original" (the PDF and SAV files); USE="derivative" (the TAB and R files); and USE="metadata" (the JSON file and the three metadata files from the zipped bundle).
*All of the files unpacked from the Dataverse bundle have a GROUPID attribute to indicate the relationship between them. If the transfer had consisted of more than one bundle, each set of unpacked files would have its own GROUPID.
*Three dmdSecs have been generated:
**dmdSec_1, consisting of a small number of study-level DDI terms
**dmdSec_2, consisting of an mdRef to the JSON file
**dmdSec_3, consisting of an mdRef to the DDI XML file
*In the structMap, dmdSec_1 and dmdSec_2 are linked to the study as a whole, while dmdSec_3 is linked to the TAB file. The endnote and ris files have not been made into dmdSecs because they contain small subsets of metadata which are already captured in dmdSec_1 and the DDI xml file.



[[File:METS1G.png|900px|thumb|center]]
[[File:METS2G.png|900px|thumb|center]]
[[File:METS3G.png|900px|thumb|center]]



Metadata sources for METS file



{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!style="width:15%"|'''METS element'''
!style="width:25%"|'''Information source'''
!style="width:40%"|'''Notes'''
|-
|ddi:titl
|json: citation/typeName: "title", value: [value]
|
|-
|ddi:IDNo
|json: authority, identifier
|json example: "authority": "10.5072/FK2/", "identifier": "0MOPJM"
|-
|ddi:IDNo agency attribute
|json: protocol
|json example: "protocol": "doi"
|-
|ddi:AuthEntity
|json: citation/typeName: "authorName"
|
|-
|ddi:distrbtr
|Config setting in ingest tool
|
|-
|ddi:version date attribute
|json: "releaseTime"
|
|-
|ddi:version type attribute
|json: "versionState"
|
|-
|ddi:version
|json: "versionNumber", "versionMinorNumber"
|
|-
|ddi:restrctn
|json: "termsOfUse"
|
|-
|fileGrp USE="original"
|json: datafile
|Each non-tabular data file is listed as a datafile in the files section. Each TAB file derived by Dataverse for uploaded tabular file formats is also listed as a datafile, with the original file uploaded by the researcher indicated by "originalFileFormat".
|-
|fileGrp USE="derivative"
|All files that are included in a bundle, except for the original file and the metadata files (see below).
|
|-
|fileGrp USE="metadata"
|Any files with .json or .ris extension, any -ddi.xml files and -endnote.xml files
|
|-
|CHECKSUM
|json: datafile/"md5": [value]
|
|-
|CHECKSUMTYPE
|json: datafile/"md5"
|
|-
|GROUPID
|Generated by ingest tool. Each file unpacked from a bundle is given the same group id.
|
|-
|}



== AIP METS file ==

=== Basic METS file structure ===

The Archival Information Package (AIP) METS file will follow the basic structure for a standard Archivematica AIP METS file described at [[METS]]. A new fileGrp USE="derivative" will be added to indicate TAB, RData and other derivatives generated by Dataverse for uploaded tabular data format files.

=== dmdSecs in AIP METS file ===

The dmdSecs in the transfer METS file will be copied over to the AIP METS file.

=== Additions to PREMIS for derivative files ===

In the PREMIS Object entity, relationships between original and derivative tabular format files from Dataverse will be described using PREMIS relationship semantic units. A PREMIS derivation event will be added to indicate the derivative file was generated from the original file, and a Dataverse Agent will be added to indicate the Event were carried out by Dataverse prior to ingest, rather than by Archivematica.

'''Note''' We originally considered adding a creation event for the derivative files as well, but decided that it's not necessary as the event can be inferred from the derivation event and the PREMIS object relationships.

'''Note''' "Derivation" is not an event type on the Library of Congress controlled vocabulary list at http://id.loc.gov/vocabulary/preservation/eventType.html. However, we have submitted it as a proposed new term (November 2015) at http://premisimplementers.pbworks.com/w/page/102413902/Preservation%20Events%20Controlled%20Vocabulary - a list of new terms that is being considered by the PREMIS Editorial Committee.

'''Update''' ''April 2018'': The most recently available Event Type Controlled List (June 2017) does not yet have derivation as a controlled type, https://www.loc.gov/standards/premis/v3/preservation-events.pdf

Example:

Original SPSS SAV file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>is source of</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[TAB file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
...
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>derivation</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://dataverse.scholarsportal.info/dvn/
</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
...
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://dataverse.scholarsportal.info/dvn/</premis:agentIdentifierValue>
</premis:agentIdentifier>
<premis:agentName>SP Dataverse Network</premis:agentName>
<premis:agentType>organization</premis:agentType>
</pre>

Derivative TAB file
<pre>
<premis:relationship>
<premis:relationshipType>derivation</premis:relationshipType>
<premis:relationshipSubType>has source</premis:relationshipSubType>
<premis:relatedObjectIdentification>
<premis:relatedObjectIdentifierType>UUID</premis:relatedObjectIdentifierType>
<premis:relatedObjectIdentifierValue>[SPSS SAV file UUID]</premis:relatedObjectIdentifierValue>
<premis:relationship>
</pre>

=== Fixity check for checksums received from Dataverse ===

<pre>
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>[Event UUID assigned by Archivematica]</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>fixity check</premis:eventType>
<premis:eventDateTime>2015-08-21</premis:eventDateTime>
<premis:eventDetail>program="python"; module="hashlib.sha256()"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>Pass</premis:EventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>Dataverse checksum 91b65277959ec273763d28ef002e83a6b3fba57c7a3[...]
verified</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
<premis:eventOutcomeInformation>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica 1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</pre>

== AIP structure ==

An Archival Information Package derived from a Dataverse ingest will have the same basic structure as a generic Archivematica AIP, described at [[AIP_structure]]. There are additional metadata files that are included in a Dataverse-derived AIP, and each zipped bundle that is included in the ingest will result in a separate directory in the AIP. The following is a sample structure.

'''Bag structure'''

The Archival Information Package (AIP) is packaged in the Library of Congress BagIt format, and may be stored compressed or uncompressed:

<pre>
Pacific_weather_patterns_study-dfb0b75d-6555-4e99-a8d8-95bed0f6303f.7z
├── bag-info.txt
├── bagit.txt
├── manifest-sha512.txt│
├── tagmanifest-md5.txt
└── data [standard bag directory containing contents of the AIP]</pre>

'''AIP structure'''

All of the contents of the AIP reside within the data directory:

<pre>

├── data
│ ├── logs [log files generated during processing]
│ │ ├── fileFormatIdentification.log
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ └── logs
│ │ ├── extractContents.log
│ │ ├── fileFormatIdentification.log
│ │ └── filenameCleanup.log
│ ├── METS.dfb0b75d-6555-4e99-a8d8-95bed0f6303f.xml [the AIP METS file]
│ ├── objects [a directory containing the digital objects being preserved, plus their metadata]
│ ├── chelan_052.jpg [an original file from Dataverse]
│ ├── Weather_data.sav [an original file from Dataverse]
│ ├── Weather_data [a bundle retrieved from Dataverse]
│ │ ├── Weather_data.xml
│ │ ├── Weather_data.ris
│ │ ├── Weather_data-ddi.xml
│ │ └── Weather_data.tab [a TAB derivative file generated by Dataverse]
│ ├── metadata
│ │ └── transfers
│ │ └── Pacific_weather_patterns_study-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ │ ├── agents.json [information about the source of the data, used to populate the
PREMIS Dataverse agent in the AIP METS file]
│ │ ├── dataset.json [the full json file retrieved from Dataverse]
│ │ └── METS.xml [the METS file generated by the ingest script to prepare
Dataverse contents for ingest into Archivematica]
│ └── submissionDocumentation
│ └── transfer-58-1a0f309a-d3ec-43ee-bb48-a868cd5ca85c
│ └── METS.xml [a standard transfer METS file generated to list all contents of
an Archivematica transfer]
</pre>

'''AIP METS file structure'''

The AIP METS file records information a bout the contents of the AIP, and indicates the relationships between the various files in the AIP. A sample AIP METS file would be structured as follows:

<pre>
METS header
-Date METS file was created
METS dmdSec [descriptive metadata section]
-DDI XML metadata taken from the METS transfer file, as follows
--ddi:title
--ddi:IDno
--ddi:authEnty
--ddi:distrbtr
--ddi:version
--ddi:restrctn
METS dmdSec [descriptive metadata section]
-link to dataset.json
METS dmdSec [descriptive metadata section]
-link to DDI.XML file created for derivative file as part of bundle
METS amdSec [administrative metadata section, one for each original, derivative and normalized file in the AIP]
-techMD [technical metadata]
--PREMIS technical metadata about a digital object, including file format information and extracted metadata
-digiprovMD [digital provenance metadata]
--PREMIS event: derivation (for derived formats)
-digiprovMD [digital provenance metadata]
--PREMIS event:ingestion
-digiprovMD [digital provenance metadata]
--PREMIS event: unpacking (for bundled files)
-digiprovMD [digital provenance metadata]
--PREMIS event: message digest calculation
-digiprovMD [digital provenance metadata]
--PREMIS event: virus check
-digiprovMD [digital provenance metadata]
--PREMIS event: format identification
-digiprovMD [digital provenance metadata]
--PREMIS event: fixity check (if file comes from Dataverse with a checksum)
-digiprovMD [digital provenance metadata]
--PREMIS event: normalization (if file is normalized to a preservation format during Archivematica processing)
-digiprovMD [digital provenance metadata]
--PREMIS event: creation (if file is a normalized preservation master generated during Archivematica processing)
-digiprovMD
--PREMIS agent: organization
-digiprovMD
--PREMIS agent: software
-digiprovMD
--PREMIS agent: Archivematica user
METS fileSec [file section]
-fileGrp USE="original" [file group]
--original files uploaded to Dataverse
-fileGrp USE="derivative"
--derivative tabular files generated by Dataverse
-fileGrp USE="submissionDocumentation"
--METS.XML (standard Archivematica transfer METS file listing contents of transfer)
-fileGrp USE="preservation"
--normalized preservation masters generated during Archivematica processing
-fileGrp USE="metadata"
--dataset.json
--DDI.XML
--xcitation-endnote.xml
--xcitation-ris.ris
METS structMap [structural map]
-directory structure of the contents of the AIP</pre>

Community/Camps/Baltimore2018

2018-04-15T21:13:45Z

Rspencer: background information for ross spencer

The lovely city of Baltimore, Maryland is the location of our next eastern seaboard Archivematica Camp, cohosted by the Sheridan Libraries at [https://www.library.jhu.edu/ Johns Hopkins University].

Inspired by similar events like Hydra Camp, Islandora Camp and Fedora Camp, Archivematica Camp provides a space for anyone interested in or currently using Archivematica to come together, learn about the platform from other users, and share their experiences.

== Dates and Location ==

'''WHEN: Wednesday, June 20 to Friday, June 22, 2018'''

'''WHERE: [https://goo.gl/maps/jagXhcVHK232 Brody Learning Commons Room 5015/5017,]''' Johns Hopkins University, Baltimore, MD.

The Brody Learning Commons is connected to the Milton S. Eisenhower Library, the combination of which is generally known as "the Library." Security guards will be advised that the conference is happening and can direct you to the room. Please bring photo ID.

=== Getting there ===

The Johns Hopkins University campus is located in north Baltimore, approximately 15 minutes by car and 30 minutes by transit from downtown Baltimore. Parking information is here: http://ts.jhu.edu/Parking/Visitors/

Two free bus lines operate that drop riders off at the corner of 33rd Street and Charles Street, just in front of the Library.

* JHU Shuttle: http://ts.jhu.edu/Shuttles/index.html

* Charm City Circulator Purple Line: https://www.charmcitycirculator.com/route/purple

Both of these bus services have apps that we'd recommend downloading to receive real-time updates about timetables. Both buses run roughly every 15 minutes.

=== Accessibility ===

The Brody Learning Commons is wheelchair-accessible from both entrances, but we'd recommend the south entrance from the service road, next to a building called Merrick Barn. Contact info@artefactual.com for more information and we'll put you in touch with JHU staff. We're happy to help!

== Registration ==

[https://www.eventbrite.com/e/archivematica-camp-baltimore-registration-45045169314 '''Register here!''']

Registration will open on April 16.

=== Full registration ===

'''Early bird''': $400 USD, available until May 20th

'''Full price''': $450 USD, available until June 17th (or we sell out)

Full registration gets you three full days of training, discussion, and deep dive sessions into related topics like METS visualization and preservation planning, taught by experienced Archivematica users and developers! Full registration also includes coffee and tea breaks for all three days and free food at the group social event on Wednesday night.

=== Day 1 only ===

$150 USD - available from May 20th until space is filled

On Day 1 of the camp, we'll focus on basic Archivematica knowledge - what it is, what it does, and how to preserve simple data using common workflows. This is a great opportunity for those who want to get a taste of what Archivematica can do, and don't need the intensive 3-day experience. The 1-day registration includes coffee and tea breaks as well as free food at the group social event on Wednesday night.

Note that we won't know how many one-day registrations we can accommodate until we know how many full registrants don't intend to attend Day 1. It's a bit complex for us to manage, so please be patient! If you really want to attend Day 1 only, please email us at info@artefactual.com with the subject line "Archivematica Camp Baltimore" and let us know - we'll make sure you're notified before tickets go on sale.

==Know before you go==

'''Digital preservation knowledge expectations'''

If you've registered because you are curious about Archivematica but not sure what a SIP, DIP, or AIP is, you might want to do some basic digital preservation reading before Day 1. We recommend Ashley Blewer's [http://training.ashleyblewer.com/presentations/oais.html OAIS slidedeck] as a good starting place to learn about the OAIS reference model, on which Archivematica is based. If you want to dive deeper, there are a lot of OAIS and digital preservation primers out there!

'''Bring a laptop'''

We have hands-on sessions planned and Archivematica pipelines ready for learning. You may want to set up your own pipeline using vagrant and VirtualBox - this isn't required, but if you're keen then more power to you! You'll find instructions to do that [https://www.archivematica.org/en/docs/archivematica-1.7/getting-started/quick-start/quick-start/#quick-start-install here] (please do this before you arrive at camp!).

'''Food'''

Coffee and tea breaks will be catered. However, the rooms where we're hosting the camp do not permit food, so only beverages will be provided. There is a cafe two floors up from the training rooms where you can grab a snack.

As the campus is within an easy walking distance of many restaurants, cafes, and food courts, we decided to keep registration costs lower by not catering in lunches. We'll provide plenty of time for you to seek out nourishment, and have provided a list of nearby dining options below.

'''Code of conduct'''

We are committed to providing a welcoming and positive experience for everyone, whether you are in a session, online, or in a social setting. This means that Archivematica Camp Amsterdam is governed by a Code of Conduct, which you can read at the bottom of this page.

'''Questions or concerns?'''

Please email us at info@artefactual.com with "Archivematica Camp Baltimore" in the subject line and we'll get back to you as soon as possible.

== Camp staff ==

'''Tim Walsh''' is our amazing community counsellor for this camp! Community counsellors are experienced Archivematica users who have a proven track record of working with the software in innovative ways and engaging with the Archivematica community. Tim has worked with Archivematica since 2015 and lead the effort of implementing a largely automated multi-pipeline Archivematica setup at the Canadian Centre for Architecture. Tim is also the developer of METSFlask, a web application for exploring the contents of Archivematica METS files. He'll primarily be talking about his practical experience of using Automation Tools to automate transfer and ingest as well as visualization and post-ingest use of METS files, but is happy to discuss anything related to Archivematica.

'''Sara Allain''' started working at Artefactual on both the AtoM and Archivematica projects in 2016, doing requirements analysis, user documentation, QA testing and community support. Prior to her work at Artefactual Sara worked in academic libraries as an archivist and/or librarian, tackling wide-ranging subjects like archival description, data migration, Islandora implementation, digitization management, and library communications. She'll be teaching introductory Archivematica, preservation planning, and probably several other bits and pieces along the way.

'''Ross Spencer''' is a software developer for Artefactual Systems Inc. Focused primarily on the Archivematica project, Ross has a background in digital preservation research and analysis. Having previously worked for Archives New Zealand, and before then, The National Archives, UK, he has experience with the unique requirements a government workflow. Ross also has a much broader interest in all things digital preservation, archives, and code. Ross will be discussing APIs, optimisation, contributing code, and issues! And much much more.

'''Evelyn McLellan'''

== Schedule (subject to change) ==

=== Wednesday, June 20: Day One (optional) - Archivematica for Beginners ===

Archivematica Camp kicks off with an optional day of training aimed at attendees who are new to Archivematica. We'll introduce the project, go over the basics of technical architecture, and look at the core functionality. The goal of this day is to ensure that everyone is on solid ground before we dive into more complex discussions. Note: participants are welcome to join the morning session, afternoon session, or both sessions as they wish. Participants may also wish to join the Camp on Day Two.

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff;"|'''Time'''
|style="white-space: nowrap; width: 90%; background-color: #93c9ff"|'''Streams 1 & 2'''

|-
|style="white-space: nowrap;"|9:00am - 10:15pm
|'''Module 1: Archivematica in the digital preservation landscape'''
Topics covered include: where does Archivematica fit into the digital preservation landscape, what does Archivematica promise to do, what doesn't Archivematica do

|-
|style="white-space: nowrap;"|10:15am - 10:30pm
|'''Coffee break'''

|-
|style="white-space: nowrap;"|10:30am-12:00pm
|'''Module 2: How does Archivematica actually work?'''
Topics covered include: Archivematica's conceptual model, packages (transfers, bags, SIPs, AIPs, DIPs), microservices as implementations of preservation actions, workflows, storage spaces (how the storage service works and how to get at your content)

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|'''Lunch'''

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|'''Module 3: Archivematica's core functionality'''
Topics covered include: understanding Archivematica's dashboard, including the Transfer tab, Ingest tab, Administration tab, and Preservation planning tab plus a first look at an AIP produced by Archivematica. This afternoon session includes hands-on exercises.

|-
|style="white-space: nowrap;"|3:00pm - 3:15pm
|'''Coffee break'''

|-
|style="white-space: nowrap;"|3:15pm - 4:00pm
|'''Module 3: Archivematica's core functionality'''
Continuation of Module 3.
|-
|style="white-space: nowrap;"|5:30pm - 8:30pm
|'''Group social event'''
Happy hour and food at [http://www.peabodyheightsbrewery.com/ Peabody Heights Brewery]! Peabody Heights Brewery is a 15 minute walk from the Library. We'll be ordering in food (pizza? tacos? we haven't decided yet!) and we'll make sure there are nonalcoholic alternatives as well. If the weather is nice, the brewery has a large beer garden where we can relax, chat, and gear up for two more days of learning.
|}


=== Thursday, June 21: Day Two ===

Days Two and Three are divided into two streams. Stream 1 is intended for front-end users who are responsible for day-to-day operations. Stream 2 is intended for back-end users who are responsible for development and maintenance of the software. Streams occasionally converge, and we encourage attendees to choose the session that most interests them - feel free to move around!

{| border="1" style="border-collapse:collapse;"
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Stream 1'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Stream 2'''

|-
|style="white-space: nowrap;"|9:00am - 9:30am
|colspan="2"|'''Arrival and registration'''

|-
|style="white-space: nowrap;"|9:30am - 10:45am
|'''Module 4: Building on core: Archivematica's specialized workflows'''
Topics covered include transfers of zipped files, disk images, metadata.csv, manual normalization, and a look at automation tools. This module includes hands-on exercises.
|'''Module 4: Supporting Archivematica workflows'''
Topics covered include how to install automation tools, REST APIs, logs, maintenance, and upgrades.

|-
|style="white-space: nowrap;"|10:45am - 11:00am
|'''Coffee break'''

|-
|style="white-space: nowrap;"|11:00am - 12:00pm
|colspan="2"|'''Community profile: Practical experience with automation tools (Tim Walsh)'''

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|'''Lunch'''

|-
|style="white-space: nowrap;"|1:00pm - 2:45pm
|'''Module 5: Archivematica's non-core functionality'''
Topics covered include a look at the Backlog and Appraisal tab, AIP reingest, AIP encryption, transcription, and using Mediaconch. This module includes hands-on exercises.
|'''Module 5: Understanding Archivematica's logs and peformance evaluation'''

|-
|style="white-space: nowrap;"|2:45pm - 3:00pm
|'''Coffee break'''

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Community profile: Johns Hopkins University'''

|-
|style="white-space: nowrap;"|4:00pm - 5:00pm
|colspan="2"| '''Agenda adjustment for Day 3'''
What do you want to talk about tomorrow? Also, wrap-up.

|-
|style="white-space: nowrap;"|6:00pm - late
|colspan="2"| '''Small group dinners'''
We'll create a sign-up sheet containing a list of restaurants nearby. Campers can sign up to attend dinner at whichever restaurant piques their interest! This is a great way to meet new people and get out on the town. We ask that you limit the group size to 4-6 people, so that the restaurants aren't overwhelmed!
|}


=== Friday, June 22: Day Three ===

Day Three is a bit more loosely-structured than Days One and Two. Based on camper interest, the schedule may be adjusted to accommodate new topics. Plus, there's plenty of time to talk about how Archivematica works - or could work - in your institution.

{| border="1" style="border-collapse:collapse; "
|style="white-space: nowrap; width: 10%; background-color: #93c9ff"|'''Time'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Stream 1'''
|style="white-space: nowrap; width: 45%; background-color: #93c9ff"|'''Stream 2'''

|-
|9:00am - 9:30am
|colspan="2" style="white-space: nowrap; width: 90%;"|'''How this day is going to unfold'''

|-
|9:30pm - 10:30pm
|'''Module 6: AIPs for Archivists'''
Topics covered include how the AIP is structured, and a close look at the METS file
|'''Module 6: AIPs for developers/systems administrators'''
Topics covered include how the METS file is written from the database

|-
|style="white-space: nowrap;"|10:30-12:00
|colspan="2"|'''Community topics'''
Based on topics raised from campers at registration and as the camp has progressed. Led by Artefactual staff with lots of input from the community. Breakout into 2 rooms as needed. Informal and fun!

|-
|style="white-space: nowrap;"|12:00pm - 1:00pm
|colspan="2"|Lunch

|-
|style="white-space: nowrap;"|1:00pm - 3:00pm
|colspan="2"|'''Module 7: Implementation roundtable'''
Discussion of various ways Archivematica can be implemented. Campers are invited to share their particular Archivematica implementations, real or imagined, in 10-15 minute informal talks. We'll post a sign-up list on the first day of camp.

|-
|style="white-space: nowrap;"|3:00pm-4:00pm
|colspan="2"|'''Module 8: Archivematica in the community'''
Topics covered include how the code is written, collaborative digital preservation, and a look at the Archivematica roadmap.
Wrap-up and safe travels!
|}

== Local Arrangements ==

=== Hotels ===

As there are few hotel options within walking distance of Johns Hopkins, we've secured room blocks at two hotels: the Inn at the Colonnade and the Radisson at Cross Keys.

====Inn at The Colonnade Baltimore====

[https://www.google.ca/maps/dir/Inn+at+The+Colonnade+Baltimore+-+A+DoubleTree+by+Hilton+Hotel,+West+University+Parkway,+Baltimore,+MD,+USA/Milton+S+Eisenhower+Library,+North+Charles+Street,+Baltimore,+MD,+USA/@39.3313599,-76.6213098,17z/data=!3m2!4b1!5s0x89c804dff3aee8c7:0x5030d558bfc93488!4m13!4m12!1m5!1m1!1s0x89c80521e8b11795:0x9d5a52f5ed8dfc78!2m2!1d-76.619735!2d39.334029!1m5!1m1!1s0x89c804dfa8c04b1f:0xf0f3792b68b355da!2m2!1d-76.619425!2d39.329053 4 West University Parkway, Baltimore, Maryland 21218-2306]

The Inn at the Colonnade is just across the street from Johns Hopkins campus, an 8 minute walk to the Milton S. Eisenhower Library.

To book a room at the Inn at the Colonnade, use [http://doubletree.hilton.com/en/dt/groups/personalized/B/BWICUDT-AMC-20180619/index.jhtml?WT.mc_id=POG this link].

====Radisson at Cross Keys====

[https://www.google.ca/maps/dir/Radisson+Hotel+at+Cross+Keys,+Baltimore,+5100+Falls+Rd,+Baltimore,+MD+21210,+USA/Johns+Hopkins+University,+Baltimore,+MD,+USA/@39.3371967,-76.6496263,14z/data=!3m1!4b1!4m13!4m12!1m5!1m1!1s0x89c81ab45a596f8d:0x73744f706126b216!2m2!1d-76.6467539!2d39.3551217!1m5!1m1!1s0x89c804df8502f88d:0x303d58494fa04c66!2m2!1d-76.6205177!2d39.3299013 5100 Falls Rd, Baltimore, MD 21210]

The Radisson at Cross Keys is a 10 minute drive or 30 minute walk/transit trip to the Milton S. Eisenhower Library (20 minutes walking, 10 minutes on the bus).

To book a room at the Radisson at Cross Keys, follow these steps:

#Go to [https://www.radisson.com/baltimore https://www.radisson.com/baltimore]
#Under Modify Search Select the dates you are checking in and checking out and click More Search Options
#Please enter in the Promotional Code box - '''JHUGRP'''
#Click Search Again
#Select the hotel (only your group’s name should appear) by selecting Book Now
#A Room Options screen will appear. Please select the bed type that you are requesting which is located at the bottom of the page. Once you select the bed type you are requesting you are then prompted to your reservation profile screen where you would input your contact and payment information.

If you have questions, please contact the hotel directly by calling Reservations at 1-800-333-3333 and say you are making a reservation for "The Archives Matika Group" (yes, we're aware this is incorrect!) & the dates you will be staying at the hotel.

====AirBnB====

Any AirBnBs within a 15 walk from campus in all directions are perfectly safe.

The city has two free bus lines (http://ts.jhu.edu/Shuttles/TransLoc/ and https://www.charmcitycirculator.com/route/purple) so you can also stay anywhere between campus and the Inner Harbor and get here within 30 minutes.

=== Food & drink ===

All are within walking distance of the Library.

*[https://r.housebaltimore.com/ R House] - Fancy food court and bar

*[http://www.partsandlaborbutchery.com/ Parts and Labor] - “Everything but the snout”-style restaurant for meat eaters

*[http://places.singleplatform.com/one-world-cafe/menu One World] - Unpretentious vegetarian-ish restaurant dating back to before vegetarian was hip

*[http://www.sweet27.com/ Sweet 27] - Amusing variety of cuisines, but good vegan bakery and Indian food

*[http://www.tambersrestaurant.com/ Tamber’s] - Diner and Indian — a very Baltimore mash-up

*[http://www.charlesvillagepub.net/index.html Charles Village Pub] - Venerable Baltimore/Charles Village dive with straightforward grill fare

*[https://www.eatthb.com/ Towson Hot Bagel] - Omelettes, sandwiches, and (of course) bagels

*[http://www.pekopekoramen.com/ Peko Peko Ramen] - Chicken broth ramen, rice bowls, mochi ice cream

*[https://www.chipotle.com/ Chipotle] - Burritos and the like

*[http://eddiesmarket.com/ Eddies Market of Charles Village] - Supermarket with a deli in the back

==Code of Conduct==

The Archivematica community is dedicated to providing a welcoming and positive experience for everyone, whether they are in a formal session or a social setting related to an Archivematica event, or are taking part in activities online. Archivematica community participants come from all over the world and bring with them a wide variety of professional, personal and social backgrounds; whatever these may be, we treat colleagues with dignity and respect. We are sensitive to the fact that the international nature of the Archivematica community means that we span many different social norms around language and behaviour and we strive to conduct ourselves in ways that are unlikely to cause offense. In the event that someone’s conduct is causing offense or distress, the Archivematica Camp has a detailed Anti-Harassment Policy, details below, which can be applied to address the problem.

The first step in dealing with any serious misconduct is to contact a member of the organizing group, or a counsellor at an Archivematica Camp (please see counsellors, above). These people will be made known before and during any Archivematica Camp event.

'''Anti-Harassment Policy'''

The Archivematica community is dedicated to providing a harassment-free conference/camp experience for everyone. If you have been harassed, please consult this protocol for conflict resolution which makes clear how we can help.

The Archivematica community has established the following policy to make it clear that we do not tolerate harassment in any form. Sexual or discriminatory language and imagery are not appropriate for any event venue, including talks, or any other communication channel used during the conference (such as social media).

Harassment includes:

* offensive verbal comments related to sex, gender, ethnicity, nationality, socioeconomic status, sexual orientation, disability, physical appearance, body size, age, race, religion;
* sexual or discriminatory images in public spaces;
* deliberate intimidation;
* stalking;
* harassing photography or recording;
* sustained disruption of talks or other events;
* inappropriate physical contact; and
* unwelcome sexual attention.

The lingua franca of Archivematica Camp is English; however, English may not be the native language of Archivematica Camp participants. Further, cultural norms around what may be considered obscene and offensive communication will vary among Archivematica Camp attendees. Campers are asked to please be mindful of the international character of Archivematica Camp, and to use respectful and clear language free of slang to facilitate communication.

Participants asked to stop any harassing behavior are expected to comply immediately.

If a participant engages in harassing behavior, event organizers and Archivematica representatives (or their designees) may take any action they deem appropriate, including warning the offender, expulsion from Archivematica Camp, or contacting a higher authority such as a representative from the offender’s institution.

If a participant raises a concern relating to actions covered by this policy in good faith, there will be no retaliation for bringing forward their concern. Threatening or taking action against someone for invoking this policy or for participating in any related investigation will be considered a violation of this policy.

Participants are expected to follow the anti-harassment policy at all Camp-related venues, Camp-related social events, and online communication channels.

We expect attendees and speakers past and present to be respectful to each other, and we will deal with any incidents that arise, including on social media.

We value your participation in the Archivematica community and your support in keeping the Archivematica community a safe, welcoming, and friendly space for fellow participants.

'''Acknowledgments'''

This policy is based on the [http://www.pasignyc.org/#code-of-conduct-section PASIG Code of Conduct], portions of which were modified from the [http://or2016.net/code-of-conduct/ Open Repositories’ code of conduct], which was itself modified from [https://www.seattleattic.com/codeofconduct Seattle Attic’s code of conduct] and the [http://adainitiative.org/what-we-do/events/ada-initiative-event-anti-harassment-policy/ Ada Initiative’s event harassment policy].

'''License'''

This document is licensed under [https://creativecommons.org/licenses/by-sa/3.0/ Creative Commons BY-SA].

AIP pointer file

2018-03-19T20:46:12Z

Rspencer: Summary entry

AIP pointer file

2018-03-12T03:48:30Z

Rspencer: Ways of finding the pointer file

Documentation in progress

== Access ==

Accessing pointer files inside Archivematica can be done by navigating the Dashboard to '''Archival storage''' finding a suitable AIP, e.g. at time of writing a compressed AIP, and under '''Pointer file''' clicking '''View'''. Archivematica will make a call to the Storage Service API to return the file for that AIP.

* [https://wiki.archivematica.org/Storage_Service_API#Get_pointer_file Storage Service API reference to return a Pointer File]

== Developer Notes ==

=== Location inside Storage Service ===

Pointer file are stored in internal Storage Service directories, e.g.

/var/archivematica/storage_service/d5c3/3360/7b92/4a2f/9002/32f5/a143/daaf/pointer.d5c33360-7b92-4a2f-9002-32f5a143daaf.xml

You can see where they are stored looking up the Storage Service database:

<source lang="sql">
select locations_location.uuid,
locations_location.relative_path,
locations_location.description,
locations_location.purpose
from locations_location
inner join locations_package
on locations_location.uuid =
locations_package.pointer_file_location_id
</source>

== Sample Pointer Files ==

* [https://wiki.archivematica.org/Dataset_preservation#Sample_pointer.xml_file Sample XML Pointer File]
* [https://wiki.archivematica.org/LOCKSS_Integration#Sample_pointer_files LOCKSS Integration Pointer File]

== External Links ==

* [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ Justin Simpson on the structure and purpose of pointer files at Google Groups]
* [https://projects.artefactual.com/issues/5159 Feature #5159 The original tranche of work resulting in the implementation of pointer files]

AIP pointer file

2018-03-12T02:25:57Z

Rspencer: Wiki links

Documentation in progress

== Samples ==

* [https://wiki.archivematica.org/Dataset_preservation#Sample_pointer.xml_file Sample XML Pointer File]
* [https://wiki.archivematica.org/LOCKSS_Integration#Sample_pointer_files LOCKSS Integration Pointer File]

== External Links ==

* [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ Justin Simpson on the structure and purpose of pointer files at Google Groups]
* [https://projects.artefactual.com/issues/5159 Feature #5159 The original tranche of work resulting in the implementation of pointer files]

AIP pointer file

2018-03-12T02:17:00Z

Rspencer: Redmine ticket for pointer files

Documentation in progress

== External Links ==

* [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ Justin Simpson on the structure and purpose of pointer files at Google Groups]
* [https://projects.artefactual.com/issues/5159 Feature #5159 The original tranche of work resulting in the implementation of pointer files]

AIP pointer file

2018-03-12T02:15:54Z

Rspencer:

AIP pointer file

2018-03-12T02:15:38Z

Rspencer: Adding external links to pointer files

Documentation in progress

== External Links ==

* [https://groups.google.com/d/msg/archivematica/VPQGd4s7hI8/x3WFdqO9AQAJ link Justin Simpson on the structure and purpose of pointer files at Google Groups]

Improvements/Avalon

2018-03-09T01:50:08Z

Rspencer:

This page will document potential areas of improvement in Archivematica that will allow for better workflows with [http://www.avalonmediasystem.org/ Avalon].

== Project Thesis ==

Artefactual and Avalon will seek to create:

* support for activities such as file characterization and preservation metadata extraction:
** digitized and born-digital AV materials.
* Integration with Archivematica or its components into Avalon ingestion workflows to provide:
** preservation metadata
** better packaging for long-term digital preservation activities

== Use Cases ==

=== In scope ===

Two clear use-cases based on previous discussion (up to 8 March 2018), for which we can focus on and keep scope narrow. These are:

*Avalon first
** Process in Avalon and then send the package to Archivematica to generate an AIP

'''Questions'''

- Do we ignore the Fedora metadata entirely?
- Do we transfer Fedora metadata as a digital object, but perform no special conversion mapping
- Out-of-scope: Mapping/cross-walk to try and see Fedora metadata mapped into Archivematica

* AM first
** Collection of assets described in a batch manifest and they are processed and then picked up by Avalon

Two useful phrasings, that could be considered nuances on these, that came from the same meeting:

* ''As an user of Avalon send the output of a project to a repository and have institutional digital preservation policies applied to that collectionffor long-term preservation''
* ''As a preservation specialist, in a catastrophic scenario, be able to round-trip from AM back into an instance of Avalon''**

**'''NB''' Recognised that there is no hard-requirement to round-trip back into Avalon from Archivematica, though it was noted that it would be possible to learn quite a lot.

=== Out of scope ===

'''Versioning'''
This is a very broad summary, but because of the potential for work to continue being completed on a 'unit' in Avalon, data will become out of date, and there may be a wish to see that updated in Archivematica. This is one example of where data might need updating or to be re-versioned. Others:
* AIP Update following updated in Avalon
* Data supplied as an asset to Avalon/Archivematica, e.g. VTT Subtitle Information, (addition of other languages/corrections, etc.)
* Other cases

== What do we need? ==

* Northwest to work with Sarah to discuss use-cases
* Do we need more sample files/batches of data?
* Understand the current transfer issue being experienced with the initial setup - In progress.

== Questions ==

=== METS ===

* Is it required that the metadata in the batch ingest manifest is represented in the AIP METS file (in dmdSecs for example?)
* Will users have a need to populate the batch ingest manifest with metadata entered into Archivematica?
** Descriptive metadata?
** Rights metadata?
* When Avalon is being used to disseminate different quality files, which file(s) should be stored in the AIP?
** How should mulitple versions be modeled in the METS?
* Should the structure of files be modeled in the METS when structure XML files are included?
** Or is describing the presence and use of structure files sufficient?
* How should the presence of caption files be noted in the METS?

=== Exploration of Archivematica ===

* Progress and first impressions
* Ingest progress
* Exploration of the AIP generated from Avalon Batch Packages
* Immediate observations:
** Workflow questions
** Perceived or real gaps
** Questions Artefactual can answer about what Archivematica is doing/should be doing
* Relationship between Archivematica and Avalon, e.g position in workflow, pre-Avalon? post-Avalon? Something else?

Improvements/Avalon

2018-03-09T01:38:35Z

Rspencer:

This page will document potential areas of improvement in Archivematica that will allow for better workflows with [http://www.avalonmediasystem.org/ Avalon].

== Project Thesis ==

Artefactual and Avalon will seek to create:

* support for activities such as file characterization and preservation metadata extraction:
** digitized and born-digital AV materials.
* Integration with Archivematica or its components into Avalon ingestion workflows to provide:
** preservation metadata
** better packaging for long-term digital preservation activities

== Use Cases ==

=== In scope ===

Two clear use-cases based on previous discussion (up to 8 March 2018), for which we can focus on and keep scope narrow. These are:

*Avalon first
** Process in Avalon and then send the package to Archivematica to generate an AIP

'''Questions'''

- Do we ignore the Fedora metadata entirely?
- Do we transfer Fedora metadata as a digital object, but perform no special conversion mapping
- Out-of-scope: Mapping/cross-walk to try and see Fedora metadata mapped into Archivematica

* AM first
** Collection of assets described in a batch manifest and they are processed and then picked up by Avalon

Two useful phrasings, that could be considered nuances on these, that came from the same meeting:

* ''As an user of Avalon send the output of a project to a repository and have institutional digital preservation policies applied to that collectionffor long-term preservation''
* ''As a preservation specialist, in a catastrophic scenario, be able to round-trip from AM back into an instance of Avalon''**

**'''NB''' Recognised that there is no hard-requirement to round-trip back into Avalon from Archivematica, though it was noted that it would be possible to learn quite a lot.

=== Out of scope ===

'''Versioning'''
This is a very broad summary, but because of the potential for work to continue being completed on a 'unit' in Avalon, data will become out of date, and there may be a wish to see that updated in Archivematica. This is one example of where data might need updating or to be re-versioned. Others:
* AIP Update following updated in Avalon
* Data supplied as an asset to Avalon/Archivematica, e.g. VTT Subtitle Information, (addition of other languages/corrections, etc.)
* Other cases

== What do we need? ==

- Northwest to work with Sarah to discuss use-cases
- Do we need more sample files/batches of data?
- Understand the current transfer issue being experienced with the initial setup - In progress.

== Questions ==

=== METS ===

* Is it required that the metadata in the batch ingest manifest is represented in the AIP METS file (in dmdSecs for example?)
* Will users have a need to populate the batch ingest manifest with metadata entered into Archivematica?
** Descriptive metadata?
** Rights metadata?
* When Avalon is being used to disseminate different quality files, which file(s) should be stored in the AIP?
** How should mulitple versions be modeled in the METS?
* Should the structure of files be modeled in the METS when structure XML files are included?
** Or is describing the presence and use of structure files sufficient?
* How should the presence of caption files be noted in the METS?

=== Exploration of Archivematica ===

* Progress and first impressions
* Ingest progress
* Exploration of the AIP generated from Avalon Batch Packages
* Immediate observations:
** Workflow questions
** Perceived or real gaps
** Questions Artefactual can answer about what Archivematica is doing/should be doing
* Relationship between Archivematica and Avalon, e.g position in workflow, pre-Avalon? post-Avalon? Something else?

Improvements/Avalon

2018-03-09T01:37:49Z

Rspencer: ross-spencer notes from meeting: 8 March 2018

This page will document potential areas of improvement in Archivematica that will allow for better workflows with [http://www.avalonmediasystem.org/ Avalon].

== Project Thesis ==

Artefactual and Avalon will seek to create:

* support for activities such as file characterization and preservation metadata extraction:
** digitized and born-digital AV materials.
* Integration with Archivematica or its components into Avalon ingestion workflows to provide:
** preservation metadata
** better packaging for long-term digital preservation activities

== Use Cases ==

=== In scope ===

Two clear use-cases based on previous discussion (up to 8 March 2018), for which we can focus on and keep scope narrow. These are:

*Avalon first
** Process in Avalon and then send the package to Archivematica to generate an AIP

'''Questions'''

- Do we ignore the Fedora metadata entirely?
- Do we transfer Fedora metadata as a digital object, but perform no special conversion mapping
- Out-of-scope: Mapping/cross-walk to try and see Fedora metadata mapped into Archivematica

* AM first
** Collection of assets described in a batch manifest and they are processed and then picked up by Avalon

Two useful phrasings, that could be considered nuances on these, that came from the same meeting:

* ''As an user of Avalon send the output of a project to a repository and have institutional digital preservation policies applied to that collectionffor long-term preservation''
* ''As a preservation specialist, in a catastrophic scenario, be able to round-trip from AM back into an instance of Avalon''**

**'''NB''' Recognised that there is no hard-requirement to round-trip back into Avalon from Archivematica, though it was noted that it would be possible to learn quite a lot.

==== Out of scope ===

'''Versioning'''
This is a very broad summary, but because of the potential for work to continue being completed on a 'unit' in Avalon, data will become out of date, and there may be a wish to see that updated in Archivematica. This is one example of where data might need updating or to be re-versioned. Others:
* AIP Update following updated in Avalon
* Data supplied as an asset to Avalon/Archivematica, e.g. VTT Subtitle Information, (addition of other languages/corrections, etc.)
* Other cases

== What do we need? ==

- Northwest to work with Sarah to discuss use-cases
- Do we need more sample files/batches of data?
- Understand the current transfer issue being experienced with the initial setup - In progress.

== Questions ==

=== METS ===

* Is it required that the metadata in the batch ingest manifest is represented in the AIP METS file (in dmdSecs for example?)
* Will users have a need to populate the batch ingest manifest with metadata entered into Archivematica?
** Descriptive metadata?
** Rights metadata?
* When Avalon is being used to disseminate different quality files, which file(s) should be stored in the AIP?
** How should mulitple versions be modeled in the METS?
* Should the structure of files be modeled in the METS when structure XML files are included?
** Or is describing the presence and use of structure files sufficient?
* How should the presence of caption files be noted in the METS?

=== Exploration of Archivematica ===

* Progress and first impressions
* Ingest progress
* Exploration of the AIP generated from Avalon Batch Packages
* Immediate observations:
** Workflow questions
** Perceived or real gaps
** Questions Artefactual can answer about what Archivematica is doing/should be doing
* Relationship between Archivematica and Avalon, e.g position in workflow, pre-Avalon? post-Avalon? Something else?

Improvements/Avalon

2018-03-08T02:04:41Z

Rspencer: Added notes about things to observe now that Archivematica has been deployed