Archivematica - User contributions [en]

Merging

2017-03-31T22:26:13Z

Hbecker: /* Code review & approval */ Update code review link

[[Main Page]] > [[Development]] > Merging & Branches

== Branching guidelines ==

Guidelines on when to create branches for new development in Archivematica projects. Short answer: always create branches!

* All development work happens in branches - nothing is committed directly to master, qa/ or stable/ branches
* Development branches should follow the naming format: <code>dev/issue-####-short-description</code>
** <code>dev/</code> marks it as being a development branch, not for QA integration or a stable release
** <code>issue-####</code> is the Redmine ticket that the work is mostly related to. If the branch expands in scope to include multiple tickets later, that's fine, and don't worry about renaming branches
** <code>short-description</code> is a couple word description of the branch, to make it easier to remember what the topic of the branch is.
** Example: <code>dev/issue-8161-dashboard-i18n</code> or <code>dev/issue-8693-dataverse</code>
* Development branches should generally be create from the branch they will be merged into. In most cases, this is master, qa/1.x (Archivematica) or qa/0.x (Storage Service). If something is a patch for an old release, it may be more appropriate to branch from the old stable branch (e.g. stable/1.5.x)
* All branches should undergo code review before being merged into the target branch. If something was developed for an old release, it should be cherry-picked to the current development branch, and a comment with the commit hash should be left in the pull request for tracking.

== Code review & approval ==

All code changes merged into a stable or qa branch must go through code review. We use Github's pull request feature for code review. Every change being submitted to an Artefactual project should be submitted as a pull request to the appropriate repository.

Another core contributor (usually an Artefactual employee) will review the code for content, style and suitability in the project. A more extensive guideline to code review is [https://github.com/artefactual/archivematica/blob/qa/1.x/code_review.md also available]. After the reviewer is satisfied with the code, they will approve it on github. If at least one reviewer approves, and there are no disapprovals, the branch can be merged.

In general, the original contributor will [[#Merge process | merge the branch]]. For external contributions, the reviewer can merge or cherry-pick the commits. This will leave the Author correct in git, but update the committer.

== Merge process ==

We use a rebase-based workflow, where branches are brought up to date with the latest stable by rebasing them. (This is compared to a merge-based workflow, where branches are merged, and conflicts resolved in the merge commit). This shows the process for merging development branch <code>dev/test</code> into target branch <code>master</code>. The target branch is usually master, qa/0.x or qa/1.x.

The instructions are formatted: <code>(current branch) $ <command to run></code>

# Check your remote is set to be gitolite
#* <code>$ git remote -v</code> should show <code>origin git@git.artefactual.com:<repository-name>.git</code>
#* Not all repositories have gitolite as the canonical repo - ask if you're unsure
# Ensure your copy of the target branch is up to date
#* <code>$ git checkout master</code>
#* <code>(master) $ git pull --rebase </code>
# Ensure your development branch is based off the latest target branch
#* <code>$ git checkout dev/test</code>
#* <code>(dev/test) $ git rebase master</code>
# Check that your changes still work after rebasing
# Check tests pass (see https://wiki.archivematica.org/Getting_started#Tests for more details)
#* <code>(dev/test) $ py.test</code>
# Push your newly-rebased branch to remote. Because this is a rebase, which changes history, you'll need --force. The rebased dev branch has to be pushed before the master branch is or github gets confused.
#* '''Never force push on a stable/, qa/ or master branch!'''
#* <code>(dev/test) $ git push -f</code>
# Fast-forward merge
#* <code>$ git checkout master</code>
#* <code>(master) $ git merge dev/test</code>
# Push merge!
#* <code>(master) $ git push</code>
# Delete dev branch
#* Locally: <code>(master) $ git branch -d dev/test</code>
#* Remote: <code>(master) $ git push origin --delete dev/test</code>
# Update the related ticket, if needed
# You're done!

== Proposed changes ==

=== Forks ===

* '''Current status''': All development branches are created in the artefactual (or artefactual-labs) owned repository, and core contributors (usually Artefactual employees) commit directly to the official repository
* '''Proposed change''': Development happens in contributor's forks, and the official repo contains only stable & QA branches
* '''Pros''':
** Fewer branches in the official repository, so there is less clutter and noise for people watching activity
** Handles more contributors, since there won't be dozens or hundreds of branches of uncertain development status
** Contribution process is the same for external and internal contributors
** Contributors can handle their repository branches however they want - naming conventions, deleting old branches or not, etc
* '''Cons''':
** Potential confusion about which repository is the correct official one
** Potential to push changes to the wrong repo (e.g. push a development branch to the official repo instead of the contributor's fork)
** Developers have to keep 2 repositories up to date: upstream and their fork
** Harder for analysts to check out development code, since they would have to set their remote to the contributor's fork

=== Hybrid merge/rebase model ===

* '''Current status''': All branches are rebased onto their target branch before being (fast-forward) merged
* '''Proposed change''': Small or minor branches are rebased, larger feature branches are true merged
* '''Pros''':
** Commit history clearly shows 'feature', and which commits came in together as a unit
** Rebasing small branches preserves clean history for minor changes
** Better reflects nature of development - shows minor changes as happening inline, and major changes as happening in parallel
** Merge commits are clear places to look for integration issues
* '''Cons''':
** Confusion about when to merged vs rebase
** Developers have to learn two workflows

[[Category:Process documentation]]

Archivematica API

2017-03-31T19:13:13Z

Hbecker: Add warning on waiting endpoint

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

[[Category:Development documentation]]

SIP Structure

2017-03-31T00:05:42Z

Hbecker: Links to features

[[Main Page]] > [[Development]] > [[:Category:Development documentation]] > SIP Structure

A Transfer or SIP in Archivematica has a number of special directories and files. Many of these are created by Archivematica, and should not be submitted by users. Others will trigger specific workflows if included in the transfer.

{|
|-
!Path
!File or Folder?
!Source?
!Usage
|-
|<code>/SIP_Folder</code>
|folder
|User
|Top level container for the the SIP
|-
|<code>/SIP_Folder/DIP</code>
|folder
|Archivematica
|This will become the [[DIP_structure|DIP]]
|-
|<code>/SIP_Folder/DIP/objects</code>
|folder
|Archivematica
|Contains the access derivatives
|-
|<code>/SIP_Folder/DIP/thumbnails</code>
|folder
|Archivematica
|Contains thumbnails for the DIP
|-
|<code>/SIP_Folder/logs</code>
|folder
|Archivematica
|Log files generated by Archivematica
|-
|<code>/SIP_Folder/logs/fileMeta</code>
|folder
|User
|Uncertain, used in the TRIM workflow
|-
|<code>/SIP_Folder/metadata</code>
|folder
|User
|Metadata files provided by the user. Depending on the file, these may be processed by Archivematica.
|-
|<code>/SIP_Folder/metadata/checksum.md5</code>
|text file
|User
|External checksums for original files. Format is the same as the output of <code>md5deep -rl objects/</code>. Also supported: <code>checksums.sha1</code> and <code>checksums.sha256</code>
|-
|<code>/SIP_Folder/metadata/file_labels.csv</code>
|CSV file
|User
|Labels for files in the METS structMap
|-
|<code>/SIP_Folder/metadata/metadata.csv</code>
|CSV file
|User
|File-level [[Metadata_import|DublinCore metadata]]
|-
|<code>/SIP_Folder/metadata/metadata.json</code>
|JSON file
|User
|File-level DublinCore metadata formatted as JSON. Will be converted to a CSV during processing
|-
|<code>/SIP_Folder/metadata/rights.csv</code>
|CSV file
|User
|[[Object-level_PREMIS_rights_import|File-level PREMIS Rights]] metadata.
|-
|<code>/SIP_Folder/METS.sip-uuid.xml</code>
|XML file
|Archivematica
|Contains all of the metadata on the METS
|-
|<code>/SIP_Folder/objects</code>
|folder
|User or Archivematica
|Original objects provide by the user. If this directory doesn't exist, all files in <code>/SIP_Folder</code> will be moved here by Archivematica
|-
|<code>/SIP_Folder/objects/access</code>
|folder
|User
|Triggers manually created access derivatives workflow, which is deprecated in favor of [[Manual_normalization_requirements_and_testing_procedures|manual normalization]] workflow.
|-
|<code>/SIP_Folder/objects/manualNormalization/normalization.csv</code>
|CSV file
|User
|CSV that provides a mapping for original objects of the same name to their access and preservation derivatives in [[Manual_normalization_requirements_and_testing_procedures|manual normalization]].
|-
|<code>/SIP_Folder/objects/manualNormalization/access</code>
|folder
|User
|Access derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|<code>/SIP_Folder/objects/manualNormalization/preservation</code>
|folder
|User
|Preservation derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|<code>/SIP_Folder/objects/service</code>
|folder
|User
|Service files - cleaned up versions of the original that are not derivatives. Normalization can be performed on these files instead of the originals
|-
|<code>/SIP_Folder/processingMCP.xml</code>
|XML file
|Archivematica or user
|Automates workflow decisions. Created by Archivematica, but can be submitted as part of the Transfer to override the default pipeline workflow choices.
|-
|<code>/SIP_Folder/submissionDocumentation</code>
|folder
|User
|Folder for files that are not original objects but should still be in the AIP
|-
|<code>/SIP_Folder/thumbnails</code>
|folder
|Archivematica
|Contains thumbnail derivatives
|}

[[Category:Development documentation]]

SIP Structure

2017-03-30T23:57:00Z

Hbecker: More details, formatting

[[Main Page]] > [[Development]] > [[:Category:Development documentation]] > SIP Structure

A Transfer or SIP in Archivematica has a number of special directories and files. Many of these are created by Archivematica, and should not be submitted by users. Others will trigger specific workflows if included in the transfer.

{|
|-
!Path
!File or Folder?
!Source?
!Usage
|-
|<code>/SIP_Folder</code>
|folder
|User
|Top level container for the the SIP
|-
|<code>/SIP_Folder/DIP</code>
|folder
|Archivematica
|This will become the DIP
|-
|<code>/SIP_Folder/DIP/objects</code>
|folder
|Archivematica
|Contains the access derivatives
|-
|<code>/SIP_Folder/DIP/thumbnails</code>
|folder
|Archivematica
|Contains thumbnails for the DIP
|-
|<code>/SIP_Folder/logs</code>
|folder
|Archivematica
|Log files generated by Archivematica
|-
|<code>/SIP_Folder/logs/fileMeta</code>
|folder
|User
|Uncertain, used in the TRIM workflow
|-
|<code>/SIP_Folder/metadata</code>
|folder
|User
|Metadata files provided by the user. Depending on the file, these may be processed by Archivematica.
|-
|<code>/SIP_Folder/metadata/checksum.md5</code>
|text file
|User
|External checksums for original files. Format is the same as the output of <code>md5deep -rl objects/</code>. Also supported: <code>checksums.sha1</code> and <code>checksums.sha256</code>
|-
|<code>/SIP_Folder/metadata/file_labels.csv</code>
|CSV file
|User
|Labels for files in the METS structMap
|-
|<code>/SIP_Folder/metadata/metadata.csv</code>
|CSV file
|User
|File-level DublinCore metadata
|-
|<code>/SIP_Folder/metadata/metadata.json</code>
|JSON file
|User
|File-level DublinCore metadata formatted as JSON. Will be converted to a CSV during processing
|-
|<code>/SIP_Folder/METS.sip-uuid.xml</code>
|XML file
|Archivematica
|Contains all of the metadata on the METS
|-
|<code>/SIP_Folder/objects</code>
|folder
|User or Archivematica
|Original objects provide by the user. If this directory doesn't exist, all files in <code>/SIP_Folder</code> will be moved here by Archivematica
|-
|<code>/SIP_Folder/objects/access</code>
|folder
|User
|Triggers manually created access derivatives workflow, which is deprecated in favor of manual normalization workflow.
|-
|<code>/SIP_Folder/objects/manualNormalization/normalization.csv</code>
|CSV file
|User
|CSV that provides a mapping for original objects of the same name to their access and preservation derivatives in manual normalization.
|-
|<code>/SIP_Folder/objects/manualNormalization/access</code>
|folder
|User
|Access derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|<code>/SIP_Folder/objects/manualNormalization/preservation</code>
|folder
|User
|Preservation derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|<code>/SIP_Folder/objects/service</code>
|folder
|User
|Service files - cleaned up versions of the original that are not derivatives. Normalization can be performed on these files instead of the originals
|-
|<code>/SIP_Folder/processingMCP.xml</code>
|XML file
|Archivematica or user
|Automates workflow decisions. Created by Archivematica, but can be submitted as part of the Transfer to override the default pipeline workflow choices.
|-
|<code>/SIP_Folder/submissionDocumentation</code>
|folder
|User
|Folder for files that are not original objects but should still be in the AIP
|-
|<code>/SIP_Folder/thumbnails</code>
|folder
|Archivematica
|Contains thumbnail derivatives
|}

[[Category:Development documentation]]

Release Process

2017-03-30T23:37:33Z

Hbecker: /* Update dependencies */ Combine metsrw & agentarchives

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

Release Process

2017-03-30T23:35:18Z

Hbecker: /* agentarchives */ Packaging instructions

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Update dependencies ==

=== metsrw ===

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, pull request, merge
# Tag new release, push tag
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

=== agentarchives ===

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, pull request, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

Release Process

2017-03-30T23:31:02Z

Hbecker: /* metsrw */ Add packaging instructions

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Update dependencies ==

=== metsrw ===

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, pull request, merge
# Tag new release, push tag
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

=== agentarchives ===

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

Release Process

2017-03-30T22:08:34Z

Hbecker: Stub dependencies section

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Update dependencies ==

=== metsrw ===

=== agentarchives ===

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

Release Process

2017-03-30T22:07:52Z

Hbecker: /* Overview */ Add dependencies

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

SIP Structure

2017-03-30T21:37:14Z

Hbecker: Expand magic files list

[[Main Page]] > [[Development]] > [[:Category:Development documentation]] > SIP Structure

A Transfer or SIP in Archivematica has a number of special directories and files that will trigger specific workflows.

{|
|-
!Path
!File or Folder?
!Usage
|-
|/SIP_Folder
|folder
|Top level container for the the SIP
|-
|/SIP_Folder/DIP
|folder
|Created by Archivematica for the DIP
|-
|/SIP_Folder/DIP/objects
|folder
|Created by Archivematica and contains the access objects
|-
|/SIP_Folder/DIP/thumbnails
|folder
|Created by Archivematica for thumbnails for the DIP objects
|-
|/SIP_Folder/logs
|folder
|Created by Archivematica for log files generated by Archivematica
|-
|/SIP_Folder/logs/fileMeta
|folder
|???
|-
|/SIP_Folder/metadata
|folder
|Metadata files provided by the user or created by Archivematica
|-
|/SIP_Folder/metadata/checksums.md5
|text file
|User-provided checksums for original files. Format is the same as the output of <code>md5deep -rl objects/</code>. Also supported: <code>checksums.sha1</code> and <code>checksums.sha256</code>
|-
|/SIP_Folder/metadata/file_labels.csv
|CSV file
|???
|-
|/SIP_Folder/metadata/metadata.csv
|CSV file
|User-provided file-level DublinCore metadata
|-
|/SIP_Folder/metadata/metadata.json
|JSON file
|User-provided file-level DublinCore metadata formatted as JSON. Will be converted to a CSV during processing
|-
|/SIP_Folder/METS.sip-uuid.xml
|XML file
|Created by Archivematica with all of the metadata on the METS
|-
|/SIP_Folder/objects
|folder
|Original objects provide by the user. If this directory doesn't exist, all files in /SIP_Folder will be moved here by Archivematica
|-
|/SIP_Folder/objects/access
|folder
|Triggers access workflow. Deprecated in favor of manual normalization workflow.
|-
|/SIP_Folder/objects/manualNormalization/normalization.csv
|CSV file
|User-provided CSV that provides a mapping for original objects of the same name to their access and preservation derivatives
|-
|/SIP_Folder/objects/manualNormalization/access
|folder
|Access derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|/SIP_Folder/objects/manualNormalization/preservation
|folder
|Preservation derivatives created outside of Archivematica. Should have the same filename as the original, but may have a different extension
|-
|/SIP_Folder/service
|folder
|User-provided service files. Normalization can be performed on these files instead of the originals
|-
|/SIP_Folder/submissionDocumentation
|folder
|Folder for user-provided files that are not original objects but should still be in the AIP
|-
|/SIP_Folder/thumbnails
|folder
|Created by Archivematica for thumbnail derivatives
|}

[[Category:Development documentation]]

Release Process

2017-03-30T20:40:31Z

Hbecker: Link to package creation in overview, move tagging section below packages

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# [[#Build deb/rpm packages]]
# [[#Tag Release]]
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/fido/Makefile</code>, and <code>deb/fido/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in dev/packages-1.6-docker
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)
This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

[[Category:Process documentation]]

Release Process

2017-03-28T23:45:34Z

Hbecker: Add branch update notes

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Create packages
# [[#Tag Release]]
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

Add packaging instructions here...

=== Package Siegfried ===

Add packaging instructions here...

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch

[[Category:Process documentation]]

Release Process

2017-03-28T23:39:37Z

Hbecker: Add PRONOM instructions

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Create packages
# Tag release
# Announce release

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request. This may trigger a release from FIDO

=== Package FIDO ===

Add packaging instructions here...

=== Package Siegfried ===

Add packaging instructions here...

=== Update FPR ===

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Install the latest stable Archivematica release
# Generate a JSON with the current version of the FPR (for use later)
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json</code>
# Install the latest development version of Archivematica
# Generate a blank data migration in the FPR-admin module
#* E.g. <code>./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt</code>
# Copy the output into the blank migration, add RunPython operation
#* <code>migrations.RunPython(data_migration),</code>
# Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports
#* Remove the <code>apps.get_model</code> lines
#* Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>
# Commit
# Deploy on testing pipeline
# (Analysts) Update the new entries
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
#* Remove the direct imports from the bug
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge

[[Category:Process documentation]]

Release Process

2017-03-28T23:29:35Z

Hbecker: Create page

This is an outline/checklist of the process to create Archivematica & Storage service releases.

[[Category:Process documentation]]

MCPClient

2017-03-27T23:48:12Z

Hbecker: /* Client scripts */ Expand client script info

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCPClient

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

Archivematica has one or more MCPClient instances to perform the actual work. They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.

== Client scripts ==
Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.

New scripts are defined in <code>[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/archivematicaClientModules src/MCPClient/lib/archivematicaClientModules]</code>, which is what is registered with Gearman on MCPClient startup.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: archivematicaClientModules lists both 'supportedCommandSpecial' and 'supportedCommands'. This distinction may have once been based on scripts that relied on external services, but serves no purpose now and should be removed.
|}

The name is what the [[MCPServer#StandardTasksConfigs | StandardTasksConfig]] table will refer to them as, and the value is the script that will be run. Some are defined as shell builtins (eg copy_v0.0 is cp). Most are paths to a script in the clientScripts directory, using the <code>%clientScriptsDirectory%</code> replacement variable. The name of the client script is usually the same as the name in archivematicaClientModules, but for very old scripts may have ‘archivematica’ at the beginning (eg createMETS_v2.0 = archivematicaCreateMETS2.py) or be named more pythonically (eg parseExternalMETS = parse_external_mets.py). Entries are added alphabetically.

The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen. Newer scripts may not have the version defined.

The list of client scripts is sorted roughly in order of appearance during processing

=== moveTransfer_v0.0 ===

* '''Purpose''': Move a Transfer & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveTransfer.py archivematicaMoveTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole Transfer and updates the database with the new location relative to the shared directory.

=== assignFileUUIDs_v0.0 ===

* '''Purpose''': Starts tracking files new to Archivematica
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaAssignFileUUID.py archivematicaAssignFileUUID.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': ingestion/reingestion, possibly registration
* '''FPR?''': No

This creates an entry in the Files table with the file's UUID, current & original paths and file group. It also creates an 'ingestion' Event and an 'registration' Event if an accession ID was specified. Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.

In ingest, is used on manually normalized files which may have been newly added, metadata and submission documentation.

On reingest, it parses the METS file instead of generating the file UUID, path & group. The Event type is 'reingestion'.

=== updateSizeAndChecksum_v0.0 ===

* '''Purpose''': Set file's size & checksum
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaUpdateSizeAndChecksum.py archivematicaUpdateSizeAndChecksum.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': message digest calculation
* '''FPR?''': No

Updates the entry in the Files table with a size and checksum. IT also generates a 'message digest calculation' Event.

On reingest, it parses the METS file instead of generating the checksums & sizes. It also re-adds Derivation & Format links.

Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].

=== createMETS_v0.0 ===

* '''Purpose''': Generate the transfer METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS.py archivematicaCreateMETS.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Creates the Transfer METS file. This will contain all the information generated on the transfer during processing, and is especially useful for backlogged transfers.

Not to be confused with [[#createMETS_v2.0]] for the AIP METS.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The Transfer METS file & related backlog functionality needs to be expanded. See [[Transfer_backlog_requirements#Proposed_improvements]] for details.
|}

=== createEvent_v0.0 ===

* '''Purpose''': Create an Event outside of other scripts
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/createEvent.py createEvent.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': as parameter: quarantine, unquarantine, placement in backlog, removal from backlog
* '''FPR?''': No

Used to generate events for when a file is placed in or removed from quarantine or backlog. These do not have scripts associated with them, so the event creation is handled here.

=== archivematicaClamscan_v0.0 ===

* '''Purpose''': Check for viruses in incoming files
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaClamscan.py archivematicaClamscan.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': virus check
* '''FPR?''': No

Runs clamscan on the file and generates a 'virus scan' event. If a scan has been run, it is not run again on the same file.

This is run on incoming files, files after extraction, metadata files and submission documentation. It is not run on normalized files.

=== sanitizeObjectNames_v0.0 ===

* '''Purpose''': Strip problematic characters from filenames
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeObjectNames.py sanitizeObjectNames.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': name cleanup
* '''FPR?''': No
* '''Tests''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_sanitize.py test_sanitize.py]

Sanitize object names by replacing anything that isn't an ASCII letter, number, hyphen, underscore, period or parenthesis. This runs on both files and directories, though only files have Events generated. An Event is generated for a file even if only a parent directory is sanitized.

This used code from [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeNames.py sanitizeNames.py]

=== identifyFileFormat_v0.0 ===

* '''Purpose''': Identify a file's format
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/identifyFileFormat.py identifyFileFormat.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': format identification
* '''FPR?''': IDCommand & IDRule

One of the most important scripts in Archivematica. Since the file format is used to determine many later actions (extraction, characterization, normalization etc), if this fails many important command later will also fail. This is the only script that uses the FPR that doesn't use the file format as a key for looking up what command to run. Instead, an IDCommand is selected and the output is matched to an IDRule to find the FormatVersion.

There is a short circuit handling of PRONOM ID (PUID) outputs. Since many FormatVersions have PUIDs, and both FIDO & Siegfried output PUIDs, this script looks for a FormatVersion with a given PUID. This reduces the number of IDRules that have to be created.

This also populates the legacy but still required FilesIDs table.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Only one identification tool can be run at a time currently. It would be better to allow a cascading of tools. E.g. if a file is identified as a video to subsequently run a tool specialized in identifying different types of video. Similarly, if the default tool failed, we could run a backup tool for a second opinion.
|}

=== extractContents_v0.0 ===

* '''Purpose''': Extract files from packages
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/extractContents.py extractContents.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': unpacking, registration, deletion, message digest calculation
* '''FPR?''': extract

Extracts files from a package (e.g. zip, tar, 7z etc) and optionally deletes the package. Generates Events for the new files.

=== characterizeFile_v0.0 ===

* '''Purpose''': Collects characterization information on a file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/characterizeFile.py characterizeFile.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': No
* '''FPR?''': characterization, default_characterization

Collects characterization commands for the provided file, then either
# Inserts the tool's XML output into the database, or
# Prints the tool's stdout, for tools which do not output XML

If a tool has no defined characterization commands, then the default will be run instead (currently FITS). Can run multiple characterization commands and log the output of all of them.

=== validateFile_v1.0 ===

* '''Purpose''': Validate a file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/validateFile.py validateFile.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': validation
* '''FPR?''': validation, default_validation

Validates files are correct, where correctness is defined by the file format.

=== examineContents_v0.0 ===

* '''Purpose''': Run bulk extractor for a detailed analysis of contents
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/examineContents.py examineContents.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': No
* '''FPR?''': No

Runs bulk extractor and stores the outputs in the logs directory.

=== elasticSearchIndex_v0.0 ===

* '''Purpose''': Index the Transfer METS into ElasticSearch when sending files to backlog
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/elasticSearchIndexProcessTransfer.py elasticSearchIndexProcessTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

The data in ElasticSearch is used by the Backlog tab, SIP Arrangement and the Appraisal tab when dealing with files from backlog. Note that this is not run if the transfer is not sent to backlog (since AM 1.5).

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The client config 'disableElasticsearchIndexing' can disable indexing, but this should be removed, since searching for files in backlog is required functionality.
|}

=== moveSIP_v0.0 ===

* '''Purpose''': Move a SIP & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveSIP.py archivematicaMoveSIP.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole SIP and updates the database with the new location relative to the shared directory.

=== normalize_v1.0 ===

* '''Purpose''': Generate a preservation or access derivative
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/normalize.py normalize.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': normalization
* '''FPR?''': Rules: access, preservation, thumbnail, default_access, default_thumbnail. Commands: normalization, verification

One of the most important scripts in Archivematica. This generates the preservation and access derivatives, as well as thumbnails. If manual normalization happens, this instead finds the manually normalized files and links them as derivatives.

The same script is used for generating preservation, access or thumbnail derivatives, controlled by the parameter passed in. In general, normalization is only performed on original files, but may be done on service files. This reads the FPR for the specific command, and may fall back to a default access or thumbnail command.

The output is used to generate the normalization report about normalization attempted and success/failure rates.

For historical reasons, much of the functionality is also implemented in [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/transcoder.py transcoder.py]. It runs an event detail & verification command associated with the normalization command.

On reingest, a 'deletion' Event is created for replaced derivatives.

=== transcribeFile_v0.0 ===

* '''Purpose''': Generate OCR of files.
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaTranscribeFile.py archivematicaTranscribeFile.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': transcription
* '''FPR?''': transcription

Optionally generates an OCR file for original files based on FPR entries for transcription. If the original file has no transcription rules, runs on the derivative. The new file is a derivation of the original, has a group of 'text/ocr' and is updated with a UUID, checksum, size etc.

=== createMETS_v2.0 ===

* '''Purpose''': Generate the AIP METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py archivematicaCreateMETS2.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No
* '''Tests''':
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_create_aip_mets.py test_create_aip_mets.py]
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_reingest_mets.py test_reingest_mets.py]

Perhaps the most important script in Archivematica: it creates the AIP METS which contains all the archival metadata generated by previous client scripts.

This script imports from several other files for additional functionality:
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSMetadataCSV.py archivematicaCreateMETSMetadataCSV]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRights.py archivematicaCreateMETSRights]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRightsDspaceMDRef.py archivematicaCreateMETSRightsDspaceMDRef]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSTrim.py archivematicaCreateMETSTrim]

On reingest, it short-circuits and runs [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSReingest.py archivematicaCreateMETSReingest] to update the METS file instead.

Not to be confused with [[#createMETS_v0.0]] for the transfer METS.

=== storeAIP_v0.0 ===

* '''Purpose''': Send the completed AIP to the storage service
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/storeAIP.py storeAIP.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Sends the currently processing AIP to the storage service. The Location is selected from the list of AIP Storage Locations associated with the Pipeline in previous tasks.

=== ===

* '''Purpose''':
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/]
* '''Used in''':
* '''Task type''':
* '''Event?''':
* '''FPR?''':
* '''Tests''':

== Config File ==

Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.

Variables in the MCPClient section:

{|
|-
! Variable !! Description !! Default value
|-
| MCPArchivematicaServer || URL of the MCP gearman server. Must match the [[MCPServer#Config File | server config file]]. || localhost:4730
|-
| sharedDirectoryMounted || Directory structure owned by Archivematica and shared between the MCPServer & MCPClient. Must match the [[MCPServer#Config File | server config file]]. || /var/archivematica/sharedDirectory/
|-
| archivematicaClientModules || Path to the list of jobs to register with Gearman || /usr/lib/archivematica/MCPClient/archivematicaClientModules
|-
| clientScriptsDirectory || Path to the directory where client scripts are installed. Used when parsing archivematicaClientModules || /usr/lib/archivematica/MCPClient/clientScripts/
|- style="background: #ffeecc;"
| LoadSupportedCommandsSpecial || Whether or not to register the SupportedCommandsSpecial section of archivematicaClientModules. This should be removed. || True
|-
| numberOfTasks || Number of MCPClient workers to created. 0 detects the number of cores and uses that. || 0
|-
| elasticsearchServer || URL of the ElasticSearch server. || localhost:9200
|- style="background: #ffeecc;"
| disableElasticsearchIndexing || If true, do not index AIPs or Transfers in backlog. This should be removed, since ElasticSearch indexing is required || False
|-
| temp_dir || Path to the temporary usage directory. Should be in the shared directory || /var/archivematica/sharedDirectory/tmp
|- style="background: #ffeecc;"
| kioskMode || Dashboard setting that disables editing users. This should be removed, or at least moved to dashboard settings || False
|-
| removableFiles || List of filenames that are not archivally significant and can be removed. || Thumbs.db, Icon, Icon\r, .DS_Store
|-
| django_settings_module || Name of the Django settings module, so the client scripts can access the database via the Django ORM. || settings.common
|-
|}

[[Category:Development documentation]]

MCPClient

2017-03-27T23:37:20Z

Hbecker: /* Client scripts */ Expand client script info

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCPClient

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

Archivematica has one or more MCPClient instances to perform the actual work. They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.

== Client scripts ==
Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.

New scripts are defined in <code>[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/archivematicaClientModules src/MCPClient/lib/archivematicaClientModules]</code>, which is what is registered with Gearman on MCPClient startup.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: archivematicaClientModules lists both 'supportedCommandSpecial' and 'supportedCommands'. This distinction may have once been based on scripts that relied on external services, but serves no purpose now and should be removed.
|}

The name is what the [[MCPServer#StandardTasksConfigs | StandardTasksConfig]] table will refer to them as, and the value is the script that will be run. Some are defined as shell builtins (eg copy_v0.0 is cp). Most are paths to a script in the clientScripts directory, using the <code>%clientScriptsDirectory%</code> replacement variable. The name of the client script is usually the same as the name in archivematicaClientModules, but for very old scripts may have ‘archivematica’ at the beginning (eg createMETS_v2.0 = archivematicaCreateMETS2.py) or be named more pythonically (eg parseExternalMETS = parse_external_mets.py). Entries are added alphabetically.

The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen. Newer scripts may not have the version defined.

The list of client scripts is sorted roughly in order of appearance during processing

=== moveTransfer_v0.0 ===

* '''Purpose''': Move a Transfer & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveTransfer.py archivematicaMoveTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole Transfer and updates the database with the new location relative to the shared directory.

=== assignFileUUIDs_v0.0 ===

* '''Purpose''': Starts tracking files new to Archivematica
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaAssignFileUUID.py archivematicaAssignFileUUID.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': ingestion/reingestion, possibly registration
* '''FPR?''': No

This creates an entry in the Files table with the file's UUID, current & original paths and file group. It also creates an 'ingestion' Event and an 'registration' Event if an accession ID was specified. Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.

In ingest, is used on manually normalized files which may have been newly added, metadata and submission documentation.

On reingest, it parses the METS file instead of generating the file UUID, path & group. The Event type is 'reingestion'.

=== updateSizeAndChecksum_v0.0 ===

* '''Purpose''': Set file's size & checksum
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaUpdateSizeAndChecksum.py archivematicaUpdateSizeAndChecksum.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': message digest calculation
* '''FPR?''': No

Updates the entry in the Files table with a size and checksum. IT also generates a 'message digest calculation' Event.

On reingest, it parses the METS file instead of generating the checksums & sizes. It also re-adds Derivation & Format links.

Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].

=== createEvent_v0.0 ===

* '''Purpose''': Create an Event outside of other scripts
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/createEvent.py createEvent.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': as parameter: quarantine, unquarantine, placement in backlog, removal from backlog
* '''FPR?''': No

Used to generate events for when a file is placed in or removed from quarantine or backlog. These do not have scripts associated with them, so the event creation is handled here.

=== archivematicaClamscan_v0.0 ===

* '''Purpose''': Check for viruses in incoming files
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaClamscan.py archivematicaClamscan.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': virus check
* '''FPR?''': No

Runs clamscan on the file and generates a 'virus scan' event. If a scan has been run, it is not run again on the same file.

This is run on incoming files, files after extraction, metadata files and submission documentation. It is not run on normalized files.

=== sanitizeObjectNames_v0.0 ===

* '''Purpose''': Strip problematic characters from filenames
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeObjectNames.py sanitizeObjectNames.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': name cleanup
* '''FPR?''': No
* '''Tests''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_sanitize.py test_sanitize.py]

Sanitize object names by replacing anything that isn't an ASCII letter, number, hyphen, underscore, period or parenthesis. This runs on both files and directories, though only files have Events generated. An Event is generated for a file even if only a parent directory is sanitized.

This used code from [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/sanitizeNames.py sanitizeNames.py]

=== identifyFileFormat_v0.0 ===

* '''Purpose''': Identify a file's format
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/identifyFileFormat.py identifyFileFormat.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': format identification
* '''FPR?''': IDCommand & IDRule

One of the most important scripts in Archivematica. Since the file format is used to determine many later actions (extraction, characterization, normalization etc), if this fails many important command later will also fail. This is the only script that uses the FPR that doesn't use the file format as a key for looking up what command to run. Instead, an IDCommand is selected and the output is matched to an IDRule to find the FormatVersion.

There is a short circuit handling of PRONOM ID (PUID) outputs. Since many FormatVersions have PUIDs, and both FIDO & Siegfried output PUIDs, this script looks for a FormatVersion with a given PUID. This reduces the number of IDRules that have to be created.

This also populates the legacy but still required FilesIDs table.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Only one identification tool can be run at a time currently. It would be better to allow a cascading of tools. E.g. if a file is identified as a video to subsequently run a tool specialized in identifying different types of video. Similarly, if the default tool failed, we could run a backup tool for a second opinion.
|}

=== extractContents_v0.0 ===

* '''Purpose''': Extract files from packages
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/extractContents.py extractContents.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': unpacking, registration, deletion, message digest calculation
* '''FPR?''': extract

Extracts files from a package (e.g. zip, tar, 7z etc) and optionally deletes the package. Generates Events for the new files.

=== validateFile_v1.0 ===

* '''Purpose''': Validate a file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/validateFile.py validateFile.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': validation
* '''FPR?''': validation, default_validation

Validates files are correct, where correctness is defined by the file format.

=== createMETS_v0.0 ===

* '''Purpose''': Generate the transfer METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS.py archivematicaCreateMETS.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Creates the Transfer METS file. This will contain all the information generated on the transfer during processing, and is especially useful for backlogged transfers.

Not to be confused with [[#createMETS_v2.0]] for the AIP METS.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The Transfer METS file & related backlog functionality needs to be expanded. See [[Transfer_backlog_requirements#Proposed_improvements]] for details.
|}

=== elasticSearchIndex_v0.0 ===

* '''Purpose''': Index the Transfer METS into ElasticSearch when sending files to backlog
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/elasticSearchIndexProcessTransfer.py elasticSearchIndexProcessTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

The data in ElasticSearch is used by the Backlog tab, SIP Arrangement and the Appraisal tab when dealing with files from backlog. Note that this is not run if the transfer is not sent to backlog (since AM 1.5).

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The client config 'disableElasticsearchIndexing' can disable indexing, but this should be removed, since searching for files in backlog is required functionality.
|}

=== moveSIP_v0.0 ===

* '''Purpose''': Move a SIP & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveSIP.py archivematicaMoveSIP.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole SIP and updates the database with the new location relative to the shared directory.

=== normalize_v1.0 ===

* '''Purpose''': Generate a preservation or access derivative
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/normalize.py normalize.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': normalization
* '''FPR?''': Rules: access, preservation, thumbnail, default_access, default_thumbnail. Commands: normalization, verification

One of the most important scripts in Archivematica. This generates the preservation and access derivatives, as well as thumbnails. If manual normalization happens, this instead finds the manually normalized files and links them as derivatives.

The same script is used for generating preservation, access or thumbnail derivatives, controlled by the parameter passed in. In general, normalization is only performed on original files, but may be done on service files. This reads the FPR for the specific command, and may fall back to a default access or thumbnail command.

The output is used to generate the normalization report about normalization attempted and success/failure rates.

For historical reasons, much of the functionality is also implemented in [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/transcoder.py transcoder.py]. It runs an event detail & verification command associated with the normalization command.

On reingest, a 'deletion' Event is created for replaced derivatives.

=== transcribeFile_v0.0 ===

* '''Purpose''': Generate OCR of files.
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaTranscribeFile.py archivematicaTranscribeFile.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': transcription
* '''FPR?''': transcription

Optionally generates an OCR file for original files based on FPR entries for transcription. If the original file has no transcription rules, runs on the derivative. The new file is a derivation of the original, has a group of 'text/ocr' and is updated with a UUID, checksum, size etc.

=== createMETS_v2.0 ===

* '''Purpose''': Generate the AIP METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py archivematicaCreateMETS2.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No
* '''Tests''':
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_create_aip_mets.py test_create_aip_mets.py]
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_reingest_mets.py test_reingest_mets.py]

Perhaps the most important script in Archivematica: it creates the AIP METS which contains all the archival metadata generated by previous client scripts.

This script imports from several other files for additional functionality:
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSMetadataCSV.py archivematicaCreateMETSMetadataCSV]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRights.py archivematicaCreateMETSRights]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRightsDspaceMDRef.py archivematicaCreateMETSRightsDspaceMDRef]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSTrim.py archivematicaCreateMETSTrim]

On reingest, it short-circuits and runs [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSReingest.py archivematicaCreateMETSReingest] to update the METS file instead.

Not to be confused with [[#createMETS_v0.0]] for the transfer METS.

=== storeAIP_v0.0 ===

* '''Purpose''': Send the completed AIP to the storage service
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/storeAIP.py storeAIP.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Sends the currently processing AIP to the storage service. The Location is selected from the list of AIP Storage Locations associated with the Pipeline in previous tasks.

=== ===

* '''Purpose''':
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/]
* '''Used in''':
* '''Task type''':
* '''Event?''':
* '''FPR?''':
* '''Tests''':

== Config File ==

Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.

Variables in the MCPClient section:

{|
|-
! Variable !! Description !! Default value
|-
| MCPArchivematicaServer || URL of the MCP gearman server. Must match the [[MCPServer#Config File | server config file]]. || localhost:4730
|-
| sharedDirectoryMounted || Directory structure owned by Archivematica and shared between the MCPServer & MCPClient. Must match the [[MCPServer#Config File | server config file]]. || /var/archivematica/sharedDirectory/
|-
| archivematicaClientModules || Path to the list of jobs to register with Gearman || /usr/lib/archivematica/MCPClient/archivematicaClientModules
|-
| clientScriptsDirectory || Path to the directory where client scripts are installed. Used when parsing archivematicaClientModules || /usr/lib/archivematica/MCPClient/clientScripts/
|- style="background: #ffeecc;"
| LoadSupportedCommandsSpecial || Whether or not to register the SupportedCommandsSpecial section of archivematicaClientModules. This should be removed. || True
|-
| numberOfTasks || Number of MCPClient workers to created. 0 detects the number of cores and uses that. || 0
|-
| elasticsearchServer || URL of the ElasticSearch server. || localhost:9200
|- style="background: #ffeecc;"
| disableElasticsearchIndexing || If true, do not index AIPs or Transfers in backlog. This should be removed, since ElasticSearch indexing is required || False
|-
| temp_dir || Path to the temporary usage directory. Should be in the shared directory || /var/archivematica/sharedDirectory/tmp
|- style="background: #ffeecc;"
| kioskMode || Dashboard setting that disables editing users. This should be removed, or at least moved to dashboard settings || False
|-
| removableFiles || List of filenames that are not archivally significant and can be removed. || Thumbs.db, Icon, Icon\r, .DS_Store
|-
| django_settings_module || Name of the Django settings module, so the client scripts can access the database via the Django ORM. || settings.common
|-
|}

[[Category:Development documentation]]

MCPClient

2017-03-27T22:51:49Z

Hbecker: /* Client scripts */ Expand client script info

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCPClient

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

Archivematica has one or more MCPClient instances to perform the actual work. They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.

== Client scripts ==
Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.

New scripts are defined in <code>[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/archivematicaClientModules src/MCPClient/lib/archivematicaClientModules]</code>, which is what is registered with Gearman on MCPClient startup.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: archivematicaClientModules lists both 'supportedCommandSpecial' and 'supportedCommands'. This distinction may have once been based on scripts that relied on external services, but serves no purpose now and should be removed.
|}

The name is what the [[MCPServer#StandardTasksConfigs | StandardTasksConfig]] table will refer to them as, and the value is the script that will be run. Some are defined as shell builtins (eg copy_v0.0 is cp). Most are paths to a script in the clientScripts directory, using the <code>%clientScriptsDirectory%</code> replacement variable. The name of the client script is usually the same as the name in archivematicaClientModules, but for very old scripts may have ‘archivematica’ at the beginning (eg createMETS_v2.0 = archivematicaCreateMETS2.py) or be named more pythonically (eg parseExternalMETS = parse_external_mets.py). Entries are added alphabetically.

The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen. Newer scripts may not have the version defined.

The list of client scripts is sorted roughly in order of appearance during processing

=== moveTransfer_v0.0 ===

* '''Purpose''': Move a Transfer & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveTransfer.py archivematicaMoveTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole Transfer and updates the database with the new location relative to the shared directory.

=== assignFileUUIDs_v0.0 ===

* '''Purpose''': Starts tracking files new to Archivematica
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaAssignFileUUID.py archivematicaAssignFileUUID.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': ingestion/reingestion, possibly accession
* '''FPR?''': No

This creates an entry in the Files table with the file's UUID, current & original paths and file group. It also creates an 'ingestion' Event and an 'accession' Event if an accession ID was specified. Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.

In ingest, is used on manually normalized files which may have been newly added, metadata and submission documentation.

On reingest, it parses the METS file instead of generating the file UUID, path & group. The Event type is 'reingestion'.

=== updateSizeAndChecksum_v0.0 ===

* '''Purpose''': Set file's size & checksum
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaUpdateSizeAndChecksum.py archivematicaUpdateSizeAndChecksum.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': message digest calculation
* '''FPR?''': No

Updates the entry in the Files table with a size and checksum. IT also generates a 'message digest calculation' Event.

On reingest, it parses the METS file instead of generating the checksums & sizes. It also re-adds Derivation & Format links.

Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].

=== archivematicaClamscan_v0.0 ===

* '''Purpose''': Check for viruses in incoming files
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaClamscan.py archivematicaClamscan.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': virus check
* '''FPR?''': No

Runs clamscan on the file and generates a 'virus scan' event. If a scan has been run, it is not run again on the same file.

This is run on incoming files, files after extraction, metadata files and submission documentation. It is not run on normalized files.

=== identifyFileFormat_v0.0 ===

* '''Purpose''': Identify a file's format
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/identifyFileFormat.py identifyFileFormat.py]
* '''Used in''': Transfer, Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': format identification
* '''FPR?''': IDCommand & IDRule

One of the most important scripts in Archivematica. Since the file format is used to determine many later actions (extraction, characterization, normalization etc), if this fails many important command later will also fail. This is the only script that uses the FPR that doesn't use the file format as a key for looking up what command to run. Instead, an IDCommand is selected and the output is matched to an IDRule to find the FormatVersion.

There is a short circuit handling of PRONOM ID (PUID) outputs. Since many FormatVersions have PUIDs, and both FIDO & Siegfried output PUIDs, this script looks for a FormatVersion with a given PUID. This reduces the number of IDRules that have to be created.

This also populates the legacy but still required FilesIDs table.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Only one identification tool can be run at a time currently. It would be better to allow a cascading of tools. E.g. if a file is identified as a video to subsequently run a tool specialized in identifying different types of video. Similarly, if the default tool failed, we could run a backup tool for a second opinion.
|}

=== createMETS_v0.0 ===

* '''Purpose''': Generate the transfer METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS.py archivematicaCreateMETS.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Creates the Transfer METS file. This will contain all the information generated on the transfer during processing, and is especially useful for backlogged transfers.

Not to be confused with [[#createMETS_v2.0]] for the AIP METS.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The Transfer METS file & related backlog functionality needs to be expanded. See [[Transfer_backlog_requirements#Proposed_improvements]] for details.
|}

=== elasticSearchIndex_v0.0 ===

* '''Purpose''': Index the Transfer METS into ElasticSearch when sending files to backlog
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/elasticSearchIndexProcessTransfer.py elasticSearchIndexProcessTransfer.py]
* '''Used in''': Transfer
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

The data in ElasticSearch is used by the Backlog tab, SIP Arrangement and the Appraisal tab when dealing with files from backlog. Note that this is not run if the transfer is not sent to backlog (since AM 1.5).

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: The client config 'disableElasticsearchIndexing' can disable indexing, but this should be removed, since searching for files in backlog is required functionality.
|}

=== moveSIP_v0.0 ===

* '''Purpose''': Move a SIP & update database
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveSIP.py archivematicaMoveSIP.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Moves the whole SIP and updates the database with the new location relative to the shared directory.

=== transcribeFile_v0.0 ===

* '''Purpose''': Generate OCR of files.
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaTranscribeFile.py archivematicaTranscribeFile.py]
* '''Used in''': Ingest
* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
* '''Event?''': transcription
* '''FPR?''': transcription

Optionally generates an OCR file for original files based on FPR entries for transcription. If the original file has no transcription rules, runs on the derivative. The new file is a derivation of the original, has a group of 'text/ocr' and is updated with a UUID, checksum, size etc.

=== createMETS_v2.0 ===

* '''Purpose''': Generate the AIP METS file
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS2.py archivematicaCreateMETS2.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No
* '''Tests''':
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_create_aip_mets.py test_create_aip_mets.py]
** [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/tests/test_reingest_mets.py test_reingest_mets.py]

Perhaps the most important script in Archivematica: it creates the AIP METS which contains all the archival metadata generated by previous client scripts.

This script imports from several other files for additional functionality:
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSMetadataCSV.py archivematicaCreateMETSMetadataCSV]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRights.py archivematicaCreateMETSRights]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSRightsDspaceMDRef.py archivematicaCreateMETSRightsDspaceMDRef]
[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSTrim.py archivematicaCreateMETSTrim]

On reingest, it short-circuits and runs [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETSReingest.py archivematicaCreateMETSReingest] to update the METS file instead.

Not to be confused with [[#createMETS_v0.0]] for the transfer METS.

=== storeAIP_v0.0 ===

* '''Purpose''': Send the completed AIP to the storage service
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/storeAIP.py storeAIP.py]
* '''Used in''': SIP
* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
* '''Event?''': No
* '''FPR?''': No

Sends the currently processing AIP to the storage service. The Location is selected from the list of AIP Storage Locations associated with the Pipeline in previous tasks.

=== ===

* '''Purpose''':
* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/]
* '''Used in''':
* '''Task type''':
* '''Event?''':
* '''FPR?''':
* '''Tests''':

== Config File ==

Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.

Variables in the MCPClient section:

{|
|-
! Variable !! Description !! Default value
|-
| MCPArchivematicaServer || URL of the MCP gearman server. Must match the [[MCPServer#Config File | server config file]]. || localhost:4730
|-
| sharedDirectoryMounted || Directory structure owned by Archivematica and shared between the MCPServer & MCPClient. Must match the [[MCPServer#Config File | server config file]]. || /var/archivematica/sharedDirectory/
|-
| archivematicaClientModules || Path to the list of jobs to register with Gearman || /usr/lib/archivematica/MCPClient/archivematicaClientModules
|-
| clientScriptsDirectory || Path to the directory where client scripts are installed. Used when parsing archivematicaClientModules || /usr/lib/archivematica/MCPClient/clientScripts/
|- style="background: #ffeecc;"
| LoadSupportedCommandsSpecial || Whether or not to register the SupportedCommandsSpecial section of archivematicaClientModules. This should be removed. || True
|-
| numberOfTasks || Number of MCPClient workers to created. 0 detects the number of cores and uses that. || 0
|-
| elasticsearchServer || URL of the ElasticSearch server. || localhost:9200
|- style="background: #ffeecc;"
| disableElasticsearchIndexing || If true, do not index AIPs or Transfers in backlog. This should be removed, since ElasticSearch indexing is required || False
|-
| temp_dir || Path to the temporary usage directory. Should be in the shared directory || /var/archivematica/sharedDirectory/tmp
|- style="background: #ffeecc;"
| kioskMode || Dashboard setting that disables editing users. This should be removed, or at least moved to dashboard settings || False
|-
| removableFiles || List of filenames that are not archivally significant and can be removed. || Thumbs.db, Icon, Icon\r, .DS_Store
|-
| django_settings_module || Name of the Django settings module, so the client scripts can access the database via the Django ORM. || settings.common
|-
|}

[[Category:Development documentation]]

Category:Development documentation

2017-03-27T21:48:08Z

Hbecker: Clarify section explanations

[[Main Page]] > [[Development]] > Development documentation

Archivematica is a web- and standards-based, open-source application which allows institutions to preserve long-term access to trustworthy, authentic and reliable digital content. Our target users are archivists, librarians, and anyone working to preserve digital objects.

The Archivematica project consists of several projects working together:
* Archivematica: Main repository containing the user-facing dashboard, task manager MCPServer and clients scripts for the MCPClient. Also referred to as a pipeline
** [[Dashboard]]: User-facing web GUI written in Django. This handles external systems & workflow configuration, Transfer and SIP process status & decisions, and gives access to the backlog and archival storage.
** [[MCPServer]]: Task manager using gearman that assigns tasks to MCPClients and determines workflow.
** [[MCPClient]]: Gearman worker that dispatches jobs to the various scripts.
*** Client scripts: The actual scripts that do the processing. Most are written in Python, some in bash or other tools.
* [[Storage Service]]: Responsible for moving files to Archivematica for processing, and from Archivematica into storage. It is the more permanent record of AIPs & backlogged transfers and where they’re stored.
* [[ Format_policy_registry_requirements | Format Policy Registry]]: Submodule shared between Archivematica and the Format Policy Registry (FPR) server that displays and updates FPR rules and commands

Detailed feature descriptions live in the subcategory [[:Category:Feature requirements]]. These may be complete, in progress, or just ideas. Documentation for old versions that is no longer valid in current Archivematica is moved to [[:Category:Deprecated]].

See also [[:Category:Process documentation]]

Transfer backlog requirements

2017-03-27T21:02:24Z

Hbecker: /* Proposed improvements */ Update cross-pipeline backlog

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer backlog requirements

[[Category:Feature requirements]]

== Proposed improvements ==

=== Handle cross-pipeline backlog ===

March 2017

'''Summary''': Transfers put into backlog should be able to be started as SIP (through the Appraisal tab, SIP arrange, or Backlog tab) on a different pipeline than they were put in backlog from.

'''Problem''': Much of the information needed during Ingest is stored only in the database. If a Transfer is put into backlog and ingest is started on a different pipeline that information is not there.

'''Proposed fix''': The Transfer METS file should be improved to contain all information needed, which could be parsed back into the database at the start of ingest.

Currently there are 4 places where backlogged transfer data lives: pipeline database, Elasticsearch transfers index, storage service database & transfer METS file. This should be consolidated to a single location and the other sources could be rebuilt from the canonical one.

The METS file is treated as the canonical metadata store in the AIP and is a good choice for the canonical metadata store for a transfer in backlog. Since all of this information is needed in ingest is for the AIP METS file, we know all the information can be stored as a METS file. We already parse the AIP METS file into the database on full reingest, so there is precedent & examples for doing this.

Tables needed include:
* Transfers
* Files
* DublinCore
* RightsStatement & related
* Events
* Events_agents (link between Event & Agent)
* Agents (older version of AM?)
* FilesIdentifiedIDs (file ID info)
* FilesIDs (more file ID info)
* main_fpcommandoutput (characterization)
* Jobs or Tasks may also be required for status checking?

This also needs to handle transfers put in backlog before the transfer METS was updated. It should be straightforward to handle backlogged transfers where the pipeline still exists, as the data is all in the database. For backlogged transfers where the pipeline no longer exists, truncated, stub or default data could be done, or require that it be re-run through transfer.

== Original requirements ==

Added in Archivematica 1.0

=== Transfer Backlog Management ===
* Related issues: Issue 951, Issue 1220, Issue 1141, Issue 1225, Issue 1257

=== Requirements for transfer backlog search ===

* Add ability to search transfer backlog and send one or more transfers to Ingest
* Add ability to download and/or view files/transfers (via right click)
* Search the following fields: Any field, transfer name, file name, accession number, PUID, Mimetype, Date - Ingest

=== Mockup of transfer backlog search ===

[[File:1.0_TransferBacklogSearch.png|680px|thumb|center|]]

[[File:1.0_TransBacklogSearchResults.png|680px|thumb|center|]]

=== Transfer Workflow ===

* Administration - allow MCP access to media or storage where transfer is located
* Assign accession number to transfer
* Remove transfer backup from workflow - no long processing configuration option
* Add Send transfer to backlog microservice
* Add Search transfer backlog tab from Ingest in Dashboard
* Add ability to download and/or view transfers and files from Search tab
* Add ability to send transfers from backlog search to Ingest/Create SIP (checkboxes, send button)
* see workflow diagrams below

'''0.9 Transfer workflow'''

* grey steps are automated, white are manual

[[File:transferWorkflow0.9.png|680px|thumb|center|]]
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]

=== Administration Tab in Dashboard ===

* Assign permission and access to the MCPServer to copy from transfer media (hard drives, optical media, USB, etc.) or network location.
* Assign transfer backlog locations (configuration is done outside of AM)
* Assign source directories
* Define transfer types
* Assign report locations (post-1.0)
* Set AIP storage location
* Set DIP upload location

=== Accession metadata ===

* PREMIS Event = Registration
<event>
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>35cbe00d-d661-4174-b11a-e203f5608008</eventIdentifierValue>
</eventIdentifier>
<eventType>registration</eventType>
<eventDateTime>2012-03-14</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>accession#2012-029</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>archivist</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Courtney Mumma</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>

* Manually input metadata in template on dashboard (See [[File_Browser_Requirements]]) : accession number
* Agent is the archivist logged in at the time doing the accession (post-1.0, for 1.0 this will still be repository)
* Event name is "registration" (to be added to PREMIS events master list should we decide to implement)
* UUID

* Also see Issue 787 on the Archivematica issues list

=== Microservices Completed Before Move to Backlog ===

* All transfer microservices
* Indexing: See [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]

=== Handling of Submission Documentation ===

* [http://sites.tufts.edu/dca/about-us/research-initiatives/taper-tufts-accessioning-program-for-electronic-records/| TAPER]?
* Normalized with objects in AIP (0.8)
* Upload submission documentation with transfer in transfer tab - Issue 1255

=== Search transfers from Archival Storage ===

New sponsored development planned for Archivematica 1.6 or later will allow users to manage the transfer backlog through the Archival Storage tab, as outlined in general workflows described in these diagrams:

[[File:Transfer_management_workflows.png|700px]]

As outlined above, users will be able to:

*Search transfers from archival storage tab
*Download copies of transfers or selected files from archival storage tab
*Be able to perform transfer deletion requests from archival storage tab

'''Transfer search user stories'''

As an archivist, I need to find transfers by searching...

* by the name of the transfer
* by the date the transfer was stored in backlog
* by names of files within the transfer
* by....?

'''Mockups: Version 1'''

Search transfers from Archival Storage:

[[File:Archival_Storage_Transfer_Search.png|800px]]

'''Notes:'''

*Click on "Show transfers" to search AIPs as well as Transfers
*A new column in the table indicates whether a package is a Transfer or an AIP.
*To trigger transfer deletion, click on red "remove" icon (same functionality as AIP deletion)

Search files from transfers in Archival Storage:

[[File:Archival_Storage_Transfer_file_search.png|800px]]

'''Notes:'''

*Clicking on both "Show files" and "Show transfers" before searching will load preview of files from transfer backlog.
*The UUID of the package and an indication of whether the file is from a Transfer or an AIP is in the right column.

'''Mockups: Version 2'''

In this version, toggling between searching for AIPs/Transfers is done through a tab at the top. This makes the development significantly less complicated, as we would not need to combine the Elasticsearch indexes for transfer METS and AIP METS.

Transfer search:

[[File:Transfer_search_v2.png|800px]]

Files within transfer search:

[[File:Transfer_search_files_v2.png|800px]]

Transfer backlog requirements

2017-03-27T18:59:15Z

Hbecker: Add proposed changes

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer backlog requirements

[[Category:Feature requirements]]

== Proposed improvements ==

=== Handle cross-pipeline backlog ===

March 2017

'''Summary''': Transfers put into backlog should be able to be started as SIP (through the Appraisal tab, SIP arrange, or Backlog tab) on a different pipeline than they were put in backlog from.

'''Problem''': Much of the information needed during Ingest is stored only in the database. If a Transfer is put into backlog and ingest is started on a different pipeline that information is not there.

'''Proposed fix''': The Transfer METS file should be improved to contain all information needed, which could be parsed back into the database at the start of ingest.

We already parse a METS file into the database on full reingest, so there is precedent & examples for doing this.

Tables needed include:
* Transfers
* Files
* DublinCore
* RightsStatement & related
* Events
* Agents (older version of AM?)
* FilesIdentifiedIDs (file ID info)
* FilesIDs (more file ID info)
* main_fpcommandoutput (characterization)

== Original requirements ==

Added in Archivematica 1.0

=== Transfer Backlog Management ===
* Related issues: Issue 951, Issue 1220, Issue 1141, Issue 1225, Issue 1257

=== Requirements for transfer backlog search ===

* Add ability to search transfer backlog and send one or more transfers to Ingest
* Add ability to download and/or view files/transfers (via right click)
* Search the following fields: Any field, transfer name, file name, accession number, PUID, Mimetype, Date - Ingest

=== Mockup of transfer backlog search ===

[[File:1.0_TransferBacklogSearch.png|680px|thumb|center|]]

[[File:1.0_TransBacklogSearchResults.png|680px|thumb|center|]]

=== Transfer Workflow ===

* Administration - allow MCP access to media or storage where transfer is located
* Assign accession number to transfer
* Remove transfer backup from workflow - no long processing configuration option
* Add Send transfer to backlog microservice
* Add Search transfer backlog tab from Ingest in Dashboard
* Add ability to download and/or view transfers and files from Search tab
* Add ability to send transfers from backlog search to Ingest/Create SIP (checkboxes, send button)
* see workflow diagrams below

'''0.9 Transfer workflow'''

* grey steps are automated, white are manual

[[File:transferWorkflow0.9.png|680px|thumb|center|]]
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]

=== Administration Tab in Dashboard ===

* Assign permission and access to the MCPServer to copy from transfer media (hard drives, optical media, USB, etc.) or network location.
* Assign transfer backlog locations (configuration is done outside of AM)
* Assign source directories
* Define transfer types
* Assign report locations (post-1.0)
* Set AIP storage location
* Set DIP upload location

=== Accession metadata ===

* PREMIS Event = Registration
<event>
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>35cbe00d-d661-4174-b11a-e203f5608008</eventIdentifierValue>
</eventIdentifier>
<eventType>registration</eventType>
<eventDateTime>2012-03-14</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>accession#2012-029</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>archivist</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Courtney Mumma</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>

* Manually input metadata in template on dashboard (See [[File_Browser_Requirements]]) : accession number
* Agent is the archivist logged in at the time doing the accession (post-1.0, for 1.0 this will still be repository)
* Event name is "registration" (to be added to PREMIS events master list should we decide to implement)
* UUID

* Also see Issue 787 on the Archivematica issues list

=== Microservices Completed Before Move to Backlog ===

* All transfer microservices
* Indexing: See [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]

=== Handling of Submission Documentation ===

* [http://sites.tufts.edu/dca/about-us/research-initiatives/taper-tufts-accessioning-program-for-electronic-records/| TAPER]?
* Normalized with objects in AIP (0.8)
* Upload submission documentation with transfer in transfer tab - Issue 1255

=== Search transfers from Archival Storage ===

New sponsored development planned for Archivematica 1.6 or later will allow users to manage the transfer backlog through the Archival Storage tab, as outlined in general workflows described in these diagrams:

[[File:Transfer_management_workflows.png|700px]]

As outlined above, users will be able to:

*Search transfers from archival storage tab
*Download copies of transfers or selected files from archival storage tab
*Be able to perform transfer deletion requests from archival storage tab

'''Transfer search user stories'''

As an archivist, I need to find transfers by searching...

* by the name of the transfer
* by the date the transfer was stored in backlog
* by names of files within the transfer
* by....?

'''Mockups: Version 1'''

Search transfers from Archival Storage:

[[File:Archival_Storage_Transfer_Search.png|800px]]

'''Notes:'''

*Click on "Show transfers" to search AIPs as well as Transfers
*A new column in the table indicates whether a package is a Transfer or an AIP.
*To trigger transfer deletion, click on red "remove" icon (same functionality as AIP deletion)

Search files from transfers in Archival Storage:

[[File:Archival_Storage_Transfer_file_search.png|800px]]

'''Notes:'''

*Clicking on both "Show files" and "Show transfers" before searching will load preview of files from transfer backlog.
*The UUID of the package and an indication of whether the file is from a Transfer or an AIP is in the right column.

'''Mockups: Version 2'''

In this version, toggling between searching for AIPs/Transfers is done through a tab at the top. This makes the development significantly less complicated, as we would not need to combine the Elasticsearch indexes for transfer METS and AIP METS.

Transfer search:

[[File:Transfer_search_v2.png|800px]]

Files within transfer search:

[[File:Transfer_search_files_v2.png|800px]]

Transfer backlog requirements

2017-03-27T18:47:55Z

Hbecker: Move original reqs to subheading

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer backlog requirements

[[Category:Feature requirements]]

== Original requirements ==

Added in Archivematica 1.0

=== Transfer Backlog Management ===
* Related issues: Issue 951, Issue 1220, Issue 1141, Issue 1225, Issue 1257

=== Requirements for transfer backlog search ===

* Add ability to search transfer backlog and send one or more transfers to Ingest
* Add ability to download and/or view files/transfers (via right click)
* Search the following fields: Any field, transfer name, file name, accession number, PUID, Mimetype, Date - Ingest

=== Mockup of transfer backlog search ===

[[File:1.0_TransferBacklogSearch.png|680px|thumb|center|]]

[[File:1.0_TransBacklogSearchResults.png|680px|thumb|center|]]

=== Transfer Workflow ===

* Administration - allow MCP access to media or storage where transfer is located
* Assign accession number to transfer
* Remove transfer backup from workflow - no long processing configuration option
* Add Send transfer to backlog microservice
* Add Search transfer backlog tab from Ingest in Dashboard
* Add ability to download and/or view transfers and files from Search tab
* Add ability to send transfers from backlog search to Ingest/Create SIP (checkboxes, send button)
* see workflow diagrams below

'''0.9 Transfer workflow'''

* grey steps are automated, white are manual

[[File:transferWorkflow0.9.png|680px|thumb|center|]]
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]

=== Administration Tab in Dashboard ===

* Assign permission and access to the MCPServer to copy from transfer media (hard drives, optical media, USB, etc.) or network location.
* Assign transfer backlog locations (configuration is done outside of AM)
* Assign source directories
* Define transfer types
* Assign report locations (post-1.0)
* Set AIP storage location
* Set DIP upload location

=== Accession metadata ===

* PREMIS Event = Registration
<event>
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>35cbe00d-d661-4174-b11a-e203f5608008</eventIdentifierValue>
</eventIdentifier>
<eventType>registration</eventType>
<eventDateTime>2012-03-14</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>accession#2012-029</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>archivist</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Courtney Mumma</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>

* Manually input metadata in template on dashboard (See [[File_Browser_Requirements]]) : accession number
* Agent is the archivist logged in at the time doing the accession (post-1.0, for 1.0 this will still be repository)
* Event name is "registration" (to be added to PREMIS events master list should we decide to implement)
* UUID

* Also see Issue 787 on the Archivematica issues list

=== Microservices Completed Before Move to Backlog ===

* All transfer microservices
* Indexing: See [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]

=== Handling of Submission Documentation ===

* [http://sites.tufts.edu/dca/about-us/research-initiatives/taper-tufts-accessioning-program-for-electronic-records/| TAPER]?
* Normalized with objects in AIP (0.8)
* Upload submission documentation with transfer in transfer tab - Issue 1255

=== Search transfers from Archival Storage ===

New sponsored development planned for Archivematica 1.6 or later will allow users to manage the transfer backlog through the Archival Storage tab, as outlined in general workflows described in these diagrams:

[[File:Transfer_management_workflows.png|700px]]

As outlined above, users will be able to:

*Search transfers from archival storage tab
*Download copies of transfers or selected files from archival storage tab
*Be able to perform transfer deletion requests from archival storage tab

'''Transfer search user stories'''

As an archivist, I need to find transfers by searching...

* by the name of the transfer
* by the date the transfer was stored in backlog
* by names of files within the transfer
* by....?

'''Mockups: Version 1'''

Search transfers from Archival Storage:

[[File:Archival_Storage_Transfer_Search.png|800px]]

'''Notes:'''

*Click on "Show transfers" to search AIPs as well as Transfers
*A new column in the table indicates whether a package is a Transfer or an AIP.
*To trigger transfer deletion, click on red "remove" icon (same functionality as AIP deletion)

Search files from transfers in Archival Storage:

[[File:Archival_Storage_Transfer_file_search.png|800px]]

'''Notes:'''

*Clicking on both "Show files" and "Show transfers" before searching will load preview of files from transfer backlog.
*The UUID of the package and an indication of whether the file is from a Transfer or an AIP is in the right column.

'''Mockups: Version 2'''

In this version, toggling between searching for AIPs/Transfers is done through a tab at the top. This makes the development significantly less complicated, as we would not need to combine the Elasticsearch indexes for transfer METS and AIP METS.

Transfer search:

[[File:Transfer_search_v2.png|800px]]

Files within transfer search:

[[File:Transfer_search_files_v2.png|800px]]

Merging

2017-03-23T23:33:48Z

Hbecker: Add code review section

[[Main Page]] > [[Development]] > Merging & Branches

== Branching guidelines ==

Guidelines on when to create branches for new development in Archivematica projects. Short answer: always create branches!

* All development work happens in branches - nothing is committed directly to master, qa/ or stable/ branches
* Development branches should follow the naming format: <code>dev/issue-####-short-description</code>
** <code>dev/</code> marks it as being a development branch, not for QA integration or a stable release
** <code>issue-####</code> is the Redmine ticket that the work is mostly related to. If the branch expands in scope to include multiple tickets later, that's fine, and don't worry about renaming branches
** <code>short-description</code> is a couple word description of the branch, to make it easier to remember what the topic of the branch is.
** Example: <code>dev/issue-8161-dashboard-i18n</code> or <code>dev/issue-8693-dataverse</code>
* Development branches should generally be create from the branch they will be merged into. In most cases, this is master, qa/1.x (Archivematica) or qa/0.x (Storage Service). If something is a patch for an old release, it may be more appropriate to branch from the old stable branch (e.g. stable/1.5.x)
* All branches should undergo code review before being merged into the target branch. If something was developed for an old release, it should be cherry-picked to the current development branch, and a comment with the commit hash should be left in the pull request for tracking.

== Code review & approval ==

All code changes merged into a stable or qa branch must go through code review. We use Github's pull request feature for code review. Every change being submitted to an Artefactual project should be submitted as a pull request to the appropriate repository.

Another core contributor (usually an Artefactual employee) will review the code for content, style and suitability in the project. A more extensive guideline to code review is [https://github.com/artefactual/archivematica/pull/547 also available]. (TODO update after PR is merged). After the reviewer is satisfied with the code, they will approve it on github. If at least one reviewer approves, and there are no disapprovals, the branch can be merged.

In general, the original contributor will [[#Merge process | merge the branch]]. For external contributions, the reviewer can merge or cherry-pick the commits. This will leave the Author correct in git, but update the committer.

== Merge process ==

We use a rebase-based workflow, where branches are brought up to date with the latest stable by rebasing them. (This is compared to a merge-based workflow, where branches are merged, and conflicts resolved in the merge commit). This shows the process for merging development branch <code>dev/test</code> into target branch <code>master</code>. The target branch is usually master, qa/0.x or qa/1.x.

The instructions are formatted: <code>(current branch) $ <command to run></code>

# Check your remote is set to be gitolite
#* <code>$ git remote -v</code> should show <code>origin git@git.artefactual.com:<repository-name>.git</code>
#* Not all repositories have gitolite as the canonical repo - ask if you're unsure
# Ensure your copy of the target branch is up to date
#* <code>$ git checkout master</code>
#* <code>(master) $ git pull --rebase </code>
# Ensure your development branch is based off the latest target branch
#* <code>$ git checkout dev/test</code>
#* <code>(dev/test) $ git rebase master</code>
# Check that your changes still work after rebasing
# Check tests pass (see https://wiki.archivematica.org/Getting_started#Tests for more details)
#* <code>(dev/test) $ py.test</code>
# Push your newly-rebased branch to remote. Because this is a rebase, which changes history, you'll need --force. The rebased dev branch has to be pushed before the master branch is or github gets confused.
#* '''Never force push on a stable/, qa/ or master branch!'''
#* <code>(dev/test) $ git push -f</code>
# Fast-forward merge
#* <code>$ git checkout master</code>
#* <code>(master) $ git merge dev/test</code>
# Push merge!
#* <code>(master) $ git push</code>
# Delete dev branch
#* Locally: <code>(master) $ git branch -d dev/test</code>
#* Remote: <code>(master) $ git push origin --delete dev/test</code>
# Update the related ticket, if needed
# You're done!

== Proposed changes ==

=== Forks ===

* '''Current status''': All development branches are created in the artefactual (or artefactual-labs) owned repository, and core contributors (usually Artefactual employees) commit directly to the official repository
* '''Proposed change''': Development happens in contributor's forks, and the official repo contains only stable & QA branches
* '''Pros''':
** Fewer branches in the official repository, so there is less clutter and noise for people watching activity
** Handles more contributors, since there won't be dozens or hundreds of branches of uncertain development status
** Contribution process is the same for external and internal contributors
** Contributors can handle their repository branches however they want - naming conventions, deleting old branches or not, etc
* '''Cons''':
** Potential confusion about which repository is the correct official one
** Potential to push changes to the wrong repo (e.g. push a development branch to the official repo instead of the contributor's fork)
** Developers have to keep 2 repositories up to date: upstream and their fork
** Harder for analysts to check out development code, since they would have to set their remote to the contributor's fork

=== Hybrid merge/rebase model ===

* '''Current status''': All branches are rebased onto their target branch before being (fast-forward) merged
* '''Proposed change''': Small or minor branches are rebased, larger feature branches are true merged
* '''Pros''':
** Commit history clearly shows 'feature', and which commits came in together as a unit
** Rebasing small branches preserves clean history for minor changes
** Better reflects nature of development - shows minor changes as happening inline, and major changes as happening in parallel
** Merge commits are clear places to look for integration issues
* '''Cons''':
** Confusion about when to merged vs rebase
** Developers have to learn two workflows

[[Category:Process documentation]]

Category:Development documentation

2017-03-23T22:49:36Z

Hbecker: Link other categories

[[Main Page]] > [[Development]] > Development documentation

Archivematica is a web- and standards-based, open-source application which allows institutions to preserve long-term access to trustworthy, authentic and reliable digital content. Our target users are archivists, librarians, and anyone working to preserve digital objects.

The Archivematica project consists of several projects working together:
* Archivematica: Main repository containing the user-facing dashboard, task manager MCPServer and clients scripts for the MCPClient. Also referred to as a pipeline
** [[Dashboard]]: User-facing web GUI written in Django. This is where Transfer and SIP process can be monitored and directed, and gives access to the backlog and archival storage.
** [[MCPServer]]: Task manager using gearman that determines what scripts to run next and what to do on error
** [[MCPClient]]: Gearman worker that dispatches jobs to the various scripts
*** Client scripts: The actual scripts that do the processing. Most are written in Python, some in bash or other tools.
* [[Storage Service]]: Responsible for moving files to Archivematica for processing, and from Archivematica into storage
* [[ Format_policy_registry_requirements | Format Policy Registry]]: Submodule shared between Archivematica and the Format Policy Registry (FPR) server that displays and updates FPR rules and commands

Detailed feature descriptions live in the subcategory [[:Category:Feature requirements]]. These may be complete, in progress, or just ideas. Documentation for old versions that is no longer valid in current Archivematica is moved to [[:Category:Deprecated]].

See also [[:Category:Process documentation]]

Development

2017-03-23T22:47:13Z

Hbecker: /* Developer Resources */ Add link to process docs category

[[Main Page]] > Development

This page lists the resources available for project contributors:

==Project Communication==
* [http://groups.google.ca/group/archivematica Discussion list]
* Archivematica.org wiki: [[Special:UserLogin|create an account]] to correct and add content on this wiki
* Issue tracking
** [https://projects.artefactual.com/projects/archivematica Archivematica]
** [https://projects.artefactual.com/projects/archivematica-storage-service Archivematica Storage Service]
** [https://projects.artefactual.com/projects/preservation-planning-module Format Policy Registry common]

==Developer Resources==
* [[Getting started]]
* [https://github.com/artefactual/archivematica Code repository]
* [[:Category:Development documentation|Development documentation]]
* [[:Category:Process documentation|Process documentation]] & contributing code
** [[Contributor Agreement]]
** [[License]]
** [[Trademark]]
* [[Development_roadmap:_Archivematica|Development roadmap]]

==Project Management==
Archivematica software development, release management, and community support is managed by [http://artefactual.com Artefactual Systems] in collaboration with its contract clients and a growing network of Archivematica users and service partners.

[[Category:Development documentation]]

Merging

2017-03-23T22:46:20Z

Hbecker: Copied from internal wiki docs

[[Main Page]] > [[Development]] > Merging & Branches

== Archivematica branching guidelines ==

Guidelines on when to create branches for new development in Archivematica projects. Short answer: always create branches!

* All development work happens in branches - nothing is committed directly to master, qa/ or stable/ branches
* Development branches should follow the naming format: <code>dev/issue-####-short-description</code>
** <code>dev/</code> marks it as being a development branch, not for QA integration or a stable release
** <code>issue-####</code> is the Redmine ticket that the work is mostly related to. If the branch expands in scope to include multiple tickets later, that's fine, and don't worry about renaming branches
** <code>short-description</code> is a couple word description of the branch, to make it easier to remember what the topic of the branch is.
** Example: <code>dev/issue-8161-dashboard-i18n</code> or <code>dev/issue-8693-dataverse</code>
* Development branches should generally be create from the branch they will be merged into. In most cases, this is master, qa/1.x (Archivematica) or qa/0.x (Storage Service). If something is a patch for an old release, it may be more appropriate to branch from the old stable branch (e.g. stable/1.5.x)
* All branches should undergo code review before being merged into the target branch. If something was developed for an old release, it should be cherry-picked to the current development branch, and a comment with the commit hash should be left in the pull request for tracking.

== Archivematica merge process ==

We use a rebase-based workflow, where branches are brought up to date with the latest stable by rebasing them. (This is compared to a merge-based workflow, where branches are merged, and conflicts resolved in the merge commit). This shows the process for merging development branch <code>dev/test</code> into target branch <code>master</code>. The target branch is usually master, qa/0.x or qa/1.x.

The instructions are formatted: <code>(current branch) $ <command to run></code>

# Check your remote is set to be gitolite
#* <code>$ git remote -v</code> should show <code>origin git@git.artefactual.com:<repository-name>.git</code>
#* Not all repositories have gitolite as the canonical repo - ask if you're unsure
# Ensure your copy of the target branch is up to date
#* <code>$ git checkout master</code>
#* <code>(master) $ git pull --rebase </code>
# Ensure your development branch is based off the latest target branch
#* <code>$ git checkout dev/test</code>
#* <code>(dev/test) $ git rebase master</code>
# Check that your changes still work after rebasing
# Check tests pass (see https://wiki.archivematica.org/Getting_started#Tests for more details)
#* <code>(dev/test) $ py.test</code>
# Push your newly-rebased branch to remote. Because this is a rebase, which changes history, you'll need --force. The rebased dev branch has to be pushed before the master branch is or github gets confused.
#* '''Never force push on a stable/, qa/ or master branch!'''
#* <code>(dev/test) $ git push -f</code>
# Fast-forward merge
#* <code>$ git checkout master</code>
#* <code>(master) $ git merge dev/test</code>
# Push merge!
#* <code>(master) $ git push</code>
# Delete dev branch
#* Locally: <code>(master) $ git branch -d dev/test</code>
#* Remote: <code>(master) $ git push origin --delete dev/test</code>
# Update the related ticket, if needed
# You're done!

== Proposed changes ==

=== Forks ===

* '''Current status''': All development branches are created in the artefactual (or artefactual-labs) owned repository, and core contributors (usually Artefactual employees) commit directly to the official repository
* '''Proposed change''': Development happens in contributor's forks, and the official repo contains only stable & QA branches
* '''Pros''':
** Fewer branches in the official repository, so there is less clutter and noise for people watching activity
** Handles more contributors, since there won't be dozens or hundreds of branches of uncertain development status
** Contribution process is the same for external and internal contributors
** Contributors can handle their repository branches however they want - naming conventions, deleting old branches or not, etc
* '''Cons''':
** Potential confusion about which repository is the correct official one
** Potential to push changes to the wrong repo (e.g. push a development branch to the official repo instead of the contributor's fork)
** Developers have to keep 2 repositories up to date: upstream and their fork
** Harder for analysts to check out development code, since they would have to set their remote to the contributor's fork

=== Hybrid merge/rebase model ===

* '''Current status''': All branches are rebased onto their target branch before being (fast-forward) merged
* '''Proposed change''': Small or minor branches are rebased, larger feature branches are true merged
* '''Pros''':
** Commit history clearly shows 'feature', and which commits came in together as a unit
** Rebasing small branches preserves clean history for minor changes
** Better reflects nature of development - shows minor changes as happening inline, and major changes as happening in parallel
** Merge commits are clear places to look for integration issues
* '''Cons''':
** Confusion about when to merged vs rebase
** Developers have to learn two workflows

[[Category:Process documentation]]

Category:Process documentation

2017-03-23T22:43:45Z

Hbecker: Initial stub

[[Main Page]] > [[Development]] > Process documentation

This category describes the processes Archivematica and related projects use.

For development documentation, see [[:Category:Development documentation]]

Contributor Agreement

2017-03-23T22:41:25Z

Hbecker: Add to process documentation category

[[Main Page]] > [[Development]] > [[Contribute code]] > Archivematica Contributor's Agreement

In order for the Archivematica development team to accept any patches or code commits, contributors must first sign this '''[[Media:Contributor_agreement.txt|Contributor Agreement]]'''.

==Why do I have to sign a Contributor's Agreement?==
One of the key challenges for open source software is to support a collaborative development environment while protecting the rights of contributors and users over the long-term. Unifying Archivematica copyrights through contributor agreements is the best way to protect the availability and sustainability of Archivematica over the long-term as free and open-source software. In all cases, contributors who sign the Contributor's Agreement retain full rights to use their original contributions for any other purpose outside of Archivematica, while enabling Artefactual Systems, any successor Foundation which may eventually take over responsibility for Archivematica, and the wider Archivematica community to benefit from their collaboration and contributions in this open source project.

[http://artefactual.com Artefactual Systems] has made the decision and has a proven track record of making our intellectual property available to the community at large. By standardizing contributions on these agreements the Archivematica intellectual property position does not become too complicated. This ensures our resources are devoted to making our project the best they can be, rather than fighting legal battles over contributions.

==Do you have examples of Contributor's Agreements from other projects?==
The Archivematica contributor's agreement is based almost verbatim on the [http://apache.org Apache Foundation]'s individual [http://www.apache.org/licenses/icla.txt contributor license].

Examples from other projects include: [https://www.irods.org/index.php/iRODS_Contributor%E2%80%99s_Agreement iRODS] and [http://www.canonical.com/sites/default/files/active/images/Canonical-HA-CLA-ANY-I.pdf Ubuntu] (Canonical).

[[Category:Process documentation]]

Contribute code

2017-03-23T22:40:59Z

Hbecker: Add to process documentation category

[[Main Page]] > [[Development]] > Contribute code

Expanded contribution guidelines can also be found in the Archivematica [https://github.com/artefactual/archivematica/blob/qa/1.x/CONTRIBUTING.md contributing document].

[[Category:Process documentation]]

==Patches==

If you find a bug in this project or would like to make an enhancement, please be encouraged to contribute code via a pull request. Archivematica code is available on github. We will accept git pull requests from individuals who have submitted a [[Contributor Agreement]].

For more information please visit:
* https://help.github.com/articles/creating-a-pull-request
* https://help.github.com/articles/using-pull-requests
* https://help.github.com/articles/fork-a-repo

==Commit access==
Anyone can contribute code patches to this project. Project collaborators and regular patch contributors will be given access to commit directly to the git code repository.

==Contributor's Agreement==
In order to accept any patches or code commits, contributors must first sign the [[Contributor Agreement|Archivematica Contributor's Agreement]].

==Standards==

=== Code Style Guide For Archivematica ===

Archivematica follows common Python coding standards, linked below. We encourage installing a Python [http://stackoverflow.com/questions/8503559/what-is-linting linter] to help with this. [https://pypi.python.org/pypi/flake8 flake8] is recommended because it wraps three common linters (pep8, pyflakes, mccabe).
* [https://www.python.org/dev/peps/pep-0008/ PEP8] for styling
** Exception is line length, which should be wrapped at 79 characters or left long for the IDE to wrap.
** Imports are sorted alphabetically within their grouping to reduce duplicate imports
* [https://www.python.org/dev/peps/pep-0257/ PEP257] for docstring structure
** Parameters and return values should be specified in [http://sphinx-doc.org/domains.html#info-field-lists Sphinx-style].
* See also [https://github.com/artefactual/archivematica/pull/218 this pull request] and associated discussion.

Example docstrings with Sphinx markup below. Other attributes that can be used in a docstring can be found [http://sphinx-doc.org/domains.html#the-python-domain on the Sphinx website].

<pre>def get_unit_status(unit_uuid, unit_type):
"""
Get status for a SIP or Transfer.

Returns a dict with status info. Keys will always include 'status' and
'microservice', and may include 'sip_uuid'.

Status is one of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING.
Microservice is the name of the current microservice.
SIP UUID is populated only if the unit_type was unitTransfer and status is
COMPLETE. Otherwise, it is None.

:param str unit_uuid: UUID of the SIP or Transfer
:param str unit_type: unitSIP or unitTransfer
:return: Dict with status info.
"""</pre>

<pre>def each_child(path, file_group_identifier, base_path, base_path_name, sip_uuid):
"""
Iterates over entries in a filesystem, beginning at `path`.

At each entry in the filesystem, yields either a File model instance
(for files) or a string (for directories).

When iterating, makes two passes: first iterating over files, then
directories. Does not iterate over directories; consumers should
call this function again on directory strings to recurse.

:param string path: path to a directory on disk to recurse into.
:raises ValueError: if the specified path does not exist, or is not a directory.
"""
</pre>

=== File Structure ===
The file structure in Archivematica will comply with the Filesystem Hierarchy Standard (FHS).
<br/>More information on this standard is available at:
<br/>http://www.pathname.com/fhs/

Improvements/warc

2017-03-23T19:50:11Z

Hbecker: Move to feature requirements category

[[Category:Feature requirements]]

== Synopsis ==

Improvements to Archivematica's handling of WARC files could go in a number of directions, most of which involve better extraction of technical and provenance metadata to Archivematica's METS file, which would improve the understanding and preservation of the WARC files overtime.

== User story ==

As an archivist I want an external agent to be recorded in the METS file to records the software agent that created the WARC file.

As an archivist I want a creation event for the creation of the WARC file to be recorded in the METS file.

As an archivist, I want to record relevant Dublin core in the dmdSec (identifier, isPartOf, rights) of the METS file.

As an archivist I want there to be a validation rule in FPR using Warcat so I get more meaningful validation output.

== Status ==

Some code is in a development branch of Archivematica (https://github.com/artefactual/archivematica/tree/dev/issue-8634-warc-mets) which will read certain elements of the WARC header. This lays the groundwork for parsing this descriptive information to the METS file. This code is based on an Archivematica branch that introduces external agents to the METS file, which lays the ground work for describing the software agent that created the WARC (e.g. ArchiveIt, wget, chrome extension, etc)

== Analysis ==

=== WARCat ===

We tested and evaluated the tool WARCat for verifying, validating and extracting content from WARC files: https://github.com/chfoo/warcat

Here's what we found about how warcat verifies WARC files:

* Iterates through a (possibly gzipped) WARC file
** During iteration, uses the Content-Length and looking for delimiters (typically newlines) to verify that it's reading each block correctly.
** archive-it.warc, chrome.warc and wget.warc all fail this correct-iteration-checking

* The verify command checks lots of things, mostly related to the various headers.

* Checks 'WARC-Record-ID', 'Content-Length', 'WARC-Date', 'WARC-Type' in headers
** If 'WARC-Block-Digest', checks block checksum
** If 'WARC-Payload-Digest', checks payload checksum
** Checks record ID has not been seen before in this WARC file
** Checks no whitespace in record ID
** Checks 'Content-Length' also has 'Content-Type'
** If 'WARC-Concurrent-To' or 'WARC-Refers-To', checks 'WARC-Type' not 'warcinfo', 'conversion' or 'continuation' and that concurrent/refers to record ID has been seen before
** If 'WARC-Type' is 'response', 'resource', 'request', 'revisit', 'conversion' or 'continuation', checks 'WARC-Target-URI'
** If 'WARC-Type' is 'warc_info', checks no 'WARC-Target-URI' *
** If 'WARC-Target-URI' checks no whitespace
** If 'WARC-Type' is 'warcinfo', checks no 'WARC-Filename'
** If 'WARC-Type' is 'revisit', checks no 'WARC-Profile'
** If 'WARC-Type' is 'continuation', checks 'WARC-Segment-Origin-ID' and 'WARC-Segment-Total-Length'
** If 'WARC-Type' is not 'continuation', checks no 'WARC-Segment-Origin-ID' or 'WARC-Segment-Total-Length'

We think there's a typo in this check, because other places refer to a 'WARC-Type' of 'warcinfo', and this is the only place that refers to a 'warc_info'

The code for the checks is found here: https://github.com/chfoo/warcat/blob/master/warcat/tool.py#L262-L406

and the checksum verification is here: https://github.com/chfoo/warcat/blob/master/warcat/verify.py#L38-L67

Iterating through the records is done here https://github.com/chfoo/warcat/blob/master/warcat/model/warc.py#L62-L89

and here https://github.com/chfoo/warcat/blob/master/warcat/model/warc.py#L62-L89

== Changes to METS ==

Below is a mock-up of an AIP METS file with enhancements for recording WARC files.

<pre>

<?xml version='1.0' encoding='ASCII'?>
<mets:mets xmlns:mets="http://www.loc.gov/METS/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<mets:metsHdr CREATEDATE="2015-11-27T17:17:29"/>
<mets:dmdSec ID="dmdSec_1">
<mets:mdWrap MDTYPE="DC">
<mets:xmlData>
<dcterms:dublincore xmlns:dcterms="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xsi:schemaLocation="http://purl.org/dc/terms/ http://dublincore.org/schemas/xmls/qdc/2008/02/11/dcterms.xsd">
<dc:identifier>urn:uuid:33bf5acf-4584-446f-b187-ce4f6ad79af9</dc:identifier>

<dc:isPartOf>Sooke Artworks Exhibit 2014</dc:isPartOf>

<dc:isPartOf>4867-20141008190114161</dc:isPartOf>

<dc:rights>collectionPublic=false</dc:rights>

</dcterms:dublincore>
</mets:xmlData>
</mets:mdWrap>
</mets:dmdSec>
<mets:amdSec ID="amdSec_1">
<mets:techMD ID="techMD_1">
<mets:mdWrap MDTYPE="PREMIS:OBJECT">
<mets:xmlData>
<premis:object xmlns:premis="info:lc/xmlns/premis-v2" xsi:type="premis:file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<premis:objectIdentifier>
<premis:objectIdentifierType>UUID</premis:objectIdentifierType>
<premis:objectIdentifierValue>9a6db35f-b444-4295-a1b9-c0c94665c778</premis:objectIdentifierValue>
</premis:objectIdentifier>
<premis:objectCharacteristics>
<premis:compositionLevel>0</premis:compositionLevel>
<premis:fixity>
<premis:messageDigestAlgorithm>sha256</premis:messageDigestAlgorithm>
<premis:messageDigest>b8ed228653bbe2fc73f5a4711daaab3b427bc57920fc00778b9b96da35d5cbd9</premis:messageDigest>
</premis:fixity>
<premis:size>77038680</premis:size>
<premis:format>
<premis:formatDesignation>
<premis:formatName>WARC (Web ARChive)</premis:formatName>
<premis:formatVersion>ISO 28500</premis:formatVersion>
</premis:formatDesignation>
<premis:formatRegistry>
<premis:formatRegistryName>PRONOM</premis:formatRegistryName>
<premis:formatRegistryKey>fmt/289</premis:formatRegistryKey>
</premis:formatRegistry>
</premis:format>
<premis:objectCharacteristicsExtension>

</premis:objectCharacteristicsExtension>
</premis:objectCharacteristics>
<premis:originalName>%transferDirectory%objects/ARCHIVEIT-4867-NONE-15219-20141008190130659-00000-wbgrp-crawl052.us.archive.org-6442.warc</premis:originalName>
</premis:object>
</mets:xmlData>
</mets:mdWrap>
</mets:techMD>
<mets:digiprovMD ID="digiprovMD_1">
<mets:mdWrap MDTYPE="PREMIS:EVENT">
<mets:xmlData>
<premis:event xmlns:premis="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>670799cf-5ca0-4869-b0ba-7d1d951e3857</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>creation</premis:eventType>
<premis:eventDateTime>2015-11-27T17:14:59</premis:eventDateTime>
<premis:eventDetail>software: Heritrix/3.3.0-SNAPSHOT-20140912-0039 http://crawler.archive.org
ip: 207.241.226.89
hostname: wbgrp-crawl052.us.archive.org
format: WARC File Format 1.0
conformsTo: http://bibnum.bnf.fr/WARC/WARC_ISO_28500_version1_latestdraft.pdf
isPartOf: 4867-20141008190114161
description: recurrence=NONE, maxDuration=3600, maxDocumentCount=null, isTestCrawl=false, isPatchCrawl=false, oneTimeSubtype=CRAWL_SELECTED_SEEDS, seedCount=1, accountId=739, accountType=SUBSCRIBER, organizationName="Not a Real Institution", collectionId=4867, collectionName="Sooke Artworks Exhibit 2014", collectionPublic=false
robots: obey
http-header-user-agent: Mozilla/5.0 (compatible; archive.org_bot; Archive-It; +http://archive-it.org/files/site-owners.html)
</premis:eventDetail>

<premis:eventOutcomeInformation>
<premis:eventOutcome></premis:eventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote></premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
</premis:eventOutcomeInformation>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>URI</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>http://crawler.archive.org</premis:linkingAgentIdentifierValue>

</premis:linkingAgentIdentifier>
</premis:event>
</mets:xmlData>
</mets:mdWrap>
</mets:digiprovMD>
<mets:digiprovMD ID="digiprovMD_7">
<mets:mdWrap MDTYPE="PREMIS:AGENT">
<mets:xmlData>
<premis:agent xmlns:premis="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<premis:agentIdentifier>
<premis:agentIdentifierType>URI</premis:agentIdentifierType>
<premis:agentIdentifierValue>http://crawler.archive.org</premis:agentIdentifierValue>

</premis:agentIdentifier>
<premis:agentName>Heritrix/3.3.0-SNAPSHOT-20140912-0039</premis:agentName>

<premis:agentType>software</premis:agentType>
</premis:agent>
</mets:xmlData>
</mets:mdWrap>
</mets:digiprovMD>
</mets:amdSec>
<mets:fileSec>
<mets:fileGrp USE="original">
<mets:file GROUPID="Group-9a6db35f-b444-4295-a1b9-c0c94665c778" ID="file-9a6db35f-b444-4295-a1b9-c0c94665c778" ADMID="amdSec_1" DMDID="dmdSec_1">
<mets:FLocat xlink:href="objects/ARCHIVEIT-4867-NONE-15219-20141008190130659-00000-wbgrp-crawl052.us.archive.org-6442.warc" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>
</mets:file>
</mets:fileGrp>
</mets:fileSec>
<mets:structMap ID="structMap_1" LABEL="Archivematica default" TYPE="physical">
<mets:div LABEL="WARC_file-b681af4b-8e17-479e-8a1f-0e9443415d5e" TYPE="Directory">
<mets:div LABEL="objects" TYPE="Directory">
<mets:div LABEL="ARCHIVEIT-4867-NONE-15219-20141008190130659-00000-wbgrp-crawl052.us.archive.org-6442.warc" TYPE="Item">
<mets:fptr FILEID="file-9a6db35f-b444-4295-a1b9-c0c94665c778"/>
</mets:div>
</mets:div>
</mets:div>
</mets:structMap>
</mets:mets>

</pre>

Improvements/Islandora

2017-03-23T19:49:45Z

Hbecker: Move to feature requirements category

Sections of this page have been copied and adapted from the Islandora Foundation's [https://wiki.duraspace.org/display/ISLANDORA/Archidora Archidora documentation] under a [http://creativecommons.org/licenses/by-sa/3.0/ Creative Commons Attribution-Share Alike 3.0 Unported License]. These sections are appended by a hyperlink to the original content in the Islandora wiki. We are grateful to the Islandora Foundation for both writing and sharing this documentation.

[[Category:Feature requirements]]

== Synopsis ==

Archidora is a module that integrates the digital preservation functionality of Archivematica with [http://islandora.ca/ Islandora]. It was developed by [http://www.artefactual.com/ Artefactual Systems] and [http://www.discoverygarden.ca/ Discovery Garden], sponsored by the [http://library.usask.ca/ University of Saskatchewan Library].

== User story ==

The goal of the Archidora module is to allow Islandora users to seamlessly preserve content that is ingested into Islandora using Archivematica's suite of digital preservation micro-services, creating preservation copies of that content for long-term storage. The Islandora user ingesting the content should not be required to mediate the transfer to Archivematica in any way. Upon completion of the transfer and ingest into Archivematica, a notification is sent back to Islandora indicating that the storage was successful.

[[File:Archidora-1.png|500px]]

== Status ==

The Archidora module was developed in 2014 and has been deployed at the University of Saskatchewan Library since 2015. Testing is ongoing.

The code is currently held [https://github.com/Islandora-Labs/archidora in Github by the Islandora Foundation], but is not being actively maintained.

== Analysis ==

Building on the basic workflow described in the [User story], above, the following detailed workflow was developed to describe how content is ingested from Islandora into Archivematica and stored.

# Content Upload
## Content is ingested into Islandora
## Drupal cron and Fedora content validation are used to trigger content upload to Archivematica
## One upload is created per Fedora Object
## Metadata includes an AIP ID generated either from user input or based on the collection metadata
## METS.xml is posted to the Archivematica URI
# Create Transfer
## Archivematica parses and validates the METS.xml
## If the METS is valid, Archivematica sends either a 201 (Created) or 412 (Precondition failed) response to Islandora; Islandora saves the EM-IRI from the response and uses it to notify the user if the transfer was successful
## Archivematica identifies which files to request from Islandora
## The transfer object is created
## The file URIs are passed to Archivematica for asynchronous processing
# Collect Files
## Files are retrieved from the Fedora REST API using HTTP GET and added to the transfer; if there is an error, an error response is returned
## Checksums are confirmed
# Transfer and Ingest
## Files are moved to the watchedDirectory
## Transfer and Ingest are completed in Archivematica, either manually or automatically
## A blank HTTP POST is sent to SE-IRI; if the HTTP POST is false, the transfer or ingest are still in progress
# Check Archivematica status
## Islandora uses GET statement IRI to request status of transfer
# Status Response
## If the ingest is successful, Archivematica sends either a 201 (Created) or 412 (Precondition failed) response to Islandora, letting Islandora know that the last object in the AIP has been uploaded
# Content deleted from Islandora
## The Hi-Res datastream is deleted from Islandora (configurable), preserving only the access copy/ies of the content
## Islandora sends an HTTP POST to EM-IRI to indicate that the content has been deleted
# Log Islandora deletion
## Archivematica marks objects as deleted from the access system
## Search index is updated
## Archivematica sends either a 200 (OK) or 400 (error) response to Islandora

[[File:Archidora-2.png|500 px]]

== Download ==

* Islandora module: https://github.com/discoverygarden/archidora
* Archivematica: Archivematica 1.4 and Storage Service 0.7 or later is required; download from http://www.archivematica.org.
* This integration is currently considered a beta feature.

- ''[https://wiki.duraspace.org/display/ISLANDORA/Archidora Islandora Archidora documentation]''

== Install ==

Installation and testing is similar to any Drupal module. Please see [https://wiki.duraspace.org/display/ISLANDORA715/Installing+the+Islandora+Enhancement+Modules Installing the Islandora Enhancement Modules] for details.

- ''[https://wiki.duraspace.org/display/ISLANDORA/Archidora Islandora Archidora documentation]''

== Configure ==

=== In the Archivematica Storage Space: ===

* Create a Space with access protocol FEDORA via SWORD2.
* Create a Location within that Space (purpose = FEDORA deposits)
* Enter the Fedora URL, username and password.
** See [https://www.archivematica.org/en/docs/storage-service-0.7/administrators/#fedora Archivematica Storage Service documentation] for more details.
* Archivematica may also be configured to call back to Islandora to delete the high-res "OBJ" datastreams - this is done in the Storage Service > Administration > Service callbacks
** URI: http://islandora-base-url/islandora/object/<source_id>/archidora/{Islandora API}/delete
*** Where the API key is the "Islandora Archivematica integration API key" listed/generated on the Archidora admin screen
** Event: post-store
** Method: post
** Expected status: 200

* Note: the OBJ datastreams are not deleted automatically, but rather are listed at the collection level (or compound object level) on the Manage | Archivematica tab. They can be deleted individually or in bulk.

=== On the Archivematica dashboard: ===

* On the administration tab, add the IP address of the storage service to the IP whitelist for the REST API

=== In Islandora ===

* Enable cron; configure it to run at reasonably frequent intervals (e.g. every five minutes), otherwise the expected callbacks may not be triggered often enough.

* Configure Archidora at admin/islandora/archidora.
** Archivematica Storage Service Base URL - normally http://archivematica-url:8000
** Deposit Location - will be configured automatically once storage service URL is entered
** Archivematica User - Archivematica dashboard user to be used for Islandora integration (not storage service)
** Archivematica API Key - API key for the Archivematica dashboard user listed above
** EM-IRI Solr field - used for constructing Sword API call (default is "RELS_EXT_edit_media_uri_ms")
** AIP max age - new objects will not be added to a deposit after the specified time has elapsed
** AIP max size - new objects will not be added to a deposit after the specified size has been reached. Note that this is really the transfer size; the ** AIP could be larger due to normalized objects
** Cron time - the amount of time for which the queue of items will be allowed to process, at each cron invocation. Setting a higher time is recommended if compound objects are being ingested (especially manually), otherwise the relationships may not be included in the METS file sent to Archivematica

* You may also need to add a rule to the firewall on the Fedora server to allow access from the Archivematica storage service (e.g. to port 8080)

* Collection-level configuration:
** Check off "Don't Archive Children" to stop objects from being sent to Archivematica for a particular collection.

- ''[https://wiki.duraspace.org/display/ISLANDORA/Archidora Islandora Archidora documentation]''

== Scope ==

Archidora is still considered a beta feature. As such, further development is likely required to bring it to stability.

Other proposed improvements include the following:

* Ingest Premis from Islandora into Archivematica
* Ingest other metadata (e.g., DDI)
* Ingest Bags
* Format Policy Registry Integration
* Asynchronous derivative generation
* Integrate with checksum checker
* Provide more information back to Islandora (e.g., aip url)
* Support other workflows (e.g. upload from Archivematica to Islandora)
* Support Fedora for AIP Storage
* Improve reporting/logging in Archidora

No estimates have been prepared for the above.

== Interest ==

Please feel free to add your or your organization's name and any comments to this section if you have an interest in improving this module.

Artefactual would like to see active development on Archidora. We are able to do the development work, for a fee. We are also willing to assist others to complete all or part of the work required.

Interested in talking to us about sponsoring development of Archidora? Get in touch with Artefactual at info@artefactual.org or with Discovery Garden at info@discoverygarden.ca.

Interested in contributing code to the Archidora project? [https://github.com/Islandora-Labs/archidora Create a pull request]!

Email preservation

2017-03-23T19:48:57Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Email preservation

[[Category:Feature requirements]]

==Functional Requirements==

These are requirements for email preservation in Archivematica:
*Preservation format
**Proprietary closed formats such as PST should be converted to open preservation format
**Preservation format should be text or xml based
**Email messages, calendars, contacts and other related entities should be normalized to the preservation format
**Preservation format should preserve the [[Significant characteristics of email|significant characteristics]] of the email messages
**Preservation format should be capable of being viewed as an access format or should be capable of generating an access format
*Attachments
**Attachments should be converted to preservation and access formats
**Converted attachments should retain links to emails to which they were attached
*Access format
**Access format should be human-readable and should be recognizable as email
**Access format should allow simple, intuitive navigation between messages, attachments, email boxes, contacts, calendars
**Access format should allow navigation to normalized access copies of attachments
</br>

==Formats==

===EML===

From http://www.coolutils.com/Formats/EML:
*EML, which stands for ‘E-mail’, is the file extension of the Outlook Express Saved Mail Messages files. It belongs to the Microsoft range of e-mail management programs and is used for saving e-mails for storage and forwarding purposes.
*Since the object of an EML file is to store e-mail messages, it is a plain text file, and as a result, has a standard file structure. It consists of a short header and the main body. The header contains the e-mail addresses of the sender and the recipient, the subject, and the time and date of the message. The main message of the e-mail is in the body of the file. EML files can also contain hyperlinks and attachments.
*Since EML files are created to comply with the industry RFC 822 standard, they can be used with most e-mail clients, servers and applications. Besides the Microsoft Outlook Express, EML files can be opened using most e-mail clients, such as Microsoft Outlook, Microsoft Entourage, Mozilla Thunderbird, Apple Mail, and IncrediMail. Since EML files are plaintext and formatted much like MHT (MIME HTML) files, they can also be opened directly in the Internet Explorer, Mozilla Firefox and Opera, by first changing the file extension from ‘.eml’ to ‘.mht’. It is also possible to view EML files using notepad or any other text editor.

[http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&id=1018 PRONOM]:
*Defined in RFC5322 the Internet Message Format provides a syntax for text messages that are sent among computer users, within the framework of "electronic mail". Consisting of header fields and an optional body, encoded using ASCII characters. The Internet Message Format (*.eml) is a format used by numerous email systems. The format does not prevent the use of alternative formats for storage of electronic mail on those systems.
</br>

===MBOX===

From [http://www.significantproperties.org.uk/email-testingreport.html Inspect: Significant Properties Testing Report: Electronic Mail] Gareth Knight, 30 March 2009:
*"The mbox family refers to four related, but only semi-compatible formats for the storage of one or more email messages and attachments. The four formats - mboxo, mboxrd, mboxcl, and mboxcl2 – originate from different versions of Unix. Each mbox file represents a set of email messages that are ordered sequentially and grouped into a ‘folder ‘. Email messages are stored in their source format, e.g. plain text may be stored as ASCII or Unicode, binary data is stored as Base64-encoded text. The format is well supported by a number of email applications and, thanks to its text-based composition can be processed, rendered and converted by a wide range of text processing software."
</br>

===Maildir===

From http://wiki.dovecot.org/MailboxFormat/Maildir:
*"This format debuted with the qmail server in the mid-1990s. Each mailbox folder is a directory and each message a file. This improves efficiency because individual emails can be modified, deleted and added without affecting the mailbox or other emails, and makes it safer to use on networked file systems such as NFS."
From [http://www.significantproperties.org.uk/email-testingreport.html Inspect: Significant Properties Testing Report: Electronic Mail] Gareth Knight, 30 March 2009:
*"Maildir is an organizational structure for the storage of one or more emails on a file system. Each email is stored as a distinct file in one of three sub-directories: the ‘tmp ‘ sub-directory temporarily stores emails during processing; ‘new ‘ contains newly delivered emails; and ‘cur ‘ contains emails that have been processed by the client ‘s mail-reader software. The storage of each email as a distinct file in the file structure is cited as workaround to file locking issues that affect compound formats, such as mbox that update the mail data file that the user is accessing. However, the filename convention used for the storage of emails may cause incompatibility in implementations of Maildir for Unix-compatible and Microsoft Windows operating systems. The colon character is an illegal character in Microsoft Windows. However, there is no standard on the alternative character that may be used in the environment."
Notes:
*Archivematica sanitizes filenames to remove characters that are prohibited in various operating systems. In the case of maildir, a filename such as ''1330052116_1.7424.dell-desktop,U=153,FMD5=f72fc74533f0b9f432010a357af35516:2,'' would be changed to ''1330052116_1.7424.dell-desktop_U_153_FMD5_f72fc74533f0b9f432010a357af35516_2_''. If we do ingest and preserver maildir files we may need to skip the name cleanup micro-service if email systems depend on the format of the filename to recognize maildir files.
</br>

==Tools==
*[http://www.five-ten-sg.com/libpst/rn01re01.html Readpst], which can be downloaded from http://www.five-ten-sg.com/libpst/packages/, converts PST (MS Outlook Personal Folders) files to mbox and other formats.
*[http://offlineimap.org/ OfflineImap] connects to IMAP accounts and saves the contents locally as maildir backups.
*[https://gist.github.com/1709069 md2mb.py] is a python script that converts maildir to mbox format.
</br>

===Muse===
[http://mobisocial.stanford.edu/muse/ Muse] is a social media tool that can render mbox files as plain text messages with attachments.
*Features:
**Ability to tag restricted messages and export the untagged messages as an mbox file. Note that tagging one message also tags any threads containing the same message.
**Ability to keyword search email messages.
**Ability to key quickly through messages by using forward and back arrows on the keyboard.
**Ability to click on sender or receiver to view all the messages sent or received by that individual.
**Ability to click on selected terms in the email to view other messages with the same terms.
*Drawbacks:
**In alpha development, somewhat brittle in Linux environment.
**Poorly designed user interface.
**Poor user documentation.
</br>

==Email preservation research==
[http://dx.doi.org/10.7207/twr11-01 Preserving Email] (DPC Technology Watch Report 11-01, ISSN 2048-7916, Digital Preservation Coalition 2011), Chris Prom, University of Illinois.
*"In general, if an institution can get email into one of the MBOX or EML formats, it has taken a very big step on the road toward preserving email." (p.23)
*"XML conversion tools...can be very useful in achieving format neutrality. However, the author is aware of no general-purpose tools that are intended to facilitate the access, display, searching, or visualization of messages that have been migrated to XML. Until such tools have been developed – if they ever are – institutions will be forced to provide access to migrated messages using an email client of their choice or the user’s choice, recognizing that specific tools support different functionality." (p.23)
*"Institutions are beginning to implement the Email Account Schema [CERP], but few tools exist to query, display and render messages that are stored in the format. If the digital preservation community were to develop tools that support the Email Account Schema or a different XML standard for email, that XML format would be a likely candidate for adoption as an International Council on Archives or even an ISO standard." (p.24)
**"Until such applications are developed, it may seem that there is relatively little immediate benefit to be gained by migrating email into an XML-based format. Therefore, institutions that decide to keep email in an XML format should also keep a copy of messages in one of the IETF formats, preferable EML, since it allows attachments to be written as separate files." (p.24)
*"Once messages have been migrated outside their native client/server architecture, they can be searched, retrieved and displayed by loading them back into another server or Preserving Email client application. Therefore, repositories should keep the original email format or MBOX/EML files as an access copy, which can be imported into email clients as needed." (pp.28-29)
*"At present, several tools, including Hypermail and Aid4Mail, can convert messages to a static HTML format. In addition, a tool currently under development at Stanford University shows exciting potential for making preserved email useful. The Muse program, which can capture messages from several server environments to a local computer, also includes a search, browsing, visualization and analysis tool (Stanford University, Mobisocial Laboratory 2011)." (p.29)
</br>
[http://www.ifs.tuwien.ac.at/dp/ipres2010/schedule.html 2010. Reshaping the Repository: The Challenge of Email Archiving], Goethals, A. and Gogel, W. In 7th International Conference on Preservation of Digital Objects], (iPRES2010). Vienna, Austria.
*"We have preliminarily chosen to use the CERP/EMCAP schema, because we think it strikes the right balance between fully supporting the complexities of email headers and structure with a welcome lack of manipulation of the message bodies and attachments. Unlike most of the other schemas, it uses generic <Header> elements to store the names and values of the message headers. The advantage of this approach is that it can accommodate unanticipated headers, for example custom headers added by client systems, or those that will be added to future revisions of the email RFCs. It can support multiple message bodies per email, including HTML, and pointers to externally-stored attachments. They also have a separate schema for wrapping base64-encoded attachments, however we will likely decode attachments and store them in their original formats. While the CERP/EMCAP schema is designed to contain all the email messages for an account, we anticipate that it will work equally well at storing a single email message, which is how we intend to use it." (p.3)
</br>

==Archivematica 0.9 maildir ingest requirements==

*The Maildir directory as a whole forms the transfer
*The Maildir directory is the preservation master in the AIP
*Attachments must be extracted and normalized
*Extracted attachments and their normalized versions will reside in a directory outside the Maildir directory
*Maildir message names must NOT be sanitized (they contain colons, commas and equal signs)
*The Maildir messages DO NOT have to go through FITS
*The extracted attachments DO have to go through FITS
*Each Maildir subdirectory is normalized to an mbox file for access purposes
*Question: how will we link the mbox file back to the directory in the AIP? May not have to do this for 0.9.

Creating Custom Workflows

2017-03-23T19:48:25Z

Hbecker: Move back to dev doc category

[[Main Page]] > [[Development]] > Creating Custom Workflows

* [[ Creating Custom Workflows 0.9 alpha ]]
* [[ Creating Custom Workflows 0.8 alpha ]]

[[Category:Development_documentation]]

Creating Custom Workflows

2017-03-23T19:47:02Z

Hbecker: Move to deprecated category

[[Main Page]] > [[Development]] > Creating Custom Workflows

* [[ Creating Custom Workflows 0.9 alpha ]]
* [[ Creating Custom Workflows 0.8 alpha ]]

[[Category:Deprecated]]

AIP integrity checking

2017-03-23T19:46:20Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > AIP integrity checking

This is critical requirement because it is required by OAIS, page 4-8, section 4.1.1.3: "The '''Error Checking''' function provides statistically acceptable assurance that no components of the AIP are corrupted during any internal Archival Storage data transfer. This function requires that all hardware and software within the archive provide notification of potential errors and that these errors are routed to standard error logs that are checked by the Archival Storage staff. The PDI Fixity Information provides some assurance that the Content Information has not been altered as the AIP is moved and accessed. Similar information is needed to protect the PDI itself...The storage facility procedures should provide for random verification of the integrity of data objects using CRCs or some other error checking mechanism."

From [[Development_roadmap#AIP_storage|0.8 Development roadmap]]:
* AIP storage integrity checking and logging to confirm that:
** write to storage was successful (completed)
** ongoing integrity of the AIP is maintained and can be demonstrated. ([https://wiki.umiacs.umd.edu/adapt/index.php/Ace ACE]?)

See also [[Fixity checking and reporting]]

[[Category:Feature requirements]]

Adding Format Identification Tools

2017-03-23T19:44:49Z

Hbecker: Move to feature requirements category

{| class="wikitable" style="background-color:#ffcccc; font-size: 120%; font-weight: bold; " cellpadding="10"
| Warning: These instructions are out of date as of Archivematica 1.0. Please see documentation on the [[Format_policy_registry_requirements | FPR]] and [[MCP | MCPServer]]
|}

This will show a developer how to add a new tool new tool to identify file formats (file ids), and allow normalization based on the newly identified file formats.

The workflow this section is looking to implement are:
* Make the fileID tool a selectable choice
* Specify that normalization will be using that fileIDType
* Identify files using that tool, with valid archivematica fileIDs (format IDs)
* Use those format IDs and associated commands to normalize files.

Since each link in the workflow depends on the next one, we need to start at the end and work our way back.

All pk's (primary keys) in the database are UUIDs generated with 'uuid -v4' on the command line

=Add tool=
To include a new tool in the archivematica packages, it should be a dependency package itself.

==FileIDTypes==
First, an entry for FileIDTypes is required for the new tool.

<pre>
INSERT INTO
`FileIDTypes` (`pk`, `description`, `replaces`)
VALUES
('afdbee13-eec5-4182-8c6c-f5638ee290f3','FileIDByFIDO',NULL);
</pre>

==Identify Files Script==

The script to identify the file must be created, and an entry for it added to the FilesIdentifiedIDs table. For Fido this script is [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/lib/clientScripts/archivematicaFido.py archivematicaFido.py] All the arguments the script accepts will be listed as arguments in StandardTasksConfigs, below.

This script must also be added to the client as a supported module by adding it to [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]. Please keep the list in alphabetical order.
<pre>
archivematicaFido_v0.0 = %clientScriptsDirectory%archivematicaFido.py
</pre>

==Run tool==
Now, we set up the chain link that will run the script.

FIDO workflow link example:
<pre>
-- Set up the UUIDs to be used as pks
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';

--Where to go once completed is the same for both
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';
SET @defaultNextChainLink = @YLink;
SET @NextMicroServiceChainLink = @YLink;
</pre>
<pre>
INSERT INTO
StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
VALUES
(@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');
</pre>
* This creates the configuration for running the script
* filterFileEnd, filterFileStart, filterSubDir: What files to run the identification script on. In this case, all the files in the objects/ subdirectory
* execute: The script to execute, as it was defined in [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]
* arguments: Anything that should be passed on the command line to the script. It should include both the argument, and the value to pass to the argument.

<pre>
INSERT INTO
TasksConfigs (pk, taskType, taskTypePKReference, description)
VALUES
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');
</pre>
* This maps between the task (inserted below), and its configuration
* taskType: We want this to run on every file, and a6b1c323-7d36-428e-846a-e7e819423577 is the pk for that
* taskTypePKReference: is semantically a foreign key to StandardTasksConfigs, where our config is stored

<pre>
INSERT INTO
MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)
VALUES
(@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);
</pre>
* This creates the actual link to run
* microserviceGroup:
* currentTask: Points to the TasksConfigs we just inserted
* defaultNextChainLink: where to go if we can't figure out where else to go, usually on failure

<pre>
INSERT INTO
MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)
VALUES
(@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);
</pre>
* This configures where to go once the MicroServiceChainLink is complete, based on exit code
* microServiceChainLink: the chain link we just defined
* exitCode: the exit code this is valid for. In this case, 0=success. These should obviously match what the script generates
* nextMicroServiceChainLink: where to go next. The last link, to return to the processing workflow already exists, and must be defined as the next chain to process. The link is 83484326-7be7-4f9f-b252-94553cd42370

<pre>
SELECT
MicroServiceChainLinks.pk, TasksConfigs.description
FROM
MicroServiceChainLinks
JOIN TasksConfigs ON TasksConfigs.pk = MicroServiceChainLinks.currentTask
WHERE
MicroServiceChainLinks.pk='83484326-7be7-4f9f-b252-94553cd42370';
+--------------------------------------+---------------------------------------------------------------+
| pk | description |
+--------------------------------------+---------------------------------------------------------------+
| 83484326-7be7-4f9f-b252-94553cd42370 | Resume after normalization file identification tool selected. |
+--------------------------------------+---------------------------------------------------------------+
</pre>

=Add workflow=

==Set Selection==
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a piece of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.

For FIDO:
<pre>
INSERT INTO
`MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`)
VALUES
('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed');
</pre>
<pre>
INSERT INTO
`TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`)
VALUES
('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.');
</pre>
<pre>
INSERT INTO
`TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`)
VALUES
('f130c16d-d419-4063-8c8b-2e4c3ad138bb', 'normalizationFileIdentificationToolIdentifierTypes', 'FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'', NULL);
</pre>
<pre>
INSERT INTO
`MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`)
VALUES
('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully');
</pre>

And the chain that starts running the chain links we defined above:
<pre>
INSERT INTO
MicroServiceChains (pk, startingLink, description)
VALUES
('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');
</pre>
* MicroServiceChains.startingLink: the chain link we defined above
* MicroServiceChains.description: name of the tool

==Chain Choice==
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.

Additional choices can be added by adding entries.
<pre>
SELECT
chainAvailable, startingLink, description
FROM
MicroServiceChainChoice
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk
WHERE
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';
+--------------------------------------+--------------------------------------+---------------------+
| chainAvailable | startingLink | description |
+--------------------------------------+--------------------------------------+---------------------+
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |
+--------------------------------------+--------------------------------------+---------------------+
</pre>

Finally, we add the actual choice:
<pre>
INSERT INTO
MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)
VALUES
('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');
</pre>
* MicroServiceChainChoice.choiceAvailableAtLink: MicroServiceChainLink where the choice is made available. We know this is f4dea20e-f3fe-4a37-b20f-0e70a7bc960e
* MicroServiceChainChoice.chainAvailable: The chain we just added

=Add FPR rules=
Populating the local FPR rules can be done by creating a temporary script, and using a set of sample files with extensions. Use the FileIDs for the extensions to populate values for already in preservation/access status, and related commands.

Fido example: [https://github.com/artefactual/archivematica/commit/ea02f7d6ffc420d5675fcd3bf261c63870881a65]

[[Category:Deprecated]]

Access tab

2017-03-23T19:44:16Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Access tab

= Requirements =
*Allow navigation from AIP to DIP(s) and DIP(s) to AIP
*Link AIP to uploaded DIP(s)
</br>

[[File:Access.png|680px|thumb|center|]]

[[Category:Feature requirements]]

TRIM exports

2017-03-23T19:43:36Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > TRIM exports

This page documents ingest of TRIM exports based on requirements for VanDocs ingest at City of Vancouver Archives.

[[Category:Feature requirements]]

==TRIM export contents==

A TRIM export consists of
*1 or more containers
*A manifest of the transfer (manifest.txt)
*XML schema documentation for all xml files in the transfer (container, location and document xml metadata)
*Location metadata (Location.xml)
*Container metadata (ContainerMetadata.xml)
*Document metadata (eg DOC_2012_000100_Metadata.xml)
*Documents (eg DOC_2012_000100.docx)

</br>

[[File:VanDocs1g.png|680px|thumb|center|]]

</br>

==Processing a TRIM export==

===Parsing contents to the SIP===

*Each transfer is broken into one SIP per container
*manifest.txt is copied to metadata/submissionDocumentation/
*Location.xml is copied to metadata/
*All schema documentation is copied to metadata/
*The relevant ContainerMetadata.xml is copied to metadata/
*The relevant document metadata files are copied to metadata/
*All documents are copied to objects/

</br>

[[File:VanDocs2g.png|680px|thumb|center|A SIP generated from a TRIM export]]

</br>

===Verifying manifest===

The contents of the transfer must be verified against the manifest.txt file during the "Verify transfer compliance" micro-service.
Associated PREMIS event: manifest check. See below for details.

==Manifest check==
{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''Semantic unit'''
!style="width:20%"|'''Semantic component'''
!style="width:20%"|'''Sample value(s)'''
!style="width:20%"|'''Notes'''
|-
|eventIdentifier
|eventIdentifierType
|UUID
|
|-
|eventIdentifier
|eventIdentifierValue
|21h50321-6d7b-3855-89ag-a8b0fhc1f256
|
|-
|eventType
|none
|manifest check
|
|-
|eventDateTime
|none
|2011-08-01T09:08:46-01:00
|
|-
|eventDetail
|none
|
|
|-
|eventOutcomeInformation
|eventOutcome
|{pass; fail}
|
|-
|eventOutcomeDetail
|eventOutcomeDetailNote
|
|
|-
|linkingAgentIdentifier
|linkingAgentIdentifierType
|preservation system
|
|-
|linkingAgentIdentifier
|linkingAgentIdentifierValue
|Archivematica-1.0
|
|-
|}

<br>

===Verifying checksums===

Each document metadata file contains an md5 checksum for the document:

</br>

[[File:checksumg.png|680px|thumb|center|]]

</br>

These checksums must be verified during the "Verify transfer checksums" micro-service.
Associated PREMIS event: fixity check

</br>

==Fixity check==

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''Semantic unit'''
!style="width:20%"|'''Semantic component'''
!style="width:20%"|'''Sample value(s)'''
!style="width:20%"|'''Notes'''
|-
|eventIdentifier
|eventIdentifierType
|UUID
|
|-
|eventIdentifier
|eventIdentifierValue
|73f87321-6d7b-3855-89ag-a8b0fhc1f256
|
|-
|eventType
|none
|fixity check
|
|-
|eventDateTime
|none
|2010-08-01T09:08:46-01:00
|
|-
|eventDetail
|none
|program="MD5Deep"; version="3.6"
|
|-
|eventOutcomeInformation
|eventOutcome
|{pass; fail}
|
|-
|eventOutcomeDetail
|eventOutcomeDetailNote
|
|
|-
|linkingAgentIdentifier
|linkingAgentIdentifierType
|preservation system
|
|-
|linkingAgentIdentifier
|linkingAgentIdentifierValue
|Archivematica-1.0
|
|-
|}

<br>

==The AIP METS file==

===dmdSecs===

*Each container will have one dmdSec consisting of Dublin Core metadata derived from the TRIM export metadata (''ContainerMetadata.xml'')
*Each file will have one dmdSec consisting of Dublin Core metadata derived from the TRIM export metadata (eg ''DOC_2012_000100_Metadata.xml'')

</br>

[[File:dmdSecsg.png|680px|thumb|center|]]

</br>

====Container metadata mapping====

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''TRIM element'''
!'''DC element'''
!'''RAD/AtoM element'''
!'''Comments'''
|-
|<TitleFreeTextPart>
|<dcterms:title>
|'''Title proper'''
|
|-
|<Department>
|<dcterms:creator>
|'''Name'''
|AtoM adds a Name field linked to the Date(s) of creation field
|-
|<DateModified>
|<dcterms:date>
|'''Date(s) of creation'''
|Date range based on earliest and latest DateModified in document metadata
|-
|<OPR>
|<dcterms:provenance>
|'''Immediate source of acquisition'''
|
|-
|<RecordNumber>
|<dc:identifier>
|'''Identifier'''
|Only the numbers to the right of the slash in this field are used - eg 04-4000/0000070 --> 0000070
|-
|n/a
|<dcterms:extent>
|'''Physical description'''
|Count of documents in the SIP plus fixed text: "digital objects"
|-
|n/a
|n/a
|'''Level of description'''
|Level of description taken from METS structMap div TYPE
|-
|<FullClassificationNumber>
|<dcterms:isPartOf>
|n/a
|Field does not map to RAD but is used along with <OPR> to determine DIP upload destination
|-
|}

</br>

'''Sample container description'''

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''TRIM'''
!'''AtoM'''
|-
|'''<TitleFreeTextPart>''' PCI Compliance
|'''Title proper''': PCI Compliance
|-
|'''<Department>''' IT Strategy, Business Relationships and Projects - IT
|'''Name''': IT Strategy, Business Relationships and Projects - IT
|-
|'''<DateModified>''' 2010-03-01T18:20:15-08:00 / 2012-05-01T19:26:23-08:00
|'''Date(s) of creation''': 2010-03-01 - 2012-05-01
|-
|'''<OPR>''' IT Business Strategies
|'''Immediate source of acquisition''': IT Business Strategies
|-
|'''<RecordNumber>''' 04-4000/0000070
|'''Identifier''': 0000070
|-
|n/a
|'''Physical description''': 184 digital objects
|-
|-
|n/a
|'''Level of description''': File
|-
|'''<FullClassificationNumber>'''04-4000-20
|
|-
|}

</br>

====Document metadata mapping====

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''TRIM element'''
!'''DC element'''
!'''RAD/AtoM element'''
!'''Comments'''
|-
|<TitleFreeTextPart>
|<dc:title>
|'''Title proper'''
|
|-
|<DateModified>
|<dc:date>
|'''Date(s) of creation'''
|
|-
|<RecordNumber>
|<dc:identifier>
|'''Identifier'''
|
|-
|n/a
|n/a
|'''Level of description'''
|Level of description will be obtained from METS StructMap div TYPE
|-
|}

</br>

'''Sample document description'''

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''TRIM'''
!'''AtoM'''
|-
|'''<TitleFreeTextPart>''' MCPP Project Report
|'''Title proper''': MCPP Project Report
|-
|'''<DateModified>''' 2010-03-01T18:20:15-08:00
|'''Date(s) of creation''': 2010-03-01
|-
|'''<RecordNumber>''' DOC/2010/000100
|'''Identifier''': DOC/2010/000100
|-
|-
|n/a
|'''Level of description''': Item
|}

</br>

===amdSecs===
*Each container will have an amdSec consisting of:
**A digiprovMD with an xlink reference to metadata/ContainerMetadata.xml

</br>

[[File:TRIMamdSec1g.png|680px|thumb|center|Sample amdSec for a container]]

</br>

*Each file will have an amdSec consisting of:
**A rightsMD populated with PREMIS rights (see '''Flagging closed AIPs''', below)
**A digiprovMD with an xlink reference to the the relevant document metadata xml file
**A techMD and digiprovMDs generated by Archivematica during processing

</br>

[[File:TRIMamdSec2g.png|680px|thumb|center|Sample amdSec for a file]]

</br>

===fileSec and structMaps===
*Each METS file will have two structMaps, the Archivematica default structMap and a logical structMap for hierarchically arranging the container into a file and its child items
*The container and file div TYPE elements in the logical structMap will map to the RAD Level of description field in AtoM
*The structMap contains the links between containers and files and their relevant dmdSecs
*The structMap also contains the link between the container and its amdSec
*The files are linked to their amdSecs in the fileSec

</br>

[[File:structMapg.png|680px|thumb|center|]]

</br>

==Flagging closed AIPs==

*The container metadata file (ContainerMetadata.xml) has two fields whose values will be used to populate the PREMIS rights entity in the SIP (in the METS <rightsMD> element), DateClosed and RetentionSchedule. Examples are:
**<DateClosed>2012-08-17T16:13:31-08:00</DateClosed>
**<RetentionSchedule>EV2.3.A</RetentionSchedule>
*The DateClosed field will be used to populate the termOfRestriction startDate in the PREMIS rights entity
*The DateClosed and RetentionSchedule fields will be used to calculate the termOfRestriction endDate in the PREMIS rights entity. For the examples provided above, Archivematica would calculate 5 years from the end of 2012-08-17 and then to the end of the calendar year, for a result of 2017-12-31.
*The closure period would also be captured as a standardized free text entry in the rightsGrantedNote field of the PREMIS rights entity, for example: Closed until 2012-12-31.
*Other PREMIS fields would be auto-populated for every VanDocs ingest as shown in the screenshot below.

</br>

[[File:VanDocs_rights.png|680px|thumb|center|]]

==DIP upload==

*Upon DIP upload to AtoM, the container will become a file-level description, with level of description populated by the structMap div label for the container ("file"). Each object in the DIP will become a child level with the level of description populated by the structMap div label for the object ("item").
*Descriptive metadata in RAD will be populated by the appropriate dmdSec for each container and object (see container and document metadata mapping, above).

Transfer and SIP creation

2017-03-23T19:42:53Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer and SIP creation

[[Category:Feature requirements]]

= Use Cases =
* Standard (i.e. unstructured)
* Bag (zipped and unzipped)
* Maildir
* TRIM export
* [[Dspace exports]]

= Mockups =

1.0 Mockup for Create SIP

[[File:1.0_CreateSIP.png|1000px|thumb|center|]]

= Workflow (Standard Transfer) =
In these workflow diagrams, the white ovals are manual steps and the grey ovals are automated steps.
===Transfer and create SIPs===

'''0.9 Transfer workflow'''

[[File:transferWorkflow0.9.png|680px|thumb|center|]]
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]

'''0.9 Create SIP workflow'''

[[File:createSIPWorkflow0.9.png|680px|thumb|center|]]
[[File:createSIPWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:createSIPWorkflow0.9.pdf]]

'''0.8 Transfer workflow'''

[[File:transfer1.png|680px|thumb|center|]]

====Transfer indexing requirements 0.9 and beyond====

* Full text content
* File embedded metadata
* Formats - by folder, by transfer
* Keyword & pattern matching for privacy/security sensitive information (e.g. social insurance numbers/social security numbers, credit card numbers, email addresses security keywords like 'private', 'confidential' - find or generate list)
* list of PDFs that have not been OCR'ed
* list of password protected / encrypted files
* duplicates with their file paths

Reports for these indexing requirements will be available via the Examine Contents windows, accessible from the Create SIP browser window in the Ingest tab

= Examine contents splash page =

[[File:1.0_ExamineContents.png|1000px|thumb|center|]]

** information about the transfer or selected file group (number of files, size, name, uuid, accession #, and?)
** pie graph showing file type distribution overall and bargraph showing file type by folder and ordered by size)
***clickable links: file type opens into new tab with file browser interface of all of specified format), folders opens into new tab with file browser interface of entire folder in context of rest of transfer)
** search box to search index (opens in new tab)
** Report options (each opens in new tab):
*** See duplicates with locations
*** See Security keywords, CC numbers, SIN/SS#s, Email addresses (with distribution graph)
*** See password protected files with distribution (graph?)

===Ingest SIPs===
[[File:ingest.png|680px|thumb|center|]]

= Workflow (DSpace export) =
In these workflow diagrams, the white ovals are manual steps and the grey ovals are automated steps.

===Transfer and create SIPs===
[[File:Dspace1.png|680px|thumb|center|]]
[[File:DSpace2.png|680px|thumb|center|]]

===Ingest SIPs===
[[File:DSpace3.png|680px|thumb|center|]]

= Dashboard design =
===Dashboard showing list of transfers===
[[File:TransferDashboard1.png|680px|thumb|center|]]
===Click the metadata icon to add transfer metadata===
[[File:TransferDashboard2.png|680px|thumb|center|]]
===Add transfer metadata===
[[File:TransferDashboard3.png|680px|thumb|center|]]

===A completed transfer===
[[File:TransferDashboard4.png|680px|thumb|center|]]
===File visualization reporting page===
[[File:reports.png|680px|thumb|center|]]

= Create SIPs using file manager =
In this mockup, the user has completed a transfer, has created SIP folders in ingestSIP and is preparing to move objects from the completed transfer into the SIP folders.

[[File:thunar.png|680px|thumb|center|]]

= Add Dublin Core metadata to a SIP during ingest =
*All fields are simple Dublin Core except for Relation: Is Part Of, which is designed to upload the SIP to a pre-existing archival description in the access system.
*Multi-value entries are separated by a backward slash: \.

[[File:Ingest1a.png|680px|thumb|center|]]

= The dashboard showing a SIP that has completed the ingest process =

[[File:Ingest2.png|680px|thumb|center|]]

Transfer backlog requirements

2017-03-23T19:42:25Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer backlog requirements

Release 1.0

[[Category:Feature requirements]]

== Transfer Backlog Management ==
* Related issues: Issue 951, Issue 1220, Issue 1141, Issue 1225, Issue 1257

= Requirements for transfer backlog search =

* Add ability to search transfer backlog and send one or more transfers to Ingest
* Add ability to download and/or view files/transfers (via right click)
* Search the following fields: Any field, transfer name, file name, accession number, PUID, Mimetype, Date - Ingest

= Mockup of transfer backlog search =

[[File:1.0_TransferBacklogSearch.png|680px|thumb|center|]]

[[File:1.0_TransBacklogSearchResults.png|680px|thumb|center|]]

= Transfer Workflow =

* Administration - allow MCP access to media or storage where transfer is located
* Assign accession number to transfer
* Remove transfer backup from workflow - no long processing configuration option
* Add Send transfer to backlog microservice
* Add Search transfer backlog tab from Ingest in Dashboard
* Add ability to download and/or view transfers and files from Search tab
* Add ability to send transfers from backlog search to Ingest/Create SIP (checkboxes, send button)
* see workflow diagrams below

'''0.9 Transfer workflow'''

* grey steps are automated, white are manual

[[File:transferWorkflow0.9.png|680px|thumb|center|]]
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]

[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]

= Administration Tab in Dashboard =

* Assign permission and access to the MCPServer to copy from transfer media (hard drives, optical media, USB, etc.) or network location.
* Assign transfer backlog locations (configuration is done outside of AM)
* Assign source directories
* Define transfer types
* Assign report locations (post-1.0)
* Set AIP storage location
* Set DIP upload location

= Accession metadata =

* PREMIS Event = Registration
<event>
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>35cbe00d-d661-4174-b11a-e203f5608008</eventIdentifierValue>
</eventIdentifier>
<eventType>registration</eventType>
<eventDateTime>2012-03-14</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>accession#2012-029</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>archivist</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Courtney Mumma</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>

* Manually input metadata in template on dashboard (See [[File_Browser_Requirements]]) : accession number
* Agent is the archivist logged in at the time doing the accession (post-1.0, for 1.0 this will still be repository)
* Event name is "registration" (to be added to PREMIS events master list should we decide to implement)
* UUID

* Also see Issue 787 on the Archivematica issues list

= Microservices Completed Before Move to Backlog =

* All transfer microservices
* Indexing: See [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]

= Handling of Submission Documentation =

* [http://sites.tufts.edu/dca/about-us/research-initiatives/taper-tufts-accessioning-program-for-electronic-records/| TAPER]?
* Normalized with objects in AIP (0.8)
* Upload submission documentation with transfer in transfer tab - Issue 1255

= Search transfers from Archival Storage =

New sponsored development planned for Archivematica 1.6 or later will allow users to manage the transfer backlog through the Archival Storage tab, as outlined in general workflows described in these diagrams:

[[File:Transfer_management_workflows.png|700px]]

As outlined above, users will be able to:

*Search transfers from archival storage tab
*Download copies of transfers or selected files from archival storage tab
*Be able to perform transfer deletion requests from archival storage tab

'''Transfer search user stories'''

As an archivist, I need to find transfers by searching...

* by the name of the transfer
* by the date the transfer was stored in backlog
* by names of files within the transfer
* by....?

'''Mockups: Version 1'''

Search transfers from Archival Storage:

[[File:Archival_Storage_Transfer_Search.png|800px]]

'''Notes:'''

*Click on "Show transfers" to search AIPs as well as Transfers
*A new column in the table indicates whether a package is a Transfer or an AIP.
*To trigger transfer deletion, click on red "remove" icon (same functionality as AIP deletion)

Search files from transfers in Archival Storage:

[[File:Archival_Storage_Transfer_file_search.png|800px]]

'''Notes:'''

*Clicking on both "Show files" and "Show transfers" before searching will load preview of files from transfer backlog.
*The UUID of the package and an indication of whether the file is from a Transfer or an AIP is in the right column.

'''Mockups: Version 2'''

In this version, toggling between searching for AIPs/Transfers is done through a tab at the top. This makes the development significantly less complicated, as we would not need to combine the Elasticsearch indexes for transfer METS and AIP METS.

Transfer search:

[[File:Transfer_search_v2.png|800px]]

Files within transfer search:

[[File:Transfer_search_files_v2.png|800px]]

Sword API

2017-03-23T19:41:47Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Documentation]] > [[Requirements]] > Sword API

[[Category:Feature requirements]]

== Overview ==

One of the 1.1 release features is a sponsored project to integrate Archivematica with Islandora. This integration will be accomplished by creating a [http://swordapp.org SWORD] 2.0 API for the Archivematica [https://www.archivematica.org/wiki/Administrator_manual_1.0#Storage_service storage service]. Islandora development work will add functionality to Islandora to use this API to deposit digital objects into Archviematica.

As described [[Overview|elsewhere]] on this wiki, ''"the primary function of Archivematica is to process digital transfers (accessioned digital objects), turn them into SIPs, apply format policies and create high-quality, repository-independent Archival Information Packages (AIP)"''.

The Archivematica Sword API allows 3rd party applications to automate the process of creating Transfers.
#Create Transfer - set the name and other metadata about a Transfer
#Populate Transfer - add/edit/update digital objects in a Transfer, and associated metadata
#Finalize Transfer - indicates that the Transfer is ready to start processing.

After content has been deposited, if users have access to the deposit directory they can also manually manipulate the deposit directory aside from any manipulation using the API.

Once a Transfer has been created, populated and finalized, Archivematica will begin processing that Transfer.

== Configuration ==

The SWORD API is part of the Archivematica storage service REST API. The storage service also talks to the Archivematica dashboard, however, to notify it that a, transfer created from a SWORD deposit, has been approved for processing.

To use the SWORD API you must install the Archivematica dashboard 1.0 branch and the Archivematica storage service dev/issue-5980-islandora branch.

Dashboard configuration:

# In dashboard REST API administration, add the IP of the storage service server to IP whitelist
# In dashboard user administration, take note of username and API key for a dashboard user (example: user "demo" and API key "4e5f32ab2aefd3543e1b19a2de554dd65f90108a")
# In dashboard general admininstration, take note of the dashboard's UUID

Storage service configuration:

# In the storage service web interface's Pipelines tab, find the dashboard pipeline (using dashboard's UUID noted during dashboard configuration)
# Enter the dashboard's IP address into the "Remote name" field and the user and API key noted earlier into the "Api username" and "Api key" fields. Click the button to submit these changes.

After configuration you'll need to find the space UUID of the SWORD server. To do so, in the web interface's Spaces tab, click "View Details and Locations" for the SWORD server. The UUID will be shown at the top of the screen. The UUID is used to figure out the initial API URL to access (for example: "http://localhost:8000/api/v1/space/c0bee7c8-3e9b-41e3-8600-ee9b2c475da2/sword/collection/").

== Example session ==

Below is an example session using curl to manipulate the API. In the example a deposit is created, a file is added to it, and it is finalized.

NOTE: The SWORD API is in the process of being moved to the storage service. These URLs may not work.

<pre># create new deposit, providing a METS file specifying digital object URLs to download in the background
curl -v -H "In-Progress: true" --data-binary @mets.xml --request POST http://localhost:8000/api/v1/location/c0bee7c8-3e9b-41e3-8600-ee9b2c475da2/sword/collection/
# response XML includes endpoints for adding additional files, etc.

# add a single file to the deposit
curl -v -H "Content-Disposition: attachment; filename=cat.jpg" --request POST \
--data-binary "@cat.jpg" \
http://127.0.0.1:8000/api/v1/file/9c8b4ac0-0407-4360-a10d-af6c62a48b69/sword/media/

# start another background download of resources specified in a METS
curl -v -H "In-Progress: true" -H "Packaging: METS" --data-binary @mets.xml --request POST http://127.0.0.1:8000/api/v1/file/9c8b4ac0-0407-4360-a10d-af6c62a48b69/sword/media/

# finalize transfer and approve processing
curl -v -H "In-Progress: false" --request POST http://127.0.0.1:8000/api/v1/file/149cc29d-6472-4bcf-bee8-f8223bf60580/sword/
</pre>

You might, at some point, want to list the transfers that have been started. The following curl command will do so.

<pre>curl -v http://localhost:8000/api/v1/location/c0bee7c8-3e9b-41e3-8600-ee9b2c475da2/sword/collection/</pre>

If you're working on a transfer and want to list what files are included in it, the following curl command will list them.

<pre># list files in transfer
curl -v http://127.0.0.1:8000/api/v1/file/9c8b4ac0-0407-4360-a10d-af6c62a48b69/sword/media/</pre>

If you've started a deposit, but want to discard it, you can delete the deposit. The following curl command shows an example.

<pre>curl -v -XDELETE http://127.0.0.1:8000/api/v1/file/149cc29d-6472-4bcf-bee8-f8223bf60580/sword/</pre>

== Single step deposit ==

If you wanted to deposit a transfer then immediately finalize it, after background downloading is complete, you could use something like the following curl command.

<pre>curl -v -H "In-Progress: false" --data-binary @mets.xml --request POST http://localhost:8000/api/v1/location/c0bee7c8-3e9b-41e3-8600-ee9b2c475da2/sword/collection/</pre>

== Endpoints ==

=== Service document ===

The SWORD service document provides information about the SWORD provider's capacities and lists SWORD collections. Archivematica currently includes one SWORD collection to which SWORD deposits can be made: transfers.

The SWORD service document endpoint is located at /api/v1/sword/.

Service document example:
<pre>
<service xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:sword="http://purl.org/net/sword/terms/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns="http://www.w3.org/2007/app">

<sword:version>2.0</sword:version>

<workspace>
<atom:title>Archivematica storage service</atom:title>

<collection href="http://192.168.1.231:8000/api/v1/space/8f24ef5f-19d5-4b77-8a38-b30d034b11e7/sword/collection/">
<atom:title>Collection</atom:title>
<accept>*/*</accept>
<accept alternate="multipart-related">*/*</accept>
<sword:mediation>false</sword:mediation>
</collection>

</workspace>
</service>
</pre>

=== List existing transfers ===
* HTTP Get to the Collection IRI (defined by default as: /api/v1/space/[space UUID]/sword/collection/)
* Transfers are listed as an Atom feed
* Each Atom feed entry contains an "atom:link" element whose href attribute contains the URL needed to get details about the transfer
* optional filters via get parameters (not yet implemented)

Example response:
<pre>
<atom:feed xmlns="http://www.w3.org/2005/Atom">
<atom:id>http://192.168.1.231:8000/api/v1/space/8f24ef5f-19d5-4b77-8a38-b30d034b11e7/sword/collection/</atom:id>
<atom:title type="text">Deposits</atom:title>
<atom:link href="http://192.168.1.231:8000/api/v1/space/8f24ef5f-19d5-4b77-8a38-b30d034b11e7/sword/collection/" rel="self">
</atom:link>
<atom:entry>
<atom:id>http://192.168.1.231:8000/api/v1/location/88e70cd2-9258-4a84-8938-16e74be032e6/sword/</atom:id>
<atom:title type="text">Cinderella</atom:title>
<atom:link href="http://192.168.1.231:8000/api/v1/location/88e70cd2-9258-4a84-8938-16e74be032e6/sword/" rel="self">
</atom:link>
</atom:entry>
</atom:feed>
</pre>

=== Create a new transfer ===
* used to start a new transfer in Archivematica
* HTTP POST of an Atom Entry Document to the Collection IRI (defined by default as: /api/v1/space/[space UUID]/sword/collection/)

'''Possible HTTP Response Codes'''
* HTTP 200 OK - transfer already exists (not yet implemented)
* HTTP 201 Created - transfer has been accepted
* HTTP 412 Precondition Failed - required metadata missing or invalid

Valid requests will receive a Sword Deposit Receipt in the body of the response.

'''Required HTTP Headers'''
The HTTP POST must include certain specific http headers.

Required by http 1.1 specification:
* Host: Must be set to archivematica host name.
* Content-Length: Must be set to the length of the atom entry document.
* Content-Type: Must be set to "application/atom+xml;type=entry".

Required by Archivematica.
* Authorization: Must include the api key and username assigned by Archivematica.

Required by SWORD 2.0 protocol:
* In-Progress: Must be set to "true"

Optional in SWORD 2.0 protocol:
* On-Behalf-Of: not implemented by Archivematica
* Slug: not implemented by Archivematica

'''Required in Body'''
The Body of the request must be a valid METS Document. The required fields in an Atom Entry Document are:
* LABEL: used as the transfer name
* id: used as an accession id (not yet implemented)
* author: used as the source of acquisition (not yet implemented)
* summary: not implemented yet. Could be used in the future as a description for the Transfer
* updated: should be set to the current timestamp.

'''Example'''
Example HTTP POST request:
<pre>
POST /api/v1/space/96606387-cc70-4b09-b422-a7220606488d/sword/collection/ HTTP/1.1
Host: localhost
Authorization: Archivematica-API api_key="XXXXXXXXXXXXXXXXXXXX", username="timh"
Content-Length: 213
Content-Type: application/atom+xml;type=entry
In-Progress: true

<?xml version="1.0" encoding="UTF-8"?>
<METS:mets EXT_VERSION="1.1" OBJID="islandora:72" LABEL="Cinderella"
xmlns:METS="http://www.loc.gov/METS/"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.loc.gov/METS/ http://www.fedora.info/definitions/1/0/mets-fedora-ext1-1.xsd">
<METS:fileSec>
<METS:fileGrp ID="DATASTREAMS">
<METS:fileGrp ID="OBJ">
<METS:file ID="OBJ.0">
<METS:FLocat xlink:title="Cinderella_(1865).pdf" LOCTYPE="URL"
CHECKSUM="fad0d2f982f317a15804d2e5c4b9efcc" CHECKSUMTYPE="MD5"
xlink:href="http://localhost:8080/fedora/get/islandora:72/OBJ/2013-08-12T14:46:43.454Z" />
</METS:file>
</METS:fileGrp>
</METS:fileGrp>
</METS:fileSec>
</METS:mets>
</pre>

Example Response:
<pre>
201 Created
Location: http://localhost/api/v1/location/91d4f258-0050-4914-9d0c-6a74815358d2/sword/

<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:sword="http://purl.org/net/sword/"
xmlns:dcterms="http://purl.org/dc/terms/">

<title>Cinderella</title>
<id>91d4f258-0050-4914-9d0c-6a74815358d2</id>
<updated></updated>
<generator uri="https://www.archivematica.org" version="1.0"/>

<summary>A deposit was started.</summary>
<sword:treatment>A deposit directory was created and, optionally, asynchronous download of digital objects.</sword:treatment>

<link rel="alternate" href="http://www.swordserver.ac.uk/col1/mydeposit.html"/>
<link rel="edit-media" href="http://localhost/api/v1/location/91d4f258-0050-4914-9d0c-6a74815358d2/sword/media/"/>
<link rel="edit" href="http://localhost/api/v1/location/91d4f258-0050-4914-9d0c-6a74815358d2/sword/" />
<link rel="http://purl.org/net/sword/terms/add" href="http://192.168.1.76:8000/api/v1/location/91d4f258-0050-4914-9d0c-6a74815358d2/sword/" />
<link rel="http://purl.org/net/sword/terms/statement" type="application/atom+xml;type=feed"
href="http://localhost/api/v1/location/91d4f258-0050-4914-9d0c-6a74815358d2/sword/state/" />
</entry>
</pre>

=== Add Files to a Transfer ===

Post file data:
* It is possible to also POST the actual file to a deposit location's EM-IRI (/api/v1/location/[location UUID]/sword/media/)
* The file should be posted as the request body and the filename specified using the Content-Disposition header (as per RFC 6266): for example "Content-Disposition: Attachment; filename=dog.jpg"
* An MD5 checksum for an uploaded file can be provided via the Content-MD5 header (as per RFC 1544): for example "Content-MD5: Q2hlY2sgSW50ZWdyaXR5IQ=="
* If only an Atom Entry Document is POST'ed, Archivematica will look for uri's in the file StructMap section of the METS file, and attempt to GET each file listed, to include in the Transfer. (not yet implemented)

=== Finalize a Transfer ===
* POST with no body to the SE-IRI
* set In-Progress HTTP header to : false
* This will tell Archivematica that no further content will be added or removed from the Transfer.
* Archivematica will finish fetching any files that were added previously, and once they have all been downloaded, the Transfer will start through the Archivematica pipeline.

'''Example'''
<pre>
POST http://localhost/api/v1/location/55ce8053-113a-4954-8aa3-fc9771da0bda/sword/ HTTP/1.1
Host: localhost
Authorization: Archivematica-API api_key="XXXXXXXXXXXXXXXXXXXX", username="timh"
Content-Length: 0
Content-Type: application/atom+xml;type=entry
In-Progress: false
</pre>

The response will be HTTP 200/OK or 400/Error (400 if the Transfer was already finalized). If the transfer doesn't exist the response will be HTTP 404.

=== Get Status of Transfer ===
To check Status:

GET the State-IRI
in this example:

GET http://localhost/api/v1/location/09224ddb-4c7d-404c-9218-2f2e1b5a599e/sword/state/ HTTP/1.1

response will include:
<pre><atom:feed xmlns:sword="http://purl.org/net/sword/terms/"
xmlns:atom="http://www.w3.org/2005/Atom">

<atom:category scheme="http://purl.org/net/sword/terms/state"
term="failed"
label="State">
Deposit initiation: Failed
</atom:category>

</atom:feed></pre>
the list of possible descriptions is not finalized.

The state term value will either be "processing" (asynchronous deposit still working), "complete" (ready to be finalized), or "failed" (asynchronous deposit encountered an error).

== Additional Details ==

=== Transfer Details ===
* redirect to Edit-IRI
* used to get basic info about a Transfer with HTTP GET
* can update basic info using HTTP PUT (not yet implemented)
* can allow deletion with HTTP DELETE
* follows the form: /api/v1/location/[deposit location UUID]/sword/
* example: http://192.168.1.231:8000/api/v1/location/09224ddb-4c7d-404c-9218-2f2e1b5a599e/sword/

=== EM-IRI ===
* Edit-Media IRI
* used to add new objects to an existing transfer
* follows the form: [archivematica hostname]/api/v2/transfer/[uuid of transfer]/media
* example: /api/v1/location/[deposit location UUID]/sword/media/

An HTTP POST to the em-iri for a transfer should include a single file as the body. If the HTTP "Packaging" header is set to "METS" then METS XML can be sent as the body, specifying a list of digital object URLs to download in the background.

An HTTP GET should return a list of files in the transfer

An HTTP DELETE to the em-iri will remove all digital objects from the Transfer. This is a valid operation only while the Transfer is being assembled. Once the Transfer has been finalized, attempting to DELETE will return an error.

=== Edit-IRI ===
The client can replace the metadata of a resource by performing an HTTP PUT of a new Atom Entry on the Edit-IRI. (not yet implemented)

This would be used to update metadata about a transfer, such as the transfer name.

* example: http://localhost/api/v2/transfer/1225c695-cfb8-4ebb-aaaa-80da344efa6a/edit

=== SE-IRI ===
* example: http://localhost/api/v2/transfer/add/1225c695-cfb8-4ebb-aaaa-80da344efa6a (not yet implemented)

=== State-IRI ===
* used to retrieve status of transfer
* implemented as Atom document
* example: /api/v1/location/[deposit location UUID]/sword/state/
* should be able to subscribe like RSS feed

=== Service document ===

Object-level PREMIS rights import

2017-03-23T19:41:19Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[Requirements]] > Object-level PREMIS rights import

Page text is a work in progress.

[[Category:Feature requirements]]

== Background ==

The Rockefeller Archives Centre is a heavy user of the PREMIS rights functionality in Archivematica and therefore sponsored the development of this foundational feature. This feature will allow users to apply PREMIS rights to individual objects by including a CSV file with the transfer, rather than having all of the objects in the transfer inherit rights applied to the SIP via the Add Rights form in the metadata module.

== Requirements ==

* Allow a user to include object-level PREMIS rights with an Archivematica transfer via a CSV file.
* Enable those rights to be passed to the appropriate ArchivesSpace resources.

This work mirrors the method of passing object-level descriptive metadata to a transfer by including a CSV file

To further support RAC's Archivematica workflow, this feature also works with Archivematica's [https://www.archivematica.org/docs/archivematica-1.6/user-manual/access/access/#upload-as ArchivesSpace integration].

== Workflow ==

Once the feature is merged into a QA branch of the Archivematica project for public release, this content will be moved to the official [https://www.archivematica.org/en/docs/ Archivematica documentation].

1. '''Create PREMIS rights CSV file (rights.csv)'''

User creates a CSV file containing PREMIS rights information for the desired digital objects. The user will be able to enter multiple acts for the same rights basis. Rows for the same object with the same rights basis will be treated as separate acts for the basis and merged. For example, the first two rows below will be merged, while the third row will be separate.

{|
|file
|basis
|status
|determination_date
|jurisdiction
|start_date
|end_date
|terms
|citation
|note
|grant_act
|grant_restriction
|grant_start_date
|grant_end_date
|grant_note
|doc_id_type
|doc_id_value
|doc_id_role
|-
|image1.tif
|copyright
|copyrighted
|2011-01-01
|ca
|2011-01-01
|2013-12-31
|Terms of copyright.
|Citation of copyright.
|Note about copyright.
|disseminate
|disallow
|2011-01-01
|2013-12-31
|Grant note
|Copyright documentation identifier type.
|Copyright documentation identifier value.
|Copyright documentation identifier role.
|-
|document.pdf
|license
|
|
|
|2000-09-09
|2010-09-08
|Terms of license.
|
|Note about license.
|migrate
|allow
|2000-09-08
|
|Grant note
|License documentation identifier type.
|License documentation identifier value.
|License documentation identifier role.
|}

2. '''Add rights.csv to transfer directory'''

The rights.csv file should be placed inside your transfer's metadata folder. Note that the sample folder structure below may not exactly reflect your directory structure, depending on your transfer requirements.

: /my-transfer
:: image1.tif
:: document.pdf
:: /metadata
::: rights.csv
::: archivesspaceids.csv (optional)

Metrics requirements

2017-03-23T19:40:50Z

Hbecker: Move to feature requirements category

[[Documentation]] > [[Requirements]] > Metrics requirements

[[Category:Feature requirements]]

== Description ==

* The Archivematica team is currently gathering metrics requirements to support digital repository management. The question we would like to answer is: what kinds of metrics would be useful to Archivematica users (e.g. performance/usage statistics, AIP/DIP monitoring, file format statistics, etc.).

* Our goal is to get a sense of the foundational elements necessary to fulfill future reporting requirements that we will need to build into Archivematica ... eventually. We are of course limited by available funding and time. We will prioritize the metrics and do our best to get as much as we can into upcoming releases given available funding and time.

* Related issues: [[https://projects.artefactual.com/issues/1537 #1537]]

= Requirements =

== Use Cases ==

The following are some use cases generated by the Archivematica team and contributed by Archivematica users.

* Output reports on the following:

# number of file format X ingested per month
# total disk size of ingests added per year
# number of failed normalizations per format x
# name of person who processed an AIP
# inventory of shares & mount points (e.g. SIP Source, /var/archivematica, AIP Destination, DIP destination) including disk space (total, available, used & quota) and availability
# total count of file formats
# total disk usage per file format
# growth: monthly report showing growth by file size, file format. Could also tap into descriptive metadata to show "collection growth"
# AIP Monitoring: Bit errors detected, bit errors corrected
# AIP average, median & max directory depth and directory breadth
# File size distribution, file size distribution per format
# Preservation and access normalization monitoring: Display not just file format but also version/specification

== Mockups ==

== Data Model ==

-Logged-in user (should be captured as PREMIS agent)
-UUID of the Archivematica instance (should be captured as PREMIS agent)
-Possibly also environment data: what machines did Archivematica live on, what versions of all the tools were installed (already in PREMIS events), what version of Archivematica was used (already in software agent).

For AIPs:
* AIP name - In METS structMap: <div TYPE="directory" LABEL="[AIPname]-[UUID]">
* AIP UUID - In METS structMap: <div TYPE="directory" LABEL="[AIPname]-[UUID]">
* AIP storage location - Not in METS file
* Date placed in storage - Not in METS file
* Date updated (when we have AIP versioning) - will be in METS header <metsHdr CREATEDATE="2013-05-09T15:00:00" LASTMODDATE=”2014-02-09T21:00:00>
* AIP size - Not in METS file
* Related DIP - Not in METS file
* DIP storage location - Not in METS file
* DIP size - Not in METS file
* Date DIP uploaded - Not in METS file

For transfer backups:
* Transfer name <div TYPE="directory" LABEL="[Transfername]-[UUID]">
* Transfer UUID <div TYPE="directory" LABEL="[Transfername]-[UUID]">
* Transfer size - Not in METS file
* Transfer storage location - Not in METS file
* Date placed in storage - Not in METS file

Metadata import

2017-03-23T19:38:31Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Metadata import

This page documents the workflow and METS content for lower-level metadata import - i.e. metadata to be attached to subdirectories and files within a SIP.

[[Category:Feature requirements]]

==Workflow==

#For simple objects, the user places files in the objects directory, with or without intervening subdirectories. The imported metadata are attached to each object.
#For compound objects, the user creates one or more subdirectories in the objects directory, each containing the files that form a compound object. The imported metadata are attached to each subdirectory.
#*'''The subdirectory names must not contain spaces or other forbidden characters.'''
#The user adds a csv file to the metadata folder for the transfer entitled ''metadata.csv''
#*The first row of the csv file consists of field names.
#*Dublin Core field names must contain the "dc" element in the name, eg "dc.title"
#*Each subsequent row contains the complete set of field values for a single directory or file
#*For multi-value fields (such as dc.subject), the entire column is repeated and each column contains a single value
#*If the metadata are for simple objects, the csv file must contain a "filename" column listing the filepath and filename of each objects: eg "objects/BrocktonOval.jp2"
#*If the metadata are for compound objects, the csv file must contain a "parts" column listing the names of the directories containing the items that form the compound object: eg "objects/Jan021964"
#*Note that filenames can be duplicates of filenames in other subdirectories if desired. For example, the name "page01.jp2" can occur in multiple subdirectories.
#At the generate METS micro-service, Archivematica parses the metadata in ''metadata.csv'' to the METS file, as follows:
#*All Dublin Core elements are used to generate a dmdSec for each directory or file with MDTYPE="DC"
#*All non-Dublin Core elements are used to generate a dmdSec for each directory or file with MDTYPE="OTHER" OTHERMDTYPE="CUSTOM"
#*The dmdSecs are linked to their directories or files in the structMap

</br>

==Simple objects==

This section provides csv file and METS file examples for simple objects - i.e. individual files that are not pages in a compound object such as a book or a newspaper issue.

===CSV file===

'''Sample headings and values'''

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''filename'''
!'''dc.title'''
!'''dcterms.issued'''
!'''dc.publisher'''
!'''dc.contributor'''
!'''dc.subject'''
!'''dc.subject'''
!'''dc.date'''
!'''dc.description'''
!'''notes'''
!'''dcterms.isPartOf'''
!'''repository'''
!'''dc.rights'''
!'''project_website'''
!'''dc.format'''
|-
|objects/BrocktonOval.jp2
|Stanley Park in December
|1996-01-17
|Riley Studios, Vancouver BC
|Don Langfield, photographer
|Vancouver (B.C.)--Parks
|Landscapes
|1992/12/04
|Image shows Brockton Oval after light snowfall
|Originally part of series entitled "Winter in Vancouver"
|Riley Studios collection
|New Caledonia Public Library
|Copyright held by Riley Studios
|http://www.ncpl/donlangfieldphotographs.ca
|image/jp2
|-
|objects/QE Park sunset.jp2
|Sunset in Queen Elizabeth Park
|
|Riley Studios, Vancouver BC
|Don Langfield, photographer
|Vancouver (B.C.)--Parks
|
|1994/07/13
|
|
|Riley Studios collection
|New Caledonia Public Library
|Copyright held by Riley Studios
|http://www.ncpl/donlangfieldphotographs.ca
|image/jp2
|-
|}

</br>

===METS file===

[[File:mets_1g.png|980px|thumb|center|]]
[[File:mets_2g.png|980px|thumb|center|]]
[[File:mets_3g.png|980px|thumb|center|]]

==Compound objects==

This section provides csv file and METS file examples for compound objects - i.e. multi-page digital objects such as newspapers and books.

===CSV file===

'''Sample headings and values'''

{| border="1" cellpadding="10" cellspacing="0" width="100%"
|-
!'''parts'''
!'''dc.title'''
!'''alternative_title'''
!'''dc.publisher'''
!'''dates_of_publication'''
!'''dc.subject'''
!'''dc.date'''
!'''dc.description'''
!'''frequency'''
!'''dc.language'''
!'''forms_part_of'''
!'''repository'''
!'''project_website'''
!'''digital_file_format'''
|-
|objects/Jan021964
|Coast News, January 02, 1964
|Sunshine Coast News
|Fred Cruice
|1945-1995
|Gibsons (B.C.)--Newspapers
|1964/01/02
|Serving the Growing Sunshine Coast
|Weekly
|English
|British Columbia Historical Newspapers collection
|Sunshine Coast Museum and Archives
|http://historicalnewspapers.library.ubc.ca
|image/jp2
|-
|objects/Jan091964
|Coast News, January 09, 1964
|Sunshine Coast News
|Fred Cruice
|1945-1995
|Gibsons (B.C.)--Newspapers
|1964/01/09
|Serving the Growing Sunshine Coast
|Weekly
|English
|British Columbia Historical Newspapers collection
|Sunshine Coast Museum and Archives
|http://historicalnewspapers.library.ubc.ca
|image/jp2
|-
|}

</br>

</br>

===METS file===

[[File:mets_4g.png|980px|thumb|center|]]
[[File:mets_5g.png|980px|thumb|center|]]
[[File:mets_6g.png|980px|thumb|center|]]

Requirements/MediaConch integration

2017-03-23T19:37:46Z

Hbecker: Move to feature requirements category

Page for requirements gathering for integration of [https://mediaarea.net/MediaConch/ Media Conch] in Archivematica.

[[Category:Feature requirements]]

=Use cases=

==MKV conformance checking==

===Pre-normalization===

'''User Story:''' As a digital preservationist, I need to know whether or not mkv files which I am preserving are valid before I process them, because if they are invalid it might impact further preservation decisions.

'''Specification'''

''Feature:'' Pre-ingest conformance check

Archivists need to know whether or not mkv files which they are preserving are valid before they process them, because if they are invalid it might impact further preservation decisions.

''Background:'' the file has been identified as an .mkv

''Scenario outline:'' Jamie wants to confirm that their file is a valid .mkv </p>
''When:'' file validation is run </p>
''And:'' the file is <file_validity> .mkv </p>
''Then:'' validation micro-service output is <microservice_output> </p>
''And:'' <am_action> </p>
''And:'' a PREMIS validation event is generated with eventOutcome = <eventOutcome> </p>

''Examples:''

{|
|''file_validity''
|''microservice_output''
|''am_action''
|''eventOutcome''
|-
|valid
|"Completed successfully"
|Archivematica continues processing
|pass
|-
|not valid
|"Failed"
|Archivematica continues processing
|fail
|}

<p>

'''Process:''' </p>

1. In the Archivematica FPR, a new command will be written under Validation to run Mediaconch.
Possible command: <pre> mediaconch -mc -iv 4 -fx nameOfFile.mkv </pre>

Where iv defines the verbosity of the output and -fx prints the output in xml. Note that presently MediaConch has the same verbosity for anything under 5. More granularity of verbosity will be available later in the development project.

Outcome of above:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<MediaConch xmlns="https://mediaarea.net/mediaconch" version="0.1">
<media ref="testFiles/MakeUp-029f3287-4f5d-4399-8330-8b31f2b749c8.mkv">
<implementationChecks>
<name>MediaConch EBML Implementation Checker</name>
</implementationChecks>
<implementationChecks>
<name>MediaConch FFV1 Implementation Checker</name>
<check icid="FFV1-VALID-VERSION-VALUE" version="1">
<context field="Valid Values" value="1 3"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 version"/>
</test>
</check>
<check icid="FFV1-VALID-CODERTYPE-VALUE" version="1">
<context field="Valid Values" value="0 1 2"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 coder type"/>
</test>
</check>
<check icid="FFV1-VALID-COLORSPACETYPE-VALUE" version="1">
<context field="Valid Values" value="0 1"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 colorspace type"/>
</test>
</check>
</implementationChecks>
</media>
</MediaConch>
</pre>

2. In the Archivematica FPR, a new validation rule is written to use the above command to validate any original file in mkv format.

3. The command runs in the Validation micro-service. If the outcome of any of the checks is fail, the job fails and indicates failure by colour. In the case of either success or failure, the process should proceed to the next micro-service (Examine contents). The user has the opportunity to reject the Transfer in the Create SIP from Transfer micro-service, a decision they can make based on the outcome of Validation.

4. Archivematica writes a PREMIS event for validation to the AIP METS file.

'''Diagram'''

[[File:Pre_norm_checker.png]]

'''Mockups'''

PREMIS:

<pre>
<mets:digiprovMD ID="digiprovMD_15">
<mets:mdWrap MDTYPE="PREMIS:EVENT">
<mets:xmlData>
<premis:event xmlns:premis="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>ceb56635-f114-46d7-af3d-5766c8f6c75f</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>validation</premis:eventType>
<premis:eventDateTime>2016-05-25T20:55:38</premis:eventDateTime>
<premis:eventDetail>program="MediaConch"; version="16.03"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>pass</premis:eventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>version="1" outcome="pass"; coder type="1" outcome="pass"; colorspace="1" outcome="pass"</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
</premis:eventOutcomeInformation>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>Archivematica user pk</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica-1.4.1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>repository code</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>ORG</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</premis:event>
</mets:xmlData>
</mets:mdWrap>
</mets:digiprovMD>
</pre>

===Post-normalization===

'''User Story:''' As a digital preservationist, I need to verify that the preservation copies that Archivematica has created through normalization are in conformance with the FFV1 specification so that I know I am placing preservation copies in storage which are reliable.

'''Specification'''

''Feature:''Post-normalization conformance check

Archivists need to verify that the preservation and access copies that Archivematica has created through normalization are in conformance with the .mkv specification so that they know they are placing derivatives in storage/repositories which are reliable.

''Background:'' file has been normalized to .mkv

''Scenario outline:'' Jamie wants to confirm that normalization to .mkv is successful </p>
''When:'' file validation is run </p>
''And:'' the file is <file_validity> .mkv </p>
''Then:'' validation microservice output is <microservice_output> </p>
''And:'' Archivematica outputs a <validation_result> validation in the normalization report </p>

''Examples:''

{|
|''file_validity''
|''microservice_output''
|''validation_result''
|-
|valid
|"Completed successfully"
|successful
|-
|not valid
|"Failed"
|failed
|}

<p>

''Scenario outline:'' Jamie wants to approve normalization of a .mkv for preservation </p>
''Given:'' normalized file is <file_validity> .mkv </p>
''When:'' Jamie approved normalization </p>
''Then:'' A PREMIS validation event is generated with eventOutcome = <eventOutcome> </p>
''And:'' Archivematica continues processing </p>

''Examples:''

{|
|''file_validity''
|''eventOutcome''
|-
|valid
|pass
|-
|not valid
|fail
|}

<p>

'''Process:''' </p>

1. In the Archivematica FPR, a new command will be written under Verfication to run MediaConch.
Possible command: <pre> mediaconch -mc -iv 4 -fx nameOfFile.mkv </pre>

Where iv defines the verbosity of the output and -fx prints the output in xml. Note that presently MediaConch has the same verbosity for anything under 5. More granularity of verbosity will be available later in the development project.

Outcome of above:

<pre>
<?xml version="1.0" encoding="UTF-8"?>
<MediaConch xmlns="https://mediaarea.net/mediaconch" version="0.1">
<media ref="testFiles/MakeUp-029f3287-4f5d-4399-8330-8b31f2b749c8.mkv">
<implementationChecks>
<name>MediaConch EBML Implementation Checker</name>
</implementationChecks>
<implementationChecks>
<name>MediaConch FFV1 Implementation Checker</name>
<check icid="FFV1-VALID-VERSION-VALUE" version="1">
<context field="Valid Values" value="1 3"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 version"/>
</test>
</check>
<check icid="FFV1-VALID-CODERTYPE-VALUE" version="1">
<context field="Valid Values" value="0 1 2"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 coder type"/>
</test>
</check>
<check icid="FFV1-VALID-COLORSPACETYPE-VALUE" version="1">
<context field="Valid Values" value="0 1"/>
<test outcome="pass">
<value offset="@offset" name="FFV1 colorspace type"/>
</test>
</check>
</implementationChecks>
</media>
</MediaConch>
</pre>

2. In the Archivematica FPR, a rule is written for validation to run the above command when an mkv file has been created for preservation or access.

3. A new job in the Normalization micro-service runs the command (must be before Job: Move to approve normalization directory). If the outcome of any of the checks is fail, the job fails and indicates failure by colour. The normalization micro-service should proceed to the Approval step. Failures could also be indicated in the Approve normalization report.

4. Archivematica writes a PREMIS event in the METS to record the outcomes of validation.

'''Diagram'''

[[File:Post_norm_checker.png]]

'''Mockups'''

Normalization report:

[[File:conformance_norm_report.png|900 px]]

Columns for checking validity of both preservation and access derivatives have been added to the current Archivematica normalization report.

PREMIS:

<pre>
<mets:digiprovMD ID="digiprovMD_428512" CREATED="2016-04-05T18:55:56">
<mets:mdWrap MDTYPE="PREMIS:EVENT">
<mets:xmlData>
<premis:event xmlns:premis="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<premis:eventIdentifier>
<premis:eventIdentifierType>UUID</premis:eventIdentifierType>
<premis:eventIdentifierValue>e2a85831-8dcd-43be-ba3d-9b7cae0f42b5</premis:eventIdentifierValue>
</premis:eventIdentifier>
<premis:eventType>verification</premis:eventType>
<premis:eventDateTime>2016-05-19T18:03:03</premis:eventDateTime>
<premis:eventDetail>program="MediaConch"; version="16.03"</premis:eventDetail>
<premis:eventOutcomeInformation>
<premis:eventOutcome>pass</premis:eventOutcome>
<premis:eventOutcomeDetail>
<premis:eventOutcomeDetailNote>version="1" outcome="pass"; coder type="1" outcome="pass"; colorspace="1" outcome="pass"</premis:eventOutcomeDetailNote>
</premis:eventOutcomeDetail>
</premis:eventOutcomeInformation>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>Archivematica user pk</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>1</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>preservation system</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Archivematica-1.7</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifier>
<premis:linkingAgentIdentifierType>repository code</premis:linkingAgentIdentifierType>
<premis:linkingAgentIdentifierValue>Example Repository</premis:linkingAgentIdentifierValue>
</premis:linkingAgentIdentifier>
</premis:event>
</mets:xmlData>
</mets:mdWrap>
</mets:digiprovMD>
</pre>

==Policy checking==

'''User stories'''

* As a digital preservationist, I want to check vendor-digitized videos against a policy to ensure that the agreed-upon specifications were followed during digitization.
* As a digital preservationist, I want to ensure that Archivematica is normalizing for access according to my policy so that the access files are optimized for my institution's access system.
* As a digital preservationist, I want to ensure that Archivematica is normalizing for preservation according to my policy so that preservation files meet my institution's specifications.

'''Specification'''

''Feature:'' Ingest policy check </p>
Archivists want to check that the files that are defined for access or preservation conform to a pre-defined policy/policies. The access and preservation derivatives may have been created by Archivematica through normalization or may have been predefined by the user prior to transfer.

''Scenario Outline:'' Jamie has preservation derivatives and they need to know whether they conform to their preservation policy</p>
''Given:'' a policy has been provided for preservation</p>
''And:'' <does_file_conform> to the policy</p>
''When:'' file verification is run</p>
''Then:'' verification microservice output is <microservice_output></p>
''And:'' a PREMIS verification event is generated with eventOutcome = <eventOutcome></p>
''And:'' Archivematica outputs a <verification_result> verification in the normalization report</p>

''Examples:''

{|
|''does_file_conform''
|''microservice_output''
|''eventOutcome''
|''validation_result''
|-
|conforms
|pass
|pass
|successful
|-
|does not conform
|fail
|fail
|failed
|}

<p>

Manual normalization requirements and testing procedures

2017-03-23T19:37:17Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Manual normalization requirements and testing procedures

[[Category:Feature requirements]]

The following procedures describe functionality available in the 0.10-beta release of Archivematica.

===Procedures for manual normalization===

There are two manual normalization workflow options. First, the user can do some normalization before bringing any objects into Archivematica. Second, the user can process objects through to Ingest and then choose to do manual normalization on some of the objects. Below are the workflow steps to test each of these options.

These requirements and workflows were developed as a result of the following related issues: [https://projects.artefactual.com/issues/4656 #4656], [https://projects.artefactual.com/issues/4511 #4511]. [https://projects.artefactual.com/issues/4510 #4510]

==Option 1: Manual normalization prior to transfer into Archivematica==

The manual normalization feature allows the user to prepare transfers that already contain manually normalized files. The user adds a /manualNormalization folder and its subdirectories to the transfer prior to processing. Archivematica recognizes that this folder already exists and will not add one during processing. This workflow requires that there must be a one-to-one relationship between the transfer and the SIP.

To test

1. Clone the newest sampledata to your local dev environment

You should find the sample to test in the following location:
/archivematica-sampledata-2013-xx-xx-xx-xx/TestTransfers/manNormAtBeginning

Should you wish to create your own tests, you must create transfers with a structure that mimics the one in this sample. That is:

/topDirectory

: /manualNormalization

:: /access

::: /example.bmp

:: /preservation

::: /example.prk
:: /file.tif
:: /file2.doc
:: /file3.docx

'''Where'''
*the /topDirectory is named whatever you want (you can still change this when you upload the transfer to the dashboard)
*/manualNormalization contains two directories, /access and /preservation which include in them the files you've already normalized for access and/or preservation prior to transfer
*other objects in the /topDirectory are to be treated as regular transfers

2. Upload the transfer to the dashboard from your source directories in the Transfer Tab.

For instructions detailing how to set up source directories, see the Administrator's manual here: [[Administrator_manual_0.9#Transfer_source_directories]]

3. Continue processing as you would a regular SIP. For instructions, see the online User manual here: [[User_manual_0.9]] through to the Ingest tab.

4. In the Ingest tab, at normalization step, choose whether to normalize for access, preservation, or both. If you choose to review normalization results prior to accepting, you will find your /manualNormalization folder and its contents in the objects directory of your transfer, with it's /access and/or /preservation folders and their content intact. If you're happy with the results here, approve normalization.

Continue processing according to instructions in the User manual. Stop when you reach the Store AIP micro-service.

5. At Store AIP, you have the option to review your AIP. If you download the AIP from the review tab by simply clicking on the file with the 7z extension, you can download, unzip and view your AIP in your file browser. Your AIP will consist of a top level folder with the name you assigned plus a UUID. In that folder, you'll find a /data folder which contains your objects for preservation, logs, thumbnails and the METS xml. That data folder will contain your originals and normalized files, and the METS will include PREMIS events with "manual normalization" noted in the eventDetail of the appropriate normalization events. A discussion of the overall AIP structure can be found here: [[AIP_structure]] (Note that this was the 0.9-beta description and will change a bit for the next release: see related task at [https://projects.artefactual.com/issues/4597 #4597])

Additionally, you might note that the manualNormalization folder and subfolders used during the manual normalization workflow have been deleted once normalization was approved rather than being packaged into the AIP.

6. If you are satisfied with everything here, continue to store your AIP and/or upload your DIP as per the User manual instructions.

*This process describes the basic test; however, we recommend putting the system through it's paces. Test with/without extra files in the transfer outside of the manualNormalization folder; test with only an access folder or preservation folder in the /manualNormalization folder; get creative!

==Option 2: Manual normalization during Ingest==

The manual normalization feature allows users to choose to do manual normalization at the normalization micro-service in the Ingest tab of the dashboard. The user does not have to manually normalize all of the objects for preservation and/or access; any files not manually normalized will be normalized according the regular Archivematica workflow.

To test

1. Start with any regular transfer in the Transfer tab of the dashboard. Follow the instructions in the user manual until you get to the normalization micro-service in the Ingest tab.

2. Select "Normalize manually" from the dropdown menu.

3. In your file browser, open /archivematica/src/MCPServer/sharedDirectoryStructure/watchedDirectories/approveNormalization/preservation/yourTransfer123abc/objects/

4. Choose which files you'd like to normalize for preservation and/or access

5. Do your normalization. Or, for testing purposees, you can simply copy and change the extension(s) of some of the objects in the transfer.

Once you've done all the normalization for preservation and access, place your preservation copies in the /manualNormalization/preservation folder and your access copies in the /manualNormalization/access folder. Note that any objects you haven't normalized manually will not be normalized. However, if you do not include anything in the /access folder, no DIP will be generated.

6. Return to the dashboard and approve normalization.

7. Continue processing according to instructions in the User manual. Stop when you reach the Store AIP micro-service.

8. At the Store AIP micro-service, you have the option to review your AIP. If you download the AIP from the review tab by simply clicking on the file with the 7z extension, you can download, unzip and view your AIP in your file browser. Your AIP will consist of a top level folder with the name you assigned plus a UUID. In that folder, you'll find a /data folder which contains your objects for preservation, logs, thumbnails and the METS xml. That data folder will contain your originals and normalized files, and the METS will include PREMIS events with "manual normalization" noted in the eventDetail of the appropriate normalization events. A discussion of the overall AIP structure can be found here: https://www.archivematica.org/wiki/AIP_structure (Note that this was the 0.9-beta description and will change a bit for the next release: see related task at [https://projects.artefactual.com/issues/4597 #4597])

Additionally, you might note that the manualNormalization folder and subfolders used during the manual normalization workflow have been deleted once normalization was approved rather than being packaged into the AIP.

If you are satisfied with everything here, continue to store your AIP and/or upload your DIP as per the User manual instructions.

*This process describes the basic test; however, we recommend putting the system through it's paces. Test with/without access files in the manualNormalization folder; test with no objects outside of the /manualNormalization folder; get creative!

LOCKSS Integration

2017-03-23T19:36:55Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > LOCKSS Integration

[[Category:Feature requirements]]

==Status==

'''Sponsored''': SFU Library<br>
'''Development Status''': Complete<br>
'''Public Release''': September 2014

==Overview==

LOCKSS (Lots of Copies Keeps Stuff Safe) http://www.lockss.org/ is treated by Archivematica as a storage sub-system. The Archivematica Storage Service can be configured to store AIPs in a LOCKSS network. The procedure for storing AIPs follows the SWORD v2 protocol: http://swordapp.org/sword-v2/ .

The Archivematica Storage Service contains an implementation of a SWORD client, which communicates with a SWORD server. The implementation of the SWORD Server is provided by LOCKSS-O-Matic https://github.com/mjordan/lockss-o-matic. LOCKSS-O-Matic is responsible for communicating with the Private LOCKSS Network (PLN)

===Basic Workflow===

* Digital Objects are processed and packaged into AIPs in an Archivematica pipeline
* Archivematica pipeline sends the final AIP to the Storage Service using the 'Store AIP' micro-service
* Storage Service keeps a local copy of the AIP
* Storage Service POSTs the list of files (e.g., a .7z file or parts of one created with split) that make up the AIP to LOCKSS-O-Matic. This list is in the form of an Atom document.
* LOCKSS-O-Matic parses the list of files in the Atom document and registers them in its database.
* LOCKSS-O-Matic edits the configuration files of all the LOCKSS boxes in the Private LOCKSS Network
* LOCKSS boxes harvest the AIP files from the Storage Service
* Storage Service polls LOCKSS-O-Matic (GETs the State document) to determine when the AIP has been stored in LOCKSS
* If/When the Storage Service wants to remove its local copy of the AIP, it first POSTs a metadata update to LOCKSS-O-Matic
* LOCKSS-O-Matic then updates the configuration files in LOCKSS boxes so they will stop harvesting the AIP

==Technical Details==

This document outlines a minimalist SWORD API where LOCKSS-O-Matic is the server, and the Archivematica Storage Service is the client. @todos and questions appear in  throughout.

In the examples below, Archivematica is at http://archivematica.example.org and LOCKSS-O-Matic is at http://lockssomatic.example.org. The content file being managed by the SWORD deposit is an Archivematica AIP with the UUID dd3e3247-8466-4f2a-bb32-22a210cfce60. LOCKSS-O-Matic is managing a small Private LOCKSS Network containing two boxes, http://lockss1.example.org and http://lockss2.example.org.

===Service Document===
Archivematica issues a GET (with an HTTP header of ‘On-Behalf-Of’ with the value of the Archivematica instance’s content provider ID in the target LOM instance) to the SD-IRI (http://lockssomatic.example.org/api/sword/2.0/sd-iri). LOCKSS-O-Matic responds with a Service Document like:

''On-Behalf-Of: 12''

<pre>
<service xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:sword="http://purl.org/net/sword/terms/"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:lom="http://lockssomatic.info/SWORD2"
xmlns="http://www.w3.org/2007/app">

<sword:version>2.0</sword:version>

<sword:maxUploadSize>102400</sword:maxUploadSize>


<lom:uploadChecksumType>md5</lom:uploadChecksumType>

<workspace>
<atom:title>LOCKSS-O-Matic at Simon Fraser University</atom:title>

<collection href="http://lockssomatic.example.org/api/sword/2.0/col-iri/12">
<atom:title>SFU Archivematica content provider</atom:title>
<accept>application/atom_xml;type=entry</accept>
<sword:mediation>true</sword:mediation>
</collection>
</workspace>
</service>
</pre>

===Creating a Resource with an Atom Entry===
Archivematica issues a POST to the Col-IRI, ensuring that the <id> element in the Atom entry contains the UUID of the AIP, and that the URLs to the files to be preserved are listed in lom:content elements:

<pre>
<entry xmlns="http://www.w3.org/2005/Atom"
xmlns:dcterms="http://purl.org/dc/terms/"
xmlns:lom="http://lockssomatic.info/SWORD2">
<title>Some AIP</title>
<id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</id>
<updated>2013-10-07T17:17:08Z</updated>
<author><name>Name of PREMIS agent owning AIP</name></author>
<summary type="text">The AIP’s dc:description if it has one. If not, use a generic summary.</summary>

<lom:content size="102400" checksumType="md5" checksumValue="bd4a9b642562547754086de2dab26b7d">
http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-22a210cfce60.001
</lom:content>
<lom:content size="46899" checksumType="md5" checksumValue="226190d94b21d1b0c7b1a42d855e419d">
http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-22a210cfce60.002
</lom:content>
</entry>
</pre>

LOCKSS-O-Matic responds with a 201 Created and the Location header with the entry’s Edit-IRI, and the deposit receipt.

===Deposit Receipt===
Contains the Cont-IRI, the EM-IRI, and the State-IRI.

<pre>
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:sword="http://purl.org/net/sword/">
<sword:treatment>Stored in LOCKSS via LOCKSS-o-matic</sword:treatment>
<content type="application/x-7z-compressed" src="http://lockssomatic.example.org/api/sword/2.0/cont-iri/12/dd3e3247-8466-4f2a-bb32-22a210cfce60" />


<link rel="edit-media" href="http://lockssomatic.example.org/api/sword/2.0/cont-iri/12/dd3e3247-8466-4f2a-bb32-22a210cfce60" />



<link rel="http://purl.org/net/sword/terms/add" href="http://lockssomatic.example.org/api/sword/2.0/cont-iri/12/dd3e3247-8466-4f2a-bb32-22a210cfce60/edit" />

<link rel="edit" href="http://lockssomatic.example.org/api/sword/2.0/cont-iri/12/dd3e3247-8466-4f2a-bb32-22a210cfce60/edit" />

<link rel="http://purl.org/net/sword/terms/statement" type="application/atom+xml;type=feed"
href="http://lockssomatic.example.org/api/sword/2.0/cont-IRI/12/dd3e3247-8466-4f2a-bb32-22a210cfce60/state" />
</entry>
</pre>

===SWORD Statement===
State-IRI is defined in the deposit receipt. GET requests to the State-IRI for a resource will return an Atom feed as described below.

These state terms apply to the Allowed state term and message values, using the namespace http://lockssomatic.info/SWORD2, are:

{| class="wikitable"
|-
! Term
! Message (one of)
|-
| failed
| Content cannot be harvested by LOCKSS.
|-
| disagreement
| LOCKSS network is not in agreement on content checksums.
|-
| agreement
| LOCKSS network agrees internally on content checksums.
|}

If LOCKSS-O-Matic reports ‘agreement’, Archivematica may delete the AIP from local storage.

Sample SWORD statement serialized as an Atom feed:
<pre>
<atom:feed xmlns:sword="http://purl.org/net/sword/terms/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:lom="http://lockssomatic.info/SWORD2">
<atom:entry>
<atom:category scheme="http://purl.org/net/sword/terms/" term="http://purl.org/net/sword/terms/originalDeposit" label="Orignal Deposit"/>


<lom:content id="http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-22a210cfce60.001">
<lom:serverlist>
<lom:server id="1" state="agreement" src="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001" checksumType="md5" checksumValue="bd4a9b642562547754086de2dab26b7d" />
<lom:server id="2" state="failed" src="http://lockss2.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001" checksumType="md5" checksumValue="bd4a9b642562547754086de2dab26b7d" />
</lom:serverlist>
</lom:content>

<lom:content id="http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-22a210cfce60.002">
<lom:serverlist>
<lom:server id="1" state="agreement" src="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce602.002" checksumType="md5" checksumValue="226190d94b21d1b0c7b1a42d855e419d" />
<lom:server id="2" state="disagreement" src="http://lockss2.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce602.002" checksumType="md5" checksumValue="226190d94b21d1b0c7b1a42d855e419d" />
</lom:serverlist>
</lom:content>

</atom:entry>
</atom:feed>
</pre>

===Updating Metadata===
Archivematica informs LOCKSS-O-Matic that it wants to delete the local copy of a file by PUTing to the SE-IRI an Atom document containing lom:content elements with a recrawl="false" attribute for each file. LOCKSS-O-Matic responds with a 202 Accepted and updates the LOCKSS configuration files to indicate that the AU the file is in should not be reharvested. After the configuration files have been updated, LOCKSS-O-Matic will respond with a 200 OK to the same request.

PUT SE-IRI
<pre>
<?xml version="1.0"?>
<entry xmlns="http://www.w3.org/2005/Atom" xmlns:dcterms="http://purl.org/dc/terms/" xmlns:lom="http://lockssomatic.info/SWORD2">
<id>urn:uuid:dd3e3247-8466-4f2a-bb32-22a210cfce60</id>
<lom:content recrawl="false">http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-
+22a210cfce60.001</lom:content>
<lom:content recrawl="false">http://archivematica.example.org/archival-storage/download/aip/dd3e3247-8466-4f2a-bb32-22a210cfce60.002</lom:content>
</entry>
</pre>
LOM can return:
*- HTTP 202 (Accepted) meaning LOM is updating the LOCKSS config files saying not to harvest this, but it is not done yet.
*- HTTP 200 (OK) meaning all config updates are complete
*- HTTP 204 (No Content) if there is no matching AIP
*- HTTP 409 (Conflict) There are files in the LOCKSS Archival Unit (AU) that do not have ‘recrawl=false’.

“Once the server has processed the request it MUST return an HTTP status code 200 (OK) or 204 (No Content), or an appropriate error code. “

===Comments/questions===

;'''Can an AIP be deleted from LOCKSS-O-Matic? How?
:Not really, since LOCKSS doesn’t let you delete content, not directly anyway (it can be done on the boxes’ command lines). In LOM we could have a flag that specifies that content should not be harvested into the LOCKSS network. In the Archivematica dashboard, an AIP deletion request can be made. This deletion request goes to the Storage Service, which would tell LOM to stop harvesting the AIP, and then delete the local copy. The copies in the PLN would not be deleted, and would have to be manually deleted by a LOCKSS administrator.

;'''Can an AIP be retrieved from LOCKSS via the Archivematica Dashboard?
:Yes - with the following caveat. All AIP's stored in the Storage Service are searchable via the Archival Storage tab in the dashboard. The Storage Service keeps a local copy of AIP's that are stored in LOCKSS, so when a user clicks on an AIP in the Archival Storage tab, that local copy will be delivered. If the local copy has been deleted (which will eventually happen, exactly when is based on Storage Service configuration settings) then the dashboard user will get a 'local copy not available, contact your storage administrator' message. From the Storage Service, it is possible for a storage administrator to click a button to download a copy of the AIP directly from one of the LOCKSS boxes, to the storage service. Once that is done, the AIP will then be retrievable from a dashboard.

;'''Is there any sort of authentication/authorization on the API?
: This should support HTTP Basic Auth (which is a “SHOULD” in SWORD). At a minimum IP whitelisting.
LOCKSS itself relies on both whitelisting and HTTP Basic

;'''Voting
:Each LOCKSS box initiates a ‘poll’ at random intervals bounded by two box-specific configuration settings:

<pre>
<property name="contentpoll.min" value="30m" />
<property name="contentpoll.max" value="2d" />
</pre>

;'''How long does it take for 100% agreement on an Archival Unit (AU)?
:Because each box recrawls AUs (looking for changes/additions) on its own schedule, AUs that are constantly changing likely won't achieve 100% agreement until they stop changing. If all the copies of the AU are the same when it settles down, polling will reach 100% agreement once all boxes have called and completed a poll. If the copies aren't identical (usually because of random crawl errors) it may take a couple cycles of polling to resolve the differences.

== Sample pointer files ==

The pointer file will be updated to track the current state of files in LOCKSS (more details at #6212). Here are some sample pointer files representing different states of the package stored in LOCKSS.

=== Base pointer file ===

The pointer file as it arrives in the storage service, unmodified for LOCKSS.

<pre>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<metsHdr CREATEDATE="2014-01-14T14:35:15"/>
<amdSec ID="amdSec_1">
<techMD ID="techMD_1">
<mdWrap MDTYPE="PREMIS:OBJECT">
<xmlData>
<object version="2.2" xmlns="info:lc/xmlns/premis-v2" xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd">
<objectIdentifier>
<objectIdentifierType>UUID</objectIdentifierType>
<objectIdentifierValue>e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6</objectIdentifierValue>
</objectIdentifier>
<objectCharacteristics>
<compositionLevel>0</compositionLevel>
<fixity>
<messageDigestAlgorithm>sha256</messageDigestAlgorithm>
<messageDigest>7d4ede07b9288960d82c2ee146f6b4368d7dd065a99fae3523e3aa0d932bdb66</messageDigest>
</fixity>
<size>190989</size>
<format>
<formatDesignation>
<formatName>7Zip format</formatName>
<formatVersion/>
</formatDesignation>
<formatRegistry>
<formatRegistryName>PRONOM</formatRegistryName>
<formatRegistryKey>fmt/484</formatRegistryKey>
</formatRegistry>
</format>
<creatingApplication>
<creatingApplicationName>7-Zip</creatingApplicationName>
<creatingApplicationVersion>9.20</creatingApplicationVersion>
<dateCreatedByApplication>2014-01-14T14:35:15</dateCreatedByApplication>
</creatingApplication>
</objectCharacteristics>
</object>
</xmlData>
</mdWrap>
</techMD>
<digiprovMD ID="digiprovMD_1">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>7b16bct8-389x-4c0e-815c-98df287b234k</eventIdentifierValue>
</eventIdentifier>
<eventType>compression</eventType>
<eventDateTime>2014-01-14T22:31:22</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>preservation system</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_2">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>preservation system</agentIdentifierType>
<agentIdentifierValue>Archivematica-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
</amdSec>
<fileSec>
<fileGrp USE="Archival Information Package">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6">
<FLocat OTHERLOCTYPE="SYSTEM" LOCTYPE="OTHER" xlink:href="/filepath/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<transformFile TRANSFORMALGORITHM="bzip2" TRANSFORMORDER="1" TRANSFORMTYPE="decompression"/>
</file>
</fileGrp>
</fileSec>
<structMap TYPE="physical">
<div TYPE="Archival Information Package" ADMID="amdSec_1">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6"/>
</div>
</structMap>
</mets>
</pre>

=== Package unsplit, local copy kept ===

The package was smaller than the max file size and therefore unsplit. The package is fully stored, and the local copy was retained.

<pre>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<metsHdr CREATEDATE="2014-01-14T14:35:15"/>
<amdSec ID="amdSec_1">
<techMD ID="techMD_1">
<mdWrap MDTYPE="PREMIS:OBJECT">
<xmlData>
<object version="2.2" xmlns="info:lc/xmlns/premis-v2" xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd">
<objectIdentifier>
<objectIdentifierType>UUID</objectIdentifierType>
<objectIdentifierValue>e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6</objectIdentifierValue>
</objectIdentifier>
<objectCharacteristics>
<compositionLevel>1</compositionLevel>
<fixity>
<messageDigestAlgorithm>sha256</messageDigestAlgorithm>
<messageDigest>7d4ede07b9288960d82c2ee146f6b4368d7dd065a99fae3523e3aa0d932bdb66</messageDigest>
</fixity>
<size>190989</size>
<format>
<formatDesignation>
<formatName>7Zip format</formatName>
<formatVersion/>
</formatDesignation>
<formatRegistry>
<formatRegistryName>PRONOM</formatRegistryName>
<formatRegistryKey>fmt/484</formatRegistryKey>
</formatRegistry>
</format>
<creatingApplication>
<creatingApplicationName>7-Zip</creatingApplicationName>
<creatingApplicationVersion>9.20</creatingApplicationVersion>
<dateCreatedByApplication>2014-01-14T14:35:15</dateCreatedByApplication>
</creatingApplication>
</objectCharacteristics>
</object>
</xmlData>
</mdWrap>
</techMD>
<digiprovMD ID="digiprovMD_1">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>7b16bct8-389x-4c0e-815c-98df287b234k</eventIdentifierValue>
</eventIdentifier>
<eventType>compression</eventType>
<eventDateTime>2014-01-14T22:31:22</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>preservation system</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_2">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>preservation system</agentIdentifierType>
<agentIdentifierValue>Archivematica-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
</amdSec>
<fileSec>
<fileGrp USE="Archival Information Package">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6">
<FLocat OTHERLOCTYPE="SYSTEM" LOCTYPE="OTHER" xlink:href="/filepath/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<transformFile TRANSFORMALGORITHM="bzip2" TRANSFORMORDER="1" TRANSFORMTYPE="decompression"/>
</file>
</fileGrp>
</fileSec>
<structMap TYPE="physical">
<div TYPE="Archival Information Package" ADMID="amdSec_1">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6"/>
</div>
</structMap>
</mets>
</pre>

=== Package unsplit, local copy deleted ===

The package was smaller than the max file size and therefore unsplit. The package is fully stored, and the local copy was deleted.

<pre>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<metsHdr CREATEDATE="2014-01-14T14:35:15"/>
<amdSec ID="amdSec_1">
<techMD ID="techMD_1">
<mdWrap MDTYPE="PREMIS:OBJECT">
<xmlData>
<object version="2.2" xmlns="info:lc/xmlns/premis-v2" xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd">
<objectIdentifier>
<objectIdentifierType>UUID</objectIdentifierType>
<objectIdentifierValue>e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6</objectIdentifierValue>
</objectIdentifier>
<objectCharacteristics>
<compositionLevel>1</compositionLevel>
<fixity>
<messageDigestAlgorithm>sha256</messageDigestAlgorithm>
<messageDigest>7d4ede07b9288960d82c2ee146f6b4368d7dd065a99fae3523e3aa0d932bdb66</messageDigest>
</fixity>
<size>190989</size>
<format>
<formatDesignation>
<formatName>7Zip format</formatName>
<formatVersion/>
</formatDesignation>
<formatRegistry>
<formatRegistryName>PRONOM</formatRegistryName>
<formatRegistryKey>fmt/484</formatRegistryKey>
</formatRegistry>
</format>
<creatingApplication>
<creatingApplicationName>7-Zip</creatingApplicationName>
<creatingApplicationVersion>9.20</creatingApplicationVersion>
<dateCreatedByApplication>2014-01-14T14:35:15</dateCreatedByApplication>
</creatingApplication>
</objectCharacteristics>
</object>
</xmlData>
</mdWrap>
</techMD>
<digiprovMD ID="digiprovMD_1">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>7b16bct8-389x-4c0e-815c-98df287b234k</eventIdentifierValue>
</eventIdentifier>
<eventType>compression</eventType>
<eventDateTime>2014-01-14T22:31:22</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>preservation system</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_2">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>5e16bccb-384a-4c0e-815c-01df286b235b</eventIdentifierValue>
</eventIdentifier>
<eventType>deletion</eventType>
<eventDateTime>2014-01-14T22:33:48</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>AIP deleted from local storage</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>storage service</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica Storage Service-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_3">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>preservation system</agentIdentifierType>
<agentIdentifierValue>Archivematica-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_4">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>storage service</agentIdentifierType>
<agentIdentifierValue>Archivematica Storage Service-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica Storage Service</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
</amdSec>
<fileSec>
<fileGrp USE="Archival Information Package">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6">
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<transformFile TRANSFORMALGORITHM="bzip2" TRANSFORMORDER="1" TRANSFORMTYPE="decompression"/>
</file>
</fileGrp>
</fileSec>
<structMap TYPE="physical">
<div TYPE="Archival Information Package" ADMID="amdSec_1">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6"/>
</div>
</structMap>
</mets>
</pre>

=== Package split, local copy kept ===

The package was larger than the max file size and therefore split into several smaller chunks. The package is fully stored, and the local copy was retained.

<pre>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<metsHdr CREATEDATE="2014-01-14T14:35:15"/>
<amdSec ID="amdSec_1">
<techMD ID="techMD_1">
<mdWrap MDTYPE="PREMIS:OBJECT">
<xmlData>
<object version="2.2" xmlns="info:lc/xmlns/premis-v2" xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd">
<objectIdentifier>
<objectIdentifierType>UUID</objectIdentifierType>
<objectIdentifierValue>e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6</objectIdentifierValue>
</objectIdentifier>
<objectCharacteristics>
<compositionLevel>1</compositionLevel>
<fixity>
<messageDigestAlgorithm>sha256</messageDigestAlgorithm>
<messageDigest>7d4ede07b9288960d82c2ee146f6b4368d7dd065a99fae3523e3aa0d932bdb66</messageDigest>
</fixity>
<size>190989</size>
<format>
<formatDesignation>
<formatName>7Zip format</formatName>
<formatVersion/>
</formatDesignation>
<formatRegistry>
<formatRegistryName>PRONOM</formatRegistryName>
<formatRegistryKey>fmt/484</formatRegistryKey>
</formatRegistry>
</format>
<creatingApplication>
<creatingApplicationName>7-Zip</creatingApplicationName>
<creatingApplicationVersion>9.20</creatingApplicationVersion>
<dateCreatedByApplication>2014-01-14T14:35:15</dateCreatedByApplication>
</creatingApplication>
</objectCharacteristics>
</object>
</xmlData>
</mdWrap>
</techMD>
<digiprovMD ID="digiprovMD_1">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>7b16bct8-389x-4c0e-815c-98df287b234k</eventIdentifierValue>
</eventIdentifier>
<eventType>compression</eventType>
<eventDateTime>2014-01-14T22:31:22</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>preservation system</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_2">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>5e16bccb-384a-4c0e-815c-01df286b235b</eventIdentifierValue>
</eventIdentifier>
<eventType>division</eventType>
<eventDateTime>2014-01-14T22:33:48</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>3 LOCKSS chunks created</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>storage service</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica Storage Service-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_3">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>preservation system</agentIdentifierType>
<agentIdentifierValue>Archivematica-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_4">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>storage service</agentIdentifierType>
<agentIdentifierValue>Archivematica Storage Service-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica Storage Service</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
</amdSec>
<fileSec>
<fileGrp USE="Archival Information Package">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6">
<FLocat OTHERLOCTYPE="SYSTEM" LOCTYPE="OTHER" xlink:href="/filepath/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6.7z"/>
<transformFile TRANSFORMALGORITHM="bzip2" TRANSFORMORDER="1" TRANSFORMTYPE="decompression"/>
</file>
</fileGrp>
<fileGrp USE="LOCKSS chunk">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_001" SIZE="19383838" CHECKSUM="8g4ede07b9288960d82c2ee146f6b4368dqad065a99fae3523e3aa0d932bda87" CHECKSUMTYPE="SHA-256">
<FLocat OTHERLOCTYPE="SYSTEM" LOCTYPE="OTHER" xlink:href="/filepath/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_001"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss2.example.org:8085/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss3.example.org:8080/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
</file>
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_002" SIZE="19273844" CHECKSUM="9h4ede07b9288960d82c2ee146f6b4368hmad065a99fae3523e3aa0d932bda56" CHECKSUMTYPE="SHA-256">
<FLocat OTHERLOCTYPE="SYSTEM" LOCTYPE="OTHER" xlink:href="/filepath/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6/FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_002"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss2.example.org:8085/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss3.example.org:8080/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
</file>
</fileGrp>
</fileSec>
<structMap TYPE="logical">
<div TYPE="Archival Information Package" ADMID="amdSec_1">
<div TYPE="Local copy">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6"/>
</div>
<div TYPE="LOCKSS chunk" ORDER="1">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_001"/>
</div>
<div TYPE="LOCKSS chunk" ORDER="2">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_002"/>
</div>
</div>
</structMap>
</mets>
</pre>

=== Package split, local copy deleted ===

The package was larger than the max file size and therefore split into several smaller chunks. The package is fully stored, and the local copy was deleted.

<pre>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
<metsHdr CREATEDATE="2014-01-14T14:35:15"/>
<amdSec ID="amdSec_1">
<techMD ID="techMD_1">
<mdWrap MDTYPE="PREMIS:OBJECT">
<xmlData>
<object version="2.2" xmlns="info:lc/xmlns/premis-v2" xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd">
<objectIdentifier>
<objectIdentifierType>UUID</objectIdentifierType>
<objectIdentifierValue>e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6</objectIdentifierValue>
</objectIdentifier>
<objectCharacteristics>
<compositionLevel>1</compositionLevel>
<fixity>
<messageDigestAlgorithm>sha256</messageDigestAlgorithm>
<messageDigest>7d4ede07b9288960d82c2ee146f6b4368d7dd065a99fae3523e3aa0d932bdb66</messageDigest>
</fixity>
<size>190989</size>
<format>
<formatDesignation>
<formatName>7Zip format</formatName>
<formatVersion/>
</formatDesignation>
<formatRegistry>
<formatRegistryName>PRONOM</formatRegistryName>
<formatRegistryKey>fmt/484</formatRegistryKey>
</formatRegistry>
</format>
<creatingApplication>
<creatingApplicationName>7-Zip</creatingApplicationName>
<creatingApplicationVersion>9.20</creatingApplicationVersion>
<dateCreatedByApplication>2014-01-14T14:35:15</dateCreatedByApplication>
</creatingApplication>
</objectCharacteristics>
</object>
</xmlData>
</mdWrap>
</techMD>
<digiprovMD ID="digiprovMD_1">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>7b16bct8-389x-4c0e-815c-98df287b234k</eventIdentifierValue>
</eventIdentifier>
<eventType>compression</eventType>
<eventDateTime>2014-01-14T22:31:22</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>preservation system</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_2">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>5e16bccb-384a-4c0e-815c-01df286b235b</eventIdentifierValue>
</eventIdentifier>
<eventType>division</eventType>
<eventDateTime>2014-01-14T22:33:48</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome>3 LOCKSS chunks created</eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote></eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>storage service</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica Storage Service-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_3">
<mdWrap MDTYPE="PREMIS:EVENT">
<xmlData>
<event xmlns="info:lc/xmlns/premis-v2" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2">
<eventIdentifier>
<eventIdentifierType>UUID</eventIdentifierType>
<eventIdentifierValue>5e16bccb-384a-4c0e-815c-01df286b235b</eventIdentifierValue>
</eventIdentifier>
<eventType>deletion</eventType>
<eventDateTime>2014-01-14T22:33:48</eventDateTime>
<eventDetail></eventDetail>
<eventOutcomeInformation>
<eventOutcome></eventOutcome>
<eventOutcomeDetail>
<eventOutcomeDetailNote>AIP deleted from local storage</eventOutcomeDetailNote>
</eventOutcomeDetail>
</eventOutcomeInformation>
<linkingAgentIdentifier>
<linkingAgentIdentifierType>storage service</linkingAgentIdentifierType>
<linkingAgentIdentifierValue>Archivematica Storage Service-1.0</linkingAgentIdentifierValue>
</linkingAgentIdentifier>
</event>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_4">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>preservation system</agentIdentifierType>
<agentIdentifierValue>Archivematica-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
<digiprovMD ID="digiprovMD_5">
<mdWrap MDTYPE="PREMIS:AGENT">
<xmlData>
<agent>
<agentIdentifier>
<agentIdentifierType>storage service</agentIdentifierType>
<agentIdentifierValue>Archivematica Storage Service-1.0</agentIdentifierValue>
</agentIdentifier>
<agentName>Archivematica Storage Service</agentName>
<agentType>software</agentType>
</agent>
</xmlData>
</mdWrap>
</digiprovMD>
</amdSec>
<fileSec>
<fileGrp USE="LOCKSS chunk">
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_001" SIZE="19383838" CHECKSUM="8g4ede07b9288960d82c2ee146f6b4368dqad065a99fae3523e3aa0d932bda87" CHECKSUMTYPE="SHA-256">
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss2.example.org:8085/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss3.example.org:8080/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.001"/>
</file>
<file ID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_002" SIZE="19273844" CHECKSUM="9h4ede07b9288960d82c2ee146f6b4368hmad065a99fae3523e3aa0d932bda56" CHECKSUMTYPE="SHA-256">
<FLocat LOCTYPE="URL" xlink:href="http://lockss1.example.org:8083/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss2.example.org:8085/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
<FLocat LOCTYPE="URL" xlink:href="http://lockss3.example.org:8080/ServeContent?url=http://archivematicastorage.example.com/lockssomatic/dd3e3247-8466-4f2a-bb32-22a210cfce60.002"/>
</file>
</fileGrp>
</fileSec>
<structMap TYPE="logical">
<div TYPE="Archival Information Package" ADMID="amdSec_1">
<div TYPE="LOCKSS chunk" ORDER="1">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_001"/>
</div>
<div TYPE="LOCKSS chunk" ORDER="2">
<fptr FILEID="FLocatTest-e0ff9a0e-a7f5-4bee-b9c2-7e2321dde7a6_002"/>
</div>
</div>
</structMap>
</mets>
</pre>

Format policy registry requirements

2017-03-23T19:36:27Z

Hbecker: Also list as feature requirement page

[[Documentation]] > [[Requirements]] > Format policy registry requirements

The Format Policy Registry (FPR) contains user-configurable scripts for file identification, characterization, extraction, normalization and other preservation actions that differ depending on file format. It also contains a list of formats recognized by Archivematica, and the relationship between those formats and the scripts.

== Overview ==

* The Archivematica project team is working on a better way to manage format policies for preservation events such as normalization, transcription, extraction, characterization and format identification.
* A format policy consists of the business rules and tool commands for preservation events based on format.
* The Format Policy Registry lists all of Archivematica's default format policy rules.
* Currently, users can download updates from the FPR server and replace default rules with your own local policies.
* Future funding is sought for further FPR enhancements to include statistical information about the default and custom format policies adopted and implemented by Archivematica users and the ability to upload local policies to the FPR server.
* One of the primary goals of the FPR is to eventually aggregate empirical information about institutional format policies to better identify community best practices. The FPR could provide a practical, community-based approach to OAIS preservation and access planning, allowing the Archivematica community of users to monitor and evaluate format policies as they are adopted, adapted and supplemented by real-world practitioners. The FPR APIs would be designed to share this information with the Archivematica user base as well with other interested communities and projects.
* The FPR server is hosted at fpr.archivematica.org. Sponsorship is actively being sought to develop a front-facing website at this server location. Currently, the local copy of the FPR server in the Archivematica dashboard's preservation planning tab is the only interface for the user. Artefactual manages the server from the back end. With further sponsorship, the server site would be a place to compare institutional format policies and their success rates over time.

== Description ==

* The Archivematica project team created the FPR after having recognized the need for a better way to manage preservation plans, i.e. business rules and tool commands for format-based preservation events. Since these are either implemented or altered by the institution running an Archivematica instance, these rules are referred to as policies. Format policies will change as community standards, practices and tools evolve. A format policy indicates the actions, tools and settings to apply to a file of a particular file format (e.g. conversion to preservation format, conversion to access format).

* Prior to the FPR, the Archivematica project has managed this information on the [[Media_type_preservation_plans|archivematica.org/preservation]] wiki page. These format policies were all researched as a result of related development partnerships with samples provided by funding partners. Since the FPR was first released in late 2013, all new format policies were developed in partnership with funding institutions and any new rules were included in the public FPR server, hosted by Artefactual Systems, Inc.

* The Format Policy Registry (FPR) manages this information in a structured format (SQL/JSON).
** With additional sponsored development, APIs with other serializations could be added (e.g. XML, RDF)

* The FPR includes updates from PRONOM, which are manually applied by Artefactual Systems. With funding, we could enhance and/or automate this PRONOM interface and add interfaces with other registries like UDFR and linked data registries. A web of interfaces is the best way to monitor and evaluate community-wide best practices.

* The FPR stores structured information about:
**Format identification (FIDO based on PRONOM IDs or file extension, with more tools to be added in the future)
**Normalization format policies for preservation and access. These policies identify preferred preservation and access formats by media type. The default choice of access formats is based on the ubiquity of viewers for the file format. Archivematica's default preservation formats are all open standards; additionally, the choice of preservation format is based on community best practices, availability of open-source normalization tools, and an analysis of the significant characteristics for each media type.
**Characterization (default is FITS for most formats and MediaInfo for some audiovisual formats)
**Transcription (default for OCR using Tesseract)
**Extraction - tools and commands for extracting packages and forensic disk images

* Archivematica default format policies can all be changed or enhanced by individual Archivematica implementers.

=FPR development=

== Use Cases ==
* Alternate preservation event tool than Archivematica default
* Alternate default preservation event outcome format than Archivematica default
* Alternate specification event result
* Disable default in Archivematica
* Add new format policy
* Add new tool
* Add new format
* Run format policy tools and commands on local digital acquisitions
* Proposed: Add format policies to other open source system workflows
* Proposed: Allow other open source system users to submit proposed format policies

==1.2==

*Updated PRONOM data for FIDO
*Made one Tools section rather than having tools in each event space, except for Identification tools
*Added Transcription, Extraction, Characterization and Verification event sections
*Ability to update from server in local dashboard's Preservation Planning tab

==1.0==

* Ability to add/edit tools, rules, commands and formats

==0.10-beta==

*Ability to view, add/edit local format policies
** FPR local is for superusers (preservation planning tab)
* Dashboard FPR captures usage statistics
* Central server at fpr.archivematica.org
* Ability to download most current Archivematica default format policies from fpr.archivematica.org on first installation

==Wishlist==

* Front-facing server website
**authenticated, web-based access for Artefactual and super-user updates and maintenance
* Ability to submit local changes to the central FPR server
** fpr.archivematica.org will mirror local implementation, except user won't be able to apply changes, only submit them for review and website could include the institution that submitted the policy or indicate whether it is a default Archivematica policy
** users would be able to apply changes locally and submit them for review to fpr
** nice to have: ability to add links to documentation about reasoning behind local institutional policy selection
** accepted changes would result in a new FPR server entry
* Ability to download select new Archivematica default FPR policies as well as those from other institutions from the central FPR server to local Archivematica installations
** ideally a single click download to allow for local implementation from the server or from a list viewable after selecting Update from the Preservation Planning tab of the local Dashboard
* Ability to disable a format policy in the dashboard
* Update Archivematica without losing local FPR configuration and added policies
* Integrate with PRONOM to automate PUID updates
* Integrate with other registries
* Provide a read-only RESTful Web API for accessing policies in JSON format. This is only partially implemented in 1.2, intended only to be used internally (between dashboard and FPR server) and only provides access to the full set of policies. Future development would ideally include the ability to access a subset of policies via the FPR REST API

=Legacy FPR foundation docs=

[[File:FPR overview Oct 2012.png|border|900px|FPR overview Oct 2012]]

* developer [[Format_policy_registry|notes]]

== Mockups ==

'''Preservation Planning tab in Dashboard - Local FPR'''

[[File:1.0_PreservationPlanningFPR.png|Preservation Planning tab in dashboard]]

------------------------------------------------------------------------------------------------------

* Archivematica format ID column is populated by the "description" from the File ID table in the Archivematica database; It links to the file ID table in the Archivematica database, which is composed of the tool and tool version that identified the format.
* Command description is the "description" from the Command table in the Archivematica database
* Purpose is the same as "classification" in the Archivematica database, but is clearer to the user
* Add new -links to create new (blank) form for Archivematica format ID -or- to the create new (blank) form for the Command
* Copy - links to create new form based on this one (populated with current data) for either the Archivematica format ID -or- Command
* Edit -links to edit page for the Archivematica format ID -or- to the edit page for the Command
* Make default - makes the selected Archivematica format ID format policy or the selected Command format policy default for normalization. See Feature #4503
* Show performance and show command are modeled on the old preservation planning tab in the dashboard, available in the dev version of Archivematica at /preservationplanning/old/
'''Add new Archivematica format ID'''

[[File:1.0_FormatPolicyFormatID.png]]
--------------------------------------------------------------------------------------------------------------
'''Add new Command'''

[[File:1.0_FormatPolicyCommand.png]]

* Added https://projects.artefactual.com/issues/4568 (#4568) to add a Output file format field to this form from the outputFileFormat field in the MCP Commands table
-------------------------------------------------------------------------------------------------------------------
'''Copy Archivematica format ID'''

[[File:1.0_FormatPolicyFormatIDCOPY.png]]
------------------------------------------------------------------------------------------------------------------
'''Copy Command'''

[[File:1.0_FormatPolicyCommandCOPY.png]]

== Data Model ==
Design from summer 2013 of the data structures in version 2 of the API.

[[File:Fpr design-2013-07-25.png|900px]]

Updated design as of March 2017

[[File:Fpr-design-2017-03.png|900px]]

== Workflow ==

== GUI ==

== API ==

== Proposed Changes ==

=== Replace VersionedModel with set of associated rules ===

January 2014

'''Summary:''' Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.

'''Bug:'''#6192

'''Problem:'''
Currently, we track a linear history of rule modifications. A newly modified rule is always added to the end of the chain, even if it modified an earlier rule. All rules in the chain must be kept. Example: A <- B (enabled). A is modified, resulting in C with history of: A <- B <- C (enabled) B cannot be disabled, even if it never used (eg had a typo)

We need to track variations on a rule that is fundamentally the same, but not necessarily a linear version history. With a linear history, the true relationship between a rule and where it originated from is unclear, and a version history of any sort for the rules is probably unnecessary - all we care about is the currently active rule (and any newly downloaded rules that are candidates for replacing it). The linear history also complicates things when modifications are coming from several sources (eg. Artefactual, various institutions, local modifications)

'''Proposed fix:'''
Each rule stores an Agent (the institution/person that created it) and a 'meta-rule' ID, and no longer stores what rule it replaced. All rules that are variations on each other have the same 'meta-rule' ID and are 'rule versions'. Potentially the meta-rule ID is a foreign key to an actual meta-rule table, but that may not be necessary.

In a set of rule versions, only one may be enabled at a time. Rule versions can come from several sources - originally installed and FPR updates from Artefactual, downloads from a particular institution that is reputable, a local FPR server - and are conceptually linked by having a common meta-rule ID. Agent IDs will be used to distinguish the sources, and created time used to track the most recent variation.

'''Pros:'''
* Easier to download updates to FPR rules, especially from more than one source
* Can provide the ability to delete truly unwanted rules
* ???

'''Cons:'''
* What information to display to the user? What to display if all the rule versions for a meta-rule are disabled?
* Probably backwards incompatible, or non-trivial to make it
* ???

==Early prototype==

*An early FPR prototype (called "Formatica") was developed by Heather Bowden, then Carolina Digital Curation Doctoral Fellow at the School of Information and Library Science in the University of North Carolina at Chapel Hill.

[[File:Formatica.png|border|450px|Early FPR prototype originally called Formatica]]

[[Category:Development documentation]]
[[Category:Feature requirements]]

Format identification requirements

2017-03-23T19:35:32Z

Hbecker: Move to feature requirements category

[[Documentation]] > [[Requirements]] > Format identification requirements

[[Category:Feature requirements]]

== Requirements ==

* FITS runs for characterization and md extraction in transfer tab
** Output indexed and stored in transfer backlog

* '''Select format identification tool''' job in Normalization micro-service, after SIP creation
** user selects from dropdown menu of identification tool options:
***Identify file by extension - this is already the default in Archivematica 0.9
***[http://unixhelp.ed.ac.uk/CGI/man-cgi?file file utility] - file utility identification metadata from FITS output
***[http://web.archive.org/web/20061106114156/http://schmidt.devlib.org/ffident/index.html ffident] - ffident identification metadata from FITS output
***[http://droid.sourceforge.net/ DROID] - DROID identification metadata from FITS output
***[http://sourceforge.net/projects/jhove/ JHOVE] - JHOVE identification metadata from FITS output
***[http://www.openplanetsfoundation.org/software/fido FIDO] - run FIDO
***Tika - run modified Tika
***[http://mediainfo.sourceforge.net/en mediainfo] - run mediainfo
* Duplicate results need cleared from DB - one command for each ID
* Multiple IDs - do all normalization tasks that apply
** Indicate duplicates in Norm report and viewer
** Add delete functionality to Review normalization report and viewer in Dashboard browser, user chooses one or the other
* If there is no rule, normalization should be based on extension
* Distinguish identification tool versions in DB
* Output for each tool must be restructured to comply with format policy

* Format policies should have globally unique IDs

* Format policies result in rules that are stored in the [[Format_policy_registry_requirements|Format Policy Registry (FPR)]]
* Do not split major/minor mimetypes

=== FITS output for format identification ===

* in file utility output: <format> <mimetype>
* in ffident output: <mimetype>
* in DROID output: <PUID> <MimeType>
* in JHOVE output: <format> <mimeType>

== Use Cases ==

== Workflows ==

* Insert File identification micro-service after Clean up names micro-service and before Normalize micro-service
* User selects from dropdown menu which FITS tool to trust for format identification (see tools listed above)
* Archivematica bases Normalization path / application of format policy on the tool selected
* Post 1.0 work: add tools outside of FITS that will run when selected by the user at the File identification micro-service.

== Whiteboard ==

[[File:FileID.jpg|800px]]

== Mockups ==

Note that the Normalize micro-service is included here for illustrative purposes only. In action, the user would not see the Normalize micro-service until a format identification mechanism was selected from the dropdown menu.

[[File:FileIdentificationMS.png]]

=Format policy models and table=
The [[Transcoder]] page has a lot of information on it.
<br/>This diagram is from there.

[[File:Transcoder_database_schema.png]]

I think for the [[Format_policy_registry_requirements|Format Policy Registry (FPR)]] almost every table will need to replace the pk INT with a pk UUID, and the corresponding foreign keys. This is to aid in the creation/submission of new rules.

We're also going to need to know what format the rules output to.

Speaking to future iterations: We'll need to be able to identify what formats the original file, and preservation file are in. Determine the format risk of the data with both of those considered. We may choose to normalize a to a new preservation format from the the original file, or the preservation file. The reason for this may be that both formats risk obsolescence, but there are only tools available to handle the preservation format.

To track changes/updates/new items, I would suggest the FPR keep track of the last modified date of all rows. Clients keep track of the last updated date. These dates are from the FPR.
Additional: add a marked for removal bool to each table in the FPR, to indicate to clients to remove that from their database.

Fixity checking and reporting

2017-03-23T19:35:03Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Documentation]] > [[Requirements]] > Fixity checking and reporting

This wiki page describes requirements for gathering fixity check information from the [https://github.com/artefactual/fixity | Fixity app] and reporting through the Archivematica Storage Service.

[[Category:Feature requirements]]

==Requirements==

* Modify Storage Service to record time and results of fixity checks by adding columns to packages tab

* Add script to send email alerts to administrator(s) when a fixity check fails

==Mockups==

===Storage Service packages tab===

[[File:Fixity_report_SS.png|700 px]]

Two columns are added to the Storage Service packages tab:

* Fixity Date, showing the date and time that the fixity check was run
* Fixity Result, showing the outcome (Succeeded or Failed) with links to tool output in cases of failures or errors (successful fixity scan reports are empty)

===Fixity Results: Failed===

Failures should be reported as such if they are actual fixity failures, or if there were errors in the tool.

'''In the case of a fixity failure''' the report should have the following:

* Date/time that fixity was run
* AIP UUID
* Which files (names, UUIDs) failed fixity check

'''In the case of an error''' the report should have the following:

* Date/time that fixity was run
* AIP UUID
* Nature of the error- for example:

"data/objects/no_such_file exists in manifest but not found on filesystem"

(Source: https://github.com/artefactual/archivematica-storage-service/blob/stable/0.7.x/storage_service/locations/models/package.py#L742-L762)

File Browser Requirements

2017-03-23T19:34:39Z

Hbecker: Move to feature requirements category

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > File Browser Requirements

This page describes requirements for moving all Archivematica interface to the web dashboard, including file browser functionality for transfer and SIP configuration, normalization output review, and AIP and DIP review prior to upload to storage and access. (mockups forthcoming)

[[Category:Feature requirements]]

= TRANSFER =
* Login Archivist - archivist's name will be attached to all processing actions. (PREMIS agent)

*<del>Create structured directory (create three directories: logs, metadata, objects)
-OR-
*Restructure transfer for processing (put all contents into objects directory and create an empty log and metadata directory)</del> (This should be automated)

*<del>Run checksum on transfer and place checksum in folder called checksum.md5 in the metadata folder (checkbox) This is in case the transfer hasn't arrived with a checksum already.</del> (This was discussed - currently, the checksum assigment and checksum check should occur upon beginning transfer, prior to backup. Process should be automated.) In fact, all transfers will go through all microservices for transfers before they are "backlogged"

*Backlog should be indexed

*Assign transfer type (drop-down, user configurable: Generic, DSpace export, digitization output (Issue 713), VanDocs export, BagIt package (Issue 593))

[[Image:0.9_Transfer.png]]

Transfer backup requirements:
See [[Transfer_backlog_requirements]]

= Create SIP =

'''1.0 Mockup'''

[[File:1.0_CreateSIP.png|1000px|thumb|center|]]
'''0.9 Mockup'''
This mockup is outdated.

[[Image:0.9_CreateSIP.png|300px|thumb|center]]

* Below the above contents, on the same page, would be the current 0.8 dashboard for Ingest, including popouts for tasks, etc, and decision points.
* original order and arrangement (issue 964) captured as logical <structMap> in SIP METS file.
* Report of actions in Create SIP is auto-generated

= File Viewer =
* This will allow the user to see individual documents in the transfer to get a better idea of their contents and technical metadata before assigning them to SIPs.
* Viewers are browser-dependent; viewer option is greyed out if viewer is not supported in browser
* Examine Contents window allows for viewing technical MD and other metadata available after Transfer microservices as well as indexing MD

= Examine Contents =

[[File:1.0_ExamineContents.png|1000px|thumb|center|]]

* Opens in new tab
* This will allow the user to examine contents of a SIP for keywords, use visualization tools, and identify restricted records (by keyword or by type, cc number, SIN number, etc), tag records or groups of records and apply some basic metadata that will be carried over to the description

= ADMINISTRATION TAB =
*Assign transfer backup location (what is it backing up? at which stage?) This could be a staging area, in case many transfers will become one SIP or more time is needed to begin processing ADMIN

*Go to storage reports in DB on individual storage locations - see Issue 882

*Transfer store = Backlog ; MD: name, accession #, uuid, transfer type, archivist, size of contents, number of files, date, file types? and other indexing results?, any reports? (FITS, clamAV, etc)
** Accessions managed in ICA-AtoM or other content management system, linked in DB and in tab

[[File:0.9_AdministrationTab.png]]

= More to come on this...=
* See Transfer indexing requirements [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]
* See Issue 924

*Archivematica compliant SIP creation, including:
log of original directory structure, diff to new structure (using METS <structMap> to represent both)
use METS <structMap> (or physical directory arrangement of SIP?) to represent archival arrangement for rebuild in access system (see Issue 380)

*Change permissions of directory contents recursively (or should this be integrated in every step to avoid problems?)