Archivematica 1.8 and Storage Service 0.13 release notes

Home > Release Notes > Archivematica 1.8 and Storage Service 0.13

Work in progress

Supported environments

Please see installation instructions link here.

Archivematica 1.8 and Storage Service 0.13 are supported for production use in the following environments:

• Ubuntu 16.04 64-bit Server Edition
• Ubuntu 18.04 64-bit Server Edition
• CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

Added

Dataverse integration

Archivematica can now be configured to use a Dataverse research data repository as a transfer source location. Dataverse transfer source locations can be configured to display all available datasets or a subset of them. Datasets are retrieved directly using the Dataverse API and processed using a new “Dataverse” transfer type. New Dataverse specific processing includes:

• fixity checking using checksums generated by dataverse
• retrieval of derivative and metadata files associated with tabular data files
• creation of a Dataverse METS file describing the dataset as retrieved from Dataverse
• Dataverse metadata included in the AIP METS

Some advanced or more complex use cases are not fully supported, such as handling of datasets with restricted files, versioning of datasets and reingest of datasets. For a full list of known issues and enhancement ideas, refer to the Archivematica issues repository using Dataverse label and the Archivematica-Dataverse Integration Project Wiki.

This work was sponsored by Scholars Portal, a service of the Ontario Council of University Libraries (OCUL). Thank you!

• Issue: See waffle board for all issues with the Dataverse label.
• Documentation: Dataverse Integration

Public URL

Archivematica now has a concept of a public URL in the general configuration. Archivematica usually registers itself with the Storage Service, but if you have deployed Archivematica in an environment where the URL or IP address changes frequently (i.e. in some Docker environments) this can cause issues. In these types of environments, users can now declare a stable public URL that Archivematica can use to register with the Storage Service.

This work was sponsored by Jisc. Thank you!

• Issue: 1140
• Documentation: General configuration

Package storage in DSpace via its REST API

Since Archivematica 1.6 it's been possible to store AIPs in DSpace, but they have been stored via SWORD. Some users have the need/desire to store AIPs in DSpace via the REST API. This is now possible in Storage Service 0.13.

This work was undertaken by Hrafn Malmquist at University of Edinburgh, with assistance from developers at Artefactual Systems. Thank you!

• Issue: 391
• Documentation:

Changed

Enhancements or major fixes.

Automatic approval of transfers

It is no longer necessary to approve transfers started in the user interface. By default, a checkbox is ticked for the transfer to automatically be approved- users may uncheck the box and approve their transfers manually if desired.

This enhancement was sponsored by Jisc. Thank you!

• Issue: 1139
• Documentation: Process a transfer

Streamline checksum verification

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

• Remove the "Verify checksums generated on ingest" micro-service
• Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
• Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - Problem: uncompressed AIPs need pointer files #32

This work was sponsored by Columbia University Library. Thank you!

• Issue: 918

Indexing can be enabled/disabled for Transfers and/or Archival Storage

Previously, the ElasticSearch index feature could be disabled globally as a scalability measure since indexing consumes a lot of resources. However, this also disabled Backlog and Appraisal features (which also uses indexing) and which some users still wanted to access. As of release 1.8, Archivematica can be deployed to run with indexing enabled just for Transfers (Backlog and Appraisal enabled), just for Archival Storage (Backlog and Appraisal disabled), for both indexes, or for none.

• Issue: 1172
• Documentation: Installation > Elasticsearch

Configure email settings

This change improves the ways that the email client in Archivematica can be configured, including allowing an administrator to set the sender email address for emails sent by Archivematica (i.e. normalization reports, failure reports) to comply with local IT requirements.

This work was sponsored by Jisc. Thank you!

• Issue: 1128
• Documentation: Email notification configuration

Download processing configuration and reset to default

Previous versions of Archivematica introduced the ability to add custom processing configurations, but users had to retrieve the custom configuration file via the command line to use it. There is now a download button on Administration > Processing configuration so that you can download the processing config from the user interface.

You can also reset a processing configuration to the installation pre-set by clicking on the new reset button on Administration > Processing configuration.

The documentation for using a custom processing configuration has also been updated.

This work was sponsored by Jisc. Thank you!

• Issue: 1138, 800
• Documentation: Processing configuration (user manual), Processing configuration (administrator manual), Using a custom processing configuration file

MCP batching for scalability & performance

This feature refactors how tasks are scheduled, executed & managed within Archivematica, by grouping tasks into batches. It introduces processing efficiencies that significantly decrease the processing power and time required to complete Transfer and Ingest. It includes new configuration options to further optimize processing efficiency for particular types of Transfers (e.g. few large files vs. many small files) and for different deployment patterns (e.g. installing components across multiple machines).

This work was sponsored by Jisc. Thank you!

• Issue: 938
• Documentation: Scaling Archivematica

Binder integration improvements

Archivematica has had an integration with Binder for several years. Binder is an open-source web application for managing time-based media and born-digital artworks. Binder depends on integration with both Archivematica and TMS (The Museum System).

Since Binder is built off of AtoM, much of the integration configuration was repurposed from the AtoM integration. Archivematica 1.8 makes it explicit, for example, that the job "DIP Upload to AtoM" is actually "DIP Upload to AtoM/Binder". In the Administration tab, the configuration section for AtoM has also been renamed to include Binder.

Enhancing the Binder integration itself, Archivematica's transfer tab now includes an "Access system ID" box. This allows users to pre-populate an access system ID for AtoM or Binder, so that DIPs can be automatically uploaded without having to stop at the Upload DIP microservice. Users can still use the Upload DIP popup if desired.

Finally, we've added documentation on using Binder with Archivematica.

This work was sponsored by Tate. Thank you!

• Documentation: Binder integration, Using Binder with Archivematica, Upload a DIP to Binder
• Issues: 23

Translations

First added in 1.7, translations in Archivematica are growing! In this release, we've pulled in translations for Archivematica.org, the documentation, the FPR, the Storage Service, and the Archivematica interface. The biggest part that is still missing is the Archivematica's workflow engine - that is, all of the microservice and job names in the interface. We are planning to include workflow translations in 1.9.

Thanks to our wonderful community of translators on Transifex, Archivematica resources can now be translated from English into French, Spanish, Japanese, Portuguese, Brazilian Portuguese, and Swedish. Note that the completeness of each language for each resource depends on volunteer contributions in Transifex.

This work was originally sponsored by the Canadian Council on Archives through a DHCP (Documentary Heritage Community Programs) grant. Thank you!

• Issue: 231
• Documentation: Languages, Translating Archivematica

File format identification updates

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the PRONOM release notes.

This work was sponsored by the Denver Art Museum. Thank you!

Thumbnail normalization changes

It is now easier to configure whether or not, and how, thumbnails are created. In the processing configuration, users can choose between yes, normalize for thumbnails, no, do not normalize for thumbnails, or yes, do so but only if there is a default rule in place. For users who do not need thumbnails this could make their processing faster.

This work was sponsored by Jisc. Thank you!

• Issue: 1022
• Documentation: Processing Configuration Fields

Fixed

• Validate preservation derivatives hangs Sponsored by Jisc- thank you!
• Zipped bag transfers cannot be approved via an API call Community contribution by Hillel Arnold- thank you!
• AIP verification fails for Zipped bag transfers containing .DS_Store files in object directory Community contribution by Hillel Arnold- thank you!
• Can't use package API endpoint if Transfer Source is unknown Sponsored by Jisc- thank you!
• MySQL aborting transactions under heavy load Sponsored by Jisc- thank you!
• Cannot create spaces via API Sponsored by Jisc- thank you!
• Cannot create default locations via API Sponsored by Jisc- thank you!
• premis:originalName value of unpacked packages should not be normalized Sponsored by the International Institute of Social History- thank you!
• unapproved_transfers endpoint throws a 500 error fixed with contributions by Hillel Arnold and the International Institute of Social History- thank you!
• ASCII codes can't decode when the filename contains a backtick Sponsored by the International Institute of Social History- thank you!
• AIP re-ingest fails
• PREMIS events from previous transfers are re-appearing Sponsored by Jisc- thank you!
• Metadata reingest fails when dc:type is null
• Use 7-zip without compression (Copy) mode
• Cannot store AIP in DSpace due to file extension returned
• DSpace REST login error in SS
• Unable to edit DSpace REST Space settings in SS
• Packages cannot be stored in DSpace via its REST API
• Metadata added before "Approve Transfer" disappears
• Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label Community contribution by Hillel Arnold- thank you!
• Zip files with diacritic characters are failing to extract Sponsored by the International Institute of Social History- thank you!
• restructureBagForComplianceFileUUIDsAssigned needs to create intermediate directories for Zipped bag transfers - Community contribution by Hillel Arnold. Thank you!
• Ingest fails if Archivematica isn't connected to the Internet
• Can't store encrypted uncompressed AIPs
• Can't add AIC number through metadata form
• GPG key generation doesn't work in Ubuntu 18.04
• Cannot save settings on general settings form in 1.8
• Rights.csv metadata is not imported to METS file
• Cannot create more than one SIP from a transfer
• Archivematica making multiple copies of large transfers
• Large transfers don't show up in the dashboard
• Create SIP fails when directories contain UUIDs Community contribution by Jason Jordan- thank you!
• Can't arrange a SIP from backlog with long names
• Normalization output formatting error
• Package names are being modified
• Choosing AtoM/Binder/AT/AS config is unnecessary
• Hard to know which formats are related to which PRONOM IDs in FPR

Upgraded tools and dependencies

• Fido has been upgraded to version 1.3.12
• Siegfried has been upgraded to version 1.7.10
• FITS has been upgraded to version 1.1.0
• gunicorn has been upgraded to version 19.9.0

End of life dependencies

Archivists' Toolkit integration

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a proposal deprecate this feature in Archivematica 1.9. Community response is welcome via a comment on the issue in GitHub.

Known issues

• There is a bug preventing the CONTENTdm workflow in this release. There will be a patch available with a fix.