Archivematica 1.8 and Storage Service 0.13 release notes

Home > Release Notes > Major release notes template

Work in progress

Supported environments

Link to installation instructions.

Specify supported environments.

Make special note of any changes to supported environment.

Added

Dataverse integration

Archivematica can now be configured to use a Dataverse research data repository as a transfer source location. Dataverse transfer source locations can be configured to display all available datasets or a subset of them. Datasets are retrieved directly using the Dataverse API and processed using a new “Dataverse” transfer type. New dataverse specific processing includes:

• fixity checking using checksums generated by dataverse
• retrieval of derivative and metadata files associated with tabular data files
• creation of a Dataverse METS file describing the dataset as retrieved from Dataverse
• Dataverse metadata included in the AIP METS

Some advanced or more complex use cases are not fully supported, such as handling of datasets with restricted files, versioning of datasets and reingest of datasets. For a full list of known issues and enhancement ideas, refer to the Archivematica issues repository using Dataverse label and the Archivematica Wiki.

This work was sponsored by Scholars Portal, a service of the Ontario Council of University Libraries (OCUL). Thank you!

• Issue: See waffle board for all issues with the Dataverse label.
• Documentation: Dataverse Integration

Processing configuration reset and download buttons

A new installation of Archivematica comes with a pre-set processing configuration called "default", and a second one (used only in Jisc workflows) called "automated". In testing, users are encouraged to change the configurations to suit their workflows, but may need to reset the configuration to the installation pre-sets. A reset button has been added so that users can easily change the default and automated processing configurations back to their installation pre-sets.

The second part of this feature is the addition of a download button for the processing configuration files. If you create a custom processing configuration, you can download the resulting processingMCP.xml file using the button and then include it at the top level of your transfer. Archivematica will then use this to automate your transfer selections, rather than the default configuration.

This work was sponsored by Jisc. Thank you!

• Issue: #1138
• Documentation: Processing configuration documentation

Public URL

Archivematica now has a concept of a public URL in the general configuration. Archivematica usually registers itself with the Storage Service, but if you have deployed Archivematica in an environment where the URL or IP address changes frequently (i.e. in some Docker environments) this can cause issues. In these types of environments, users can now declare a stable public URL that Archivematica can use to register with the Storage Service.

This work was sponsored by Jisc. Thank you!

• Issue: 1140
• Documentation: General configuration

Changed

Enhancements or major fixes.

Streamline checksum verification

This enhancement de-duplicates checksum verification in Archivematica, which helps to improve the performance of Archivematica in processing large transfers (many files and/or large files). This enhancement includes three changes:

• Remove the "Verify checksums generated on ingest" micro-service
• Enhance the "Verify AIP" micro-service to bulk query the database for transfer-generated checksums and then verify that they match what is documented in the bag-generated manifest-<ALGORITHM>.txt.
• Have "Verify AIP" create an AIP-level "fixity check" PREMIS:EVENT that it can pass to the Storage Service, which will document this verification in the pointer file.

This should not impact regular workflows, but it is worth noting that there is no AIP-level fixity check PREMIS event for uncompressed AIPs, which don't have pointer files. For uncompressed AIPs, there are still object-level fixity events in place. Note that there is an issue in the Archivematica Issues repository regarding this note - Problem: uncompressed AIPs need pointer files #32

This work was sponsored by Columbia University Library. Thank you!

• Issue: 918

Indexing can be enabled/disabled for Transfers and/or Archival Storage

Previously, the ElasticSearch index feature could be disabled globally as a scalability measure since indexing consumes a lot of resources. However, this also disabled Backlog and Appraisal features (which also uses indexing) and which some users still wanted to access. As of release 1.8, Archivematica can be deployed to run with indexing enabled just for Transfers (Backlog and Appraisal enabled), just for Archival Storage (Backlog and Appraisal disabled), for both indexes, or for none.

• Issue: 1172
• Documentation: Installation > Elasticsearch

Configure email settings

This change improves the ways that the email client in Archivematica can be configured, including allowing an administrator to set the sender email address for emails sent by Archivematica (i.e. normalization reports, failure reports) to comply with local IT requirements.

This work was sponsored by Jisc. Thank you!

• Issue: 1128
• Documentation: Email notification configuration

Download processing configuration and reset to default

Previous versions of Archivematica introduced the ability to add custom processing configurations, but users had to retrieve the custom configuration file via the command line to use it. There is now a download button on Administration > Processing configuration so that you can download the processing config from the user interface.

You can also reset a processing configuration to the installation pre-set by clicking on the new reset button on Administration > Processing configuration.

The documentation for using a custom processing configuration has also been updated.

This work was sponsored by Jisc. Thank you!

• Issue: 1138, 800
• Documentation: Processing configuration (user manual), Processing configuration (administrator manual), Using a custom processing configuration file

MCP batching for scalability & performance

This feature refactors how tasks are scheduled, executed & managed within Archivematica, by grouping tasks into batches. It introduces processing efficiencies that significantly decrease the processing power and time required to complete Transfer and Ingest. It includes new configuration options to further optimize processing efficiency for particular types of Transfers (e.g. few large files vs. many small files) and for different deployment patterns (e.g. installing components across multiple machines).

This feature does not impact the functionality or appearance of Archivematica.

This work was sponsored by Jisc. Thank you!

• Issue: 938
• Documentation: Scaling Archivematica [update with link when PR 182 is merged];

Binder integration improvements

Archivematica has had an integration with Binder for several years. Binder is an open-source web application for managing time-based media and born-digital artworks that also depends on TMS.

Since Binder is built off of AtoM, much of the integration configuration was repurposed from the AtoM integration. Archivematica 1.8 makes it explicit, for example, that the job "DIP Upload to AtoM" is actually "DIP Upload to AtoM/Binder". In the Administration tab, the configuration section for AtoM has also been renamed to include Binder.

Enhancing the Binder integration itself, Archivematica's transfer tab now includes an "Access system ID" box. This allows users to pre-populate an access system ID for AtoM or Binder, so that DIPs can be automatically uploaded without having to stop at the Upload DIP microservice. Users can still use the Upload DIP popup if desired.

Finally, we've added documentation on using Binder with Archivematica.

This work was sponsored by Tate. Thank you!

• Documentation: Binder integration, Using Binder with Archivematica, Upload a DIP to Binder
• Issues: 23

File format identification updates

Archivematica 1.8 is now up to date with PRONOM v.94! For more information on new data added to PRONOM, check the PRONOM release notes.

This work was sponsored by the Denver Art Museum. Thank you!

Fixed

• ASCII codes can't decode when the filename contains a backtick
• AIP re-ingest fails
• PREMIS events from previous transfers are re-appearing
• Metadata reingest fails when dc:type is null
• Use 7-zip without compression (Copy) mode
• Problem: cannot store AIP in DSpace due to file extension returned
• DSpace REST login error in SS
• Unable to edit DSpace REST Space settings in SS
• Metadata added before "Approve Transfer" disappears
• Generate AIP METS fails for bag SIPs if bag-info.txt has multiple instances of the same label
• Zip files with diacritic characters are failing to extract
• restructureBagForComplianceFileUUIDsAssigned needs to create intermediate directories for Zipped bag transfers - Community contribution by Hillel Arnold. Thank you!
• Ingest fails if Archivematica isn't connected to the Internet

Upgraded tools and dependencies

• Fido has been upgraded to version 1.3.12
• Siegfried has been upgraded to version 1.7.10
• FITS has been upgraded to version 1.1.0

End of life dependencies

Archivists' Toolkit integration

Archivists' Toolkit has been deprecated since 2013. The Archivists' Toolkit DIP upload feature has not had active development or testing since then. There are no plans to start testing or to fix any problems with the feature. As a result, there is a proposal deprecate this feature in Archivematica 1.9. Community response is welcome via a comment on the issue in GitHub.