Difference between revisions of "Archivematica 1.11 and Storage Service 0.16 release notes"

From Archivematica
Jump to navigation Jump to search
(Created page with "Home > Release Notes > Major release notes template '''Release date''' In the header, include any general notes about the release. For an exa...")
 
 
(49 intermediate revisions by 5 users not shown)
Line 1: Line 1:
−
[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Major release notes template
+
[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16
  
−
'''Release date'''
+
'''April 1, 2020'''
  
−
In the header, include any general notes about the release. For an example, see the [[Archivematica_1.6_release_notes|Archivematica 1.6 Release Notes]].
+
==Environments==
  
−
==Supported environments==
+
Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].
  
−
Link to installation instructions.
+
Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:
  
−
Specify supported environments.
+
* Ubuntu 16.04 64-bit Server Edition
 +
* Ubuntu 18.04 64-bit Server Edition
 +
* CentOS 7 64-bit
  
−
Make special note of any changes to supported environment.
+
For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.
 +
 
 +
<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>
  
 
==Added==
 
==Added==
  
−
Describe new features.
+
===PREMIS Event import===
 +
 
 +
This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.
 +
 
 +
This work was sponsored by Piql and the Norwegian Health Archives. Thank you!
 +
 
 +
* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
 +
* Issue: https://github.com/archivematica/Issues/issues/710
 +
 
 +
 
 +
===S3 as a transfer source===
 +
 
 +
This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!
 +
 
 +
* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
 +
* Issue: https://github.com/archivematica/Issues/issues/975
 +
 
 +
===Easier access to AIP METS===
  
−
===New feature 1===
+
This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.
  
−
This is a description of this amazing feature! Here's why it's a net benefit to the project and the community. Also included are any special notes, like if it's a beta feature.
+
* Documentation: pending
 +
*Issue: https://github.com/archivematica/Issues/issues/644
  
−
This work was sponsored by some amazing institution. Thank you!
+
===Zipped transfers===
  
−
* Documentation: link
+
This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!
−
* Pull requests: link
 
  
−
===New feature 2===
+
* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
 +
* Issue: https://github.com/archivematica/Issues/issues/682
  
−
Here is a description of this amazing feature! Here's why it's a net benefit to the project and the community, and here is how it will impact your workflow. Also included are any special notes, like if it's a beta feature.
+
===Add package name as configurable value to call backs===
  
−
This work was sponsored by some amazing institution. Thank you!
+
When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!
  
−
* Documentation: link
+
* Documentation: pending
−
* Pull requests: link
+
* Issue: https://github.com/archivematica/Issues/issues/978
−
* Feature files: link
 
  
 
==Changed==
 
==Changed==
  
−
Describe enhancements or major fixes.
+
===Performance and monitoring improvements===
 +
 
 +
This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.
 +
 
 +
These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!
 +
 
 +
* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
 +
* Issues:
 +
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
 +
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
 +
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
 +
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
 +
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
 +
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
 +
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199
 +
 
 +
 
 +
===Improvements for full disks===
 +
 
 +
Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
 +
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
 +
* Change the processing storage usage page to clarify storage paths/locations and improve usability
 +
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
 +
* Change Storage Service functionality to support the above changes.
 +
 
 +
* Documentation: pending
 +
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]
 +
 
 +
===Changes to default normalization for videos/images===
 +
 
 +
Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.
 +
 
 +
* Issue: https://github.com/archivematica/Issues/issues/912
 +
 
 +
===Allow users to choose whether to receive fail report emails===
 +
 
 +
Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!
 +
 
 +
* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
 +
* Issue: https://github.com/archivematica/Issues/issues/709
 +
 
 +
===Change name of sanitize names micro-service===
 +
 
 +
Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.
 +
 
 +
* Issue: https://github.com/archivematica/Issues/issues/230
 +
 
 +
===Drop-down menu orders===
 +
 
 +
In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.
 +
 
 +
* Issue: https://github.com/artefactual/archivematica/issues/891
 +
 
 +
==Removed==
  
−
===Enhancement 1===
+
===Quarantine===
  
−
We fixed this issue. Here's why it's a net benefit to the project and the community, and here is how it will impact your workflow. Also included are any special notes, like if it's a beta feature.
+
As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
 +
* Issue: https://github.com/artefactual/archivematica/issues/598
 +
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md
  
−
This work was sponsored by some amazing institution. Thank you!
+
==="Add next" disk image transfer button===
  
−
* Documentation: link
+
This button seemed redundant to the workflow so it's been removed.
−
* Pull requests: link
+
* Issue: https://github.com/archivematica/Issues/issues/290
−
* Feature files: link
 
  
 
==Fixed==
 
==Fixed==
  
−
List bugfixes with a link to the Github issue.
+
* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
 +
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
 +
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
 +
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
 +
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
 +
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
 +
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
 +
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
 +
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
 +
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
 +
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
 +
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
 +
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
 +
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
 +
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
 +
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
 +
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
 +
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
 +
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
 +
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
 +
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
 +
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
 +
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
 +
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
 +
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
 +
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
 +
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
 +
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
 +
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
 +
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
 +
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
 +
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
 +
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
 +
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
 +
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
 +
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981
 +
 
  
−
* Bugfix 1: link
+
And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.
−
* Bugfix 2: link. Sponsored by someone!
 
−
* Bugfix 3: link
 
  
 
==Upgraded tools and dependencies==
 
==Upgraded tools and dependencies==
  
−
List any tools and dependencies that have been upgraded.
 
  
−
* Tool has been updated to version X.
+
* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791
 +
 
 +
==Known issues==
 +
 
 +
Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.
  
 
==End of life dependencies==
 
==End of life dependencies==
  
−
List any dependencies that have reached end of life since the last release, as well as a note on the plan going forward.
+
Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
 +
* Dashboard: https://github.com/archivematica/Issues/issues/810
 +
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
 +
* amclient: https://github.com/archivematica/Issues/issues/817
 +
* Automation tools: https://github.com/archivematica/Issues/issues/815
 +
* Fixity: https://github.com/archivematica/Issues/issues/814
 +
* am/compose: https://github.com/archivematica/Issues/issues/804
 +
* Fido: https://github.com/archivematica/Issues/issues/847
  
−
Make note of any risks, and how users can mitigate them.
+
We will continue to work toward full Python 3 use in upcoming releases.

Latest revision as of 15:24, 30 June 2020

Home > Release Notes > Archivematica 1.11 and Storage Service 0.16

April 1, 2020

Environments[edit]

Please see the installation instructions.

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

  • Ubuntu 16.04 64-bit Server Edition
  • Ubuntu 18.04 64-bit Server Edition
  • CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the Upgrading page in the documentation.

Added[edit]

PREMIS Event import[edit]

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see sample data) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!


S3 as a transfer source[edit]

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

Easier access to AIP METS[edit]

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

Zipped transfers[edit]

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

Add package name as configurable value to call backs[edit]

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

Changed[edit]

Performance and monitoring improvements[edit]

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!


Improvements for full disks[edit]

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:

  • Change the processing storage usage page to clarify storage paths/locations and improve usability
  • Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
  • Change Storage Service functionality to support the above changes.

Changes to default normalization for videos/images[edit]

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in new or upgraded installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see this comment in the issue ticket.

Allow users to choose whether to receive fail report emails[edit]

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

Change name of sanitize names micro-service[edit]

Following reading a paper by Elvia Arroyo-Ramirez we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

Drop-down menu orders[edit]

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

Removed[edit]

Quarantine[edit]

As discussed on the community forum the quarantine micro-service has been removed from Archivematica in this release.

"Add next" disk image transfer button[edit]

This button seemed redundant to the workflow so it's been removed.

Fixed[edit]


And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

Upgraded tools and dependencies[edit]

Known issues[edit]

Please note that due to Issue 1149 the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

End of life dependencies[edit]

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:

We will continue to work toward full Python 3 use in upcoming releases.

Retrieved from "https://wiki.archivematica.org/index.php?title=Archivematica_1.11_and_Storage_Service_0.16_release_notes&oldid=13500"