Archivematica - User contributions [en]

Storage Service 0.19 Release Notes

2022-03-01T17:47:50Z

Sallain: /* User roles */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release adds a new feature to the Storage Service

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

The Storage Service now includes four different user roles: administrators, managers, reviewers and readers. Previously, users were either administrators or not administrators, the only difference being that administrators can create, edit, and delete other users. All existing users will automatically become administrators on upgrade. You may wish to review your users and adjust their permissions as needed.

Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

See [https://www.archivematica.org/en/docs/storage-service-0.19/administrators/#users Users] in the Storage Service documentation for more information.

== Fixed ==

* Storage Service 0.18.x doesn't follow symlinks: https://github.com/archivematica/Issues/issues/1515

Storage Service 0.19 Release Notes

2022-02-25T19:19:38Z

Sallain: /* Added */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release adds a new feature to the Storage Service

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

The Storage Service now includes four different user roles: administrators, managers, reviewers and readers. Previously, users were either administrators or not administrators, the only difference being that administrators can create, edit, and delete other users. All existing users will automatically become administrators on upgrade. You may wish to review your users and adjust their permissions as needed.

Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

See [https://archivematica.org/en/docs/archivematica-1.13/user-manual/administer/dashboard-admin/#dashboard-users Users] in the Archivematica documentation for more information.

== Fixed ==

* Storage Service 0.18.x doesn't follow symlinks: https://github.com/archivematica/Issues/issues/1515

Storage Service 0.19 Release Notes

2022-02-25T19:11:31Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release adds a new feature to the Storage Service

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

The Storage Service now includes four different user roles: administrators, managers, reviewers and readers. Previously, users were either administrators or not administrators, the only difference being that administrators can create, edit, and delete other users. All existing users will automatically become administrators on upgrade. You may wish to review your users and adjust their permissions as needed.

Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

See [https://archivematica.org/en/docs/archivematica-1.13/user-manual/administer/dashboard-admin/#dashboard-users Users] in the Archivematica documentation for more information.

This feature was sponsored by X. Thank you!

== Fixed ==

* Storage Service 0.18.x doesn't follow symlinks: https://github.com/archivematica/Issues/issues/1515

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2022-02-25T19:11:20Z

Sallain: /* Added */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.0 and Storage Service 0.18.0

'''Release date: July 12, 2021'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.0 and Storage Service 0.18.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Added ==

=== Disableable virus scanning ===

* Issue: https://github.com/archivematica/Issues/issues/869

Virus scanning can now be disabled from the processing configuration.

This change was '''contributed''' by Bodleian Library. Thank you!

=== Purge management command ===

* Issue: https://github.com/archivematica/Issues/issues/1239

A new [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/maintenance/maintenance/#management-commands management command] has been added to provide a mechanism to remove old processing data from the system filling up the application database and search indices.

=== Strong password validation ===

* Issue: https://github.com/archivematica/Issues/issues/1332

Archivematica and the Storage Service can now require strong passwords for user accounts. Prompts are given when creating a new user if the password doesn't meet the requirements.

This feature was '''sponsored''' by the City of Winnipeg Archives. Thank you!

=== Audit logging capability ===

* Issue: https://github.com/archivematica/Issues/issues/1341

This addition allows audit logs to be written to third party applications- Artefactual Systems' implementation of this is in a new application called [https://github.com/artefactual-labs/auditmatica Auditmatica]. The change in Archivematica and the Storage Service is only to facilitate the capture of audit logging information in other applications.

This addition was '''sponsored''' by the City of Winnipeg Archives. Thank you!

=== Customized workflow file ===

* Issue: https://github.com/archivematica/Issues/issues/1441

This feature allows the user to indicate the location/existence of a customized workflow document, in json. This would allow institutions to add to the Archivematica workflow (e.g. custom micro-services) in a way that is easier to maintain through upgrades. Developer documentation is pending.

== Changed ==

=== Python 3 ===

* Issues: https://github.com/archivematica/Issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.13.0+label%3A%22%3Asnake%3A%C2%B3+Python+3%22

All Archivematica components are now running on Python 3.6.

=== Archivematica Storage Service now uses MySQL by default ===

* Issue: https://github.com/archivematica/Issues/issues/952

MySQL is now the default database in Archivematica Storage Service. SQLite is still supported but we encourage users to migrate. We have documented the process: [https://www.archivematica.org/en/docs/storage-service-0.18/migration-sqlite-mysql/#migration-sqlite-mysql Migrating data from SQLite to MySQL].

=== Allow replicated AIPs to be packaged in a different format from the original ===

*Issue: https://github.com/archivematica/Issues/issues/1440

By using an offline replica storage space it is now possible to replicate AIPs in a different packaging format than the original AIP (helpful in use cases such as offline tape storage, etc).

This change was '''sponsored''' by Norwegian Health Archives. Thank you!

=== Tasks will open in one tab only ===

* Issue: https://github.com/archivematica/Issues/issues/85

Now instead of opening multiple tabs every time a user clicks on a task gear, they will all open in one new tab.

This change was '''contributed''' by Bodleian Library. Thank you!

=== Dev tools deprecated / Hiding active packages ===

Issue: https://github.com/archivematica/Issues/issues/68

Issue: https://github.com/archivematica/Issues/issues/1446

Due to maintainability issues, we have removed the devtools repo. The most commonly used dev tool was used for resolving hidden transfers, which should no longer be relevant since resolving [https://github.com/archivematica/Issues/issues/1446 Issue 1446]- active transfers or SIPs can no longer be hidden. However as a precaution, we have made that command available within Archivematica (https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/main/management/commands/resolve_pending_jobs.py).

=== DIP storage locations now allowed in S3 spaces ===

* Issue: https://github.com/archivematica/Issues/issues/1442

This is a '''contribution''' by Fashion Institute of Technology NYC. Thank you!

=== Other changes ===

* Archivematica now records granular time stamps for original files, Issue 1427: https://github.com/archivematica/Issues/issues/1427
* Archival storage csv file sortable by size column, Issue 1450: https://github.com/archivematica/Issues/issues/1450
* Deleting AIPs does not delete associated directories from AIP store, Issue 359: https://github.com/archivematica/Issues/issues/359
* Cannot add a processing configuration if the name has diacritics, Issue 1104: https://github.com/archivematica/Issues/issues/1104
* Cannot create a replicator location in Duracloud space, Issue 1350: https://github.com/archivematica/Issues/issues/1350
* Fixity status tab sorts alphabetically, now sorts by date, Issue 1196: https://github.com/archivematica/Issues/issues/1196
* Failure report not generated when email not configured, Issue 1033: https://github.com/archivematica/Issues/issues/1033
* Replicator does not replicate AIPs on re-ingest or delete replicated AIPs, Issue 985: https://github.com/archivematica/Issues/issues/985

== Fixed ==

Please see the 1.13 milestone in GitHub for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/17.

Storage Service 0.19 Release Notes

2022-02-25T19:09:40Z

Sallain: /* Fixed */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release...

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

The Storage Service now includes four different user roles: administrators, managers, reviewers and readers. Previously, users were either administrators or not administrators, the only difference being that administrators can create, edit, and delete other users. All existing users will automatically become administrators on upgrade. You may wish to review your users and adjust their permissions as needed.

Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

See [https://archivematica.org/en/docs/archivematica-1.13/user-manual/administer/dashboard-admin/#dashboard-users Users] in the Archivematica documentation for more information.

== Fixed ==

* Storage Service 0.18.x doesn't follow symlinks: https://github.com/archivematica/Issues/issues/1515

Storage Service 0.19 Release Notes

2022-02-25T19:09:29Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release...

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

The Storage Service now includes four different user roles: administrators, managers, reviewers and readers. Previously, users were either administrators or not administrators, the only difference being that administrators can create, edit, and delete other users. All existing users will automatically become administrators on upgrade. You may wish to review your users and adjust their permissions as needed.

Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

See [https://archivematica.org/en/docs/archivematica-1.13/user-manual/administer/dashboard-admin/#dashboard-users Users] in the Archivematica documentation for more information.

== Fixed ==

* Storage Service 0.18.x doesn't follow symlinks https://github.com/archivematica/Issues/issues/1515

Storage Service 0.19 Release Notes

2022-02-25T19:07:40Z

Sallain:

Storage Service 0.19 Release Notes

2022-02-25T19:06:58Z

Sallain:

Storage Service 0.19 Release Notes

2022-02-25T18:48:14Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release...

== Environments ==

Please see the [https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Storage Service 0.19.0 has been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the near future.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Archivematica 1.11.2 and Storage Service 0.16.1 release notes

2020-06-30T21:29:01Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.2 and Storage Service 0.16.1

'''Release date: June 12, 2020'''

This release introduces two bug fixes and one minor security fix. For installation instructions, please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation Installation documentation].

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

==Fixed==

List bugfixes with a link to the Github issue.

* Compressed AIPs break replication: [https://github.com/archivematica/Issues/issues/1149 Issue 1149]
* Responsive top menu on dashboard blocks view of content: [https://github.com/archivematica/Issues/issues/1034 Issue 1034]
* AM Dashboard does not implement Cross Site Request Forgery protection: [https://github.com/archivematica/Issues/issues/1235 Issue 1235]

Archivematica 1.11.1

2020-06-30T21:28:05Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.1

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Archivematica 1.11 and Storage Service 0.16 release notes

2020-06-30T21:24:23Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''April 1, 2020'''

==Environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

==="Add next" disk image transfer button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Release Notes

2020-06-12T22:27:59Z

Sallain: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]] (Current release)
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Release Notes

2020-06-12T22:26:57Z

Sallain:

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|'''Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes''']]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]] (Current release)
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.11.2 and Storage Service 0.16.1 release notes

2020-06-11T19:47:41Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.2 and Storage Service 0.16.1

'''Release date: June 12, 2020'''

This release introduces two bug fixes and one minor security fix.

==Fixed==

List bugfixes with a link to the Github issue.

* Compressed AIPs break replication: [https://github.com/archivematica/Issues/issues/1149 Issue #1149]
* Responsive top menu on dashboard blocks view of content: [https://github.com/archivematica/Issues/issues/1034 Issue #1034]
* AM Dashboard does not implement Cross Site Request Forgery protection: [ADD LINK HERE ONCE FILED]

Archivematica 1.11.2 and Storage Service 0.16.1 release notes

2020-06-11T19:47:29Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.2 and Storage Service 0.16.1

'''Release date: June 12, 2020'''

This release introduces two bug fixes and one minor security patch.

==Fixed==

List bugfixes with a link to the Github issue.

* Compressed AIPs break replication: [https://github.com/archivematica/Issues/issues/1149 Issue #1149]
* Responsive top menu on dashboard blocks view of content: [https://github.com/archivematica/Issues/issues/1034 Issue #1034]
* AM Dashboard does not implement Cross Site Request Forgery protection: [ADD LINK HERE ONCE FILED]

File:Archivematica-CLA-firstname-lastname-YYYY.pdf

2020-06-03T20:34:14Z

Sallain:

Release Process

2020-05-28T15:29:49Z

Sallain: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python setup.py install</code>
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Replace:
#* the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* the reference path to the container signatures in <code>fido/fido.py</code>
# Update version:
#* Update in __init__.py
#* Update in versions.xml
# Create pull request.
# Release new version of FIDO.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

When a new Siegfried version becomes available, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo and update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

==== Examples ====
<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55

There used to be a [https://projects.artefactual.com/issues/10466#change-46673 bug] in the imports model. To fix, Remove the <code>apps.get_model</code> lines and Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>. You shouldn't have to do this now.

</div>

==== Update workflow ====

This depends on FIDO having updated PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-current.json</code>
# Make a new migration (you can copy from a previous one) and update it accordingly
#* E.g. <code>cp src/dashboard/src/fpr/migrations/0022_pronom_94.py src/dashboard/src/fpr/migrations/0032_pronom_96.py</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python src/dashboard/src/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v96.xml --output-filename pronom96.txt</code>
# Copy the output into the blank migration above the Migration class. (Note: This is temporary, to create the data inside the FPR for the analyst steps below)
# Make sure the below RunPython operation is in the Migration class, in the operations list
# Deploy on testing pipeline or locally

(Analyst work)
# Update the new entries. Edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly!
#* Move new formats to the most appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analyst work)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python src/dashboard/src/manage.py get_fpr_changes fpr-current.json fpr-updated.json pronom_96.json</code>
# Update the migration to load the JSON updates (see previous migrations)
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove any direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Rebuild and test migration
# Commit, send PR, merge

=== Finally, update IDTools versions in the FPR ===
# FPR needs a migration to point to the latest and accurate versions of Fido and Siegfried, and disables the previous version of Siegfried (Fido is disabled by default. As of 1.9, there can only be one enabled identification tool). See this PR for an example of a functional migration, and heed the messy commits as a warning: https://github.com/artefactual/archivematica/pull/1547/files
# Testing the above migration can be done by running <code>make bootstrap-dashboard-db</code> to recreate the dashboard and run all associated migrations.

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via GitHub (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / GitLab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and GitLab and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

Post-release cleanup: remove any temporary VMs created for testing.

[[Category:Process documentation]]

Archivematica 1.10.2

2020-05-20T19:23:48Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.10.2

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Archivematica 1.9.3

2020-05-20T19:16:57Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.9.3

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Archivematica 1.10.2

2020-05-20T19:16:39Z

Sallain:

Archivematica 1.11.1

2020-05-20T19:16:26Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.1

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Release Notes

2020-05-20T19:07:56Z

Sallain:

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]] (Current release)
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.9.3

2020-05-20T19:05:10Z

Sallain: Created page with "Home > Release Notes > Archivematica 1.9.3 '''Release date: 20 May 2020''' This release fixes a critical security issue found in the Archivem..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.9.3

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Archivematica 1.11.1

2020-05-20T19:03:59Z

Sallain:

Archivematica 1.11.1

2020-05-20T19:03:43Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.1

'''Release date: 20 May 2020'''

These release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

==Fixed==

* [https://github.com/archivematica/Issues/milestone/14 1.11.1 milestone]

Archivematica 1.10.2

2020-05-20T19:03:08Z

Sallain: Created page with "Home > Release Notes > Archivematica 1.10.2 '''Release date: 20 May 2020''' This release fixes a critical security issue found in the Archive..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.10.2

'''Release date: 20 May 2020'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

==Fixed==

* [https://github.com/archivematica/Issues/milestone/15 1.10.2 milestone]

Archivematica 1.11.1

2020-05-20T18:56:51Z

Sallain:

Archivematica 1.11.1

2020-05-20T18:56:34Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.1

'''Release date: 20 May 2020'''

These release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

==Fixed==

* [1.11.1 milestone https://github.com/artefactual/archivematica/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.11.1]

Archivematica 1.11.1

2020-05-20T18:56:22Z

Sallain: Created page with "Home > Release Notes > Archivematica 1.11.1 and Storage Service 0.15.1 release notes '''Release date: 20 May 2020''' These release fixes a cr..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.1 and Storage Service 0.15.1 release notes

'''Release date: 20 May 2020'''

These release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from our stable branches: stable/1.9.x, stable/1.10.x or stable/1.11.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

==Fixed==

* [1.11.1 milestone https://github.com/artefactual/archivematica/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.11.1]

Release Process

2020-05-05T16:57:13Z

Sallain: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python setup.py install</code>
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Replace:
#* the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* the reference path to the container signatures in <code>fido/fido.py</code>
# Update version:
#* Update in __init__.py
#* Update in versions.xml
# Create pull request.
# Release new version of FIDO.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

When a new Siegfried version becomes available, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo and update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

==== Examples ====
<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55

There used to be a [https://projects.artefactual.com/issues/10466#change-46673 bug] in the imports model. To fix, Remove the <code>apps.get_model</code> lines and Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>. You shouldn't have to do this now.

</div>

==== Update workflow ====

This depends on FIDO having updated PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-current.json</code>
# Make a new migration (you can copy from a previous one) and update it accordingly
#* E.g. <code>cp src/dashboard/src/fpr/migrations/0022_pronom_94.py src/dashboard/src/fpr/migrations/0032_pronom_96.py</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python src/dashboard/src/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v96.xml --output-filename pronom96.txt</code>
# Copy the output into the blank migration above the Migration class. (Note: This is temporary, to create the data inside the FPR for the analyst steps below)
# Make sure the below RunPython operation is in the Migration class, in the operations list
# Deploy on testing pipeline or locally

(Analyst work)
# Update the new entries. Edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly!
#* Move new formats to the most appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analyst work)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python src/dashboard/src/manage.py get_fpr_changes fpr-current.json fpr-updated.json pronom_96.json</code>
# Update the migration to load the JSON updates (see previous migrations)
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove any direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Rebuild and test migration
# Commit, send PR, merge

=== Finally, update IDTools versions in the FPR ===
# FPR needs a migration to point to the latest and accurate versions of Fido and Siegfried, and disables the previous version of Siegfried (Fido is disabled by default. As of 1.9, there can only be one enabled identification tool). See this PR for an example of a functional migration, and heed the messy commits as a warning: https://github.com/artefactual/archivematica/pull/1547/files
# Testing the above migration can be done by running <code>make bootstrap-dashboard-db</code> to recreate the dashboard and run all associated migrations.

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via GitHub (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / GitLab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and GitLab and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Release Process

2020-05-05T16:55:37Z

Sallain: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python setup.py install</code>
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Replace:
#* the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* the reference path to the container signatures in <code>fido/fido.py</code>
# Update version:
#* Update in __init__.py
#* Update in versions.xml
# Create pull request.
# Release new version of FIDO.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

When a new Siegfried version becomes available, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo and update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

==== Examples ====
<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55

There used to be a [https://projects.artefactual.com/issues/10466#change-46673 bug] in the imports model. To fix, Remove the <code>apps.get_model</code> lines and Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>. You shouldn't have to do this now.

</div>

==== Update workflow ====

This depends on FIDO having updated PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-current.json</code>
# Make a new migration (you can copy from a previous one) and update it accordingly
#* E.g. <code>cp src/dashboard/src/fpr/migrations/0022_pronom_94.py src/dashboard/src/fpr/migrations/0032_pronom_96.py</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python src/dashboard/src/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v96.xml --output-filename pronom96.txt</code>
# Copy the output into the blank migration above the Migration class. (Note: This is temporary, to create the data inside the FPR for the analyst steps below)
# Make sure the below RunPython operation is in the Migration class, in the operations list
# Deploy on testing pipeline or locally

(Analyst work)
# Update the new entries. Edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly!
#* Move new formats to the most appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analyst work)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python src/dashboard/src/manage.py get_fpr_changes fpr-current.json fpr-updated.json pronom_96.json</code>
# Update the migration to load the JSON updates (see previous migrations)
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove any direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Rebuild and test migration
# Commit, send PR, merge

=== Finally, update IDTools versions in the FPR ===
# FPR needs a migration to point to the latest and accurate versions of Fido and Siegfried, and disables the previous version of Siegfried (Fido is disabled by default. As of 1.9, there can only be one enabled identification tool). See this PR for an example of a functional migration, and heed the messy commits as a warning: https://github.com/artefactual/archivematica/pull/1547/files
# Testing the above migration can be done by running <code>make bootstrap-dashboard-db</code> to recreate the dashboard and run all associated migrations.

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via GitHub (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in GitLab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and GitLab and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Release Process

2020-05-05T16:51:50Z

Sallain: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python setup.py install</code>
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Replace:
#* the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* the reference path to the container signatures in <code>fido/fido.py</code>
# Update version:
#* Update in __init__.py
#* Update in versions.xml
# Create pull request.
# Release new version of FIDO.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

When a new Siegfried version becomes available, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo and update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

==== Examples ====
<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55

There used to be a [https://projects.artefactual.com/issues/10466#change-46673 bug] in the imports model. To fix, Remove the <code>apps.get_model</code> lines and Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>. You shouldn't have to do this now.

</div>

==== Update workflow ====

This depends on FIDO having updated PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-current.json</code>
# Make a new migration (you can copy from a previous one) and update it accordingly
#* E.g. <code>cp src/dashboard/src/fpr/migrations/0022_pronom_94.py src/dashboard/src/fpr/migrations/0032_pronom_96.py</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python src/dashboard/src/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v96.xml --output-filename pronom96.txt</code>
# Copy the output into the blank migration above the Migration class. (Note: This is temporary, to create the data inside the FPR for the analyst steps below)
# Make sure the below RunPython operation is in the Migration class, in the operations list
# Deploy on testing pipeline or locally

(Analyst work)
# Update the new entries. Edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly!
#* Move new formats to the most appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analyst work)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python src/dashboard/src/manage.py get_fpr_changes fpr-current.json fpr-updated.json pronom_96.json</code>
# Update the migration to load the JSON updates (see previous migrations)
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove any direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Rebuild and test migration
# Commit, send PR, merge

=== Finally, update IDTools versions in the FPR ===
# FPR needs a migration to point to the latest and accurate versions of Fido and Siegfried, and disables the previous version of Siegfried (Fido is disabled by default. As of 1.9, there can only be one enabled identification tool). See this PR for an example of a functional migration, and heed the messy commits as a warning: https://github.com/artefactual/archivematica/pull/1547/files
# Testing the above migration can be done by running <code>make bootstrap-dashboard-db</code> to recreate the dashboard and run all associated migrations.

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via GitHub (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Release Process

2020-05-05T16:51:36Z

Sallain: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python setup.py install</code>
#* Run <code>python -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Replace:
#* the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* the reference path to the container signatures in <code>fido/fido.py</code>
# Update version:
#* Update in __init__.py
#* Update in versions.xml
# Create pull request.
# Release new version of FIDO.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

When a new Siegfried version becomes available, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo and update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

==== Examples ====
<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55

There used to be a [https://projects.artefactual.com/issues/10466#change-46673 bug] in the imports model. To fix, Remove the <code>apps.get_model</code> lines and Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code>. You shouldn't have to do this now.

</div>

==== Update workflow ====

This depends on FIDO having updated PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-current.json</code>
# Make a new migration (you can copy from a previous one) and update it accordingly
#* E.g. <code>cp src/dashboard/src/fpr/migrations/0022_pronom_94.py src/dashboard/src/fpr/migrations/0032_pronom_96.py</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python src/dashboard/src/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v96.xml --output-filename pronom96.txt</code>
# Copy the output into the blank migration above the Migration class. (Note: This is temporary, to create the data inside the FPR for the analyst steps below)
# Make sure the below RunPython operation is in the Migration class, in the operations list
# Deploy on testing pipeline or locally

(Analyst work)
# Update the new entries. Edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly!
#* Move new formats to the most appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analyst work)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python src/dashboard/src/manage.py dumpdata fpr > tmp/fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python src/dashboard/src/manage.py get_fpr_changes fpr-current.json fpr-updated.json pronom_96.json</code>
# Update the migration to load the JSON updates (see previous migrations)
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove any direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Rebuild and test migration
# Commit, send PR, merge

=== Finally, update IDTools versions in the FPR ===
# FPR needs a migration to point to the latest and accurate versions of Fido and Siegfried, and disables the previous version of Siegfried (Fido is disabled by default. As of 1.9, there can only be one enabled identification tool). See this PR for an example of a functional migration, and heed the messy commits as a warning: https://github.com/artefactual/archivematica/pull/1547/files
# Testing the above migration can be done by running <code>make bootstrap-dashboard-db</code> to recreate the dashboard and run all associated migrations.

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via GitHub (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin (for now; hopefully Jenkins for 1.8) builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Developer, Ops, or Analyst, creates a release branch, and release, of [https://github.com/artefactual-labs/archivematica-acceptance-tests Archivematica Automated Acceptance tests] (AMAUAT) in-line with the Archivematica versioning, e.g. for Archivematica 1.10 release a 1.10 branch and [https://github.com/artefactual-labs/archivematica-acceptance-tests/releases release] of AMAUAT.

'''Step 11''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 12''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 13''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 14''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T22:03:00Z

Sallain: /* Add next disk image button */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''April 1, 2020'''

==Environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

<blockquote style="background-color:orange;">
'''Note: Packages for CentOS/RHPM are available now. Ubuntu packages will be available soon. Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions] for more information.'''
</blockquote>

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

==="Add next" disk image transfer button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T22:02:24Z

Sallain:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''April 1, 2020'''

==Environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

<blockquote style="background-color:orange;">
'''Note: Packages for CentOS/RHPM are available now. Ubuntu packages will be available soon. Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions] for more information.'''
</blockquote>

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T21:13:02Z

Sallain: /* Supported environments */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

<blockquote style="background-color:orange;">
'''Note: Packages for CentOS/RHPM are available. Ubuntu packages will be available soon. Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions] for more information.'''
</blockquote>

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T20:59:35Z

Sallain: /* End of life dependencies */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': Artefactual is not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T20:59:19Z

Sallain: /* End of life dependencies */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''': We were not able to test some storage integrations, including Sword2, LOCKSS-o-matic and DSpace. If you can test these storage integrations and find any issues, please consider [https://github.com/archivematica/Issues/issues filing an issue].
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-04-01T20:57:46Z

Sallain: /* End of life dependencies */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team and a number of community contributors have been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''' We were not able to test some storage spaces, including Sword2, LOCKSS-o-matic and DSpace.
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-03-31T20:58:34Z

Sallain: /* Supported environments */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team along with a number of community contributors has been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''' We were not able to test some storage spaces, including Sword2, LOCKSS-o-matic and DSpace.
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Getting started

2020-02-11T22:51:31Z

Sallain:

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Getting Started

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

This wiki page describes getting started with Archivematica as a developer. For user and administrative manuals, please see http://www.archivematica.org.

== Vital Stats ==

* Language: Python (primarily)
* License: [https://en.wikipedia.org/wiki/Affero_General_Public_License AGPL]
* VCS: git
* Major libraries: [https://www.djangoproject.com/ Django], [http://gearman.org/ gearman] ([https://pythonhosted.org/gearman/ Python API])
* [[Contribute_code|Contribution guidelines]]
** [[Contribute_code#Code_Style_Guide_For_Archivematica|Coding style]]

== Projects ==

Archivematica consists of several projects working together.

* [https://github.com/artefactual/archivematica Archivematica]: Main repository containing the user-facing dashboard, task manager MCPServer and clients scripts for the MCPClient
* [https://github.com/artefactual/archivematica-storage-service Storage Service]: Responsible for moving files to Archivematica for processing, and from Archivematica into storage

There are also several smaller repositories that support Archivematica in various ways. In general, you will not need these to develop on Archivematica.

* [https://github.com/artefactual/archivematica-devtools Development tools]: Scripts to help with development. E.g. restarting services, workflow analysis
* [https://github.com/artefactual/archivematica-fpr-tools FPR tools]: All the tools, commands and rules used to populate the FPR database. Changes to the FPR should be submitted here.
* [https://github.com/artefactual/archivematica-docs Archivematica Documentation]: Documentation found at https://www.archivematica.org/en/docs/ for Archivematica
* [https://github.com/artefactual/archivematica-storage-service-docs Storage Service Documentation]: Documentation found at https://www.archivematica.org/en/docs/ for the Storage Service
* [https://github.com/artefactual/automation-tools Automation Tools]: Scripts used to automate processing material through Archivematica
* [https://github.com/artefactual/deploy-pub Deployment]: Ansible scripts for deploying and configuring Archivematica
* [https://github.com/artefactual-labs/ansible-archivematica Deployment-Archivematica]: Ansible playbook for Archivematica package install.
* [https://github.com/artefactual-labs/ansible-role-archivematica-src Deployment-Archivematica-dev]: Ansible playbook for Archivematica github install.
* [https://github.com/artefactual/fixity Fixity checker]: Commandline tool that assists in checking fixity for AIPs stored in Archivematica Storage Service instances.
* [https://github.com/artefactual-labs/mets-reader-writer METS reader/writer]: Library to create and parse METS files.
* [https://github.com/artefactual-labs/agentarchives agentarchives]: Clients to retrieve, add, and modify records from archival management systems.
* [https://github.com/artefactual/archivematica-sampledata Sample data]: Data to test and show off Archivematica's processing
* [https://github.com/artefactual/archivematica-history History]: Contains the pre-git history of Archivematica. Useful for checking the origins of code.

== Installation ==

The recommended way to install Archivematica for development is with Ansible and Vagrant.

Alternatively, you can try our environment based on Docker Compose - see https://github.com/artefactual-labs/am/tree/master/compose.

=== Ansible & Vagrant ===

The following instructions detail how to install and run Archivematica from source on a virtual machine.
# Install VirtualBox, Vagrant, and Ansible with the following commands:
#* <code>sudo apt-get install virtualbox vagrant</code> (this is the command for Ubuntu; if you use Mac or a different Linux distribution, it may be slightly different).
#** Note: Vagrant must be at least 1.5 (it can also be downloaded from [https://www.vagrantup.com/downloads.html vagrantup.com]). Check your version with <code>vagrant --version</code>.
#* <code>sudo pip install -U ansible</code>
# Checkout the deployment repo:
#* <code>git clone https://github.com/artefactual/deploy-pub.git</code>
# Download the Ansible roles:
#* <code>cd deploy-pub/playbooks/archivematica</code>
#* <code>ansible-galaxy install -f -p roles/ -r requirements.yml</code>
# (Optional) Change the branch by opening the file <code>vars-singlenode.yml</code> and modifying the following:
#* <code>archivematica_src_am_version: "branch-name"</code>
#* <code>archivematica_src_ss_version: "branch-name"</code>
# Create the virtual machine and provision it:
#* <code>vagrant up</code> (it takes a while!)
# You can now log in to your virtual machine:
#* <code>vagrant ssh</code>
# You can now access the following services in a web browser:
#* Archivematica - http://192.168.168.192
#* Archivematica Storage Service: http://192.168.168.192:8000

You may also wish to do the following.
# Provisioning (via ansible) can be re-run with vagrant to update the code on the server (for example, if new features are added):
#* <code>vagrant provision</code>
# To re-deploy a new branch to the same VM, update the branch variables in <code>vars-singlenode.yml</code>:
#* <code>archivematica_src_am_version: "branch-name"</code>
#* <code>archivematica_src_ss_version: "branch-name"</code>
#** This will probably require resetting the Archivematica installation as well. This can be done by adding variables to <code>vars-singlenode.yml</code>
#* <code>archivematica_src_reset_am_all: "true"</code> This will reset the Archivematica database, clear ElasticSearch and clear shared directories
#* <code>archivematica_src_reset_ss_db: "true"</code> This will reset the Storage Service database
#* For more variables to control deployment, see the [https://github.com/artefactual-labs/ansible-role-archivematica-src/blob/master/README.md README]
# See also the FAQ below

=== Alternative Vagrant projects ===

Community-provided alternatives have also been developed.

*https://github.com/emltech/eml-archivematica-vagrant
*https://github.com/statsbiblioteket/archivematica-vagrant

== Tests ==

Archivematica and the related projects have a small but growing test suite. We use [http://pytest.org/ py.test] to run our tests, which should be listed as a requirement in the development/local requirements file.

To run the tests, go to the repository root and run <code>py.test</code>

See below for project-specific setup or changes to running the tests.

=== Archivematica ===

Before running Archivematica tests, set the following environment variable. Archivematica does not currently have a virtualenv that needs to be activated.

<pre>
#!/usr/bin/fish
set -xg PYTHONPATH /usr/share/archivematica/dashboard/:/usr/lib/archivematica/archivematicaCommon/
</pre>

<pre>
#!/usr/bin/bash
export PYTHONPATH=$PYTHONPATH:/usr/share/archivematica/dashboard/:/usr/lib/archivematica/archivematicaCommon/
</pre>

=== Storage Service ===

Before running Storage Service tests, activate the virtualenv and set the following environment variables. The tests should be run from the <code>storage_service</code> directory. This is the same directory that contains <code>manage.py</code>. You may need to install <code>requirements/test.txt</code> to install testing dependencies.

<pre>
#!/usr/bin/fish
set -xg PYTHONPATH (pwd)/storage_service # This directory contains manage.py
set -xg DJANGO_SETTINGS_MODULE storage_service.settings.test
set -xg DJANGO_SECRET_KEY 'ADDKEY'
</pre>
<pre>
#!/usr/bin/bash
export PYTHONPATH=$(pwd)/storage_service # This directory contains manage.py
export DJANGO_SETTINGS_MODULE=storage_service.settings.test
export DJANGO_SECRET_KEY='ADDKEY'
</pre>

== FAQ ==

=== How do I restart everything in Archivematica? ===

A default install using Ansible also installs the devtools. Run <code>am restart-services</code> to restart all services related to Archivematica and the storage service.

=== How do I restart just the dashboard? ===

<code>sudo /etc/init.d/apache2 start</code> or <code>sudo service apache2 restart</code>

=== How do I restart the storage service? ===

<code>sudo service uwsgi restart</code>

=== How do I restart nginx? ===

<code>sudo service uwsgi restart</code>
<code>sudo service nginx restart</code>

=== How do I update or reset an ansible install? ===

To update an install, re-run <code>vagrant provision</code>. If you only want to run part of the ansible tasks, you can use ansible's tags, for example: <code>env ANSIBLE_ARGS="--tags=amsrc-ss-code" vagrant provision</code> More tags are documented in the [https://github.com/artefactual-labs/ansible-role-archivematica-src ansible repo]

To reset an install (delete all existing data like a fresh install) you can use ansible's role variables. For example: <code>env ANSIBLE_ARGS="--extra-vars=archivematica_src_reset_ss_db=true" vagrant provision</code> will reset the storage service database. More role variables are documented in the [https://github.com/artefactual-labs/ansible-role-archivematica-src ansible repo]

The other way to control a deployment is to modify the Vagrantfile and vars-singlenote.yml files directly. Tags can be provided in the Vagrantfile. For example:
<pre>
# ... more above
# Ansible provisioning
config.vm.provision :ansible do |ansible|
ansible.playbook = "./singlenode.yml"
ansible.host_key_checking = false
# ansible.verbose = "v"
ansible.extra_vars = {
"archivematica_src_dir" => "/srv",
"archivematica_src_environment_type" => "development",
}
ansible.raw_arguments = ENV['ANSIBLE_ARGS']
ansible.tags = ['amsrc-pipeline']
end
# ... more below
</pre>

Role variables can be modified in vars-singlenote.yml. Default values are found in the Archivematica role (in <code>roles/archivematica-src/defaults/main.yml</code>). For example:

<pre>
---

# archivematica-src role

# What to install
archivematica_src_install_devtools: "yes"
archivematica_src_install_automationtools: "yes"
# archivematica_src_install_appraisaltab: "yes"

# SS django environment variables
archivematica_src_ss_env_django_setings_module: "storage_service.settings.local"

# Branches,
archivematica_src_am_version: "qa/1.x"
archivematica_src_ss_version: "qa/0.x"
# archivematica_src_devtools_version: "master"
# archivematica_src_automationtools_version: "master"

# Reset
# archivematica_src_reset_mcpdb: "true"
# archivematica_src_reset_shareddir: "true"
# archivematica_src_reset_es: "true"
archivematica_src_reset_am_all: "true"
archivematica_src_reset_ss_db: "true"

</pre>

=== How do I activate the Storage Service virtualenv? ===

<code>source /usr/share/python/archivematica-storage-service/bin/activate</code>

The location of the virtualenv is configured as part of the ansible install, which by default is <code>/usr/share/python/archivematica-storage-service</code> Sourcing the activate script should modify the prompt to display <code>(archivematica-storage-service)vagrant@am-local</code>. Note that if you're running fish, the activate script will not work; you may want to look in to [http://virtualfish.readthedocs.io/en/latest/ virtualfish].

You will also want to set environment variables as described in the storage service testing section.

== Documentation ==

Developer facing documentation can be found in the [[:Category:Development documentation|Development documentation]] category. Notable pages include:

* [[MCPServer]]
* [[MCPClient]]
* [[Storage Service]]
* [[Storage Service API]]
* [[Archivematica API]]

[[Category:Development documentation]]

UML Activity Diagrams

2020-02-11T22:36:19Z

Sallain:

[[Main Page]] > [[Requirements]] > UML Activity Diagrams

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

These Archivematica [http://en.wikipedia.org/wiki/Unified_Modeling_Language UML] Activity Diagrams are the baseline requirements for Archivematica.

Wherever possible, these activities are assigned to software or technical tools. If it is not possible to automate these functions in the current system iteration, the functions are incorporated into a manual procedure to be carried out by the end user. This ensures that the entire set of requirements are being carried out in the system, which is an integrated whole of software, people and procedures. The goal is to improve the level of automation and sophistication with each system iteration.

These diagrams are revisions of the [[OAIS Activity Diagrams]] which, in turn, were derived from the [[OAIS Use Cases]] analysis. These revisions were based on a scenario analysis that tried to focus on the pragmatic steps necessary to accomplish each core OAIS function using existing technology and archival business processes.

Each activity is likely to generate some metadata and have one or more information entities as an input and output. However, these will be captured in a separate metadata requirement analysis and information model. Likewise, each activity may be governed or restricted by one or more policy or procedures but these are also intentionally left out of the diagram for simplicity. Except where they are explicitly noted, these diagrams assume that there no errors in the activity flow, i.e. these are 'sunny-day' scenarios.

==Ingest==
* AD1 Receive SIP ([[:File:Archivematica AD1 ReceiveSIP v1.pdf|PDF]]) ([[:File:Archivematica AD1 ReceiveSIP v1.odg|OpenDocument]])
* AD2 Audit SIP ([[:File:Archivematica AD2 AuditSIP v5.pdf|PDF]]) ([[:File:Archivematica AD2 AuditSIP v5.odg|OpenDocument]])
* AD3 Accept SIP for Ingest ([[:File:Archivematica AD3 AcceptSIPforIngest v4.pdf|PDF]]) ([[:File:Archivematica AD3 AcceptSIPforIngest v4.odg|OpenDocument]])
* AD4 Generate AIP ([[:File:Archivematica AD4 GenerateAIP v2.pdf|PDF]]) ([[:File:Archivematica AD4 GenerateAIP v2.odg|OpenDocument]])
* AD5 Transfer AIP to Archival Storage ([[:File:Archivematica AD5 TransferAIPtoArchivalStorage v3.pdf|PDF]]) ([[:File:Archivematica AD5 TransferAIPtoArchivalStorage v3.odg|OpenDocument]])

==Archival Storage==
* AD6 Receive AIP ([[:File:Archivematica AD6 ReceiveAIP v3.pdf|PDF]]) ([[:File:Archivematica AD6 ReceiveAIP v3.odg|OpenDocument]])
* AD7 Store AIP ([[:File:Archivematica AD7 StoreAIP v1.pdf|PDF]]) ([[:File:Archivematica AD7 StoreAIP v1.odg|OpenDocument]])
* AD8 Provide AIP ([[:File:Archivematica AD8 ProvideAIP v1.pdf|PDF]]) ([[:File:Archivematica AD8 ProvideAIP v1.odg|OpenDocument]])
* AD9 Recover AIP ([[:File:Archivematica AD9 RecoverAIP v1.pdf|PDF]]) ([[:File:Archivematica AD9 RecoverAIP v1.odg|OpenDocument]])

==Data Management==
* AD10 Update Database ([[:File:Archivematica AD10 UpdateDatabase v1.pdf|PDF]]) ([[:File:Archivematica AD10 UpdateDatabase v1.odg|OpenDocument]])
* AD11 Query Database ([[:File:Archivematica AD11 QueryDatabase v1.pdf|PDF]]) ([[:File:Archivematica AD11 QueryDatabase v1.odg|OpenDocument]])

==Access==
* AD12 Request DIP ([[:File:Archivematica AD12 RequestDIP v1.pdf|PDF]]) ([[:File:Archivematica AD12 RequestDIP v1.odg|OpenDocument]])
* AD13 Deliver DIP ([[:File:Archivematica AD13 DeliverDIP v1.pdf|PDF]]) ([[:File:Archivematica AD13 DeliverDIP v1.odg|OpenDocument]])
* AD14 Provide Assistance ([[:File:Archivematica AD14 ProvideAssistance v1.pdf|PDF]]) ([[:File:Archivematica AD14 ProvideAssistance v1.odg|OpenDocument]])
* AD15 Collect Feedback ([[:File:Archivematica_AD15_CollectFeedback_v1.pdf|PDF]]) ([[:File:Archivematica_AD15_CollectFeedback_v1.ODG|OpenDocument]])

==Administration==
* AD16 Update Archival Information ([[:File:Archivematica_AD16_UpdateArchivalInformation_v1.pdf|PDF]]) ([[:File:Archivematica_AD16_UpdateArchivalInformation_v1.ODG|OpenDocument]])
* AD17 Activate Requests ([[:File:Archivematica_AD17_ActivateRequests_v1.pdf|PDF]]) ([[:File:Archivematica_AD17_ActivateRequests_v1.ODG|OpenDocument]])
* AD18 Monitor Data Submission Schedule ([[:File:Archivematica_AD18_MonitorDataSubmissionSchedule_v1.pdf|PDF]]) ([[:File:Archivematica_AD18_MonitorDataSubmissionSchedule_v1.ODG|OpenDocument]])
* AD19 Negotiate Submission Agreement ([[:File:Archivematica_AD19_NegotiateSubmissionAgreement_v1.pdf|PDF]]) ([[:File:Archivematica_AD19_NegotiateSubmissionAgreement_v1.ODG|OpenDocument]])

__NOTOC__

Micro-services

2020-02-11T22:36:03Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > Micro-services

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

[[File:Micro-service.png|320px|thumb|right]]

Archivematica implements a [http://www.cdlib.org/services/uc3/curation/ micro-service] approach to digital preservation. The Archivematica micro-services are granular system tasks which operate on a conceptual entity that is equivalent to an OAIS information package: Submission Information Package (SIP), Archival Information Package (AIP), Dissemination Information Package (DIP). The physical structure of an information package will include files, checksums, logs, submission documentation, XML metadata, etc..

These information packages are processed using a series of micro-services. Micro-services are provided by a combination of Archivematica Python scripts and one or more of the free, open-source [[External tools|software tools]] bundled in the Archivematica system. Each micro-service results in a success or error state and the information package is processed accordingly by the next micro-service. There are a variety of mechanisms used to connect the various micro-services together into complex, custom workflows. Resulting in a complete ingest to access system.

Archivematica implements a [[Micro-services#Archivematica_Micro-services|default ingest to access workflow]] that is [[Requirements|compliant with the ISO-OAIS]] functional model. Micro-services can be distributed to processing clusters for highly scalable configurations.

==Current Micro-services in Archivematica==

A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).

</br>

[[Image:Micro-services1.png|600px|center|thumb|Archivematica dashboard showing a micro-service and its jobs]]

</br>

The table below shows micro-services and jobs in the current release of Archivematica ('''Updated July 2015 for release 1.4.1''').

</br>
===Transfer===
{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:30%"|'''Micro-service'''
!style="width:70%"|'''Description'''
|-
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve Transfer'''<div class="mw-collapsible-content">
</div>
</div>
| This is the approval step that moves the transfer into the Archivematica processing pipeline.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Move to processing directory
<br/>Set transfer type: (Standard, Zipped bag, Unzipped bag, DSpace, Maildir)
<br/>Remove hidden files and directories
<br/>Remove unneeded files
<br/>Attempt restructure for compliance
<br/>Verify transfer compliance
<br/>Verify mets_structmap.xml compliance
</div>
</div>
| Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename with transfer UUID'''<div class="mw-collapsible-content">
<br/>Rename with transfer UUID
</div>
</div>
| Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default Transfer processingMCP.xml'''<div class="mw-collapsible-content">
<br/>Include default Transfer processingMCP.xml
</div>
</div>
| Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Assign file UUIDs and checksums'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Assign file UUIDs to objects
<br/>Assign checksums and file sizes to objects
</div>
</div>
| Assigns a unique universal identifier and sha-256 checksum to each file in the ''/objects/'' directory and sets file permission to allow for continued processing.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer checksums'''<div class="mw-collapsible-content">
<br/>Verify metadata directory checksums
</div>
</div>
| Checks any checksum files that were placed in the ''/metadata/'' folder of the transfer prior to moving the transfer into Archivematica.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate METS.xml document'''<div class="mw-collapsible-content">
<br/>Generate METS.xml document
</div>
</div>
| Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the ''/objects/'' directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Reformat metadata files'''<div class="mw-collapsible-content">
<br/>Process transfer JSON metadata
</div>
</div>
| Transforms JSON metadata included with transfer to metadata.csv
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Quarantine'''<div class="mw-collapsible-content">
<br/>Workflow decision - send transfer to quarantine
<br/>Move to quarantine
<br/>Remove from quarantine
</div>
</div>
| Quarantines the transfer for a set duration, to allow virus definitions to update, before virus scan.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content">
<br/>Scan for viruses
</div>
</div>
| Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in ''/sharedDirectoryStructure/failed/'' and all processing on the transfer is stopped.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate transfer structure report'''<div class="mw-collapsible-content">
<br/>Generate transfer structure report
<br/>Move to generate transfer tree
</div>
</div>
| Uses Tree to create transfer structure report (original order report) in txt format
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content">
<br/>Sanitize object's file and directory names
<br/>Sanitize Transfer name
</div>
</div>
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Identify file format'''<div class="mw-collapsible-content">
<br/>Move to select file ID tool
<br/>Select file format identification command
<br/>Determine which files to identify
<br/> Identify file format
</div>
</div>
| Identifies formats of the objects in the transfer using either FIDO or file extension based on user choice. Format types are managed in the Format Policy Registry. This micro-service can be skipped and done in Ingest instead.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Extract packages'''<div class="mw-collapsible-content">
<br/>Move to processing directory
<br/>Move to extract packages
<br/>Extract packages? (preconfigured choice)
<br/>Delete package after extraction? (preconfigured choice)
<br/>Extract contents from compressed archives
</div>
</div>
| Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers. User can preconfigure choices about whether to extract and whether to keep one or both of extracted object contents and/or the package itself.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Update METS.xml document'''<div class="mw-collapsible-content">
<br/>Add processed structMap to METS.xml document
</div>
</div>
| Creates a structMap in the transfer METS to reflect structure of transfer
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content">
<br/>Characterize and extract metadata
<br/>Load labels from metadata/file_labels.csv
</div>
</div>
| Characterizes formats and extracts object metadata using File Information Tool Set (FITS), FFprobe, Exiftool and/or Mediainfo
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Validation'''<div class="mw-collapsible-content">
<br/>Validate formats
</div>
</div>
| Validates formats using JHOVE.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Examine Contents'''<div class="mw-collapsible-content">
<br/>Examine contents
</div>
</div>
| Runs Bulk Extractor on transfer contents and creates reports
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content">
<br/>Index transfer contents
<br/>Create transfer metadata XML
<br/>Move to SIP creation directory for completed transfers
</div>
</div>
| Indexes transfer contents, creates transfer METS XML, then marks the transfer as complete.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content">
<br/>Check transfer directory for objects
<br/>Updating transfer file index
<br/>Create placement in backlog PREMIS events
<br/>Move transfer to backlog
<br/>Check transfer directory for objects
<br/>Load options to create SIP
</div>
</div>
| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest) if user chooses to Create single SIP and continue processing. User can also choose to Send transfer to backlog at this time, which creates an associated PREMIS event.
|-
|}
</br>

===Ingest===
{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:30%"|'''Micro-service'''
!style="width:70%"|'''Description'''
|-
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve SIP creation'''<div class="mw-collapsible-content">
<br/>Approve SIP creation
<br/> Create removal from backlog PREMIS events
</div>
</div>
| Applies to SIPs that have been created from backlog. Approves creation of SIP, and creates PREMIS event for removal from backlog.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content">
<br/>Verify mets_structmap.xml compliance
</div>
</div>
| Verifies the METS from the transfer.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify SIP compliance'''<div class="mw-collapsible-content">
<br/>Set file permissions
<br/>Move to processing directory
<br/>Verify SIP compliance
</div>
</div>
| Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content">
<br/>Rename SIP directory with SIP UUID
<br/>Check if SIP is from Maildir Transfer
</div>
</div>
| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name and checks if SIP is from Maildir transfer type to determine workflow.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content">
<br/>Include default SIP processingMCP.xml
</div>
</div>
| Copies the processing configuration file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content">
<br/>Remove cache files
</div>
</div>
| Removes any thumbs.db files.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content">
<br/>Load Dublin Core metadata from disk
<br/>Sanitize SIP name
<br/>Set file permissions
</div>
</div>
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content">
<br/>Identify manually normalized files
<br/>Check for Service directory
<br/>Check for Access directory
<br/>Set remove preservation and access normalized files to renormalize link.
<br/>Grant normalization options for no pre-existing DIP
<br/>Move to workFlowDecisions-createDip directory
<br/>Find options to normalize as
<br/>Set resume link after tool selected
<br/>Move to select file ID tool
<br/>Select pre-normalize file format identification commant
<br/>Identify file format
<br/>Resume after normalization file identification tool selected
<br/>Normalize
<br/>Move to processing directory
<br/>Create DIP directory
<br/>Create thumbnails directory
<br/>Normalize thumbnails
<br/>Normalize access
<br/>Normalize preservation
<br/>Set file permissions
<br/>Remove files without linking information (failed normalization artifacts etc.)
<br/>Move to approve normalization directory
<br/>Approve normalization
<br/>Load post approve normalization link
<br/>Set resume link after handling any manually normalized files
<br/>Move to processing directory
<br/>Set file permissions
<br/>Load finished with manual normalized link
</div>
</div>
| Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Process manually normalized files'''<div class="mw-collapsible-content">
<br/>Check for manual normalized files
</div>
</div>
| Processed files manually normalized during processing
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Add final metadata'''<div class="mw-collapsible-content">
<br/> Move to metadata reminder
<br/>Reminder: add metadata if desired
<br/>Set file permissions
</div>
</div>
| Provides the operator a reminder to add metadata through the UI if desired.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Transcribe SIP contents'''<div class="mw-collapsible-content">
<br/>Transcribe SIP contents
<br/>Transcribe
</div>
</div>
| Runs Tesseract OCR tool on any JPG or TIFF image files in SIP.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content">
<br/>Copy transfer submission documentation
<br/>Check for submission documentation
<br/>Move submission documentation into objects directory
<br/>Assign file UUIDs to submission documentation
<br/>Assign checksums and file sizes to submissionDocumentation
<br/>Sanitize file and directory names in submission documentation
<br/>Scan for viruses in submission documentation
<br/>Select file format identification command
<br/>Identify file format
<br/>Characterize and extract metadata on submission documentation
<br/>Remove files without linking information (failed normalization artifacts etc.)
</div>
</div>
| Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Process metadata directory'''<div class="mw-collapsible-content">
<br/>Copy transfers metadata and logs
<br/>Process JSON metadata
<br/>Move metadata to objects directory
<br/>Assign file UUIDs to metadata
<br/>Assign checksums and file sizes to metadata
<br/>Sanitize file and directory names in metadata
<br/>Scan for viruses in metadata
<br/>Identify file format of metadata files
<br/>Characterize and extract metadata on metadata files
<br/>Remove empty manual normalization directories
</div>
</div>
| Processes metadata.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify checksums'''<div class="mw-collapsible-content">
<br/>Verify checksums generated on ingest
</div>
</div>
| Verifies checksums generated on ingest
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate AIP METS'''<div class="mw-collapsible-content">
<br/>Generate METS.xml document
</div>
</div>
| Generates Archivematica AIP METS.xml document
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content">
<br/>Copy thumbnails to DIP directory
<br/>Copy METS to DIP directory
<br/>Copy preconfigured choice XML to DIP directory
<br/>Generate DIP
<br/>Copy OCR data to DIP directory
<br/>Set file permissions
</div>
</div>
| Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content">
<br/>Check if DIP should be generated
<br/>Prepare AIP
<br/>Move to compressionAIPDecisions directory
<br/>Select compression algorithm
<br/>Select compression level
<br/>Compress AIP
<br/>Copy submission documentation
<br/>Create AIP pointer file
<br/>Set bag file permissions
<br/>Check if AIP is a file or directory
<br/>Removed bagged files
</div>
</div>
| Creates an AIP in Bagit format. Creates the AIP pointer file. Indexes the AIP, then losslessly compresses it.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP'''<div class="mw-collapsible-content">
<br/>Upload DIP
</div>
</div>
| Allows the user to choose to upload the DIP AtoM, CONTENTdm, Archivist's Toolkit. Also allows user to store DIP or reject DIP.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to AtoM'''<div class="mw-collapsible-content">
<br/>Upload DIP
<br/>Move to the uploadedDIPs directory
</div>
</div>
| The user uploads the DIP to a selected description in AtoM.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to CONTENTdm'''<div class="mw-collapsible-content">
<br/>Upload DIP
<br/>Restructure DIP for CONTENTdm upload
<br/>Move to the uploadedDIPs directory
</div>
</div>
| The user creates a DIP suitable for upload to CONTENTdm using Project Client.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to Archivists' Toolkit'''<div class="mw-collapsible-content">
<br/>
</div>
</div>
| The user uploads the DIP metadata to Archivists' Toolkit
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Store DIP'''<div class="mw-collapsible-content">
<br/>Upload DIP
<br/>Retrieve DIP Storage Locations
<br/>Store DIP location
<br/>Store DIP
<br/>Move to the uploadedDIPs directory
</div>
</div>
| The user stores the DIP in a location pre-configured in the Storage Service.
|-
| <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content">
<br/>Move to the store AIP approval directory
<br/>Store AIP
<br/>Retrieve AIP Storage Locations
<br/>Store AIP location
<br/>Move to processing directory
<br/>Verify AIP
<br/>Store the AIP
<br/>Index AIP
<br/>Clean up after storing AIP
<br/>Remove processing directory
</div>
</div>
| Moves the AIP to ''/sharedDirectoryStructure/www/AIPsStore/'' or another specified directory. Before the AIP has been stored, a copy of it is extracted to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.
|-
|}

=Historical Micro-services in Archivematica Releases=

*[[Archivematica 1.1 Micro-services]]
*[[Archivematica 1.0 Micro-services]]
*[[Archivematica 0.10 Micro-services]]
*[[Archivematica 0.9 Micro-services]]
*[[Archivematica 0.8 Micro-services]]
*[[Archivematica 0.7.1 Micro-services]]

Significant characteristics of word processing files

2020-02-11T22:35:48Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Significant characteristics]] > Significant characteristics of word processing files

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

*"[T]he essential characteristics of a word processing document may include the textual content; formatting such as bolded text, font type and size; layout; bulleting; colour and embedded graphics." [http://www.naa.gov.au/Images/An-approach-Green-Paper_tcm2-888.pdf An Approach to the Preservation of Digital Records, National Archives of Australia, 2002]
*[http://fclaweb.fcla.edu/uploads/Lydia%20Motyka/FDA_documentation/documentMD.pdf Document Metadata: Document Technical Metadata for Digital Preservation, Florida Digital Archive and Harvard University Library, 2009]: this document suggests technical metadata for textual records which "can be used to verify the result of document transformations, ensuring the properties of the original document are preserved and properly transformed to the new document format." The metadata in this table are adapted from that source:

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:10%"|'''Semantic unit'''
!style="width:30%"|'''Description'''
!style="width:10%"|'''Obligation'''
!style="width:20%"|'''Characteristic'''
!style="width:50%"|'''Note'''
|-
|PageCount
|Total number of pages in the document
|Mandatory
|Structure
|
|-
|WordCount
|Total number of words in the document
|Optional
|Structure
|This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element, however, because tools and applications that can determine the number of words in a document do not always use the same algorithm for determining this value.
|-
|CharacterCount
|Total number of characters in the document
|Optional
|Structure
|See note for WordCount, above
|-
|ParagraphCount
|Total number of paragraphs in the document
|Optional
|Structure
|See note for WordCount, above
|-
|LineCount
|Total number of lines in the document
|Optional
|Structure
|See note for WordCount, above
|-
|TableCount
|Total number of tables in the document
|Optional
|Structure
|See note for WordCount, above
|-
|GraphicsCount
|Total number of graphics in the document
|Optional
|Structure
|See note for WordCount, above
|-
|Language
|A language identifier specifying the natural language used in the document
|Optional
|Content
|
|-
|Fonts (FontName, isEmbedded)
|A list of fonts used in the document; An indication of whether or not a font is embedded in a document
|Mandatory
|Content, Appearance
|This element allows a repository to store the names of all fonts used in a document. Some repositories may choose to store only the non-embedded fonts. It is recommended that repositories record at least the non-embedded fonts to assist in identifying the documents with potential long-term preservation risks.
|-
|Features
|Additional document features as follows: hasLayers, hasTransparency, hasOutline, hasForms, has Annotations
|Optional
|
*hasLayers: appearance
*hasTransparency: appearance
*hasOutline: behaviour, appearance
*hasForms: content
*hasAnnotations: content
|
|}

Significant characteristics of websites

2020-02-11T22:35:36Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Significant characteristics]] > Significant characteristics of websites

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

As there is no formal default policy for websites in Archivematica, below is a summary of research done for clients that may inform future policy generation.

===Overview===

The goal of website archiving is to capture, preserve and render complete websites. An end user should be able to navigate the preserved website in the same way that the original website was navigated, and as much as possible should see the same content and experience the same functionality. Website preservation involves a number of steps, each of them requiring their own tools and procedures: capturing ("crawling" or "harvesting") a website, storing it in an archival format, applying preservation planning over time, rendering it, indexing it and providing keyword search capabilities for all of the archived content.

A number of institutions have undertaken website archiving on a large scale. Probably the best-known of these is the [http://www.archive.org/ Internet Archive], founded by Brewster Kahle in 1996. The Internet Archive gathers and makes available a vast number of websites at no charge; it also offers a third party web archiving service called [http://www.archive-it.org/ Archive-It], available for a fee. The Library of Congress has been [http://www.loc.gov/webarchiving/ preserving websites] since 2000, acquiring government and private websites based on selected themes, events and subject areas. [http://webarchives.cdlib.org/ California Digital Library] collects a wide variety of websites and makes them available on-line; like the Internet Archive, it offers a [http://webarchives.cdlib.org/p/projects third party web archiving] service for a fee. Numerous national archives and libraries also have web archiving projects, including the British Library (via the UK Web Archive), Library and Archives Canada, the National Library of New Zealand and the National Archives of Australia. (Lists of major website archiving initiatives are maintained at http://netpreserve.org/about/archiveList.php and http://en.wikipedia.org/wiki/List_of_Web_archiving_initiatives.) International efforts to develop web archiving tools, standards and practices are managed by the [http://netpreserve.org/about/index.php International Internet Preservation Consortium (IIPC)], established in 2003 by the Library of Congress, the Internet Archive and the national libraries of Australia, Canada, Denmark, Finland, France, Iceland, Italy, Norway, Sweden and the UK. (A complete list of current members is at http://netpreserve.org/about/memberList.php)

The tools and processes these large institutions use are scalable downward to smaller institutions. For example, the most popular website crawler, [https://webarchive.jira.com/wiki/display/Heritrix/Heritrix Heritrix], was developed by the Internet Archive and has been released as an open-source tool which can be used by any organization, large or small. Similarly, the [http://www.digitalpreservation.gov/formats/fdd/fdd000236.shtml WARC format], described in detail below, was developed by a consortium of large institutions, has been approved by the International Standards Organization, and can now be used by smaller institutions as their preservation format for harvested websites. A practical and pragmatic approach should be to follow in the footsteps of large institutions which have invested a great deal of time and resources into developing web archiving standards, procedures and tools, since any problems experienced by smaller institutions, now or in the future, will be the same problems being tackled by these large institutions.

===WARC for website preservation===

WARC (Web ARChive file format) is an extension of the ARC (Internet Archive ARC_IA) format which was developed by the Internet Archive in the mid-1990s. The ARC format stores simple content block sequences (representing objects such as html files and images) with additional text headers in a self-contained file. The ARC format captures responses to http requests only; WARC extends this by capturing content types such as assigned metadata and duplicate detection events (to reduce storage of identical resources). WARC also provides expanded metadata support in the text header. WARC was accepted as an international standard in 2009 (ISO 28500:2009) and is the Library of Congress' preferred format for harvested Web sites.1 Although ARC is still widely used, it is being replaced by WARC at a number of leading institutions, including the Internet Archive.

A WARC file stores multiple archived resources in a single file in order to avoid managing a large number of digital objects in numerous directories. A WARC file consists of any number of WARC records, which are single blocks of content, each with its own text header. These text headers can be extracted from the WARC file and stored separately for efficient indexing.

In this diagram, the WARC file represented on the left contains an unlimited number of WARC records: the WARC record contains the content, in this case a jpeg image, and the text header. The sample text header shows the type of resource, the URI of the image, the date of capture, a unique identifier for the WARC record, the content mime-type, checksums for the WARC record and jpeg image, and the content length in number of octets (bytes). (Sample text header content taken from WARC file format version 0.18, 2008-06-06. ISO/DIS 28500, bibnum.bnf.fr/WARC/WARC_ISO_DIS_28500.pdf.)

[[File:WARCdiagram.png|600px|thumb|center|]]

==Preservation planning==

A WARC file can contain many types of content which may require migrating to preservation-friendly formats in the future. A [https://webarchive.jira.com/wiki/display/Heritrix/Heritrix Heritrix] crawl result includes a report called mimetype-report.txt that provides a summary list of formats in the crawl. These summary reports can be used to monitor ingested formats at a high level in order to help inform decisions on at-risk formats and preservation planning. Log files included in the crawl results provide the URIs for each captured file. A preservation workflow for WARC files could consist of extracting content blocks (i.e. files such as images, pdf files etc.) from the WARC file, normalizing selected content based on a format risk assessment, and generating a new WARC file containing normalized content. (See Migrating content in WARC files , Stephan Strodl, Peter Paul Beran and Andreas Rauber, The 9th International Web Archiving Workshop proceedings, 2009, available at http://iwaw.europarchive.org/09/index.html.)

However, if crawled web content consists only of standard, widely-used formats, preservation actions need not be taken immediately. In fact, taking a wait and see approach to preservation planning for WARC files would be highly practical for a small- to medium-sized institution, since a large number of well-established institutions (such as Library of Congress, the Internet Archive and California Digital Library) use WARC and will likely start to put resources into developing preservation strategies and tools when standard web formats become at risk of format obsolescence.

Significant characteristics of video files

2020-02-11T22:35:25Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Significant characteristics]] > Significant characteristics of video files

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

*From [http://www.ahds.ac.uk/preservation/video-preservation-handbook.pdf Arts and Humanities Data Service Preservation Handbook: Moving Image, Gareth Night, 2005]: "For preservation purposes, the significant properties of digital video are the video length (e.g. 90 minutes, 20 seconds), frames per second (25 for PAL, 30 for NTSC), video dimensions (e.g. 720 pixels x 576 pixels), bit-rate (kbps), associated metadata, and file size. If audio is also provided, the audio bit-rate (kbps), frequency (kHz), and number of channels." (p.7)

*From [http://www.jisc.ac.uk/media/documents/programmes/preservation/spmovimages_report.pdf The Significant Properties of Moving Images, Mike Coyne and Mike Stapleton, 2008], adapted from table on pages 61-2. Note that this table excludes properties intrinsic to audio aspects of moving image files:

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:20%"|'''property value'''
!style="width:10%"|'''component'''
!style="width:40%"|'''property definition'''
!style="width:20%"|'''constraint unit'''
!style="width:10%"|'''datatype'''
|-
|imageStreams
|video
|the number of distinct video channels in the digital object
|channels
|positive integer
|-
|audioStreams
|audio
|the number of distinct audio channels in the digital object
|channels
|positive integer
|-
|length
|video
|the number of frames in the recording
|frames
|positive integer
|-
|width
|frame
|width of the image frame
|pixels
|positive integer
|-
|height
|frame
|height of the image frame
|pixels
|positive integer
|-
|bitDepth
|pixel
|the amount of data recorded for each pixel
|bits
|positive integer
|-
|colourModel
|pixel
|the colour model used when recording the pixel value
|enumerated list
|
|-
|colourSpace
|pixel
|the colour space used when recording the pixel value
|enumerated list
|
|-
|pixelAspectRatio
|pixel
|aspect ratio of the pixels making up the image
|ratio
|ratio
|-
|frameRate
|video
|the number of frames to be displayed in a given time interval
|frames per second
|positive integer?
|-
|interlace
|frame
|whether the source was interlaced video
|boolean
|boolean
|-
|metadata
|
|the existence of metadata associated with the digital object
|elements
|
|-
|}

Significant characteristics of vector images

2020-02-11T22:35:13Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Significant characteristics]] > Significant characteristics of vector images

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

The chief source of information for this page is [http://www.jisc.ac.uk/media/documents/programmes/preservation/vector_images.pdf Mike Coyne, David Duce, Bob Hopgood, George Mallen, Mike Stapleton. The Significant Properties of Vector Images. JISC report, 27 November 2007].
*"The hierarchy of the points, lines and areas defining the image are often Significant Properties of the image." (p.7)
*Vector images may include the following features, all of which have significant properties attached to them: points, open paths, closed paths, text, objects, inline objcts. (p.27)

Significant characteristics of email

2020-02-11T22:34:58Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Significant characteristics]] > Significant characteristics of email

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

These tables are adapted from [http://www.significantproperties.org.uk/email-testingreport.html#_Toc254545915 Significant Properties Testing Report: Email], Gareth Knight, Digital Curation Specialist, InSPECT project, 30 March 2009, Sec. 2.1, Significant properties that must be maintained.

====Agents====

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:10%"|'''name'''
!style="width:30%"|'''definition'''
!style="width:30%"|'''function description'''
|-
|Local-part
|The user account of the Agent assigned by a mail provider. The local-part is identified by alphanumeric characters prior to the @ symbol of an email address.
|Establishes the provenance (and as a result support or contradict its authenticity) of message by identifying the user account that was used to transmit the message.
|-
|Domain-part
|The host or domain name used by a DNS to indicate the mail provider that handles the email message.
|Establishes the provenance (and as a result support or contradict its authenticity) of message by identifying the domain from which the message originated.
|-
|Domain-literal
|The IP address of the source or destination domain.
|Establishes the provenance (and as a result support or contradict its authenticity) of message by identifying the machine address from which the message originated.
|-
|Display name (if present)
|A plain text indication of the Agent's name.
|Establishes the provenance (and as a result support or contradict its authenticity) of message by identifying the name of the Agent specified for the mail account.
|-
|Relationship
|The relationship that the Agent has with the email message, e.g. creator, sender, recipient (primary, CC, BCC).
|Establishes the provenance (and as a result support or contradict its authenticity) of message by identifying how each agent relates to the email.
|-
|}

</br>

====Transmission====

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:10%"|'''name'''
!style="width:30%"|'''definition'''
!style="width:30%"|'''function description'''
|-
|Message-ID
|A unique identifier created by the domain from which the email originated that is embedded within the email header. The message-id is found in received emails and is not present in local emails.
|The message-id should be used when attempting to understand the relationship between two or more emails that constitute a thread. It is beneficial when the subject line has changed. However, it has only limited use when handling a single email.
|-
|Sent-date
|The date and time that an email was completed by a Creator and/or transmitted by the Sender, or received by a Recipient.
|The sent-date is obtained from the system settings of the sender ‘s machine. It may indicate the datetime in which an idea was expressed. However, there is the potential that the datetime has been accidentally or deliberately altered, which may result in the value being untrustworthy.
|-
|Received date
|The date and time that an email was received by the recipient‘s host.
|Indicates the datetime that an email was received. However, it does not confirm that the email was downloaded or read by a recipient.
|-
|Trace-field
|Indicates the route that the email took to travel from the sender from the recipient and when it occurred. A repeatable value consisting of an optional “Return-Path” field and one or more “Received” fields.
|The trace fields are external to the control of the sender and recipient and, therefore may be thought more trustworthy than the Sent and Received date for validation.
|-
|}

</br>

====Content and structure====

{| border="1" cellpadding="10" cellspacing="0" width=90%
|-
|- style="background-color:#cccccc;"
!style="width:10%"|'''name'''
!style="width:30%"|'''definition'''
!style="width:30%"|'''function description'''
|-
|-
|Subject
|A short string that may identify the topic of the message. The subject line may be blank, indicate the content of the email to which the Sender is replying, or contain other information.
|The Subject may provide qualitative information that indicates the message purpose. Additionally, it may provide a simple method to sort several emails into a thread when used in conjunction with the received date.
|-
|Keywords (if present)
|Words and phrases that may summarise the content of the message. Keywords may be created by the person or software application that creates, receives the message, or archives the message.
|
|-
|Attachments (if present)
|An identifier that indicates one or more attachments associated with the email.
|
|-
|Message body and associated mark-up
|
|Markup may be necessary to preserve, depending on the type and format of the message. For more information, see [http://www.significantproperties.org.uk/email-testingreport.html#_Toc254545915 Significant Properties Testing Report: Email], 2.2.1.5. Message Body.
|-
|}

Significant characteristics of plain text files

2020-02-11T22:34:47Z

Sallain:

[[Main Page]] > [[Documentation]] > [[Format policies]] > [[Plain text]] > Significant characteristics of plain text files

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

From Florida Digital Archive Action plan, Plain Text (http://www.fcla.edu/digitalArchive/formatInfo.htm):

Plain text exhibits the following significant properties:
*Content: All content, including every character, will be preserved.
*Context: N/A. Plain Text does not contain metadata to describe the environment where it was originally created or used.
*Behavior: N/A. Plain Text does not provide interactive characteristics.
*Structure: The location of every linebreak will be preserved. There is no other predefined structure for Plain Text.
*Appearance: N/A. Plain Text does not store appearance attributes such as font (including font size or font style), color, alignment, etc.