Archivematica - User contributions [en]

Storage Service 0.19 Release Notes

2022-02-25T18:16:56Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

This release...

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users defined in the application database will automatically become administrators when the Django migrations are executed. If your Storage Service is configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Storage Service 0.19 Release Notes

2022-02-25T18:15:27Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: 25 Feb 2022'''

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users in the application database become administrators automatically. If you have Storage Service configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Storage Service 0.19 Release Notes

2022-02-25T18:09:46Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: TBD'''

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users in the application database become administrators automatically. If you have Storage Service configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/stable/0.19.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Storage Service 0.19 Release Notes

2022-02-25T18:09:21Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: TBD'''

== Upgrading ==

* With the new permission module (see [https://github.com/artefactual/archivematica-storage-service/pull/625 pull request #625] for more), existing users in the application database become administrators automatically. If you have Storage Service configured with an external authentication backend, you can use [https://github.com/artefactual/archivematica-storage-service/tree/stable/0.19.x/install SS_AUTH_DEFAULT_USER_ROLE] to establish a new default user role for authenticated users or tweak the [https://github.com/artefactual/archivematica-storage-service/blob/qa/0.x/storage_service/storage_service/settings/base.py authentication backend settings] to map the user roles according to your needs.

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Release Notes

2022-02-24T18:56:57Z

Sevein:

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Security patches ==

* [[Log4j|Log4j vulnerability information]] Updated December 13, 2021

== Archivematica 1.x releases ==

* [[Storage Service 0.19 Release Notes|'''Storage Service 0.19 Release Notes''']] (Upcoming release)
* [[Archivematica_1.13.2|Archivematica 1.13.2 Release Notes]] (Current release)
* [[Archivematica_1.13.1_and_Storage_Service_0.18.1_release_notes|'''Archivematica 1.13.1 and Storage Service 0.18.1 Release Notes''']]
* [[Archivematica_1.13.0_and_Storage_Service_0.18.0_release_notes|'''Archivematica 1.13.0 and Storage Service 0.18.0 Release Notes''']]
* [[Archivematica_1.12.2|Archivematica 1.12.2 Release Notes]]
* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes]]
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']]
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
* [[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
* [[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
* [[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
* [[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Storage Service 0.19 Release Notes

2022-01-15T07:03:44Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

* '''Release date: TBD'''

== Upgrading ==

* '''TODO!''' Describe transition to the new permission module and the use of SS_AUTH_DEFAULT_USER_ROLE="reader", refer to docs (https://github.com/artefactual/archivematica-storage-service/pull/625).

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Storage Service 0.19 Release Notes

2022-01-15T07:03:22Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

**Release date: ?**

== Upgrading ==

* '''TODO!''' Describe transition to the new permission module and the use of SS_AUTH_DEFAULT_USER_ROLE="reader", refer to docs (https://github.com/artefactual/archivematica-storage-service/pull/625).

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Storage Service 0.19 Release Notes

2022-01-15T07:02:55Z

Sevein: Created page with "Home > Release Notes > Storage Service 0.19 release notes **Release date: ?** == Upgrading == * **TODO!** Describe transition to the new per..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Storage Service 0.19 release notes

**Release date: ?**

== Upgrading ==

* **TODO!** Describe transition to the new permission module and the use of SS_AUTH_DEFAULT_USER_ROLE="reader", refer to docs (https://github.com/artefactual/archivematica-storage-service/pull/625).

== Added ==

=== User roles ===

* Issue: https://github.com/archivematica/Issues/issues/1486

Users can be associated to one of the following roles or permission sets: administrators, managers, reviewers and readers. Supported authentication backends like LDAP, Shibboleth or CAS include new configuration attributes to promote authenticated users.

Release Notes

2022-01-15T06:51:54Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Security patches ==

* [[Log4j|Log4j vulnerability information]] Updated December 13, 2021

== Archivematica 1.x releases ==

* [[Storage Service 0.19 Release Notes|'''Storage Service 0.19 Release Notes''']] (Upcoming release)
* [[Archivematica_1.13.2|Archivematica 1.13.2 Release Notes]] (Current release)
* [[Archivematica_1.13.1_and_Storage_Service_0.18.1_release_notes|'''Archivematica 1.13.1 and Storage Service 0.18.1 Release Notes''']] (Current release)
* [[Archivematica_1.13.0_and_Storage_Service_0.18.0_release_notes|'''Archivematica 1.13.0 and Storage Service 0.18.0 Release Notes''']]
* [[Archivematica_1.12.2|Archivematica 1.12.2 Release Notes]]
* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes]]
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']]
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
* [[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
* [[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
* [[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
* [[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.12.2

2021-12-13T16:38:48Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.12.2

'''Release date: 13 Dic 2021'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from the stable branch: stable/1.12.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

Archivematica 1.12.2

2021-12-13T10:30:50Z

Sevein:

Archivematica 1.13.2

2021-12-13T10:30:29Z

Sevein: Created page with "Home > Release Notes > Archivematica 1.13.2 '''Release date: 13 Dic 2021''' This release fixes a critical security issue found in the Archive..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.2

'''Release date: 13 Dic 2021'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from the stable branch: stable/1.13.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

Archivematica 1.12.2

2021-12-13T10:29:39Z

Sevein: Created page with "Home > Release Notes > Archivematica 1.12.2 '''Release date: 13 Dic 2021''' This release fixes a critical security issue found in the Archive..."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.12.2

'''Release date: 13 Dic 2021'''

This release fixes a critical security issue found in the Archivematica dashboard that allows unauthorized users to access some parts of the Administration tab.

This issue was discovered as a result of a security audit by Scholars Portal. It was not discovered as a result of a breach. Scholars Portal reported the issue to Artefactual privately via email. Once we became aware of the issue, we began to develop the fix. Artefactual has also implemented security reporting process documentation across Archivematica-related GitHub repositories and changed issue templates to reflect a more secure process. You can review Archivematica’s security reporting process here: https://github.com/artefactual/archivematica/security/policy.

==Upgrading==

The fix can be easily installed since this issue only affects the dashboard.

CentOS users relying on Archivematica packages should run:

<pre>
sudo yum -y update archivematica-dashboard
sudo systemctl restart archivematica-dashboard
</pre>

Automated installations using Ansible should deploy from the stable branches: stable/1.12.x.

Alternately, a fix can be applied to the web server. The following configuration snippet shows an updated Nginx server block with the additional rule added.

<pre>
server {
listen 80;
client_max_body_size 256M;
server_name _;
location / {
set $upstream_endpoint http://archivematica-dashboard:8000;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_read_timeout 172800s;
proxy_pass $upstream_endpoint;
}

# Directive to block access to admin pages in
# Archivematica v1.11.0 or older.
location ~ ^/administration/accounts/login/.+$ {
return 404;
}
}
</pre>

After the fix has been applied, please be sure to update passwords and API keys:

* Change the password and API key for the Storage Service user:
** In the Storage Service, change the password for the Storage Service user that the Archivematica dashboard uses. This will also regenerate the API key for the Storage Service user.
** In the Archivematica dashboard, under Administration > General, update the Storage Service user password and the API key to reflect the new password/key.
* Change the password for AtoM/Binder DIP upload.
* Review the PREMIS agent information to ensure that it is correct.

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

Release Notes

2021-12-13T10:28:03Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.13.2|Archivematica 1.13.2 Release Notes]] (Current release)
* [[Archivematica_1.13.1_and_Storage_Service_0.18.1_release_notes|'''Archivematica 1.13.1 and Storage Service 0.18.1 Release Notes''']] (Current release)
* [[Archivematica_1.13.0_and_Storage_Service_0.18.0_release_notes|'''Archivematica 1.13.0 and Storage Service 0.18.0 Release Notes''']]
* [[Archivematica_1.12.2|Archivematica 1.12.2 Release Notes]]
* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes]]
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']]
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.13.1 and Storage Service 0.18.1 release notes

2021-10-19T16:12:54Z

Sevein: /* Fixed */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.1 and Storage Service 0.18.1

'''Release date: October 19, 2021'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.1 and Storage Service 0.18.1 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Fixed ==

Please see the 1.13.1 milestone in GitHub for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/19?closed=1.

Archivematica 1.13.1 and Storage Service 0.18.1 release notes

2021-10-19T16:12:39Z

Sevein: Created page with "Home > Release Notes > Archivematica 1.13.1 and Storage Service 0.18.1 '''Release date: October 19, 2021''' == Environments == Please see th..."

Release Notes

2021-10-19T16:10:20Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.13.1_and_Storage_Service_0.18.1_release_notes|'''Archivematica 1.13.1 and Storage Service 0.18.1 Release Notes''']] (Current release)
* [[Archivematica_1.13.0_and_Storage_Service_0.18.0_release_notes|'''Archivematica 1.13.0 and Storage Service 0.18.0 Release Notes''']]
* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes]]
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']]
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2021-05-28T07:11:42Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.0 and Storage Service 0.18.0

'''Release date: ???'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.0 and Storage Service 0.18.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Added ==

=== Disableable virus scanning ===

* Issue: https://github.com/archivematica/Issues/issues/869

Virus scanning is now disableable from the processing configuration.

=== Purge management command ===

* Issue: https://github.com/archivematica/Issues/issues/1239

A new [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/maintenance/maintenance/#management-commands management command] has been added to provide a mechanism to remove old processing data from the system filling up the application database and search indices.

== Changed ==

=== Python 3 ===

* Issues: https://github.com/archivematica/Issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.13.0+label%3A%22%3Asnake%3A%C2%B3+Python+3%22

All Archivematica components are now running on Python 3.6.

=== Archivematica Storage Service now uses MySQL by default ===

* Issue: https://github.com/archivematica/Issues/issues/952

MySQL is now the default database in Archivematica Storage Service. SQLite is still supported but we encourage users to migrate. We have documented the process: [https://www.archivematica.org/en/docs/storage-service-0.18/migration-sqlite-mysql/#migration-sqlite-mysql Migrating data from SQLite to MySQL].

== Fixed ==

Please see the 1.13 milestone in GitHub for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/17.

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2021-05-28T07:11:22Z

Sevein: /* Fixed */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.0 and Storage Service 0.18.0

'''Release date: ???'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.0 and Storage Service 0.18.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Added ==

=== Disableable virus scanning ===

* Issue: https://github.com/archivematica/Issues/issues/869

Virus scanning is now disableable from the processing configuration.

=== Purge management command ===

* Issue: https://github.com/archivematica/Issues/issues/1239

A new [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/maintenance/maintenance/#management-commands management command] has been added to provide a mechanism to remove old processing data from the system filling up the application database and search indices.

== Changed ==

=== Python 3 ===

* Issues: https://github.com/archivematica/Issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.13.0+label%3A%22%3Asnake%3A%C2%B3+Python+3%22

All Archivematica components are now running on Python 3.6.

=== Archivematica Storage Service now uses MySQL by default ===

* Issue: https://github.com/archivematica/Issues/issues/952

MySQL is now the default database in Archivematica Storage Service. SQLite is still supported but we encourage users to migrate. We have documented the process: [https://www.archivematica.org/en/docs/storage-service-0.18/migration-sqlite-mysql/#migration-sqlite-mysql Migrating data from SQLite to MySQL].

== Fixed ==

Please see the 1.13 milestone in Github for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/17

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2021-05-28T06:22:28Z

Sevein: Some more updates in preparation for v1.13.0-rc.1

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.13.0 and Storage Service 0.18.0

'''Release date: ???'''

== Environments ==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.13.0 and Storage Service 0.18.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Please note that we've dropped support for Ubuntu Linux 16.04 since it reached the end of its five-year LTS window on April 30th 2021. We're planning to add support for Ubuntu 20.04 in the short term.

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

== Added ==

=== Disableable virus scanning ===

* Issue: https://github.com/archivematica/Issues/issues/869

Virus scanning is now disableable from the processing configuration.

=== Purge management command ===

* Issue: https://github.com/archivematica/Issues/issues/1239

A new [https://www.archivematica.org/en/docs/archivematica-1.13/admin-manual/maintenance/maintenance/#management-commands management command] has been added to provide a mechanism to remove old processing data from the system filling up the application database and search indices.

== Changed ==

=== Python 3 ===

* Issues: https://github.com/archivematica/Issues/issues?q=is%3Aopen+is%3Aissue+milestone%3A1.13.0+label%3A%22%3Asnake%3A%C2%B3+Python+3%22

All Archivematica components are now running on Python 3.6.

=== Archivematica Storage Service now uses MySQL by default ===

* Issue: https://github.com/archivematica/Issues/issues/952

MySQL is now the default database in Archivematica Storage Service. SQLite is still supported but we encourage users to migrate. We have documented the process: [https://www.archivematica.org/en/docs/storage-service-0.18/migration-sqlite-mysql/#migration-sqlite-mysql Migrating data from SQLite to MySQL].

== Fixed ==

* ...

And more! Please see the 1.13 milestone in Github for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/17

Storage Service API

2021-05-25T09:33:03Z

Sevein:

[[Main Page]] > [[Development]] > Storage Service API

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/dev-manual/ Archivematica documentation] for up-to-date information.</div>

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:

$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the location where is stored down to the lowest level.<br />E.g.: <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb/data/objects/MARBLES.TGA</code>, where <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb</code> contains both the UUID of the package and the name of the transfer which you can find using [[#Get package details | Get package details]] (see <code>current_path</code> attribute).
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

In SS0.15 and above, this endpoint has been extended to support callbacks for AIP, AIC and DIP stored events.

* '''Event''': Post-store AIP, AIC, or DIP
* '''URI''': <code>https://scope.com/api/v1/dip/<package_uuid>/stored</code>
* '''Method''': POST
* '''Headers''':
* - Authorization -> <code>Token <token></code>
* - Origin -> <code>https://ss.com</code>
* '''Expected Status''': 202

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

Example:
$ curl -d 'location_uuid=7c5c48d4-50db-4018-b4b1-7ed29d1ef9d3 ' -H"Authorization: ApiKey test:4525fd5272275caeac04a28447698c51" 'http://mysite.archivematica.org:8000/api/v2/file/8b8aa1a2-79c9-490e-b630-28f90bb7e654/move/'

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Archivematica API

2021-05-25T09:32:58Z

Sevein:

[[Main Page]] > [[Development]] > Archivematica API

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/en/docs/archivematica-1.13/dev-manual/ Archivematica documentation] for up-to-date information.</div>

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>).
<br \><br \>
''As GET parameters, format is:''<br \>
<code>?username=<''username''>&api_key=<''api_key''></code><br \>

''As a header, format is:''<br \>
<code>ApiKey <username>:<api_key></code><br \>

'''NOTE:''' The 'ApiKey' prefix is a requirement of the Tastypie library in use. See the Tastypie docs for [https://django-tastypie.readthedocs.io/en/latest/authentication.html#apikeyauthentication ApiKeyAuthentication].''<br \><br \>

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/api/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"
'''Note:''' for consumers of this endpoint, it is possible for Archivematica to return a <code>status</code> of <code>COMPLETE</code> without a <code>sip_uuid</code>. Consumers looking to use the UUID of the AIP that will be created following Ingest should therefore test for both a <code>status</code> of <code>COMPLETE</code> and the existence of <code>sip_uuid</code> that does not also equal <code>BACKLOG</code> to ensure that they retrieve it. This might mean an additional call to the status endpoint while this data becomes available.

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Return a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Unit ==

''Note: Currently, you can only query strings in English.''

=== List jobs ===
* '''URL''': <code>/api/v2beta/jobs/<unit UUID>/</code>
* '''Verb''': GET
* Return a list of jobs for the passed unit (transfer or ingest).
* '''Parameters''':
* '''Optional filters''':
*** <code>microservice</code>: Name of the microservice the jobs belong to
*** <code>link_uuid</code>: UUID of the job chain link
*** <code>name</code>: Name of the job
* '''Response''': JSON body
** '''List of dicts with keys''':
*** <code>uuid</code>: UUID of the job
*** <code>name</code>: Name of the job
*** <code>status</code>: One of USER_INPUT, PROCESSING, COMPLETE, FAILED or UNKNOWN
*** <code>microservice</code>: Microservice the job belongs to
*** <code>link_uuid</code>: UUID of the job chain link
*** <code>tasks</code>: List of dicts with information about the microservice's tasks:
*** <code>uuid</code>: UUID of the task
*** <code>exit_code</code>: Exit code of the task

=== Task ===
* '''URL''': <code>/api/v2beta/task/<task UUID>/</code>
* '''Verb''': GET
* Return information about a task.
* '''Response''': JSON body
** <code>uuid</code>: UUID of the task
** <code>exit_code</code>: Exit code of the task
** <code>file_uuid</code>: UUID of the file used for the task
** <code>file_name</code>: File used for the task,
** <code<time_created</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task was created
** <code>time_started</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task started
** <code>time_ended</code>: String (YYYY-MM-DD HH:MM:SS) representing when the task finished
** <code>duration</code>: Task duration in seconds (integer). If the duration is less than a second, this will be a < 1 string

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>true</code> or <code>false</code> to set the transfer to auto-approve, default: <code>true</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID ('''Note:''' as the package endpoint allows the caller to interact with Archivematica asynchronously it doesn't guarantee a transfer has started. The caller must use the UUID in the response to verify it has begun or any errors that were encountered initiating one.)
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransferCSV' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransferCSV' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>

=== Validate ===

Available in Archivematica 1.10+. This endpoint can be used to validate CSVs against embedded sets of rules. Currently works with Avalon Media System Manifest files.

In the future, this endpoint can be extended to support validation for <code>metadata.csv</code> or <code>rights.csv</code>, or other institutionally-based rules.

* '''URL''': <code>/api/v2beta/validate/avalon</code>
* '''Content-Type''': text/csv; charset=utf-8
* '''Parameters''': <code><document></code>

Usage example: (assuming that <code>avalon.csv</code> exists)

<pre>
curl http://127.0.0.1:62080/api/v2beta/validate/avalon \
--data-binary "@avalon.csv" \
--header "Authorization: ApiKey test:test" \
--header "Content-Type: text/csv; charset=utf-8"
</pre>

Response examples:

200 OK

<pre>
{
"valid": true
}
</pre>

400 Bad Request (expect reason to include different validation errors)

<pre>
{
"valid": false,
"reason": "Administrative data must include reference name and author."
}
</pre>

404 Not Found

<pre>
{
"error": true,
"message": "Unknown validator. Accepted values: avalon"
}
</pre>

[[Category:Development documentation]]

Archivematica 1.13.0 and Storage Service 0.18.0 release notes

2021-05-12T11:27:55Z

Sevein: Add skeleton with notes on Storage Service + MySQL

Release Notes

2021-01-26T18:09:57Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.13.0_and_Storage_Service_0.18.0_release_notes|Archivematica 1.13.0 and Storage Service 0.18.0 Release Notes]] (Upcoming release)
* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes]] (Current release)
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']]
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Update mapping and reindex Elasticsearch indices

2021-01-19T10:15:44Z

Sevein:

'''This is now part of our official upgrading docs: https://www.archivematica.org/en/docs/latest/admin-manual/installation-setup/upgrading/upgrading/.'''

---

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch''.
* Change ''ES_JAVA_OPTS'' to a bigger value, in our example:
ES_JAVA_OPTS="-Xms3g -Xmx3g".
* Restart Elasticsearch service for the changes to take effect (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
</nowiki>

'''NOTE''': The script could fail because JAVA heap size out of memory (please, check /var/log/elascticsearch.log). In this case the indices will be empty, so restore /var/lib/elasticsearch from backup, increase Elasticsearch JAVA heap size and try again.

The script uses the elasticsearch API and makes the following actions:

* Reindex the ''transfers'', ''transferfiles'', ''aips'' and ''aipfiles'' indices in new temporary indices
* Delete original indices
* Restart ''archivematica-dashboard'' service to create empty indices with new mappings
* Reindex from temporary indices
* Delete temporary indices

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Update mapping and reindex Elasticsearch indices

2021-01-13T16:06:55Z

Sevein:

'''This is now part of our official upgrading docs: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/installation-setup/upgrading/upgrading/.'''

---

= Backup Elasticsearch data =

The easiest way to backup the Elasticsearch data is copying the data directory:

sudo service elasticsearch stop
tar cvfz var_lib_elasticsearch.tgz /var/lib/elasticsearch
sudo service elasticsearch start

= List indices to resize the Elasticsearch heap size when needed =

Use the following command to list indices:

curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'

The output should show something like this:

root@archivematica-test-server:~# curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 0 0 1.2kb 1.2kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 11987 0 2.9gb 2.9gb

Take the elasticsearch heap size from ''/etc/default/elasticsearch'' (Ubuntu) or ''/etc/sysconfig/elasticsearch'' (CentOS):

root@ny-gclibrary-test-release-1:~# grep ES_JAVA_OPTS= /etc/default/elasticsearch
#ES_JAVA_OPTS=
ES_JAVA_OPTS="-Xms2g -Xmx2g"

The heap size for the example is 2G.

Ensure your Elasticsearch heap size is greater than the max ''store.size'' in the indices list. For our example, it should be greater than 3GB.

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch''.
* Change ''ES_JAVA_OPTS'' to a bigger value, in our example:
ES_JAVA_OPTS="-Xms3g -Xmx3g".
* Restart Elasticsearch service for the changes to take effect (''sudo service elasticsearch restart'')

= Run script to reindex and use new mappings =

Use the following script:

<nowiki>
#!/bin/bash

es_url="http://localhost:9200"

index_list='aips aipfiles transfers transferfiles'

echo -e "\nIndex list before reindexing:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Clone indices with _reindex API call:
for index in $index_list;do
echo "Reindex ${index} in ${index}_new..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}"'"
},
"dest": {
"index": "'"${index}_new"'"
}
}' > /dev/null
done

echo -e "\n\n"

echo -e "Index list after tmp indices creation\n"
indices_output=$(curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc")
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"

#Delete old indices
for index in $index_list;do
echo "Deleting ${index}..."
curl -s -X DELETE ${es_url}/${index} > /dev/null
done

#Restart archivematica-dashboard to create indices with new mappings
echo -e "\nRestarting archivematica-dashboard"
sudo service archivematica-dashboard restart

#Wait 30 seconds
echo "Wait 30 seconds to ensure dashboard has created the empty indices with new mapping"
sleep 30
echo -e "\n"

#When index has no docs the reindex doesn't create the new index (typically transferfiles index)
#There's a check to ensure the new index has been create before reindexing.
#Reindex fron *_new indices:
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Indexing ${index} using ${index}_new ..."
curl -s -X POST \
${es_url}/_reindex \
-H 'Content-Type: application/json' \
-d '{
"source": {
"index": "'"${index}_new"'"
},
"dest": {
"index": "'"${index}"'"
}
}' > /dev/null
fi
done

echo -e "\n"

#Delete tmp indices
for index in $index_list;do
if echo "$indices_output" | grep ${index}_new >/dev/null; then
echo "Deleting ${index}_new..."
curl -s -X DELETE ${es_url}/${index}_new > /dev/null
fi
done

echo -e "\n\nReindexing done:\n"
curl -s -X GET "${es_url}/_cat/indices/%2A?v=&s=index:desc"
echo -e "\n"
</nowiki>

For our example it takes 11 minutes, and this is the output:

<nowiki>
root@archivematica-test-server:~# time ./script_reindex_new_map.sh

Index list before reindexing:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Reindex aips in aips_new...
Reindex aipfiles in aipfiles_new...
Reindex transfers in transfers_new...
Reindex transferfiles in transferfiles_new...

Index list after tmp indices creation

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers_new gdFevH8yRdiNTdrPcfo8Lg 5 1 0 0 460b 460b
yellow open transfers lYqkYjwZRy2XG8CP_3S3PQ 5 1 3 0 11.6kb 11.6kb
yellow open transferfiles K5gnDZyOQz2JdIeZ6adJsQ 5 1 0 0 1.2kb 1.2kb
yellow open aips_new uJ-ehaYLTfe_1lOSErfu3Q 5 1 17 0 96.8mb 96.8mb
yellow open aips yAyK_koXThaZcWsBYfzN7w 5 1 17 0 101.4mb 101.4mb
yellow open aipfiles_new 00Xxu7v2QvWsq92gM247xQ 5 1 12905 0 3.1gb 3.1gb
yellow open aipfiles TVrrX8jkRhWWxGfvK_M6zg 5 1 12905 0 2.6gb 2.6gb

Deleting aips...
Deleting aipfiles...
Deleting transfers...
Deleting transferfiles...

Restarting archivematica-dashboard
Wait 30 seconds to ensure dashboard has created the empty indices with new mapping

Indexing aips using aips_new ...
Indexing aipfiles using aipfiles_new ...
Indexing transfers using transfers_new ...

Deleting aips_new...
Deleting aipfiles_new...
Deleting transfers_new...

Reindexing done:

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open transfers FC7aSVPmSmmCc_LTv1AQRA 5 1 3 0 1.2kb 1.2kb
yellow open transferfiles 5JMAft3FQwmosZQFi7eJNw 5 1 0 0 1.2kb 1.2kb
yellow open aips EtwXG3-4SO2Px-4QMRufXA 5 1 17 0 102.1mb 102.1mb
yellow open aipfiles -PFuzslgTeWJ4CWny8VZoA 5 1 12905 0 3gb 3gb

real 10m47.114s
user 0m0.068s
sys 0m0.032s
</nowiki>

'''NOTE''': The script could fail because JAVA heap size out of memory (please, check /var/log/elascticsearch.log). In this case the indices will be empty, so restore /var/lib/elasticsearch from backup, increase Elasticsearch JAVA heap size and try again.

The script uses the elasticsearch API and makes the following actions:

* Reindex the ''transfers'', ''transferfiles'', ''aips'' and ''aipfiles'' indices in new temporary indices
* Delete original indices
* Restart ''archivematica-dashboard'' service to create empty indices with new mappings
* Reindex from temporary indices
* Delete temporary indices

= Restore Elasticsearch heap size when needed=

* Edit ''/etc/default/elasticsearch'' or ''/etc/sysconfig/elasticsearch'' when needed.
* Change ''ES_JAVA_OPTS'' when needed.
* Restart Elasticsearch service when needed (''sudo service elasticsearch restart'')

Release Notes

2020-12-01T18:13:12Z

Sevein:

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.12.1_and_Storage_Service_0.17.1_release_notes|'''Archivematica 1.12.1 and Storage Service 0.17.1 Release Notes''']] (Upcoming release)
* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|'''Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes''']] (Current release)
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.12.0 and Storage Service 0.17.0 release notes

2020-10-07T06:35:28Z

Sevein: /* Environments */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.12.0 and Storage Service 0.17.0

'''Release date: still in progress!'''

==Environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.12.0 and Storage Service 0.17.0 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

Please note due to a dependency issue we have not provided RPMs for the 1.12 release. We plan to address this issue in a point release and provide RPMs then.

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

== Added ==

=== Processing configuration selector ===

This feature allows the user to choose a processing configuration at the time of transfer by way of a drop down in the "Start transfer" button.

* Issue: https://github.com/archivematica/Issues/issues/1166
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/transfer/transfer.rst

This feature has been sponsored by Simon Fraser University Archives. Thank you!

=== Column selectors for the Backlog and Archival Storage tab ===

Users can now choose the columns displayed while browsing the Backlog and Archival Storage tabs so they can see the information most relevant to them.

* Issue (Archival Storage): https://github.com/archivematica/Issues/issues/1168
* Documentation (Archival Storage): https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

* Issue (Backlog): https://github.com/archivematica/Issues/issues/1167
* Documentation (Backlog): https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/backlog/backlog/#backlog

These features have been sponsored by Simon Fraser University Archives. Thank you!

=== AIP location column on Archival Storage tab ===

* Issue: https://github.com/archivematica/Issues/issues/1214
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

This feature has been sponsored by Picturae. Thank you!

=== Downloadable CSV from Archival Storage search ===

Users can now download a CSV containing all entries in Archival Storage, containing these columns: AIP name, Size, UUID, Number of files, Date stored, Status, Encrypted, and Storage Location

* Issue: https://github.com/archivematica/Issues/issues/1213
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

This feature has been sponsored by Picturae. Thank you!

=== Central Authorization Service (CAS) for Archivematica and Storage Service ===

* Issue: https://github.com/archivematica/Issues/issues/1211
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/security/security/#cas-setup

This feature has been sponsored by Simon Fraser University Archives. Thank you!

=== OpenID Connect (OIDC) support for Archivematica and Storage Service ===

'''Experimental! Please share your feedback.'''

* Issue: https://github.com/archivematica/Issues/issues/1053
* Instructions: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/security/security/#oidc-setup

OIDC support is a community contribution by Wellcome Collection. Thank you!

== Changed ==

=== Django 1.12 support ===

* Issue: https://github.com/artefactual/archivematica/issues/1016

Archivematica has been upgraded to Django 1.12 LTS. Our next target is Django 2.2 LTS!

There are a couple of things to be aware of during the Archivematica upgrade in this respect:
* For a seamless experience for your users make sure that all active user sessions are deleted. We explain how here: [https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/maintenance/maintenance/#clear-user-sessions Maintenance > Clear user sessions].
* Django's User model now comes with a character limit of 150 characters. In Archivematica v1.11.x or older we had that limit set to 250 characters via a third-party app that we have now deleted. During the database migration (i.e. when running manage.py migrate), users may be required to fix the problem manually. See the [https://github.com/artefactual/archivematica/blob/f87d2f39acce9a59bf49f72fd7e57f9eced2dbe5/src/dashboard/src/main/migrations/0078_username_check.py#L1-L20 migration module documentation] for more details. We expect this to be highly unlikely to happen since the exception is only triggered when the 150 character limit in the username field is exceeded.

=== Replace whitelist with allowlist ===

* Issue: https://github.com/archivematica/Issues/issues/1226

We have changed "whitelist" to "allowlist" in an effort to use more inclusive language throughout Archivematica's code.

== Fixed ==

* Status doesn't revert to "Stored" after deletion requests are rejected: https://github.com/archivematica/Issues/issues/1273
* Cannot create a SIP from a Bag transfer in Appraisal tab: https://github.com/archivematica/Issues/issues/1267
* Transfer/AIP status not in sync when package is deleted within SS: https://github.com/archivematica/Issues/issues/1189
* Cannot rebuild backlog or AIP indices when location is encrypted: https://github.com/archivematica/Issues/issues/734
* Disk usage pages are not reporting accurate numbers: https://github.com/archivematica/Issues/issues/1281

And more! Please see the 1.12 milestone in Github for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/12

Archivematica 1.12.0 and Storage Service 0.17.0 release notes

2020-09-09T15:47:32Z

Sevein: Add note on username database migration

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.12.0 and Storage Service 0.17.0

'''Release date: still in progress!'''

==Environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

For development purposes, most of our developers prefer to use Docker containers. These and all above environments are linked from the installation instructions above.

Please note due to a dependency issue we have not provided RPMs for the 1.12 release. We plan to address this issue in a point release and provide RPMs then.

<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">Note: if you are upgrading from Archivematica 1.10.x or earlier, please be sure to clean up the completed transfers watched directory before upgrading. Instructions can be found on the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading] page in the documentation. </div> <p>

== Added ==

=== Processing configuration selector ===

This feature allows the user to choose a processing configuration at the time of transfer by way of a drop down in the "Start transfer" button.

* Issue: https://github.com/archivematica/Issues/issues/1166
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/transfer/transfer.rst

This feature has been sponsored by Simon Fraser University Archives. Thank you!

=== Column selectors for the Backlog and Archival Storage tab ===

Users can now choose the columns displayed while browsing the Backlog and Archival Storage tabs so they can see the information most relevant to them.

* Issue (Archival Storage): https://github.com/archivematica/Issues/issues/1168
* Documentation (Archival Storage): https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

* Issue (Backlog): https://github.com/archivematica/Issues/issues/1167
* Documentation (Backlog): https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/backlog/backlog/#backlog

These features have been sponsored by Simon Fraser University Archives. Thank you!

=== AIP location column on Archival Storage tab ===

* Issue: https://github.com/archivematica/Issues/issues/1214
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

This feature has been sponsored by Picturae. Thank you!

=== Downloadable CSV from Archival Storage search ===

Users can now download a CSV containing all entries in Archival Storage, containing these columns: AIP name, Size, UUID, Number of files, Date stored, Status, Encrypted, and Storage Location

* Issue: https://github.com/archivematica/Issues/issues/1213
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/user-manual/archival-storage/archival-storage/#archival-storage

This feature has been sponsored by Picturae. Thank you!

=== Central Authorization Service (CAS) for Archivematica and Storage Service ===

* Issue: https://github.com/archivematica/Issues/issues/1211
* Documentation: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/security/security/#cas-setup

This feature has been sponsored by Simon Fraser University Archives. Thank you!

=== OpenID Connect (OIDC) support for Archivematica and Storage Service ===

'''Experimental! Please share your feedback.'''

* Issue: https://github.com/archivematica/Issues/issues/1053
* Instructions: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/security/security/#oidc-setup

OIDC support is a community contribution by Wellcome Collection. Thank you!

== Changed ==

=== Django 1.12 support ===

* Issue: https://github.com/artefactual/archivematica/issues/1016

Archivematica has been upgraded to Django 1.12 LTS. Our next target is Django 2.2 LTS!

There are a couple of things to be aware of during the Archivematica upgrade in this respect:
* For a seamless experience for your users make sure that all active user sessions are deleted. We explain how here: [https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/maintenance/maintenance/#clear-user-sessions Maintenance > Clear user sessions].
* Django's User model now comes with a character limit of 150 characters. In Archivematica v1.11.x or older we had that limit set to 250 characters via a third-party app that we have now deleted. During the database migration (i.e. when running manage.py migrate), users may be required to fix the problem manually. See the [https://github.com/artefactual/archivematica/blob/f87d2f39acce9a59bf49f72fd7e57f9eced2dbe5/src/dashboard/src/main/migrations/0078_username_check.py#L1-L20 migration module documentation] for more details. We expect this to be highly unlikely to happen since the exception is only triggered when the 150 character limit in the username field is exceeded.

=== Replace whitelist with allowlist ===

* Issue: https://github.com/archivematica/Issues/issues/1226

We have changed "whitelist" to "allowlist" in an effort to use more inclusive language throughout Archivematica's code.

== Fixed ==

* Status doesn't revert to "Stored" after deletion requests are rejected: https://github.com/archivematica/Issues/issues/1273
* Cannot create a SIP from a Bag transfer in Appraisal tab: https://github.com/archivematica/Issues/issues/1267
* Transfer/AIP status not in sync when package is deleted within SS: https://github.com/archivematica/Issues/issues/1189
* Cannot rebuild backlog or AIP indices when location is encrypted: https://github.com/archivematica/Issues/issues/734
* Disk usage pages are not reporting accurate numbers: https://github.com/archivematica/Issues/issues/1281

And more! Please see the 1.12 milestone in Github for all issues addressed in this release: https://github.com/archivematica/Issues/milestone/12

Archivematica 1.12.0 and Storage Service 0.17.0 release notes

2020-08-28T12:20:23Z

Sevein:

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.12.0 and Storage Service 0.17.0

'''Work in progress!'''

== Supported environments ==

TBD.

== Added ==

=== OpenID Connect (OIDC) support for Archivematica and Storage Service ===

'''Experimental! Please share your feedback.'''

* Issue: https://github.com/archivematica/Issues/issues/1053
* Instructions: https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/security/security/#oidc-setup

=== Django 1.12 support ===

* Issue: https://github.com/artefactual/archivematica/issues/1016

Archivematica has finally upgraded to Django 1.12 LTS. Our next target is Django 2.2 LTS!

For a seamless experience for you users make sure that all active user sessions are deleted. We explain how here: [https://www.archivematica.org/en/docs/archivematica-1.12/admin-manual/maintenance/maintenance/#clear-user-sessions Maintenance > Clear user sessions].

Archivematica 1.12.0 and Storage Service 0.17.0 release notes

2020-08-06T13:14:04Z

Sevein: Created page with "Home > Release Notes > Archivematica 1.12.0 and Storage Service 0.17.0 '''Work in progress!''' == Supported environments == TBD. == Added =..."

Release Notes

2020-08-06T13:10:05Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.12.0_and_Storage_Service_0.17.0_release_notes|Archivematica 1.12.0 and Storage Service 0.17.0 Release Notes]] (work in progress)
* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes]] (Current release)
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]]
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Storage Service API

2020-07-30T17:57:02Z

Sevein: /* Download single file */

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:

$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the location where is stored down to the lowest level.<br />E.g.: <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb/data/objects/MARBLES.TGA</code>, where <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb</code> contains both the UUID of the package and the name of the transfer which you can find using [[#Get package details | Get package details]] (see <code>current_path</code> attribute).
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

In SS0.15 and above, this endpoint has been extended to support callbacks for AIP, AIC and DIP stored events.

* '''Event''': Post-store AIP, AIC, or DIP
* '''URI''': <code>https://scope.com/api/v1/dip/<package_uuid>/stored</code>
* '''Method''': POST
* '''Headers''':
* - Authorization -> <code>Token <token></code>
* - Origin -> <code>https://ss.com</code>
* '''Expected Status''': 202

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

Example:
$ curl -d 'location_uuid=7c5c48d4-50db-4018-b4b1-7ed29d1ef9d3 ' -H"Authorization: ApiKey test:4525fd5272275caeac04a28447698c51" 'http://mysite.archivematica.org:8000/api/v2/file/8b8aa1a2-79c9-490e-b630-28f90bb7e654/move/'

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Storage Service API

2020-07-30T17:56:05Z

Sevein: /* Download single file */

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:

$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the location where is stored down to the lowest level.<br />E.g.: <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb/data/objects/MARBLES.TGA</code>, where <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb</code> contains both the transfer name and the identifier of the package. [[#Get package details | Get package details]] can provide these (see <code>current_path</code> attribute).
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

In SS0.15 and above, this endpoint has been extended to support callbacks for AIP, AIC and DIP stored events.

* '''Event''': Post-store AIP, AIC, or DIP
* '''URI''': <code>https://scope.com/api/v1/dip/<package_uuid>/stored</code>
* '''Method''': POST
* '''Headers''':
* - Authorization -> <code>Token <token></code>
* - Origin -> <code>https://ss.com</code>
* '''Expected Status''': 202

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

Example:
$ curl -d 'location_uuid=7c5c48d4-50db-4018-b4b1-7ed29d1ef9d3 ' -H"Authorization: ApiKey test:4525fd5272275caeac04a28447698c51" 'http://mysite.archivematica.org:8000/api/v2/file/8b8aa1a2-79c9-490e-b630-28f90bb7e654/move/'

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Storage Service API

2020-07-30T17:55:30Z

Sevein: /* Download single file */

[[Main Page]] > [[Development]] > Storage Service API

The [[Storage Service]] API provides programmatic access to moving files around in storage areas that the Storage Service has access to.

The API is written using [http://django-tastypie.readthedocs.io/en/latest/ TastyPie].

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: TastyPie is less well supported than [http://www.django-rest-framework.org/ Django REST Framework], both in terms of docs & community. We should look at replacing TastyPie with DRF.
|}

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=test&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey test:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

== A note about browsing ==

A detailed schema can be found for each of the resources by adding "schema" to the get all URL.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/location/schema/?format=json
{
"allowed_detail_http_methods": [
"get",
"post"
],
"allowed_list_http_methods": [
"get"
],
"default_format": "application/json",
"default_limit": 20,
"fields": {
"description": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "description"
},
"enabled": {
"blank": true,
"default": true,
"help_text": "True if space can be accessed.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "boolean",
"unique": false,
"verbose_name": "Enabled"
},
"path": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "path"
},
"pipeline": {
"blank": false,
"default": "No default provided.",
"help_text": "Many related resources. Can be either a list of URIs or list of individually nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/pipeline/schema/",
"related_type": "to_many",
"type": "related",
"unique": false,
"verbose_name": "pipeline"
},
"purpose": {
"blank": false,
"default": "No default provided.",
"help_text": "Purpose of the space. Eg. AIP storage, Transfer source",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Purpose"
},
"quota": {
"blank": false,
"default": null,
"help_text": "Size, in bytes (optional)",
"nullable": true,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Quota"
},
"relative_path": {
"blank": false,
"default": "",
"help_text": "Path to location, relative to the storage space's path.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Relative Path"
},
"resource_uri": {
"blank": false,
"default": "No default provided.",
"help_text": "Unicode string data. Ex: \"Hello World\"",
"nullable": false,
"primary_key": false,
"readonly": true,
"type": "string",
"unique": false,
"verbose_name": "resource uri"
},
"space": {
"blank": false,
"default": "No default provided.",
"help_text": "A single related resource. Can be either a URI or set of nested resource data.",
"nullable": false,
"primary_key": false,
"readonly": false,
"related_schema": "/api/v2/space/schema/",
"related_type": "to_one",
"type": "related",
"unique": false,
"verbose_name": "space"
},
"used": {
"blank": false,
"default": 0,
"help_text": "Amount used, in bytes.",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": false,
"verbose_name": "Used"
},
"uuid": {
"blank": true,
"default": "",
"help_text": "Unique identifier",
"nullable": false,
"primary_key": false,
"readonly": false,
"type": "string",
"unique": true,
"verbose_name": "uuid"
}
},
"filtering": {
"pipeline": 2,
"purpose": 1,
"quota": 1,
"relative_path": 1,
"space": 2,
"used": 1,
"uuid": 1
}
}

This schema, among other things, describes the fields in the resource (including the schema URI of related resource fields) and the fields that allow filtering. Valid filtering values are: Django ORM filters (e.g. startswith, exact, lte, etc.) or 1 or 2. If a filtering field is set to 2 it can be filtered over the related resource fields. For example, the locations could be filtered by their pipeline UUID setting it in a request parameter formatted with two underscore chars: <code>/api/v2/location/?pipeline__uuid=<uuid></code>

For more info on how to interact with the API see:

http://django-tastypie.readthedocs.io/en/v0.13.1/interacting.html

== Pipeline ==

=== Get all pipelines ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>description</code>: Description of the pipeline
** <code>uuid</code>: UUID of the pipeline
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of pipelines. See [[#Get pipeline details]] for format

Returns information about all the pipelines in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by the description or uuid. Disabled pipelines are not returned.

Example:
$ curl -X GET -H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" 'http://localhost:8000/api/v2/pipeline/?description__startswith=Archivematica' | python -m json.tool
{
"meta": {
"limit": 20,
"next": null,
"offset": 0,
"previous": null,
"total_count": 1
},
"objects": [
{
"description": "Archivematica on alouette",
"remote_name": "127.0.0.1",
"resource_uri": "/api/v2/pipeline/dd354557-9e6e-4918-9fe3-a65b00ecb1af/",
"uuid": "dd354557-9e6e-4918-9fe3-a65b00ecb1af"
}
]
}

=== Create new pipeline ===

* '''URL''': <code>/api/v2/pipeline/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new pipeline: <code>uuid</code>, <code>description</code>, <code>api_key</code>, <code>api_username</code>
** <code>create_default_locations</code>: If True, will associated default [[Storage Service#Locations | Locations]] with the newly created pipeline
** <code>shared_path</code>: If default locations are created, create the [[Storage Service#Currently Processing | processing]] location at this path in the local filesystem
** <code>remote_name</code>: URI of the pipeline.
*** Before v0.11.0: If <code>create_default_locations</code> is set, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
*** In v0.11.0 or newer: If not provided, SS will try to guess the value using the <code>REMOTE_ADDR</code> header.
* '''Response''': JSON with data for the pipeline

If the 'Pipelines disabled on creation' setting is set, the pipeline will be disabled by default, and will not respond to queries.

Example:

$ curl -X POST \
-H"Authorization: ApiKey test:95141fc645ed97a95893f1f865d24687f89a27ad" \
-H"Content-Type: application/json" \
-d'{"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199", \
"description": "Test pipeline", "create_default_locations": true, \
"api_username": "demo", \
"api_key": "03ecb307f5b8012f4771d245d534830378a87259"}' \
'http://192.168.1.42:8000/api/v2/pipeline/'
{
"create_default_locations": true,
"description": "Test pipeline",
"remote_name": "192.168.1.42",
"resource_uri": "/api/v2/pipeline/99354557-9e6e-4918-9fe3-a65b00ecb199/",
"uuid": "99354557-9e6e-4918-9fe3-a65b00ecb199"
}

=== Get pipeline details ===

* '''URL''': <code>/api/v2/pipeline/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>description</code>: Pipeline description
** <code>remote_name</code>: IP or hostname of the pipeline. For use in API calls
** <code>resource_uri</code>: URI for this pipeline in the API
** <code>uuid</code>: UUID of the pipeline

== Space ==

=== Get all spaces ===

* '''URL''': <code>/api/v2/space/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>access_protocol</code>: Protocol that the [[Storage Service#Space | Space]] uses. Must be searched based on the database code.
** <code>path</code>: Space's path
** <code>size</code>: Maximum size in bytes. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>used</code>: Bytes stored in this space. Can use greater than (size__gt=1024), less than (size__lt=1024), and other Django [https://docs.djangoproject.com/en/1.8/ref/models/querysets/#field-lookups field lookups].
** <code>uuid</code>: UUID of the Space
* '''Response''': JSON
** <code>meta</code>: Metadata on the response: number of hits, pagination information
** <code>objects</code>: List of spaces. See [[#Get space details]] for format

Returns information about all the spaces in the system. Can be [http://django-tastypie.readthedocs.io/en/latest/resources.html#basic-filtering filtered] by several fields: access protocol, path, size, amount used, UUID and verified status. Disabled spaces are not returned.

=== Get space details ===

* '''URL''': <code>/api/v2/space/<UUID>/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': JSON
** <code>access_protocol</code>: Database code for the access protocol
** <code>last_verified</code>: Date of last verification. This is a stub feature
** <code>path</code>: Space's path
** <code>resource_uri</code>: URI to the resource in the API
** <code>size</code>: Maximum size of the space in bytes.
** <code>used</code>: Bytes stored in this space.
** <code>uuid</code>: UUID of the space
** <code>verified</code>: If the space is verified. This is a stub feature
** Other space-specific fields

=== Browse space path ===

* '''URL''': <code>/api/v2/space/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Space to look
* '''Response''': JSON
** <code>entries</code>: List of entries at path, files or directories
** <code>directories</code>: List of directories in path. Subset of `entries`.

=== Create space ===

See [https://github.com/archivematica/Issues/issues/36 Issue 36].

* '''URL''': <code>/api/v2/pipeline/space</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** Should contain fields for a new space: See the [https://www.archivematica.org/en/docs/storage-service-0.11/administrators/#id2 Storage Service Documentation] or [https://wiki.archivematica.org/Storage_Service#Space Space] for fields relevant to each type of space. Basic fields for a local file system space are listed below.
** <code>access_protocol</code>: this defines the type of space
** <code>path</code>: Absolute path to the Space on the local filesystem
** <code>size</code>: (Optional) Maximum size allowed for this space. Set to 0 or leave blank for unlimited.

Example (to create an S3 space):
<pre>
$ curl \
-X POST \
-d @payload.json \
-H "Content-Type: application/json" \
-H "Authorization: ApiKey test:test" \
http://127.0.0.1:62081/api/v2/space/

Where payload.json contains
{
"access_protocol": "S3",
"path": "",
"staging_path": "/",
"endpoint_url": "http://127.0.0.1:12345",
"access_key_id": "_Cah4cae1_",
"secret_access_key": "_Thu6Ahqu_",
"region": "us-west-2"
}
</pre>

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

== Location ==

=== Get all locations ===

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': GET

=== Create new location ===

Added in v0.12 - see [https://github.com/artefactual/archivematica-storage-service/issues/367 issue 367] and [https://github.com/archivematica/Issues/issues/37 issue 37].

This endpoint creates a location in the storage service, but it doesn't actually create the directory that the location points to.

* '''URL''': <code>/api/v2/location/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>description</code>.
** <code>pipeline</code>: URI of the pipeline.
** <code>space</code>: URI of the space.
** <code>default</code>: If 'true' this location will be the default for it's purpose.
** <code>purpose</code>: (below is a list of possible values)
*** <code>AR</code> (AIP_RECOVERY)
*** <code>AS</code> (AIP_STORAGE)
*** <code>CP</code> (CURRENTLY_PROCESSING)
*** <code>DS</code> (DIP_STORAGE)
*** <code>SD</code> (SWORD_DEPOSIT)
*** <code>SS</code> (STORAGE_SERVICE_INTERNAL)
*** <code>BL</code> (BACKLOG)
*** <code>TS</code> (TRANSFER_SOURCE)
*** <code>RP</code> (REPLICATOR)
** <code>relative_path</code>: Relative to the space's path.

Example:

<pre>
curl -s -d '{
"pipeline": ["/api/v2/pipeline/90707555-244f-47af-8271-66496a6a965b/"],
"purpose": "TS",
"relative_path": "foo/bar",
"description": "foobar",
"space": "/api/v2/space/141593ff-2a27-44a1-9de1-917573fa0f4a/"
}' \
-X POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
"http://127.0.0.1:62081/api/v2/location/"
</pre>

=== Get location details ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': GET

=== Move files to this location ===

* '''URL''': <code>/api/v2/location/<UUID>/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>origin_location</code>: URI of the Location the files should be moved from
** <code>pipeline</code>: URI of the [[Storage Service#Pipeline | pipeline]]. Both Locations must be associated with this pipeline.
** <code>files</code>: List of dicts containing <code>source</code> and <code>destination</code>. The source and destination are paths relative to their Location of the files to be moved.

Intended for use with creating Transfers, SIPs, etc and other cases where files need to be moved but not tracked by the storage service.

=== Browse location path ===

* '''URL''': <code>/api/v2/location/<UUID>/browse/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>path</code>: Path inside the Location to look
* '''Response''': JSON
** <code>entries</code>: List of entries in `path`, files or directories
** <code>directories</code>: List of directories in `path`. Subset of `entries`.

{| class="wikitable" style="background-color:#ffffcc;" cellpadding="10";
| Version 1: Returns paths as strings
Version 2: Returns all paths base64 encoded
|}

=== SWORD collection ===

* '''URL''': <code>/api/v2/location/<UUID>/sword/collection/</code>
* '''Verb''': GET, POST

See [[Sword API]] for details

== Package ==

=== Get all packages ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': GET

=== Create new package ===

* '''URL''': <code>/api/v2/file/</code>
* '''Verb''': POST
* '''Parameters''': JSON. Fields for a new package:
** <code>uuid</code>: UUID of the new package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>, <code>DIP</code>, <code>transfer</code>, <code>SIP</code>, <code>file</code>, <code>deposit</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from
** <code>related_package_uuid</code>: UUID of a package that is related to this one. E.g. UUID of a DIP when storing an AIP

Creates a database entry tracking the package (AIP, transfer, etc). If the package is an AIP, DIP or AIC and the current_location is an AIP or DIP storage location it also moves the files from the source to destination location. If the package is a Transfer and the current_location is transfer backlog, it is also moved.

This is handled through the modified <code>obj_create</code> function, which calls <code>Package.store_aip</code> or <code>Package.backlog_transfer</code>

=== Get package details ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': GET

=== Update package contents ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PUT
* '''Parameters''': JSON body
** <code>reingest</code>: Flag to mark that this is reingest. Reduces chance to accidentally modify an AIP.
** <code>uuid</code>: UUID of the existing package
** <code>origin_location</code>: URI of the Location where the package is currently
** <code>origin_path</code>: Path to the package, relative to the origin_location
** <code>current_location</code>: URI of the Location where the package should be stored
** <code>current_path</code>: Path where the package should be stored, relative to the current_location
** <code>package_type</code>: Type of package this is. One of: <code>AIP</code>, <code>AIC</code>
** <code>size</code>: Size of the package
** <code>origin_pipeline</code>: URI of the pipeline the package is from. This must be the same pipeline reingest was started on (tracked through <code>Package.misc_attributes.reingest_pipeline</code>)

Updates the contents of a package during reingest. If the package is an AIP or AIC, currently stored in an AIP storage location, and the 'reingest' parameter is set, it will call <code>Package.finish_reingest</code> and merge the new AIP with the existing one.

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>.

=== Update package metadata ===

* '''URL''': <code>/api/v2/file/<UUID>/</code>
* '''Verb''': PATCH
* '''Parameters''': JSON body
** <code>reingest</code>: Pipeline UUID or None.

Used to update metadata stored in the database for the package. Currently, this is used to update the reingest status.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Currently, this always sets Package.misc_attributes.reingest to None, regardless of what value was actually passed in.
|}

This is implemented using a modified <code>obj_update</code> which calls <code>obj_update_hook</code>. <code>update_in_place</code> also helps.

=== Delete package request ===

* '''URL''': <code>/api/v2/file/<UUID>/delete_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for deleting the AIP
** <code>pipeline</code>: UUID of the pipeline the delete request is from
** <code>user_id</code>: User ID requesting the deletion. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the deletion.

=== Recover AIP request ===

* '''URL''': <code>/api/v2/file/<UUID>/recover_aip/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>event_reason</code>: Reason for recovering the AIP
** <code>pipeline</code>: URI of the pipeline the recovery request is from
** <code>user_id</code>: User ID requesting the recovery. This is the ID of the user on the pipeline, and must be an integer greater than 0.
** <code>user_email</code>: Email of the user requesting the recovery.

=== Download single file ===

* '''URL''': <code>/api/v2/file/<UUID>/extract_file/</code>
* '''Verb''': GET, HEAD
* '''Parameters''': Query string parameters
** <code>relative_path_to_file</code>: Path to the file to download, relative to the location where is stored down to the lowest level.<br />E.g.: a valid example is <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb/data/objects/MARBLES.TGA</code>, where <code>test01-36d4efb7-83ba-46bf-a171-8c7b965d9dcb</code> contains both the transfer name and the identifier of the package. [[#Get package details | Get package details]] can provide these (see <code>current_path</code> attribute).
* '''Response''': Stream of the requested file

Returns a single file from the Package. If the package is compressed, it downloads the whole AIP and extracts it.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence. Currently, the storage service has no way to check if a file exists except by downloading and extracting this AIP
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Download package ===

* '''URL''': <code>/api/v2/file/<UUID>/download/</code>
* '''URL''': <code>/api/v2/file/<UUID>/download/<chunk number>/</code> (for [[Storage Service#LOCKSS-o-matic | LOCKSS]] harvesting)
* '''Verb''': GET, HEAD
* '''Parameters''': None
* '''Response''': Stream of the package

Returns the entire package as a single file. If the AIP is uncompressed, create one file by using `tar`.

If the download URL has a chunk number, it will attempt to serve the LOCKSS chunk specified for that package. If the package is not in LOCKSS, it will return the the whole package.

This responds to HEAD because AtoM uses HEAD to check for the existence of a file.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: HEAD and GET should not perform the same functions. HEAD should be updated to not return the file, and to only check for existence.
|}

If the package is in [[Storage Service#Arkivum | Arkivum]], the package may not actually be available. This endpoint checks if the package is locally available. If it is, it is returned as normal. If not, it returns <code>202</code> and emails the administrator about the attempted access.

=== Get pointer file ===

* '''URL''': <code>/api/v2/file/<UUID>/pointer_file/</code>
* '''Verb''': GET
* '''Parameters''': None
* '''Response''': Stream of the pointer file.

=== Check fixity ===

* '''URL''': <code>/api/v2/file/<UUID>/check_fixity/</code>
* '''Verb''': GET
* '''Parameters''': Query string parameters
** <code>force_local</code>: If true, download and run fixity on the AIP locally, instead of using the Space-provided fixity if available.
* '''Response''': JSON
** <code>success</code>: True if the verification succeeded, False if the verification failed, None if the scan could not start
** <code>message</code>: Human-readable string explaining the report; it will be empty for successful scans.
** <code>failures</code>: List of 0 or more errors
** <code>timestamp</code>: ISO-formated string with the datetime of the last fixity check. If the check was performed by an external system, this will be provided by that system. If not provided,or on error, it will be None.

=== AIP storage callback request ===

* '''URL''': <code>/api/v2/file/<UUID>/send_callback/post_store/</code>
* '''Verb''': GET

Request to call any Callbacks configured to run post-storage for this AIP.

In SS0.15 and above, this endpoint has been extended to support callbacks for AIP, AIC and DIP stored events.

* '''Event''': Post-store AIP, AIC, or DIP
* '''URI''': <code>https://scope.com/api/v1/dip/<package_uuid>/stored</code>
* '''Method''': POST
* '''Headers''':
* - Authorization -> <code>Token <token></code>
* - Origin -> <code>https://ss.com</code>
* '''Expected Status''': 202

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: This only works on locally available AIPs (AIPs stored in Spaces that are available via a UNIX filesystem layer).
|}

=== Get file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': GET
* '''Response''': JSON
** <code>success</code>: True
** <code>package</code>: UUID of the package
** <code>files</code>: List of dictionaries with file information. Each dictionary has:
*** <code>source_id</code>: UUID of the file to index
*** <code>name</code>: Relative path of the file inside the package
*** <code>source_package</code>: UUID of the SIP this file is from
*** <code>checksum</code>: Checksum of the file, or an empty string
*** <code>accessionid</code>: Accession number, or an empty string
*** <code>origin</code>: UUID of the Archivematica dashboard this is from

Returns metadata about every file within the package.

=== Update file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': PUT
* '''Parameters''': JSON list of dictionaries with information on the files to be added. Each dict must have the following attributes:
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>accessionid</code>: Accession number, or an empty string
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

Adds a set of files to a package.

=== Delete file information for package ===

* '''URL''': <code>/api/v2/file/<UUID>/contents/</code>
* '''Verb''': DELETE

Removes all file records associated with this package.

=== Query file information on packages ===

* '''URL''': <code>/api/v2/file/metadata/</code>
* '''Verb''': GET, POST
* '''Parameters''': Query string parameters. Must have at least one, but not all are required
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file
** <code>accessionid</code>: Accession number
** <code>sipuuid</code>: UUID of the SIP this file is from
* '''Response''': JSON. List of dicts with file information about the files that match the query.
** <code>accessionid</code>: Accession number, or an empty string
** <code>file_extension</code>: File extension
** <code>filename</code>: Name of the file, sans path.
** <code>relative_path</code>: Relative path of the file inside the package
** <code>fileuuid</code>: UUID of the file to index
** <code>sipuuid</code>: UUID of the SIP this file is from
** <code>origin</code>: UUID of the Archivematica dashboard this is from

=== Reingest AIP ===

* '''URL''': <code>/api/v2/file/<UUID>/reingest/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>pipeline</code>: UUID of the pipeline to reingest on
** <code>reingest_type</code>: Type of reingest to start. One of <code>METADATA_ONLY</code> (metadata-only reingest), <code>OBJECTS</code> (partial reingest), <code>FULL</code> (full reingest)
** <code>processing_config</code>: Optional. Name of the processing configuration to use on full reingest

=== Move Package ===

This endpoint is introduced in v0.14.

* '''URL''': <code>/api/v2/file/<UUID>/move/</code>
* '''Verb''': POST
* '''Parameters''': JSON body
** <code>location_uuid</code>: UUID of the location to send the package. The location must have the same Purpose as the package's current location (e.g. AIP Storage, DIP Storage).

Example:
$ curl -d 'location_uuid=7c5c48d4-50db-4018-b4b1-7ed29d1ef9d3 ' -H"Authorization: ApiKey test:4525fd5272275caeac04a28447698c51" 'http://mysite.archivematica.org:8000/api/v2/file/8b8aa1a2-79c9-490e-b630-28f90bb7e654/move/'

=== SWORD endpoints ===

* '''URL''': <code>/api/v2/file/<UUID>/sword/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/media/</code>
* '''URL''': <code>/api/v2/file/<UUID>/sword/state/</code>

See [[Sword API]] for details.

[[Category:Development documentation]]

Archivematica 1.11.2 and Storage Service 0.16.1 release notes

2020-05-28T14:58:17Z

Sevein: Created page with "Home > Release Notes > Archivematica 1.11.2 and Storage Service 0.16.1 Work in progress."

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11.2 and Storage Service 0.16.1

Work in progress.

Release Notes

2020-05-28T14:57:35Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==


* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]] (Current release)
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Release Notes

2020-05-28T14:56:34Z

Sevein: /* Archivematica 1.x releases */

[[Main_Page|Home]] > Release Notes

== About ==

Release notes are prepared by Artefactual for each release of Archivematica. Most releases also have a corresponding release of the Storage System, and may also include corresponding releases for various dependencies and libraries (i.e. FIDO, Automation Tools) - be sure to check the release notes for more information.

Major releases are '''bolded''', minor releases are not.

Features, enhancements, and bug fixes are often sponsored by members of the Archivematica community who are dedicated to funding Archivematica's ongoing development and maintenance. Code contributors are individuals who write Archivematica code and [https://github.com/artefactual/archivematica/blob/stable/1.6.x/CONTRIBUTING.md| contribute it] back to the project. Archivematica couldn't continue to grow without sponsors and contributors - thank you!

Questions about a release or the release notes? Ask on the [https://groups.google.com/forum/#!forum/archivematica| Archivematica Google Group!]

== Archivematica 1.x releases ==

* [[Archivematica_1.11.2_and_Storage_Service_0.16.1_release_notes|'''Archivematica 1.11.2 and Storage Service 0.16.1 Release Notes''']]
* [[Archivematica_1.11.1|Archivematica 1.11.1 Release Notes]] (Current release)
* [[Archivematica_1.11_and_Storage_Service_0.16_release_notes|'''Archivematica 1.11 and Storage Service 0.16 Release Notes''']]
* [[Archivematica_1.10.2|Archivematica 1.10.2 Release Notes]]
* [[Archivematica_1.10.1_release_notes|Archivematica 1.10.1 and Storage Service 0.15.1 Release Notes]]
* [[Archivematica_1.10_and_Storage_Service_0.15_release_notes|'''Archivematica 1.10 and Storage Service 0.15 Release Notes''']]
* [[Archivematica_1.9.3|Archivematica 1.9.3 Release Notes]]
* [[Archivematica_1.9.2_release_notes|Archivematica 1.9.2 Release Notes]]
* [[Archivematica_1.9.1_and_Storage_Service_0.14.1_release_notes|Archivematica 1.9.1 and Storage Service 0.14.1 Release Notes]]
* [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|'''Archivematica 1.9 and Storage Service 0.14 Release Notes''']]
* [[Archivematica_1.8.1_release_notes|Archivematica 1.8.1 Release Notes]]
* [[Archivematica_1.8_and_Storage_Service_0.13_release_notes|'''Archivematica 1.8 and Storage Service 0.13 Release Notes''']]
*[[Archivematica 1.7.2_release_notes|Archivematica 1.7.2 Release Notes]]
*[[Storage Service 0.12 Release Notes|'''Storage Service 0.12 Release Notes''']]
*[[Archivematica 1.7.1_release_notes|Archivematica 1.7.1 Release Notes]]
*[[Storage Service 0.11.1 Release Notes]]
* [[Archivematica_1.7_and_Storage_Service_0.11_release_notes|'''Archivematica 1.7 and Storage Service 0.11 Release Notes''']]
* [[Archivematica_1.6.1_release_notes|Archivematica 1.6.1 Release Notes]]
* [[Archivematica_1.6_release_notes|'''Archivematica 1.6 Release Notes''']]
* [[Archivematica_1.5.1_Release_Notes|Archivematica 1.5.1 Release Notes]]
* [[Archivematica_1.5_Release_Notes|'''Archivematica 1.5 Release Notes''']]
* [[Archivematica_1.4.1_Release_Notes|Archivematica 1.4.1 Release Notes]]
* [[Archivematica_1.4_Release_Notes|'''Archivematica 1.4 Release Notes''']]
* [[Archivematica_1.3.2_Release_Notes|Archivematica 1.3.2 Release Notes]]
* [[Archivematica_1.3.1_Release_Notes|Archivematica 1.3.1 Release Notes]]
* [[Archivematica_1.3_Release_Notes|'''Archivematica 1.3 Release Notes''']]
* [[Archivematica_1.2_Release_Notes|'''Archivematica 1.2 Release Notes''']]
* [[Archivematica_1.1_Release_Notes|'''Archivematica 1.1 Release Notes''']]
* [[Archivematica_1.0_Release_Notes|'''Archivematica 1.0 Release Notes''']]

== Archivematica 0.x releases ==

* [[Archivematica_0.10-beta_Release_Notes|'''Archivematica 0.10 Release Notes''']]
* [[Archivematica_0.9_Release_Notes|'''Archivematica 0.9 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.8 Release Notes''']]
* [[Archivematica_0.7.1_Release_Notes|Archivematica 0.7.1 Release Notes]]
* [[Archivematica_0.7_Release_Notes|'''Archivematica 0.7 Release Notes''']]
* [[Archivematica_0.8_Release_Notes|'''Archivematica 0.6 Release Notes''']]

== Templates ==

* [[Major_release_notes|Template for major releases]]
* [[Minor_release_notes|Template for minor releases]]

Archivematica 1.11 and Storage Service 0.16 release notes

2020-03-31T20:55:17Z

Sevein: /* Supported environments */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Archivematica is open source and doesn't have support. [https://www.artefactual.com/ Artefactual] provides support, and might be willing to support more (or possibly fewer) environments.

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team along with a number of community contributors has been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''' We were not able to test some storage spaces, including Sword2, LOCKSS-o-matic and DSpace.
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica 1.11 and Storage Service 0.16 release notes

2020-03-31T20:54:07Z

Sevein: /* Supported environments */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.11 and Storage Service 0.16

'''March ## 2020'''

==Supported environments==

Please see the [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/installation/installation/#installation installation instructions].

Archivematica 1.11 and Storage Service 0.16 have been tested in the following environments:

* Ubuntu 16.04 64-bit Server Edition
* Ubuntu 18.04 64-bit Server Edition
* CentOS 7 64-bit

Archivematica is open source and doesn't have support. Artefactual provides support, and might be willing to support more (or possibly fewer) environments.

For development purposes, most of our developers prefer to use Docker containers. These and all above supported environments are linked from the installation instructions above.

==Added==

===PREMIS Event import===

This feature allows the import of PREMIS events which took place prior to processing in Archivematica. The PREMIS events are written in an xml format (see [https://github.com/artefactual/archivematica-sampledata/blob/master/SampleTransfers/PremisImporter/metadata/premis.xml sample data]) and placed in the metadata folder of a transfer. The PREMIS events are then written to the AIP METS file.

This work was sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/import-metadata/#premis-xml Documentation]
* Issue: https://github.com/archivematica/Issues/issues/710

===S3 as a transfer source===

This allows an Amazon S3 space to be used as a transfer source location. This feature is a community PR from Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/storage-service-0.16/administrators/#s3-amazon Documentation]
* Issue: https://github.com/archivematica/Issues/issues/975

===Easier access to AIP METS===

This feature add a "View METS" button in the user interface when viewing an AIP in Archival Storage. The METS file is then downloaded to your desktop for your viewing pleasure.

* Documentation: pending
*Issue: https://github.com/archivematica/Issues/issues/644

===Zipped transfers===

This is a new transfer type that enables a zipped (non-bagged) package to be a transfer. Similar to the zipped bag transfer, the name of the package is used as the transfer name. This is a community contribution by Wellcome Collection. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/transfer/transfer/#transfer-types Documentation]
* Issue: https://github.com/archivematica/Issues/issues/682

===Add package name as configurable value to call backs===

When using AIP, AIC, and DIP store callbacks, the package_name is now a configurable value. This is a community contribution from Concordia University Libraries, who developed this to facilitate an EPrints to Archivematica workflow. Thank you!

* Documentation: pending
* Issue: https://github.com/archivematica/Issues/issues/978

==Changed==

===Performance and monitoring improvements===

This is a collection of issues fixed that improve performance for processing at scale, and also enable performance monitoring through external applications such as Prometheus and Grafana.

These updates have been sponsored by Piql and the Norwegian Health Archives. Thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/admin-manual/installation-setup/customization/instrumentation/ Documentation]
* Issues:
** Commonly used database tables don't have indexes: https://github.com/archivematica/Issues/issues/907
** MCPServer should reuse database connections: https://github.com/archivematica/Issues/issues/913
** Archivematica does not output metrics to analyze its performance: https://github.com/archivematica/Issues/issues/906
** MCPService must process all transfer packages sent to it at once: https://github.com/archivematica/Issues/issues/911
** Some jobs run even when disabled: https://github.com/archivematica/Issues/issues/866
** "Check transfer directory for objects" executed multiple times: https://github.com/archivematica/Issues/issues/782
** index_aip crashes elasticsearch for large transfers: https://github.com/artefactual/archivematica/issues/1199

===Improvements for full disks===

Managing workflows when various spaces on the disk fill up is a recognized pain point for Archivematica users. This project makes
three overall changes to storage space reporting in Archivematica and the Storage Service in an effort to mitigate these issues:
* Change the processing storage usage page to clarify storage paths/locations and improve usability
* Improve the transfer source location and AIP storage location pages to clarify storage paths/locations and improve usability
* Change Storage Service functionality to support the above changes.

* Documentation: pending
* [https://github.com/archivematica/Issues/issues?q=label%3A%22RED+TEAM%3A+disk+full+project%22+is%3Aclosed Issues]

===Changes to default normalization for videos/images===

Archivematica's default FPR normalization rules were creating in some cases very large video files for arguably no sound preservation reason. After discussion and community consultation, we have removed default video normalization rules. Users can still "opt in" to the rules but they are not enabled by default in '''new or upgraded''' installations. Any custom changes you have made to your own FPR will still be maintained after upgrade. We also removed default rules for preservation for PNG, JPG, GIF and DNG still images. For full details and affected formats, see [https://github.com/archivematica/Issues/issues/912#issuecomment-565197594 this comment] in the issue ticket.

* Issue: https://github.com/archivematica/Issues/issues/912

===Allow users to choose whether to receive fail report emails===

Users can now be configured to either receive fail report emails or not (previously all users received the emails). This is a community contribution from Hillel Arnold at Rockefeller Archive Center- thank you!

* [https://www.archivematica.org/en/docs/archivematica-1.11/user-manual/administer/dashboard-admin/#dashboard-users Documentation]
* Issue: https://github.com/archivematica/Issues/issues/709

===Change name of sanitize names micro-service===

Following reading a paper by [https://medium.com/on-archivy/invisible-defaults-and-perceived-limitations-processing-the-juan-gelman-files-4187fdd36759 Elvia Arroyo-Ramirez] we decided to change the name of this micro-service and align it more with the Library of Congress events vocabulary. The micro-service now displays as "Change transfer filenames" and "Change SIP filenames" in the Transfer and Ingest tabs respectively.

* Issue: https://github.com/archivematica/Issues/issues/230

===Drop-down menu orders===

In short, the order of options in drop down menus were all over the place and it was driving us nuts so we finally tried to put them in more logical orders.

* Issue: https://github.com/artefactual/archivematica/issues/891

==Removed==

===Quarantine===

As [https://groups.google.com/d/msg/archivematica/rGMsO8htF38/Pl-eLimwAgAJ discussed on the community forum] the quarantine micro-service has been removed from Archivematica in this release.
* Issue: https://github.com/artefactual/archivematica/issues/598
* ADR: https://github.com/archivematica/archivematica-architectural-decisions/blob/master/0008-remove-quarantine.md

===Add next disk image button===

This button seemed redundant to the workflow so it's been removed.
* Issue: https://github.com/archivematica/Issues/issues/290

==Fixed==

* Non-Dublin Core columns cause metadata re-ingest to fail ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1139
* RuntimeError which was causing sporadic workflow issues ('''Community contribution''' by Jorik van Kemanade- thank you!): https://github.com/archivematica/Issues/issues/1108
* Reindexing large transfer backlog error ('''Community contribution''' by Matt LaChance- thank you!): https://github.com/archivematica/Issues/issues/962
* Parallel bzip2 compression failing in am19rpm: https://github.com/archivematica/Issues/issues/606
* Fixity API endpoint and Fixity tool tail to check replicated AIPs ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1054
* Decision points break with 10 choices or more ('''Sponsored''' by Picturae- thank you!): https://github.com/archivematica/Issues/issues/850
* S3 us-east-1 fails when chosen as region in the Storage Service ('''Community contribution''' by Joseph Anderson, Fashion Institution of Technology- thank you!): https://github.com/archivematica/Issues/issues/922
* "Remove bagged files" reports failure when thumbnails aren't created: https://github.com/archivematica/Issues/issues/651
* Directories are greyed out while they still contain files available for arrangement ('''Sponsored''' by Simon Fraser University Archives- thank you!): https://github.com/archivematica/Issues/issues/822
* Dublin Core dmdSec not created if filename has diacritics: https://github.com/archivematica/Issues/issues/1073
* Cannot add metadata files through the UI ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1090
* GPG/TRANSFORMKEY being lost when reingesting an encrypted AIP: https://github.com/archivematica/Issues/issues/803
* Pointer file uses a mix of PREMIS2 and PREMIS3: https://github.com/archivematica/Issues/issues/820
* Failure to match in ArchivesSpace DIP Upload shows as success ('''Sponsored''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/258
* Allow S3 credentials to be blank ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/712
* Version of METS in mets-reader-writer is an older version: https://github.com/archivematica/Issues/issues/637
* S3 bucket name can't be configured: https://github.com/archivematica/Issues/issues/558
* Pointer files for reingested AIP has two compression events: https://github.com/archivematica/Issues/issues/1062
* Bags with metadata fail to ingest when additional metadata is added by automation tools ('''Sponsored''' by the Museum of Modern Art- thank you!): https://github.com/archivematica/Issues/issues/1022
* Transfer browser breaks if transfer source contains read protected directories: https://github.com/archivematica/Issues/issues/1019
* AIP status in dashboard does not update after AIP is deleted: https://github.com/archivematica/Issues/issues/1014
* SIPs started from ArchivesSpace pane fail when a parent object does not have a title ('''Community contribution''' by Dallas Pillen- thank you!): https://github.com/archivematica/Issues/issues/799
* Cannot create user with accented characters/diacritics: https://github.com/archivematica/Issues/issues/261
* AIP METS and pointer METS files reference outdated METS schema: https://github.com/archivematica/Issues/issues/949
* Cannot start a transfer if transfer name has diacritics: https://github.com/archivematica/Issues/issues/1051
* Non-default processing configuration is not copied over for zipped transfers ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/771
* Directory level AIP metadata is not indexed: https://github.com/archivematica/Issues/issues/888
* Descriptive metadata added via GUI is not indexed for searching: https://github.com/archivematica/Issues/issues/547
* External PIDs are not searchable in Archival storage ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1006
* Identifiers.json import fails if 'Bind PIDs' config option is not set to 'yes' ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/963
* Ldap auth fails on dashboard ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/841
* Cannot create storage service location via amclient ('''Sponsored''' by International Institute of Social History- thank you!): https://github.com/archivematica/Issues/issues/905
* It is difficult to combine status for different package types ('''Community contribution''' by Rockefeller Archive Center- thank you!): https://github.com/archivematica/Issues/issues/972
* Format identification errors are not being output from the FPR command ('''Community contribution''' by Wellcome Collection- thank you!): https://github.com/archivematica/Issues/issues/882
* Time zone setting not configurable ('''Sponsored''' by Piql/NHA- thank you!): https://github.com/archivematica/Issues/issues/1143
* Cannot store AIP with large files ('''Community contribution''' by Jorik van Kemenade- thank you!): https://github.com/archivematica/Issues/issues/981

And more! See https://github.com/archivematica/Issues/milestone/11 for full list of issues addresses in the 1.11 release.

==Upgraded tools and dependencies==

* Update to PRONOM v.96 https://github.com/archivematica/Issues/issues/791

==Known issues==

Please note that due to [https://github.com/archivematica/Issues/issues/1149 Issue 1149] the package replication functionality in the Storage Service does not work in this release. We anticipate fixing in the near future in a point release.

==End of life dependencies==

Python 2 has reached end of life. The Archivematica delivery team along with a number of community contributors has been working on upgrading this dependency. This release merges all Python 3 code that was ready in advance of the release, while still supporting Python 2. Components which have been upgraded and/or tested using Python 3 include:
* Dashboard: https://github.com/archivematica/Issues/issues/810
* Storage Service: https://github.com/archivematica/Issues/issues/806 '''Note''' We were not able to test some storage spaces, including Sword2, LOCKSS-o-matic and DSpace.
* amclient: https://github.com/archivematica/Issues/issues/817
* Automation tools: https://github.com/archivematica/Issues/issues/815
* Fixity: https://github.com/archivematica/Issues/issues/814
* am/compose: https://github.com/archivematica/Issues/issues/804
* Fido: https://github.com/archivematica/Issues/issues/847

We will continue to work toward full Python 3 use in upcoming releases.

Archivematica API

2019-07-09T16:48:26Z

Sevein: /* Status */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>).
<br \><br \>
''As GET parameters, format is:''<br \>
<code>?username=<''username''>&api_key=<''api_key''></code><br \>

''As a header, format is:''<br \>
<code>ApiKey <username>:<api_key></code><br \>

'''NOTE:''' The 'ApiKey' prefix is a requirement of the Tastypie library in use. See the Tastypie docs for [https://django-tastypie.readthedocs.io/en/latest/authentication.html#apikeyauthentication ApiKeyAuthentication].''<br \><br \>

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/api/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"
'''Note:''' for consumers of this endpoint, it is possible for Archivematica to return a <code>status</code> of <code>COMPLETE</code> without a <code>sip_uuid</code>. Consumers looking to use the UUID of the AIP that will be created following Ingest should therefore test for both a <code>status</code> of <code>COMPLETE</code> and the existence of <code>sip_uuid</code> that does not also equal <code>BACKLOG</code> to ensure that they retrieve it. This might mean an additional call to the status endpoint while this data becomes available.

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID ('''Note:''' as the package endpoint allows the caller to interact with Archivematica asynchronously it doesn't guarantee a transfer has started. The caller must use the UUID in the response to verify it has begun or any errors that were encountered initiating one.)
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>

=== Validate ===

Available in Archivematica 1.10+. This endpoint can be used to validate CSVs against embedded sets of rules. Currently works with Avalon Media System Manifest files.

In the future, this endpoint can be extended to support validation for <code>metadata.csv</code> or <code>rights.csv</code>, or other institutionally-based rules.

* '''URL''': <code>/api/v2beta/validate/avalon</code>
* '''Content-Type''': text/csv; charset=utf-8
* '''Parameters''': <code><document></code>

Usage example: (assuming that <code>avalon.csv</code> exists)

<pre>
curl http://127.0.0.1:62080/api/v2beta/validate/avalon \
--data-binary "@avalon.csv" \
--header "Authorization: ApiKey test:test" \
--header "Content-Type: text/csv; charset=utf-8"
</pre>

Response examples:

200 OK

<pre>
{
"valid": true
}
</pre>

400 Bad Request (expect reason to include different validation errors)

<pre>
{
"valid": false,
"reason": "Administrative data must include reference name and author."
}
</pre>

404 Not Found

<pre>
{
"error": true,
"message": "Unknown validator. Accepted values: avalon"
}
</pre>

[[Category:Development documentation]]

Archivematica 1.9.2 release notes

2019-06-20T16:53:16Z

Sevein: /* Upgrading */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.9.2 and Storage Service 0.14.2 release notes

'''Release date: '''

For more information about Archivematica 1.9.x, please see the release notes for [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|Archivematica 1.9.0]].

==Upgrading==

If you are upgrading from 1.9.0 to 1.9.2, please ensure that you run the following command:

<pre>
curl -XPUT 'http://localhost:9200/aips,aipfiles,transfers,transferfiles/_settings' \
-H "Content-Type: application/json" \
-d '{"index.mapping.total_fields.limit": 10000, "index.mapping.depth.limit": 1000 }'
</pre>

This is not required for upgrading from 1.8.x to 1.9.2.

For more information on upgrading, see [https://www.archivematica.org/en/docs/archivematica-1.9/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading Archivematica] in the documentation.

==Fixed==

* Index AIP errors due to asynchronous processing: [https://github.com/archivematica/Issues/issues/425 Issue 425]
* Error trying to connect to MCP server: [https://github.com/archivematica/Issues/issues/624 Issue 624]

Archivematica 1.9.2 release notes

2019-06-20T16:52:59Z

Sevein: /* Upgrading */

[[Main_Page|Home]] > [[Release_Notes|Release Notes]] > Archivematica 1.9.2 and Storage Service 0.14.2 release notes

'''Release date: '''

For more information about Archivematica 1.9.x, please see the release notes for [[Archivematica_1.9_and_Storage_Service_0.14_release_notes|Archivematica 1.9.0]].

==Upgrading==

If you are upgrading from 1.9.0 to 1.9.2, please ensure that you run the following command:

<code>
curl -XPUT 'http://localhost:9200/aips,aipfiles,transfers,transferfiles/_settings' \
-H "Content-Type: application/json" \
-d '{"index.mapping.total_fields.limit": 10000, "index.mapping.depth.limit": 1000 }'
</code>

This is not required for upgrading from 1.8.x to 1.9.2.

For more information on upgrading, see [https://www.archivematica.org/en/docs/archivematica-1.9/admin-manual/installation-setup/upgrading/upgrading/#upgrade Upgrading Archivematica] in the documentation.

==Fixed==

* Index AIP errors due to asynchronous processing: [https://github.com/archivematica/Issues/issues/425 Issue 425]
* Error trying to connect to MCP server: [https://github.com/archivematica/Issues/issues/624 Issue 624]

Archivematica API

2019-02-26T22:43:28Z

Sevein: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: Used to link to metadata sets added via the user interface. It's safe to ignore this for now since metadata can't be associated to transfer via the API at the moment.
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Archivematica API

2019-02-26T21:30:07Z

Sevein: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration], e.g. default, automated, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: TBC
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Archivematica API

2019-02-26T21:29:11Z

Sevein: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>processing_config</code>: [https://www.archivematica.org/en/docs/latest/user-manual/administer/dashboard-admin/#processing-configuration Processing configuration] to be used during, default: <code>default</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: TBC
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Archivematica API

2019-02-26T21:25:41Z

Sevein: /* Package */

[[Main Page]] > [[Development]] > Archivematica API

Endpoints require authentication with a username and API key. This can be submitted as GET parameters (eg <code>?username=demo&api_key=e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>) or as a header (eg <code>Authorization: ApiKey demo:e6282adabed84e39ffe451f8bf6ff1a67c1fc9f2</code>)

Endpoints return JSON. If there's an error, they will return a 4xx or 5xx HTTP error code and a JSON body <code>{'error': True, 'message': 'message describing error'}</code>

== Transfer ==

=== Start Transfer ===
* '''URL''': <code>/api/transfer/start_transfer/</code>
* '''Verb''': POST
* Start a transfer.
* '''Parameters''': JSON body
** <code>name</code>: Name of new transfer
** <code>type</code>: Type of the new transfer. One of: standard, unzipped bag, zipped bag, dspace
** <code>accession</code>: Accession number of new transfer
** <code>paths[]</code>: List of base64-encoded "<location_uuid>:<relative_path>" to be copied into the new transfer. Location UUIDs should be associated with this pipeline, and relative path should be relative to the location (TODO confirm relative vs absolute path). E.g. NWJiYWJjMTMtMTIyNy00MWE3LWIwY2QtZjJhYzM1MjkxZTdmOi92YWdyYW50L3NhbXBsZWRhdGEvQ1NWbWV0YWRhdGE= (decoded: 5bbabc13-1227-41a7-b0cd-f2ac35291e7f:/vagrant/sampledata/CSVmetadata)
** <code>row_ids[]</code>: ID of the associated TransferMetadataSet for disk image ingest. Can be provided as [""]
* '''Response''': JSON
** <code>message</code>: "Copy successful."
** <code>path</code>: Path the transfer was copied to on start?

=== List Unapproved Transfers ===
* '''URL''': <code>/api/transfer/unapproved</code>
* '''Verb''': GET
* Returns a list of transfers waiting for approval.
* '''Response''': JSON
** <code>message</code>: "Fetched unapproved transfers successfully."
** <code>results</code>: List of dicts with keys:
*** <code>type</code>: Transfer type. One of: standard, unzipped bag, zipped bag, dspace
*** <code>directory</code>: Directory the transfer is in currently
*** <code>uuid</code>: UUID of the transfer

=== Approve Transfer ===
* '''URL''': <code>/api/transfer/approve</code>
* '''Verb''': POST
* Approve a transfer waiting to be started.
* '''Parameters''': JSON body
** <code>type</code>: Type of the transfer to approved. One of: standard, unzipped bag, zipped bag, dspace.
** <code>directory</code>: Name of the directory for the transfer to approve
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>uuid</code>: UUID of the approved transfer

=== Status ===
* '''URL''': <code>/transfer/status/<transfer UUID>/</code>
* '''Verb''': GET
* Returns the status of the transfer.
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the transfer, e.g. "imgs"
** <code>sip_uuid</code>: If status is COMPLETE, this field will exist with either the UUID of the SIP or 'BACKLOG'
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <transfer UUID> successfully."
** <code>type</code>: "transfer"
** <code>uuid</code>: UUID of the transfer, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/transfer/<transfer UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a transfer
* '''Response''': JSON
** <code>removed</code>: True

=== Completed ===
* '''URL''': <code>/api/transfer/completed/</code>
* '''Verb''': GET
* Return list of Transfers that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed transfers successfully."
** <code>results</code>: List of UUIDs of completed Transfers
* Added in Archivematica 1.6

=== Start Reingest ===
* '''URL''': <code>/api/transfer/reingest</code>
* '''Verb''': POST
* Start a full reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested transfer

== Ingest ==

=== Status ===
* '''URL''': <code>/ingest/status/<unit UUID>/</code>
* '''Verb''': GET
* Returns the status of the SIP
* '''Response''': JSON
** <code>status</code>: One of FAILED, REJECTED, USER_INPUT, COMPLETE or PROCESSING
** <code>name</code>: Name of the SIP, e.g. "imgs"
** <code>microservice</code>: Name of the current microservice
** <code>directory</code>: Name of the directory, e.g. "imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c"
** <code>path</code>: Full path to the transfer, e.g. "/var/archivematica/sharedDirectory/currentlyProcessing/imgs-52dd0c01-e803-423a-be5f-b592b5d5d61c/"
** <code>message</code>: "Fetched status for <SIP UUID> successfully."
** <code>type</code>: "SIP"
** <code>uuid</code>: UUID of the SIP, e.g. "52dd0c01-e803-423a-be5f-b592b5d5d61c"

=== Hide ===
* '''URL''': <code>/api/ingest/<SIP UUID>/delete/</code>
* '''Verb''': DELETE
* Hide a SIP
* '''Response''': JSON
** <code>removed</code>: True

=== List SIPS Waiting for User Input ===
* '''URL''': <code>/api/ingest/waiting</code>
* '''Verb''': GET
* Returns a list of SIPs waiting for user input.
* '''Response''': JSON
** <code>message</code>: "Fetched units successfully."
** <code>results</code>: List of dicts with keys:
*** <code>sip_directory</code>: Directory the SIP is in currently
*** <code>sip_uuid</code>: UUID of the SIP
*** <code>sip_name</code>: Name of the SIP
*** <code>microservice</code>: Name of the current microservice

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement Note: Despite the URL, this currently returns both SIPs & transfers that are waiting for user input. A separate <code>/api/transfer/waiting</code> should be added for transfers.
|}

=== Completed ===
* '''URL''': <code>/api/ingest/completed/</code>
* '''Verb''': GET
* Return list of SIPs that are completed
* '''Response''': JSON
** <code>message</code>: "Fetched completed ingests successfully."
** <code>results</code>: List of UUIDs of completed SIPs
* Added in Archivematica 1.6

=== Reingest ===
* '''URL''': <code>/api/ingest/reingest</code>
* '''Verb''': POST
* Start a partial or metadata-only reingest.
* '''Parameters''': JSON body
** <code>name</code>: Name of the AIP. The AIP should also be found at <code>%sharedDirectory%/tmp/<name></code>
** <code>uuid</code>: UUID of the AIP to reingest
* '''Response''': JSON
** <code>message</code>: "Approval successful."
** <code>reingest_uuid</code>: UUID of the reingested SIP

=== Copy Metadata ===
* '''URL''': <code>/api/ingest/copy_metadata_files/</code>
* '''Verb''': POST
* Add metadata files to a SIP.
* '''Parameters''': JSON body
** <code>sip_uuid</code>: UUID of the SIP to put files in
** <code>source_paths[]</code>: List of files to be copied, base64 encoded, in the format 'source_location_uuid:full_path'
* '''Response''': JSON
** <code>error</code>: False
** <code>message</code>: "Files added successfully."

== Administration ==

=== Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/levels/</code>
* '''Verb''': GET?
* Returns a JSON-encoded set of the configured levels of description.
* ''Response'': JSON. List of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

=== Fetch Levels of Description ===
* '''URL''': <code>/api/administration/dips/atom/fetch_levels/</code>
* '''Verb''': GET?
* Fetch all levels of description from an AtoM database, replacing any previously existing.
* '''Response''': JSON. Updated list of AtoM Levels of description with key 'UUID' and value 'name of level of description'.

== Other ==

=== Path Metadata ===
* '''URL''': <code>/api/filesystem/metadata/</code>
* '''Verb''': GET, POST
* Fetch (GET) or update (POST) metadata for a path (currently only level of description).
* '''Parameters''': Query string parameters (GET) or JSON body (POST)
** <code>path</code>: Arranged path to get metadata on
* '''GET response''': JSON
** <code>level_of_description</code>: Level of description
* '''POST response''': JSON
** <code>success</code>: True

=== Processing Configuration ===
* '''URL''': <code>/api/processing-configuration/<name></code>
* '''Verb''': GET?
* Return processing configuration with <name>
* '''Response''': The processing config file as a stream.
* '''Content type''': text/xml

== Beta endpoints ==

API endpoints which are still in-flux and that could potentially change.

=== Package ===

Provides additional functionality over the start and approve transfer endpoints, for example, wrapping both those steps into a single call.

* '''URL''': <code>/api/v2beta/package</code>
* '''Verb''': POST
* Start a new transfer type.
* '''Parameters''': JSON body
** Mandatory
*** <code>name</code>: Transfer name
*** <code>path</code>: Path relative, or absolute to a storage service transfer source
**Optional
*** <code>type</code>: Transfer type, e.g. standard, dataverse, zipped bag, default: <code>standard</code>
*** <code>accession</code>: Accession ID
*** <code>access_system_id</code>: Access system ID (see [https://www.archivematica.org/en/docs/latest/user-manual/access/access docs])
*** <code>metadata_set_id</code>: TBC
*** <code>auto_approve</code>: Boolean <code>True</code> or <code>False</code> to set the transfer to auto-approve, default: <code>True</code>
* '''Response''': JSON
** <code>id</code>: Transfer UUID
'''Examples:'''
<br/>
Only a subset of these options might be needed for most use-cases. A fundamental difference between the <code>package</code> endpoint and others from which a transfer can be initiated is that a storage service transfer location UUID isnt always required. In some cases that might still be ideal.
<br/><br/>
''Starting a transfer using an absolute path:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n '/home/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_absolute\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
''Starting a transfer using an relative path with a transfer source UUID:''
<br/>
<pre>
curl -v POST \
-H "Authorization: ApiKey test:test" \
-H "Content-Type: application/json" \
-d "{\
\"path\": \"$(echo -n 'd1184f7f-d755-4c8d-831a-a3793b88f760:/archivematica/archivematica-sampledata/SampleTransfers/DemoTransfer' | base64 -w 0)\", \
\"name\": \"demo_transfer_relative\", \
\"processing_config\": \"automated\", \
\"type\": \"standard\" \
}" \
"http://<archivematica-uri>/api/v2beta/package"
</pre>
[[Category:Development documentation]]

Release Process

2019-01-10T17:58:32Z

Sevein: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python2 -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* See this pull request submitted to update to PRONOM 90: https://github.com/openpreserve/fido/pull/110
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request.
# Release new version of FIDO. Make sure to update __init__.py AND the versions XML with a new version before creating a release.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55
</div>

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python testproject/manage.py dumpdata fpr > fpr-current.json</code>
# Generate a blank data migration
#* E.g. <code>python testproject/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python testproject/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v89.xml --output-filename pronom89.txt</code>
# Copy the output into the blank migration above the Migration class
# Add the below RunPython operation into the Migration class, in the operations list
#* <code>migrations.RunPython(data_migration),</code>
# <s>Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports</s>
#* <s>Remove the <code>apps.get_model</code> lines</s>
#* <s>Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code></s>
# Commit
# Deploy on testing pipeline or locally

(Analysts)
# Update the new entries
#* '''Ask to edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly'''
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analysts)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python testproject/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python testproject/manage.py get_fpr_changes fpr-current.json fpr-updated.json fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove the direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge
# Make new FPR release. Don't forget to update fpr/__init__.py! Then tag the release (e.g. `v.1.7.9`) and push the latest tag.
# Update AM's dashboard base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/requirements/base.txt]

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via gitolite (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin (for now; hopefully Jenkins for 1.8) builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".
* https://github.com/artefactual/archivematica/releases
* https://github.com/artefactual/archivematica-storage-service/releases

'''Step 11''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 12''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 13''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Release Process

2019-01-10T17:58:16Z

Sevein: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python2 -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* See this pull request submitted to update to PRONOM 90: https://github.com/openpreserve/fido/pull/110
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request.
# Release new version of FIDO. Make sure to update __init__.py AND the versions XML with a new version before creating a release.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55
</div>

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python testproject/manage.py dumpdata fpr > fpr-current.json</code>
# Generate a blank data migration
#* E.g. <code>python testproject/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python testproject/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v89.xml --output-filename pronom89.txt</code>
# Copy the output into the blank migration above the Migration class
# Add the below RunPython operation into the Migration class, in the operations list
#* <code>migrations.RunPython(data_migration),</code>
# <s>Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports</s>
#* <s>Remove the <code>apps.get_model</code> lines</s>
#* <s>Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code></s>
# Commit
# Deploy on testing pipeline or locally

(Analysts)
# Update the new entries
#* '''Ask to edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly'''
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analysts)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python testproject/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python testproject/manage.py get_fpr_changes fpr-current.json fpr-updated.json fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove the direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge
# Make new FPR release. Don't forget to update fpr/__init__.py! Then tag the release (e.g. `v.1.7.9`) and push the latest tag.
# Update AM's dashboard base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/requirements/base.txt]

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via gitolite (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1

'''Step 3''': Sysadmin (for now; hopefully Jenkins for 1.8) builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release]. Make sure that the releases are marked as "published".

** https://github.com/artefactual/archivematica/releases
** https://github.com/artefactual/archivematica-storage-service/releases

'''Step 11''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 12''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 13''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]

Release Process

2019-01-10T17:57:23Z

Sevein: /* Release Day Checklist */

This is an outline/checklist of the process to create Archivematica & Storage service releases.

== Overview ==

# Merge new features
# Test new features
# [[#Update PRONOM]]
# Write documentation
# Update dependencies
# Update version
# [[#Build deb/rpm packages]]
# Test packages for new installs and upgrades
# [[#Tag Release]]
# Update ansible roles
# Announce release

== Translations ==

'''Needs to be improved!'''

* Determine code freeze / call for translations process
* Describe processes: push and pull - and when it needs to happen
* We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
*# We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
*# We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
* Affected repositories
** archivematica-storage-service
** archivematica-workflow
** archivematica-dashboard
*** Includes archivematica-fpr-admin
*** Includes appraisal-tab

== Update PRONOM ==

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

=== Update FIDO ===

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

# Checkout fido from https://github.com/openpreserve/fido
# Update signatures
#* Run <code>python2 -m fido.update_signatures</code> from the fido repository root
# Add:
#* New signature file <code>fido/conf/DROID_SignatureFile-v##.xml</code>
#* New formats file <code>fido/conf/formats-v##.xml</code>
#* New PRONOM zip file <code>fido/conf/pronom-xml-v##.zip</code>
#* Updated <code>fido/conf/versions.xml</code>
# Remove:
#* Old signature file
#* Old formats file
#* Old PRONOM zip file
# Update the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
#* See this pull request submitted to update to PRONOM 90: https://github.com/openpreserve/fido/pull/110
# Update <code>fido/fido.py</code> to point to the new format_files
# Create pull request.
# Release new version of FIDO. Make sure to update __init__.py AND the versions XML with a new version before creating a release.

Remember to package FIDO and Siegfried in Archivematica! See next step below.

=== Package FIDO ===

FIDO is packaged via [http://pypi.org/ PyPi] under opf-fido. The Makefile includes a `make package` command that will do the update.

MCPClient's base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/requirements/base.txt] will need to be updated with the latest version.

=== Package Siegfried ===

First, clone the [https://github.com/artefactual-labs/am-packbuild/ am-packbuild] repo. Update the Makefiles available at <code>rpm/siegfried/Makefile</code>, and <code>deb/siegfried/Makefile</code> , and run <code>make</code> in each directory to build the packages.

=== Update FPR ===

<div class="text-alert">
* In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
* At the end of the process: '''remember''' to remove "pk"s from the final migration fixture! E.g. see PR #55
* Examples
** PRONOM 92:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/63
*** ???
** PRONOM 90:
*** https://github.com/artefactual/archivematica-fpr-admin/pull/51
*** https://github.com/artefactual/archivematica-fpr-admin/pull/55
</div>

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See [[#Update FIDO]]

# Generate a JSON with the current version of the FPR (for use later)
#* <code>python testproject/manage.py dumpdata fpr > fpr-current.json</code>
# Generate a blank data migration
#* E.g. <code>python testproject/manage.py makemigrations --empty --name pronom_89 fpr</code>
# Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
#* E.g. <code>python testproject/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v89.xml --output-filename pronom89.txt</code>
# Copy the output into the blank migration above the Migration class
# Add the below RunPython operation into the Migration class, in the operations list
#* <code>migrations.RunPython(data_migration),</code>
# <s>Due to a [https://projects.artefactual.com/issues/10466#change-46673 bug], fix the imports</s>
#* <s>Remove the <code>apps.get_model</code> lines</s>
#* <s>Import the models directly with <code>from fpr.models import Format, FormatVersion, IDRule</code></s>
# Commit
# Deploy on testing pipeline or locally

(Analysts)
# Update the new entries
#* '''Ask to edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly'''
#* Move new formats to a more appropriate category
#* Create rules & commands
#* Test with data for new formats
(End Analysts)

# Generate a JSON with the updated version of the FPR on the testing pipeline
#* <code>python testproject/manage.py dumpdata fpr > fpr-updated.json</code>
# Get the updates as JSON
#* E.g. <code>python testproject/manage.py get_fpr_changes fpr-current.json fpr-updated.json fpr/migrations/pronom_89.json</code>
# Update the migration to load the JSON updates
#* Replace the contents of the data migration function with
#** <code>fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')</code>
#** <code>call_command('loaddata', fixture_file, app_label='fpr')</code>
# Review JSON -- some IDs with multipleformats are being imported and will have to be manually reviewed until bug is identified/corrected.
#* Remove the direct imports from the bug
#* '''Remove the pk's from the entries in the JSON document.'''
#* Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
# Commit
# Merge
# Make new FPR release. Don't forget to update fpr/__init__.py! Then tag the release (e.g. `v.1.7.9`) and push the latest tag.
# Update AM's dashboard base.txt [https://github.com/artefactual/archivematica/blob/qa/1.x/src/dashboard/src/requirements/base.txt]

== Update dependencies ==

=== Python Packages ===

[https://github.com/artefactual-labs/mets-reader-writer metsrw] and [https://github.com/artefactual-labs/agentarchives agentarchives] both have Makefiles that handle most of the packaging

# Check for open PRs, merge as necessary
# Update <code>setup.py</code> with the new version, create a pull request, code review, [[Merging | merge]].
# Tag new release, push tag
#* <code>git push --tags</code>
# Run <code>make package</code>
#* This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
#* The Makefile was created from instructions at https://packaging.python.org/distributing/
# <code>make clean</code> will delete packaging related files

== Update version ==

# Update PREMIS agent to Archivematica-X.X.X
# Update Dashboard-Administration-Version to X.X.X
# Update Storage services-Admin-Version to X.X.X

== Build deb/rpm packages ==

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

=== Debian packages ===

# Clone the [https://github.com/artefactual-labs/am-packbuild am-packbuild] repo. Latest work is available in master
# Put your gpg private key into <code>debs/GPG-KEYS-REPOS</code>. That's the place the Dockerfile looks for it when building the environment.
# Update the makefile at <code>debs/archivematica/Makefile</code> in order to reflect version/keys you want to use.
# Run <make>, and the packages will be available in the <code>build</code> once the building finishes.
# Upload packages to public debian repository

=== Debian reposities ===

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

#* Create folder for repo, and configuration file:
<code>
mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF
</code>

#* Go inside the repo, and import the packages previously uploaded with:
<code>
cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

</code>

The current official repo is at packages.archivematica.org

=== RPM Packages ===

# Package specs are available in [https://github.com/artefactual-labs/am-packbuild/tree/dev/packaging-1.6-docker/rpm am-packbuild/rpms]
# There are vars in the Makefiles for version/release , so update them when needed
# In order to build them, just go into the directory you want to build, and run “make”

=== RPM Repositories ===

Once the packages are built, upload them to packages.archivematica.org/<version>/centos
Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

* ''rpm --addsign *.rpm'' (already signed packages will be skipped)

* ''createrepo .'' (For packages other than archivematica , use “centos-extras” repository)

* ''gpg --detach-sign --armor repodata/repomd.xml''

The first gpg command signs the rpms, and the later signs the repository content.

=== Development stage ===

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

== Website ==

'''Needs to be improved!'''

==== Homepage ====

* Make changes in archivematica-web.git
** Update links
** Add new doc repos
* Deploy
** Log in sites-pub as archivematica-web and run update-web.sh

==== Documentation ====

* Deploy
** Log in sites-pub as archivematica-web and run update-docs.sh

==== Wiki ====

* Release notes
* Installation notes
* ...

== News ==

* Twitter
* Mailing list
* News section in artefactual.com

== Update ansible roles ==

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

== Tag Release ==

# Add release tags to repositories
#* Archivematica
#* Storage Service
#* FPR-admin
#* appraisal tab
#* transfer browser
#* Others?
# Create stable/#.x branch
# Delete qa/#.x branch if necessary

== Release Day Checklist ==

'''Step 0''': Operations Team checklist tasks are complete (as of September 2018, this is an internal Trello list).
*At this point, there should be a release candidate that has been tested. A release candidate always precedes a tagged release. If any new issues have been uncovered, a new release candidate should be made and tested before proceeding with the rest of the steps.

'''Step 1''': Release Captain decides whether the current release candidate is ready to release

* Look at all recently filed issues in GitHub -- is anything concerning / relevant for this release? Does anything need to be addressed? If an issue is uncovered that does need to be fixed, this restarts the checklist back to Step 0.
* Have automated tests passed?

'''Step 2''': Release Captain creates a new tag for the release via gitolite (e.g. `v1.7.0` or `v0.11.0` for the Storage Service) or assigns someone else to do it.

* Ensure you are adding the tag to the right commit! It should match the last commit of the final release candidate.
* You can create the tags from GitHub or from CLI: <code>git tag $VERSION $REVISION</code>and <code>git push origin refs/tags/$VERSION</code>
* Make sure that the version is valid,
** Valid values: v1.8.1, v1.8.1-rc.1
** Invalid values: 1.8.1, 1.8, 1.8.1-rc1, v1.8.1-rc1
* Publish the tags in GitHub!
** https://github.com/artefactual/archivematica/releases
** https://github.com/artefactual/archivematica-storage-service/releases
** Do not forget to include a link to the release notes! Use previous releases as a reference on how to do it.

'''Step 3''': Sysadmin (for now; hopefully Jenkins for 1.8) builds new packages using the release tag.

* See the internal wiki for steps.

'''Step 4''': Sysadmin copies new packages to the proper repository (e.g., https://packages.archivematica.org/1.7.x/)

* See the internal wiki for steps.

'''Step 5''': Analyst updates the Archivematica documentation links in the install / upgrade section with the correct package names and locations (e.g. conf.py in docs repo)

'''Step 6''': Sysadmin updates deploy-pub to use the new links.

'''Step 7''': Developer updates archivematica-web (managed in Gitolite / Gitlab) to show the new release.

* See the internal wiki for steps.

'''Step 8''': Developer changes the default branches in GitHub and Gitolite and updates references in https://gist.github.com/qubot.

'''Step 9''': Systems administrator updates am-packbuild and upgrades public and private demo sites.

* See the internal wiki for steps.

'''Step 10''': Release Captain finalizes the [https://wiki.archivematica.org/Release_Notes release notes] and adds a link to them in the [https://github.com/artefactual/archivematica/releases GitHub release].

'''Step 11''': Release Captain posts a notification to the [https://groups.google.com/forum/#!forum/archivematica Archivematica Google Group] and the News section of the Artefactual website.

'''Step 12''': Release Captain closes all release-related issues in the Verified column in Waffle.

'''Step 13''': All involved eat cake (for a major release) or cupcakes (for a minor release).

[[Category:Process documentation]]