Release Process

This is an outline/checklist of the process to create Archivematica & Storage service releases.

Overview

1. Merge new features
2. Test new features
3. #Update PRONOM
4. Write documentation
5. Update dependencies
6. #Build deb/rpm packages
7. Test packages for new installs and upgrades
8. #Tag Release
9. Update ansible roles
10. Announce release

Update PRONOM

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

Update FIDO

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

1. Checkout fido from https://github.com/openpreserve/fido
2. Update signatures
   • Run python -m fido.update_signatures from the fido repository root
3. Add:
   • New signature file fido/conf/DROID_SignatureFile-v##.xml
   • New formats file fido/conf/formats-v##.xml
   • New PRONOM zip file fido/conf/pronom-xml-v##.zip
   • Updated fido/conf/versions.xml
4. Remove:
   • Old signature file
   • Old formats file
   • Old PRONOM zip file
5. Update fido/fido.py to point to the new format_files
6. Create pull request. This may trigger a release from FIDO

Package FIDO

First, clone the am-packbuild repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at rpm/fido/Makefile, and deb/fido/Makefile , and run make in each directory to build the packages.

Package Siegfried

First, clone the am-packbuild repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at rpm/siegfried/Makefile, and deb/siegfried/Makefile , and run make in each directory to build the packages.

Update FPR

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See #Update FIDO

1. Install the latest stable Archivematica release
2. Generate a JSON with the current version of the FPR (for use later)
   • ./src/dashboard/src/manage.py dumpdata fpr > fpr-current.json
3. Install the latest development version of Archivematica
4. Generate a blank data migration in the FPR-admin module
   • E.g. ./src/dashboard/src/manage.py makemigrations --empty --name pronom_89 fpr
5. Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
   • E.g. am import-pronom-ids fido/fido/conf/formats-v89.xml --output-filename archivematica/pronom89.txt
6. Copy the output into the blank migration, add RunPython operation
   • migrations.RunPython(data_migration),
7. Due to a bug, fix the imports
   • Remove the apps.get_model lines
   • Import the models directly with from fpr.models import Format, FormatVersion, IDRule
8. Commit
9. Deploy on testing pipeline
10. (Analysts) Update the new entries
    • Move new formats to a more appropriate category
    • Create rules & commands
    • Test with data for new formats
11. Generate a JSON with the updated version of the FPR on the testing pipeline
    • ./src/dashboard/src/manage.py dumpdata fpr > fpr-updated.json
12. Get the updates as JSON
    • E.g. am get-fpr-changes fpr-current.json fpr-updated.json src/dashboard/src/fpr/migrations/pronom_89.json
13. Update the migration to load the JSON updates
    • Replace the contents of the data migration function with
      • fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')
      • call_command('loaddata', fixture_file, app_label='fpr')
    • Remove the direct imports from the bug
    • Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
14. Commit
15. Merge

Update dependencies

metsrw

1. Check for open PRs, merge as necessary
2. Update setup.py with the new version, pull request, merge
3. Tag new release, push tag
4. Run make package
   • This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
   • The Makefile was created from instructions at https://packaging.python.org/distributing/
5. make clean will delete packaging related files

agentarchives

1. Check for open PRs, merge as necessary
2. Update setup.py with the new version, pull request, merge.
3. Tag new release, push tag
   • git push --tags
4. Run make package
   • This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
   • The Makefile was created from instructions at https://packaging.python.org/distributing/
5. make clean will delete packaging related files

Build deb/rpm packages

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

Debian packages

1. Clone the am-packbuild repo. Latest work is available in dev/packages-1.6-docker
2. Put your gpg private key into debs/GPG-KEYS-REPOS. That's the place the Dockerfile looks for it when building the environment.
3. Update the makefile at debs/archivematica/Makefile in order to reflect version/keys you want to use. If you want to build the storage service, you need also to change the packbuild.py line to reflect that (TODO: create service-storage package folder)

This dockerfile uses packbuild.py script, whose function is downloadig the code from github, creating a debian/changelog file for each ubuntu version with the expected version name and description of last commit included, and build the package aftewards. It can be run on their own too, without the need of a docker environment.

1. Run <make>, and the packages will be available in the build once the building finishes.
2. Upload packages to public debian repository

Debian reposities

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

1. • Create folder for repo, and configuration file:

mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF

1. • Go inside the repo, and import the packages previously uploaded with:

cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

The current official repo is at packages.archivematica.org

RPM Packages

1. Package specs are available in am-packbuild/rpms
2. There are vars in the Makefiles for version/release , so update them when needed
3. In order to build them, just go into the directory you want to build, and run “make”

RPM Repositories

Once the packages are built, upload them to packages.archivematica.org/<version>/centos Go inside that dir, and run “createrepo .” (For packages other than archivematica , use “centos-extras” repository)

Website

Needs to be improved!

Homepage

• Make changes in archivematica-web.git
  • Update links
  • Add new doc repos
• Deploy
  • Log in sites-pub as archivematica-web and run update-web.sh

Documentation

• Deploy
  • Log in sites-pub as archivematica-web and run update-docs.sh

Wiki

• Release notes
• Installation notes
• ...

News

• Twitter
• Mailing list
• News section in artefactual.com

Update ansible roles

Check that the deploy-pub vars files for archivematica are updated, and the ansible-ansiblematica-src and ansible-archivematica-pkg roles are able to deploy the new version

Tag Release

1. Add release tags to repositories
   • Archivematica
   • Storage Service
   • FPR-admin
   • appraisal tab
   • transfer browser
   • Others?
2. Create stable/#.x branch
3. Delete qa/#.x branch if necessary