Difference between revisions of "Release Process"

Revision as of 10:34, 29 August 2018

This is an outline/checklist of the process to create Archivematica & Storage service releases.

Overview

Merge new features
Test new features
#Update PRONOM
Write documentation
Update dependencies
Update version
#Build deb/rpm packages
Test packages for new installs and upgrades
#Tag Release
Update ansible roles
Announce release

Translations

Needs to be improved!

Determine code freeze / call for translations process
Describe processes: push and pull - and when it needs to happen
We made a choice on how we're using Transifex to keep things simple: only one branch at a time pushed to Transifex. E.g. once SS 0.10.0 is released we have to decide if:
1. We move Transifex to stable/0.10.x for a while so we can work on a minor release with translation fixes (e.g. 0.10.1), or
2. We move to qa/0.11.x which would only make possible to bring new translations to SS 0.11.0.
Affected repositories
- archivematica-storage-service
- archivematica-workflow
- archivematica-dashboard
  - Includes archivematica-fpr-admin
  - Includes appraisal-tab

Update PRONOM

PRONOM needs to be updated in our file identification tools, FIDO & Siegfried, as well as in the FPR.

Update FIDO

The FPR update currently use FIDO as a source for new PRONOM, since it is formatted nicer than PRONOM offers, so we depend on FIDO having updated their PRONOM. If that has not happened, we can generate a new formats-v##.xml by updating signatures manually. Artefactual can also update PRONOM and submit a PR to FIDO.

Checkout fido from https://github.com/openpreserve/fido
Update signatures
- Run python2 -m fido.update_signatures from the fido repository root
Add:
- New signature file fido/conf/DROID_SignatureFile-v##.xml
- New formats file fido/conf/formats-v##.xml
- New PRONOM zip file fido/conf/pronom-xml-v##.zip
- Updated fido/conf/versions.xml
Remove:
- Old signature file
- Old formats file
- Old PRONOM zip file
Update fido/fido.py to point to the new format_files
Create pull request. This may trigger a release from FIDO

Also consider:

Update the container signatures (download here: https://www.nationalarchives.gov.uk/aboutapps/pronom/droid-signature-files.htm)
See this pull request submitted to update to PRONOM 90: https://github.com/openpreserve/fido/pull/110
- Notice that it also updates the container signatures
Remember to publish the wheel to pypi! https://pypi.python.org/pypi/opf-fido (see the Makefile)

Package FIDO

First, clone the am-packbuild repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at rpm/fido/Makefile, and deb/fido/Makefile , and run make in each directory to build the packages.

Package Siegfried

First, clone the am-packbuild repo. The most recent work has been done in branch dev/packages-1.6-docker, and needs to be merged into main.

Update the Makefiles available at rpm/siegfried/Makefile, and deb/siegfried/Makefile , and run make in each directory to build the packages.

Update FPR

In https://github.com/artefactual/archivematica-fpr-admin/pull/51, the devtools used here to generate the migration were included in archivematica-fpr-admin as django-admin management tools.
At the end of the process: remember to remove "pk"s from the final migration fixture! E.g. see PR #55
Examples
- PRONOM 92:
  - https://github.com/artefactual/archivematica-fpr-admin/pull/63
  - ???
- PRONOM 90:
  - https://github.com/artefactual/archivematica-fpr-admin/pull/51
  - https://github.com/artefactual/archivematica-fpr-admin/pull/55

This process is currently more convoluted than it needs to be, and should be simplified.

This depends on FIDO having updated their PRONOM files. See #Update FIDO

Generate a JSON with the current version of the FPR (for use later)
- python testproject/manage.py dumpdata fpr > fpr-current.json
Generate a blank data migration
- E.g. python testproject/manage.py makemigrations --empty --name pronom_89 fpr
Generate the FPR migration body. This also updates the local database's FPR with the new PRONOM IDs
- E.g. python testproject/manage.py import_pronom_ids path/to/fido/fido/conf/formats-v89.xml --output-filename pronom89.txt
Copy the output into the blank migration above the Migration class
Add the below RunPython operation into the Migration class, in the operations list
- migrations.RunPython(data_migration),
~Due to a bug, fix the imports~
- ~Remove the apps.get_model lines~
- ~Import the models directly with from fpr.models import Format, FormatVersion, IDRule~
Commit
Deploy on testing pipeline or locally

(Analysts)

Update the new entries
- Ask to edit ONLY entries added by the latest PRONOM update otherwise the fixture won't work properly
- Move new formats to a more appropriate category
- Create rules & commands
- Test with data for new formats

(End Analysts)

Generate a JSON with the updated version of the FPR on the testing pipeline
- python testproject/manage.py dumpdata fpr > fpr-updated.json
Get the updates as JSON
- E.g. python testproject/manage.py get_fpr_changes fpr-current.json fpr-updated.json fpr/migrations/pronom_89.json
Update the migration to load the JSON updates
- Replace the contents of the data migration function with
  - fixture_file = os.path.join(os.path.dirname(__file__), 'pronom_89.json')
  - call_command('loaddata', fixture_file, app_label='fpr')
- Remove the direct imports from the bug
- Remove the pk's from the entries in the JSON document.
- Improvement Note: Because this is using loaddata, this will have problems if the FPR models are changed. A possible solution is to update get-fpr-changes to generate a migration instead of JSON
Commit
Merge

Update dependencies

Python Packages

metsrw and agentarchives both have Makefiles that handle most of the packaging

Check for open PRs, merge as necessary
Update setup.py with the new version, create a pull request, code review, merge.
Tag new release, push tag
- git push --tags
Run make package
- This will build the package and upload it to PyPI. It will prompt for your PyPI username and password for the upload
- The Makefile was created from instructions at https://packaging.python.org/distributing/
make clean will delete packaging related files

Update version

Update PREMIS agent to Archivematica-X.X.X
Update Dashboard-Administration-Version to X.X.X
Update Storage services-Admin-Version to X.X.X

Build deb/rpm packages

The am-packbuild repository has all the code related to building packages, except the building gpg keys. The steps to follow in order to build production production packages are as follow

Debian packages

Clone the am-packbuild repo. Latest work is available in master
Put your gpg private key into debs/GPG-KEYS-REPOS. That's the place the Dockerfile looks for it when building the environment.
Update the makefile at debs/archivematica/Makefile in order to reflect version/keys you want to use.
Run <make>, and the packages will be available in the build once the building finishes.
Upload packages to public debian repository

Debian reposities

There are two debian repositories, one for archivematica packages, and one for dependencies. The procedure in order to create new ones, or upload packages to them, is the same:

- Create folder for repo, and configuration file:

mkdir -p /path/to/repos/repo/conf

cat > /path/to/repos/repo/conf/distributions << EOF

Codename: trusty

Components: main

Architectures: amd64 source

SignWith: <short gpg keyid>

EOF

- Go inside the repo, and import the packages previously uploaded with:

cd /path/to/repos/repo/

reprepro includedeb trusty /path/to/packages/*.deb

reprepro includedsc trusty /path/to/packages/*.deb

The current official repo is at packages.archivematica.org

RPM Packages

Package specs are available in am-packbuild/rpms
There are vars in the Makefiles for version/release , so update them when needed
In order to build them, just go into the directory you want to build, and run “make”

RPM Repositories

Once the packages are built, upload them to packages.archivematica.org/<version>/centos Sign the packages with rpm --addsign *.rpm (already signed packages will be skipped)

Go inside that dir, and as user ohbot run:

rpm --addsign *.rpm (already signed packages will be skipped)

createrepo . (For packages other than archivematica , use “centos-extras” repository)

gpg --detach-sign --armor repodata/repomd.xml

The first gpg command signs the rpms, and the later signs the repository content.

Development stage

In the final stages of development, the repositories for the new releases are created, but packages are signed with a development key to avoid mistakes. Once the development stage finishes, all new packages need to be rebuild using the production keys.

Development packages are built on each new commit to stable branches by Jenkins. Repositories are available at http://jenkins-ci.archivematica.org/repos/

Website