Format policy registry requirements

Documentation > Requirements > Format policy registry requirements

Overview

• The Archivematica project team is working on a better way to manage normalization format policies.
• A format policy consists of the business rules and tool commands for format normalization.
• The FPR lists all of Archivematica's default format policy rules.
• In your Archivematica 0.10-beta instance you can download updates from the FPR as well as replace default rules with your own local policies.
• Upcoming FPR enhancements will include statistical information about the default and custom format policies implemented by Archivematica users.

Description

• The Archivematica project team has recognized the need for a better way to manage preservation plans, i.e. business rules and tool commands for format transcoding. Since these are either implemented or altered by the institution running an Archivematica instance, these rules are referred to as policies. Format policies will change as community standards, practices and tools evolve. A format policy indicates the actions, tools and settings to apply to a file of a particular file format (e.g. conversion to preservation format, conversion to access format).

• Until now, the Archivematica project has managed this information on the archivematica.org/preservation wiki page.

• The Format Policy Registry (FPR) will manage this information in a structured format (SQL/JSON).
  • APIs with other serializations may be added (e.g. XML, RDF)

• It will be hosted at fpr.archivematica.org

• The FPR will also provide valuable online statistics about default format policy adoption as well as customizations amongst Archivematica users and will interface with other online registries (such as PRONOM and UDFR) to monitor and evaluate community-wide best practices.

• The FPR stores structured information about normalization format policies for preservation and access. These policies identify preferred preservation and access formats by media type. The choice of access formats is based on the ubiquity of viewers for the file format. Archivematica's preservation formats are all open standards; additionally, the choice of preservation format is based on community best practices, availability of open-source normalization tools, and an analysis of the significant characteristics for each media type.

• These default format policies can all be changed or enhanced by individual Archivematica implementers.

• Subscription to the FPR will allow the Archivematica project to notify users when new or updated preservation and access plans become available, allowing them to make better decisions about normalization and migration strategies for specific format types within their collections. It will also allow them to trigger migration processes as new tools and knowledge becomes available.

• One of the other primary goals of the FPR is to aggregate empirical information about institutional format policies to better identify community best practices. The FPR will provide a practical, community-based approach to OAIS preservation and access planning, allowing the Archivematica community of users to monitor and evaluate formats policies as they are adopted, adapted and supplemented by real-world practioners. The FPR APIs will be designed to share this information with the Archivematica user base as well with other interested communities and projects.

Release plan

0.10-beta

• Ability to view, add/edit local format policies
  • if user doesn't want to use default Archivematica normalization path for format X, user can either open the command edit page or the format id edit page to start, but BOTH must be edited to complete the new entry
  • fpr local is for superusers (preservation planning tab)
• Dashboard FPR to capture usage statistics to assist local repository managers to select format policies
• Ability to view fpr.archivematica.org
  • central FPR server will exist, but will not be editable
• Ability to download most current Archivematica default format policies from fpr.archivematica.org on first installation

1.0

• Ability to submit local changes to the central FPR server
  • fpr.archivematica.org will mirror local implementation, except user won't be able to apply changes, only submit them for review and it will include the institution that submitted the policy or indicate whether it is a default Archivematica policy
  • users would be able to apply changes locally and submit them for review to fpr
  • if the changes were accepted, they would be a new fpr entry

• Ability to download accepted FPR policies from the central FPR server to local Archivematica installations
  • if the user doesn't want default normalization path but sees something in the fpr he'd like to use locally, he would have a single click download to allow for local implementation

• Dashboard and central server FPR to capture usage statistics to assist local repository managers to select format policies

Requirements

• provide an authenticated Web based interface for creation and maintenance of policies
• provide a read-only RESTful Web API for accessing policies in JSON format
• provide an API for monitoring new and updated policies
• integrate with PRONOM to retrieve PUIDs
• model format policies so that they can be stored in a SQL (MySQL?, PostGres?, SQLlite?) dbase on both client & server
• develop iteratively with an emphasis on getting working code in front of users as quickly as possible to make them part of the design process (see #fileidhack)
• developer notes

Use Cases

• Alternate normalization tool than Archivematica default (preservation and/or access)
• Alternate default normalization outcome format than Archivematica default (preservation and/or access)
• Alternate size normalization result (preservation and/or access)
• Deselect normalization default in Archivematica to use local tool/process outside of the system (preservation and/or access)
• Add new format policy for an unrepresented or new format (preservation and/or access)
• Test format policy tools and commands on local digital acquisitions

Mockups

Preservation Planning tab in Dashboard - Local FPR

---

• Archivematica format ID column is populated by the "description" from the File ID table in the Archivematica database; It links to the file ID table in the Archivematica database, which is composed of the tool and tool version that identified the format.
• Command description is the "description" from the Command table in the Archivematica database
• Purpose is the same as "classification" in the Archivematica database, but is clearer to the user
• Add new -links to create new (blank) form for Archivematica format ID -or- to the create new (blank) form for the Command
• Copy - links to create new form based on this one (populated with current data) for either the Archivematica format ID -or- Command
• Edit -links to edit page for the Archivematica format ID -or- to the edit page for the Command
• Make default - makes the selected Archivematica format ID format policy or the selected Command format policy default for normalization. See Feature #4503
• Show performance and show command are modeled on the old preservation planning tab in the dashboard, available in the dev version of Archivematica at /preservationplanning/old/

Add new Archivematica format ID

---

Add new Command

• Added https://projects.artefactual.com/issues/4568 (#4568) to add a Output file format field to this form from the outputFileFormat field in the MCP Commands table

---

Copy Archivematica format ID

---

Copy Command

Data Model

Design from summer 2013 of the data structures in version 2 of the API.

Proposed Changes

Replace VersionedModel with set of associated rules

Summary: Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.

Bug:

Problem: Currently, we track a linear history of rule modifications. A newly modified rule is always added to the end of the chain, even if it modified an earlier rule. All rules in the chain must be kept. Example: A <- B (enabled). A is modified, resulting in C with history of: A <- B <- C (enabled) B cannot be disabled, even if it never used (eg had a typo)

We need to track variations on a rule that is fundamentally the same, but not necessarily a linear version history. With a linear history, the true relationship between a rule and where it originated from is unclear, and a version history of any sort for the rules is probably unnecessary - all we care about is the currently active rule (and any newly downloaded rules that are candidates for replacing it). The linear history also complicates things when modifications are coming from several sources (eg. Artefactual, various institutions, local modifications)

Proposed fix: Each rule stores an Agent (the institution/person that created it) and a 'meta-rule' ID, and no longer stores what rule it replaced. All rules that are variations on each other have the same 'meta-rule' ID and are 'rule versions'. Potentially the meta-rule ID is a foreign key to an actual meta-rule table, but that may not be necessary.

In a set of rule versions, only one may be enabled at a time. Rule versions can come from several sources - originally installed and FPR updates from Artefactual, downloads from a particular institution that is reputable, a local FPR server - and are conceptually linked by having a common meta-rule ID. Agent IDs will be used to distinguish the sources, and created time used to track the most recent variation.

Pros:

• Easier to download updates to FPR rules, especially from more than one source
• Can provide the ability to delete truly unwanted rules
• ???

Cons:

• What information to display to the user? What to display if all the rule versions for a meta-rule are disabled?
• Probably backwards incompatible, or non-trivial to make it
• ???

Early prototype

• An early FPR prototype (called "Formatica") was developed by Heather Bowden, then Carolina Digital Curation Doctoral Fellow at the School of Information and Library Science in the University of North Carolina at Chapel Hill.