Format policy registry requirements - Revision history

Sallain at 21:51, 11 February 2020

2020-02-11T21:51:55Z

Hbecker: Also list as feature requirement page

2017-03-23T19:36:27Z

Also list as feature requirement page

Hbecker: /* Data Model */ Add updated FPR model

2017-03-10T00:35:31Z

Data Model: Add updated FPR model

Hbecker: Add brief introduction

2017-03-08T19:17:02Z

Add brief introduction

Courtney: /* FPR development */

2014-07-28T23:02:14Z

FPR development

Courtney: /* Requirements */

2014-07-28T22:55:53Z

Requirements

Courtney at 22:54, 28 July 2014

2014-07-28T22:54:47Z

Courtney: /* Replace VersionedModel with set of associated rules */

2014-01-10T00:47:51Z

Replace VersionedModel with set of associated rules

Hbecker: /* Replace VersionedModel with set of associated rules */ add bug #

2014-01-10T00:43:09Z

Replace VersionedModel with set of associated rules: add bug #

Hbecker: /* Proposed Changes */ Replace VersionedModel proposal

2014-01-10T00:42:15Z

Proposed Changes: Replace VersionedModel proposal

← Older revision		Revision as of 21:51, 11 February 2020
Line 1:		Line 1:
	[[Documentation]] > [[Requirements]] > Format policy registry requirements		[[Documentation]] > [[Requirements]] > Format policy registry requirements
		+
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information. </div> <p>

	The Format Policy Registry (FPR) contains user-configurable scripts for file identification, characterization, extraction, normalization and other preservation actions that differ depending on file format. It also contains a list of formats recognized by Archivematica, and the relationship between those formats and the scripts.		The Format Policy Registry (FPR) contains user-configurable scripts for file identification, characterization, extraction, normalization and other preservation actions that differ depending on file format. It also contains a list of formats recognized by Archivematica, and the relationship between those formats and the scripts.

← Older revision		Revision as of 19:36, 23 March 2017
Line 178:		Line 178:


−	[[Category:Development documentation]]	+	[[Category:Development documentation]]
		+	[[Category:Feature requirements]]

← Older revision		Revision as of 00:35, 10 March 2017
Line 128:		Line 128:

	[[File:Fpr design-2013-07-25.png\|900px]]		[[File:Fpr design-2013-07-25.png\|900px]]
		+
		+	Updated design as of March 2017
		+
		+	[[File:Fpr-design-2017-03.png\|900px]]

	== Workflow ==		== Workflow ==

← Older revision		Revision as of 19:17, 8 March 2017
Line 1:		Line 1:
	[[Documentation]] > [[Requirements]] > Format policy registry requirements		[[Documentation]] > [[Requirements]] > Format policy registry requirements
		+
		+	The Format Policy Registry (FPR) contains user-configurable scripts for file identification, characterization, extraction, normalization and other preservation actions that differ depending on file format. It also contains a list of formats recognized by Archivematica, and the relationship between those formats and the scripts.

	== Overview ==		== Overview ==

@@ Line 32: / Line 32: @@
 =FPR development=
 ==1.2==
@@ Line 39: / Line 51: @@
 *Added Transcription, Extraction, Characterization and Verification event sections
 *Ability to update from server in local dashboard's Preservation Planning tab
 ==0.10-beta==
@@ Line 47: / Line 63: @@
 * Central server at fpr.archivematica.org
 * Ability to download most current Archivematica default format policies from fpr.archivematica.org on first installation
 ==Wishlist==
@@ Line 57: / Line 72: @@
 ** users would be able to apply changes locally and submit them for review to fpr
 ** nice to have: ability to add links to documentation about reasoning behind local institutional policy selection
-** if the changes were accepted, they would be a new fpr entry
+** accepted changes would result in a new FPR server entry
 * Ability to download select new Archivematica default FPR policies as well as those from other institutions from the central FPR server to local Archivematica installations
 ** ideally a single click download to allow for local implementation from the server or from a list viewable after selecting Update from the Preservation Planning tab of the local Dashboard

@@ Line 66: / Line 66: @@
 * Provide a read-only RESTful Web API for accessing policies in JSON format. This is only partially implemented in 1.2, intended only to be used internally (between dashboard and FPR server) and only provides access to the full set of policies.  Future development would ideally include the ability to access a subset of policies via the FPR REST API
-=Requirements=
+=Legacy FPR foundation docs=
 [[File:FPR overview Oct 2012.png|border|900px|FPR overview Oct 2012]]
@@ Line 72: / Line 72: @@
 * developer [[Format_policy_registry|notes]]
 == Mockups ==

@@ Line 3: / Line 3: @@
 == Overview ==
-* The Archivematica project team is working on a better way to manage normalization format policies.
+* The Archivematica project team is working on a better way to manage format policies for preservation events such as normalization, transcription, extraction, characterization and format identification.
-* A format policy consists of the business rules and tool commands for format normalization.
+* A format policy consists of the business rules and tool commands for preservation events based on format.
-* The FPR lists all of Archivematica's default format policy rules.
+* The Format Policy Registry lists all of Archivematica's default format policy rules.
-* In your Archivematica 0.10-beta instance you can download updates from the FPR as well as replace default rules with your own local policies.
+* Currently, users can download updates from the FPR server and replace default rules with your own local policies.
-* Upcoming FPR enhancements will include statistical information about the default and custom format policies implemented by Archivematica users.
+* Future funding is sought for further FPR enhancements to include statistical information about the default and custom format policies adopted and implemented by Archivematica users and the ability to upload local policies to the FPR server.
 == Description ==
-* The Archivematica project team has recognized the need for a better way to manage preservation plans, i.e. business rules and tool commands for format transcoding. Since these are either implemented or altered by the institution running an Archivematica instance, these rules are referred to as policies. Format policies will change as community standards, practices and tools evolve. A format policy indicates the actions, tools and settings to apply to a file of a particular file format (e.g. conversion to preservation format, conversion to access format).
+* The Archivematica project team created the FPR after having recognized the need for a better way to manage preservation plans, i.e. business rules and tool commands for format-based preservation events. Since these are either implemented or altered by the institution running an Archivematica instance, these rules are referred to as policies. Format policies will change as community standards, practices and tools evolve. A format policy indicates the actions, tools and settings to apply to a file of a particular file format (e.g. conversion to preservation format, conversion to access format).
-* Until now, the Archivematica project has managed this information on the [[Media_type_preservation_plans|archivematica.org/preservation]] wiki page.
+* Prior to the FPR, the Archivematica project has managed this information on the [[Media_type_preservation_plans|archivematica.org/preservation]] wiki page. These format policies were all researched as a result of related development partnerships with samples provided by funding partners. Since the FPR was first released in late 2013, all new format policies were developed in partnership with funding institutions and any new rules were included in the public FPR server, hosted by Artefactual Systems, Inc.
-* The Format Policy Registry (FPR) will manage this information in a structured format (SQL/JSON).
+* The Format Policy Registry (FPR) manages this information in a structured format (SQL/JSON).
-** APIs with other serializations may be added (e.g. XML, RDF)
+** With additional sponsored development, APIs with other serializations could be added (e.g. XML, RDF)
-* It will be hosted at fpr.archivematica.org
+* The FPR includes updates from PRONOM, which are manually applied by Artefactual Systems. With funding, we could enhance and/or automate this PRONOM interface and add interfaces with other registries like UDFR and linked data registries. A web of interfaces is the best way to monitor and evaluate community-wide best practices.
-* The FPR will also provide valuable online statistics about default format policy adoption as well as customizations amongst Archivematica users and will interface with other online registries (such as PRONOM and UDFR) to monitor and evaluate community-wide best practices.
+* The FPR stores structured information about:
-* The FPR stores structured information about normalization format policies for preservation and access. These policies identify preferred preservation and access formats by media type. The choice of access formats is based on the ubiquity of viewers for the file format. Archivematica's preservation formats are all open standards; additionally, the choice of preservation format is based on community best practices, availability of open-source normalization tools, and an analysis of the significant characteristics for each media type.
+* Archivematica default format policies can all be changed or enhanced by individual Archivematica implementers.
-* These default format policies can all be changed or enhanced by individual Archivematica implementers.
+=FPR development=
-* Subscription to the FPR will allow the Archivematica project to notify users when new or updated preservation and access plans become available, allowing them to make better decisions about normalization and migration strategies for specific format types within their collections. It will also allow them to trigger migration processes as new tools and knowledge becomes available.
+==1.2==
-*One of the other primary goals of the FPR is to aggregate empirical information about institutional format policies to better identify community best practices. The FPR will provide a practical, community-based approach to OAIS preservation and access planning, allowing the Archivematica community of users to monitor and evaluate formats policies as they are adopted, adapted and supplemented by real-world practioners. The FPR APIs will be designed to share this information with the Archivematica user base as well with other interested communities and projects.
+*Updated PRONOM data for FIDO
+*Made one Tools section rather than having tools in each event space, except for Identification tools
-=Release plan=
+*Added Transcription, Extraction, Characterization and Verification event sections
 ==0.10-beta==
 *Ability to view, add/edit local format policies
-** if user doesn't want to use default Archivematica normalization path for format X, user can either open the command edit page or the format id edit page to start, but BOTH must be edited to complete the new entry
+** FPR local is for superusers (preservation planning tab)
-** fpr local is for superusers (preservation planning tab)
+* Dashboard FPR captures usage statistics
-* Dashboard FPR to capture usage statistics to assist local repository managers to select format policies
+* Central server at fpr.archivematica.org
 * Ability to download most current Archivematica default format policies from fpr.archivematica.org on first installation
 * Ability to submit local changes to the central FPR server
-** fpr.archivematica.org will mirror local implementation, except user won't be able to apply changes, only submit them for review and it will include the institution that submitted the policy or indicate whether it is a default Archivematica policy
+** fpr.archivematica.org will mirror local implementation, except user won't be able to apply changes, only submit them for review and website could include the institution that submitted the policy or indicate whether it is a default Archivematica policy
 ** users would be able to apply changes locally and submit them for review to fpr
 ** if the changes were accepted, they would be a new fpr entry
+* Ability to download select new Archivematica default FPR policies as well as those from other institutions from the central FPR server to local Archivematica installations
-* Ability to download accepted FPR policies from the central FPR server to local Archivematica installations
+** ideally a single click download to allow for local implementation from the server or from a list viewable after selecting Update from the Preservation Planning tab of the local Dashboard
-** if the user doesn't want default normalization path but sees something in the fpr he'd like to use locally, he would have a single click download to allow for local implementation
+* Ability to disable a format policy in the dashboard
+* Update Archivematica without losing local FPR configuration and added policies
-* Dashboard and central server FPR to capture usage statistics to assist local repository managers to select format policies
+* Integrate with PRONOM to automate PUID updates
 =Requirements=
@@ Line 58: / Line 70: @@
 [[File:FPR overview Oct 2012.png|border|900px|FPR overview Oct 2012]]
-* provide an authenticated Web based interface for creation and maintenance of policies
 * developer [[Format_policy_registry|notes]]

← Older revision		Revision as of 00:47, 10 January 2014
Line 122:		Line 122:

	=== Replace VersionedModel with set of associated rules ===		=== Replace VersionedModel with set of associated rules ===
		+
		+	January 2014

	'''Summary:''' Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.		'''Summary:''' Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.

@@ Line 125: / Line 125: @@
 '''Summary:''' Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.
-'''Bug:'''
+'''Bug:'''#6192
 '''Problem:'''

← Older revision		Revision as of 00:42, 10 January 2014
Line 120:		Line 120:

	== Proposed Changes ==		== Proposed Changes ==
		+
		+	=== Replace VersionedModel with set of associated rules ===
		+
		+	'''Summary:''' Replace the VersionedModel structure of a linear history of revisions with a collection/set of rules all associated with some 'meta-rule'.
		+
		+	'''Bug:'''
		+
		+	'''Problem:'''
		+	Currently, we track a linear history of rule modifications. A newly modified rule is always added to the end of the chain, even if it modified an earlier rule. All rules in the chain must be kept. Example: A <- B (enabled). A is modified, resulting in C with history of: A <- B <- C (enabled) B cannot be disabled, even if it never used (eg had a typo)
		+
		+	We need to track variations on a rule that is fundamentally the same, but not necessarily a linear version history. With a linear history, the true relationship between a rule and where it originated from is unclear, and a version history of any sort for the rules is probably unnecessary - all we care about is the currently active rule (and any newly downloaded rules that are candidates for replacing it). The linear history also complicates things when modifications are coming from several sources (eg. Artefactual, various institutions, local modifications)
		+
		+	'''Proposed fix:'''
		+	Each rule stores an Agent (the institution/person that created it) and a 'meta-rule' ID, and no longer stores what rule it replaced. All rules that are variations on each other have the same 'meta-rule' ID and are 'rule versions'. Potentially the meta-rule ID is a foreign key to an actual meta-rule table, but that may not be necessary.
		+
		+	In a set of rule versions, only one may be enabled at a time. Rule versions can come from several sources - originally installed and FPR updates from Artefactual, downloads from a particular institution that is reputable, a local FPR server - and are conceptually linked by having a common meta-rule ID. Agent IDs will be used to distinguish the sources, and created time used to track the most recent variation.
		+
		+	'''Pros:'''
		+	* Easier to download updates to FPR rules, especially from more than one source
		+	* Can provide the ability to delete truly unwanted rules
		+	* ???
		+
		+	'''Cons:'''
		+	* What information to display to the user? What to display if all the rule versions for a meta-rule are disabled?
		+	* Probably backwards incompatible, or non-trivial to make it
		+	* ???

	==Early prototype==		==Early prototype==