Format identification requirements - Revision history

Sallain at 22:25, 11 February 2020

2020-02-11T22:25:01Z

Hbecker: Move to feature requirements category

2017-03-23T19:35:32Z

Move to feature requirements category

Courtney: /* Requirements */

2012-10-24T21:07:40Z

Requirements

Courtney: /* Requirements */

2012-10-24T21:03:48Z

Requirements

Courtney: /* Requirements */

2012-10-17T19:05:36Z

Requirements

Courtney: /* Requirements */

2012-10-17T18:44:46Z

Requirements

Courtney: /* Requirements */

2012-10-16T16:48:30Z

Requirements

Courtney: /* FITS output for format identification */

2012-10-16T16:46:03Z

FITS output for format identification

Courtney: Created page with "Documentation > Requirements > Format identification requirements == Requirements == * FITS runs for characterization and md extraction in transfer tab ** Output st..."

2012-10-16T16:36:13Z

Created page with "Documentation > Requirements > Format identification requirements == Requirements == * FITS runs for characterization and md extraction in transfer tab ** Output st..."

New page

[[Documentation]] > [[Requirements]] > Format identification requirements

== Requirements ==

* FITS runs for characterization and md extraction in transfer tab
** Output stored in transfer backlog

* '''Format identification micro-service''' in Ingest tab, after SIP creation
** user selects from dropdown menu of identification tool options:
***Identify file by extension - this is already the default in Archivematica 0.9
***[http://unixhelp.ed.ac.uk/CGI/man-cgi?file file utility] - file utility identification metadata from FITS output
***[http://web.archive.org/web/20061106114156/http://schmidt.devlib.org/ffident/index.html ffident] - ffident identification metadata from FITS output
***[http://droid.sourceforge.net/ DROID] - DROID identification metadata from FITS output
***[http://sourceforge.net/projects/jhove/ JHOVE] - JHOVE identification metadata from FITS output
***[http://www.openplanetsfoundation.org/software/fido FIDO] - run FIDO
***Tika - run modified Tika
***[http://mediainfo.sourceforge.net/en mediainfo] - run mediainfo

* Output for each tool must be restructured to comply with format policy

* Format policies should have globally unique IDs

* Format policies result in rules that are stored in the [[Format_policy_registry_requirements|Format Policy Registry (FPR)]]

=== FITS output for format identification ===

* in file utility output: <format>
* in ffident output: <mimetype>
* in DROID output: <IdentificationFile>
* in JHOVE output: <format>
* in FITS output: <identify format>

== Use Cases ==

== Workflows ==

* Insert File identification micro-service after Clean up names micro-service and before Normalize micro-service
* User selects from dropdown menu which FITS tool to trust for format identification (see tools listed above)
* Archivematica bases Normalization path / application of format policy on the tool selected
* Post 1.0 work: add tools outside of FITS that will run when selected by the user at the File identification micro-service.

== Whiteboard ==

[[File:FileID.jpg|800px]]

== Mockups ==

Note that the Normalize micro-service is included here for illustrative purposes only. In action, the user would not see the Normalize micro-service until a format identification mechanism was selected from the dropdown menu.

[[File:FileIdentificationMS.png]]

=Format policy models and table=
The [[Transcoder]] page has a lot of information on it.
<br/>This diagram is from there.

[[File:Transcoder_database_schema.png]]

I think for the [[Format_policy_registry_requirements|Format Policy Registry (FPR)]] almost every table will need to replace the pk INT with a pk UUID, and the corresponding foreign keys. This is to aid in the creation/submission of new rules.

We're also going to need to know what format the rules output to.

Speaking to future iterations: We'll need to be able to identify what formats the original file, and preservation file are in. Determine the format risk of the data with both of those considered. We may choose to normalize a to a new preservation format from the the original file, or the preservation file. The reason for this may be that both formats risk obsolescence, but there are only tools available to handle the preservation format.

To track changes/updates/new items, I would suggest the FPR keep track of the last modified date of all rows. Clients keep track of the last updated date. These dates are from the FPR.
Additional: add a marked for removal bool to each table in the FPR, to indicate to clients to remove that from their database.

[[Category:Development documentation]]

← Older revision		Revision as of 22:25, 11 February 2020
Line 1:		Line 1:
	[[Documentation]] > [[Requirements]] > Format identification requirements		[[Documentation]] > [[Requirements]] > Format identification requirements
		+
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information.</div><p>

	[[Category:Feature requirements]]		[[Category:Feature requirements]]

@@ Line 7: / Line 7: @@
 ** Output indexed and stored in transfer backlog
-* '''Select format identification tool''' in Normalization micro-service, after SIP creation
+* '''Select format identification tool''' job in Normalization micro-service, after SIP creation
 ** user selects from dropdown menu of identification tool options:
 ***Identify file by extension - this is already the default in Archivematica 0.9

@@ Line 19: / Line 19: @@
 * Duplicate results need cleared from DB - one command for each ID
 * Multiple IDs - do all normalization tasks that apply
-** User can override - select rule to default to file extension when multiple IDs occur
+** Indicate duplicates in Norm report and viewer
-** Add delete functionality to Review normalization viewer in Dashboard browser, user chooses one or the other
+** Add delete functionality to Review normalization report and viewer in Dashboard browser, user chooses one or the other
 * If there is no rule, normalization should be based on extension
+* Distinguish identification tool versions in DB
 * Output for each tool must be restructured to comply with format policy

← Older revision		Revision as of 16:48, 16 October 2012
Line 23:		Line 23:

	* Format policies result in rules that are stored in the [[Format_policy_registry_requirements\|Format Policy Registry (FPR)]]		* Format policies result in rules that are stored in the [[Format_policy_registry_requirements\|Format Policy Registry (FPR)]]
		+	* Do not split major/minor mimetypes

	=== FITS output for format identification ===		=== FITS output for format identification ===

@@ Line 1: / Line 1: @@
 [[Documentation]] > [[Requirements]] > Format identification requirements
 == Requirements ==
@@ Line 72: / Line 73: @@
 To track changes/updates/new items, I would suggest the FPR keep track of the last modified date of all rows. Clients keep track of the last updated date. These dates are from the FPR.
 Additional: add a marked for removal bool to each table in the FPR, to indicate to clients to remove that from their database.

@@ Line 5: / Line 5: @@
 * FITS runs for characterization and md extraction in transfer tab
-** Output stored in transfer backlog
+** Output indexed and stored in transfer backlog
-* '''Format identification micro-service''' in Ingest tab, after SIP creation
+* '''Select format identification tool''' in Normalization micro-service, after SIP creation
 ** user selects from dropdown menu of identification tool options:
 ***Identify file by extension - this is already the default in Archivematica 0.9
@@ Line 36: / Line 36: @@
 * in DROID output: <PUID> <MimeType>
 * in JHOVE output: <format> <mimeType>
 == Use Cases ==

@@ Line 17: / Line 17: @@
 ***Tika - run modified Tika
 ***[http://mediainfo.sourceforge.net/en mediainfo] - run mediainfo
 * Output for each tool must be restructured to comply with format policy

@@ Line 26: / Line 26: @@
 === FITS output for format identification ===
-* in file utility output: <format>
+* in file utility output: <format> <mimetype>
 * in ffident output: <mimetype>
-* in DROID output: <IdentificationFile>
+* in DROID output: <PUID> <MimeType>
-* in JHOVE output: <format>
+* in JHOVE output: <format> <mimeType>
-* in FITS output: <identify format>
+* in FITS output: <identify format> split format and mimetype
 == Use Cases ==