Dataset preservation

Workflow

• Composition of AIPs: Large datasets may be divided into multiple transfers prior to ingest, so that one dataset ultimately consists of a number of AIPs. See Hierarchical AIC/AIP structure, below.
  • note: a related, follow-up Archivematica requirement is to break up large files (e.g. video) that exceed a configurable maximum file size into multiple AIPs, to which the hierarchical structure described below would also apply
• Metadata ingest: Metadata will be created outside of Archivematica prior to ingest, and may be referenced from the dmdSec of the AIP METS file as an xlink reference. See Metadata, below.
• Normalization:Some types of data files may require manual normalization: see https://projects.artefactual.com/issues/1499.

Metadata

METS and DDI/FGDC

• DDI is Data Documentation Initiative, a metadata specification for the social and behavioral sciences; see http://www.ddialliance.org/.
• FGDC is Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]; see http://www.fgdc.gov/metadata/csdgm/
• DDI and FGDC are considered descriptive metadata (dmdSec) in METS. From http://www.loc.gov/standards/mets/METSOverview.v2.html: "Valid values for the MDTYPE element [in dmdSec] include...DDI (Data Documentation Initiative), FGDC (Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]."
  • In the Archivematica METS file, a DDI or FGDC file could be referenced from the dmdSec using mdRef, for example as follows: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="DDI" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>.

METS and other metadata standards

• Other metadata standards that could be used for ingested datasets include:
  • North American Profile (NAP) of ISO 19119, for geospatial metadata: http://www.fgdc.gov/metadata/geospatial-metadata-standards
  • SDMX for aggregate data: http://sdmx.org/?page_id=10
  • EML, the Ecological Metadata Language: http://knb.ecoinformatics.org/software/eml/eml-2.1.1/index.html
• If these standards are used, the mdRef in the METS file would need to use OTHER as MDTYPE, for example: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="OTHER" OTHERMDTYPE="SDMX" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>

Hierarchical AIC/AIP structure

• Because datasets can be large and heterogeneous, one "dataset" may be broken into multiple AIPs. In such cases, the multiple AIPs can be intellectually combined into one AIC, or Archival Information Collection, defined by the OAIS reference model as "[a]n Archival Information Package whose Content Information is an aggregation of other Archival Information Packages." (OAIS 1-9).
  • The AIC consists of a METS file containing a fileSec and a logical structMap listing all child AIPs (Note that this is based on Option 1 under Possible AIC/AIP designs, below).
  • In storage, a pointer.xml file gives storage and compression information for each AIC and AIP.
• This diagram shows a storage area with standalone AIPs, an AIC with child AIPs, and related pointer.xml files.

Archival storage area containing pointer files, AIC and AIPs

Possible AIC/AIP designs

Option 1 (preferred)

Description: An AIC consisting of only a fileSec and structMap; AIPs consisting of data files and metadata for those data files; an AIP consisting of project/program-level (i.e. dataset) metadata and documentation.

Workflow:

1. User creates X number of AIPs and puts them in archival storage
   • One of these AIPs consists only of metadata and documentation about the program/project as a whole
   • The AIPs must have one or more common metadata elements that allows them to be identified as being related
2. User searches for AIPs in archival storage tab (using the common metadata element in the AIPs in the search query)
3. Once search results are retrieved, user clicks "Create AIC" button
4. AIC is created, containing only a METS structMap listing all AIPs
5. Over time, user can add new AIPs and re-create the AIC at any time; the new AIC will either replace or update the old one
6. Over time, if needed the user either updates the existing documentation AIP or adds new documentation AIPs (i.e. there can be more than one documentation AIP per dataset)

Pros:

• Don't have to duplicate program/project-level documentation in each AIP
• Simple workflow for creating AIC
• Easy to add new AIPs
• If program/project documentation needs updating, only one AIP has to be re-processed, or user can add new documentation AIP(s)

Cons:

• There is only a one-way link between the AIC and child AIPs - i.e. the AIC has a structMap listing all child AIPs, but there is nothing in a child AIP to indicate that it belongs to a given AIC.

Sample AIC METS file

Sample pointer.xml file

Option 1A (supplied by U of A)

Description:

Workflow

1. User selects type 'AIC' on the Archivematica's Transfer page
2. User provides input about the status of this AIC: Is this a new AIC
   • Yes: User needs to create a new AIC with a new system generated or user generated AIC#
   • No: AIC already exists and the user just wants to add new AIPs or wants to update the AIC. User should be able to search this AIC through AIC# or using other search terms
3. Once a new AIC has been created OR an existing one selected, a loop starts, during which new AIPs are prepared and added to this AIC. Each AIP contains reference to the AIC through AIC#, as part of the system generated metadata. The loop ends when the user finishes adding AIPs
4. User selects "Generate AIC Struct-Map" option and it will generate a "Struct-Map" for this collection. This Struct-Map also has a reference to its corresponding AIC through AIC#. If possible, store it inside the AIC, otherwise as a seperate Struct-Map AIP
5. System prepares the whole package for preservation (AIC/AIPs/Struct-Map) and send it to an archival storage
6. Over time, user can add new AIPs. Generate AIC Struct-Map step can either replace Struct-Map or update the old one.
7. Over time, the user can add/update the existing documentation in an AIC by updating the AIC's AIP

Option 2

Description: An AIC consisting of a METS structMap and project/program-level (i.e. dataset) metadata and documentation; content AIPs consisting of data files and metadata about the data files. AIPs have information in the METS files (in the structMap?) linking them to the parent AIC.

Workflow: To be determined - probably a dashboard tab with a gui to allow users to arrange existing AIPs into an AIC

Pros:

• Don't have to duplicate program/project-level documentation in each AIP
• AIPs have a link up to the AIC, so if an AIP is orphaned the relationship to the AIC can easily be reconstructed
• If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed

Cons:

• Workflow to create this structure may be complex
• No obvious mechanism for adding new AIPs over time

Option 3

Description: An AIC with a unique identifier consisting of project/program-level (i.e. dataset) metadata and documentation only (no structMap); AIPs consisting of data files, metadata for those data files, and the same identifier as the AIC. The relationship between the AIC and AIPs in this scenario is inferred from the matching identifiers.

Workflow:

1. User creates an AIC consisting of project/program-level (i.e. dataset) metadata and documentation
   • The AIC contains an identifier that distinguishes it from other AICs
2. User creates AIPs consisting of data files and metadata for those data files
   • User includes the AIC identifier in each AIP
3. Over time, if needed the user can add more AIPs with the same identifier

Pros:

• Don't have to duplicate program/project-level documentation in each AIP
• Simple workflow
• Minimal development requirements, just new metadata field for identifier added to transfer tab, corresponding entry in AIC/AIP METS files and ability to search by AIC identifier in archival storage tab
• If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed
• Easy to add more AIPs to the same AIC over time

Cons:

• No structMap in the AIC means that there is no single source of information about how many AIPs are in the AIC

Option 4

Description: No AIC; project/program-level metadata and documentation duplicated in all AIPs; links between AIPs belonging to one dataset inferred from metadata only

Workflow: User creates any number of AIPs with complete copies of the project/program-leve (i.e. dataset) metadata and documentation in each AIP

Pros:

• Minimal Archivematica development required, just ensuring that matching metadata elements are parsed to the AIP METS files or otherwise made available to ElasticSearch index
• Easy to add new AIPs over time

Cons:

• User has to maintain copies of project/program-level metadata and documentation outside of Archivematica so they can be added to each AIP
• Updating the project/program-level metadata and documentation would require re-processing the AIPs
• Relationships between AIPs would have to be inferred from matching metadata elements alone; if an AIP were lost, there would be no list of AIPs belonging to the dataset which would reveal the loss