Dataset preservation

This page is no longer being maintained and may contain inaccurate information. Please see the Archivematica documentation for up-to-date information.

Workflow

• Composition of AIPs: Large datasets may be divided into multiple transfers prior to ingest, so that one dataset ultimately consists of a number of AIPs. See Hierarchical AIC/AIP structure, below.
  • note: a related, follow-up Archivematica requirement is to break up large files (e.g. video) that exceed a configurable maximum file size into multiple AIPs, to which the hierarchical structure described below would also apply
• Metadata ingest: Metadata will be created outside of Archivematica prior to ingest, and may be referenced from the dmdSec of the AIP METS file as an xlink reference. See Metadata, below.
• Normalization: Some types of data files may require manual normalization: see https://projects.artefactual.com/issues/1499.

Metadata

METS and DDI/FGDC

• DDI is Data Documentation Initiative, a metadata specification for the social and behavioral sciences; see http://www.ddialliance.org/.
• FGDC is Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]; see http://www.fgdc.gov/metadata/csdgm/
• DDI and FGDC are considered descriptive metadata (dmdSec) in METS. From http://www.loc.gov/standards/mets/METSOverview.v2.html: "Valid values for the MDTYPE element [in dmdSec] include...DDI (Data Documentation Initiative), FGDC (Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]."
  • In the Archivematica METS file, a DDI or FGDC file could be referenced from the dmdSec using mdRef, for example as follows: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="DDI" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>.

METS and other metadata standards

• Other metadata standards that could be used for ingested datasets include:
  • North American Profile (NAP) of ISO 19119, for geospatial metadata: http://www.fgdc.gov/metadata/geospatial-metadata-standards
  • SDMX for aggregate data: http://sdmx.org/?page_id=10
  • EML, the Ecological Metadata Language: http://knb.ecoinformatics.org/software/eml/eml-2.1.1/index.html
• If these standards are used, the mdRef in the METS file would need to use OTHER as MDTYPE, for example: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="OTHER" OTHERMDTYPE="SDMX" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>

Hierarchical AIC/AIP structure

• Because datasets can be large and heterogeneous, one "dataset" may be broken into multiple AIPs. In such cases, the multiple AIPs can be intellectually combined into one AIC, or Archival Information Collection, defined by the OAIS reference model as "[a]n Archival Information Package whose Content Information is an aggregation of other Archival Information Packages." (OAIS 1-9).
  • A basic AIC in Archivematica consists of 1) any number of related AIPs and 2) a METS file containing a fileSec and a logical structMap listing all the related AIPs.
  • In storage, a pointer.xml file gives storage and compression information for each AIC METS file and each AIP.

Workflow for preserving a dataset as an Archival Information Collection

Metadata entry

Metadata entry for the AIC

This screenshot shows sample metadata entry for the AIC (an AIP consisting of program/project-level metadata and documentation). The screenshot also shows the Dublin Core values that would be generated from the data entry. Note that that Dublin Core values would also be copied to the AIC METS file generated at the end of the workflow.

Metadata for AIPs

This screenshot shows sample metadata entry for an AIP (consisting of data files and associated metadata). The screenshot also shows the Dublin Core values that would be generated from the data entry. These values are saved in the AIP METS file but are not copied to the AIC METS file.

AIC METS file

This shows the AIC METS file which is generated at the end of the workflow. Note the Dublin Core metadata copied from the METS file in the AIC (the AIP containing program/project-level metadata and documentation).

<?xml version='1.0' encoding='ASCII'?>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
  <metsHdr CREATEDATE="2013-06-13T20:39:08"/>
  <dmdSec ID="dmdSec_1">
    <mdWrap MDTYPE="DC">
      <xmlData>
        <dublincore xmlns="http://purl.org/dc/terms/" xsi:schemaLocation="http://purl.org/dc/terms/ http://dublincore.org/schemas/xmls/qdc/2008/02/11/dcterms.xsd">
          <title>Global Warming and Arctic Marine Mammals (GWAMM)</title>
          <type>Archival Information Collection</type>
          <identifier>AIC#1774.28/D</identifier>
        </dublincore>
      </xmlData>
    </mdWrap>
  </dmdSec>
  <fileSec>
    <fileGrp>
      <file ID="GWAMM01-3b2be1ec-c70a-4270-9993-01c85d22c975"/>
      <file ID="GWAMM02-cfd9a713-6980-45cf-9db1-86f1cdcf3611"/>
      <file ID="GWAMM03-tgd9a814-4955-45cf-9db1-23f1gjkf3956"/> 
    </fileGrp>
  </fileSec>
  <structMap TYPE="logical">
    <div TYPE="Archival Information Collection" DMDID="dmdSec_1">
      <div LABEL="Global Warming and Arctic Marine Mammals (GWAMM)">
        <fptr FILEID="GWAMM01-3b2be1ec-c70a-4270-9993-01c85d22c975"/>
      </div>
      <div LABEL="Year two observations: Baffin Island">
        <fptr FILEID="GWAMM02-cfd9a713-6980-45cf-9db1-86f1cdcf3611"/>
      </div>
      <div LABEL="Beaufort Sea 2008-2009">
        <fptr FILEID="GWAMM03-tgd9a814-4955-45cf-9db1-23f1gjkf3956"/>
      </div>
    </div>     
  </structMap>
</mets>

Original screenshot

Creating an AIC structMap in the archival storage tab

Once all the AIPs have been created and stored, the user searches for them in the archival storage tab. Selecting the "Show AICs?" box at the top of the screen causes an AIC# column to appear in the search results. The user then clicks on a "Create/update AIC structMap" button below the search results table.

AICs and AIPs in archival storage

• This diagram shows a storage area with standalone AIPs, an AIC METS file with child AIPs, and related pointer.xml files. Note that from a storage/data management standpoint, Archivematica considers the AIC METS file to be the AIC.

Archival storage area containing pointer files, AIC and AIPs

Sample pointer.xml file

<mets xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd"> 
  <metsHdr CREATEDATE="2013-06-13T20:39:08"/> 
  <amdSec ID="amdSec_1" > 
    <techMD ID="techMD_1" > 
      <mdWrap MDTYPE="PREMIS:OBJECT" > 
        <xmlData> 
          <object xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" 
            version="2.2" > 
            <objectIdentifier> 
              <objectIdentifierType>UUID</objectIdentifierType> 
              <objectIdentifierValue>3b2be1ec-c70a-4270-9993-O1c85d22c975</objectIdentifierValue> 
            </objectIdentifier> 
            <objectCharacteristics> 
              <compositionLevel>0</compositionLevel> 
              <fixity> 
                <messageDigestAlgorithm>sha256</messageDigestAlgorithm> 
                <messageDigest> 
                  c7eefbc60558cc127becb290c09d5e3cb90ce886504b79f33d94bcccOOb1728c 
                </messageDigest> 
              </fixity> 
              <size>4367852658</size> 
              <format> 
                <formatDesignation> 
                  <formatName>7Zip format</formatName> 
                  <formatVersion/> 
                </formatDesignation> 
                <formatRegistry> 
                  <formatRegistryName>PRONOM</formatRegistryName> 
                  <formatRegistryKey>fmt/484</formatRegistryKey> 
                </formatRegistry> 
              </format>
              <creatingApplication> 
                <creatingApplicationName>7-Zip</creatingApplicationName> 
                <creatingApplicationVersion>9.20</creatingApplicationVersion> 
                <dateCreatedByApplication>2013-06-13T20:39:08</dateCreatedByApplication> 
              </creatingApplication> 
            </objectCharacteristics> 
          </object> 
        </xmlData> 
      </mdWrap> 
    </techMD> 
  </amdSec> 
  <fileSec> 
    <fileGrp> 
      <file ID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975" ADMID="amdSec_1" >
        <FLocat xlini:href="filepath/GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975.7z" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/> 
      </file> 
    </fileGrp> 
  </fileSec> 
  <structMap TYPE="physical"> 
    <div TYPE="Archival Information Package"> 
      <fptr FILEID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975"/> 
    </div> 
  </structMap> 
</mets>

Pre-OCR: first half of image, second half of image

Previously-discussed options and analysis

Possible AIC/AIP designs

Option 1 (preferred) Description: An AIC consisting of only a fileSec and structMap; AIPs consisting of data files and metadata for those data files; an AIP consisting of project/program-level (i.e. dataset) metadata and documentation. Workflow: User creates X number of AIPs and puts them in archival storage One of these AIPs consists only of metadata and documentation about the program/project as a whole The AIPs must have one or more common metadata elements that allows them to be identified as being related User searches for AIPs in archival storage tab (using the common metadata element in the AIPs in the search query) Once search results are retrieved, user clicks "Create AIC" button AIC is created, containing only a METS structMap listing all AIPs Over time, user can add new AIPs and re-create the AIC at any time; the new AIC will either replace or update the old one Over time, if needed the user either updates the existing documentation AIP or adds new documentation AIPs (i.e. there can be more than one documentation AIP per dataset) Pros: Don't have to duplicate program/project-level documentation in each AIP Simple workflow for creating AIC Easy to add new AIPs If program/project documentation needs updating, only one AIP has to be re-processed, or user can add new documentation AIP(s) Cons: There is only a one-way link between the AIC and child AIPs - i.e. the AIC has a structMap listing all child AIPs, but there is nothing in a child AIP to indicate that it belongs to a given AIC. Sample AIC METS file Sample pointer.xml file Option 1A (supplied by U of A) Description: Workflow User selects type 'AIC' on Archivematica's Transfer page User provides input about the status of this AIC: Is this a new AIC Yes: User needs to create a new AIC with a new system-generated or user generated AIC# No: AIC already exists and the user just wants to add new AIPs or wants to update the AIC. User should be able to search this AIC through AIC# or using other search terms Once a new AIC has been created OR an existing one selected, a loop starts, during which new AIPs are prepared and added to this AIC. Each AIP contains a reference to the AIC through AIC#, as part of the system generated metadata. The loop ends when the user finishes adding AIPs User selects "Generate AIC structMap" option and it will generate a "structMap" for this collection. This structMap also has a reference to its corresponding AIC through AIC#. If possible, store it inside the AIC, otherwise as a seperate structMap AIP System prepares the whole package for preservation (AIC/AIPs/structMap) and sends it to archival storage Over time, user can add new AIPs. Generate AIC structMap step can either replace structMap or update the old one. Over time, the user can add/update the existing documentation in an AIC by updating the AIC's AIP Pros Don't have to duplicate program/project-level documentation in each AIP AIPs and AIC are linked through AIC number Cons The process of creating an AIC first, then adding AIPs and having the system automatically add the AIC UUID to the AIP, would be complex and could not easily be built on existing code for generating AIP METS files. "System prepares the whole package for preservation (AIC/AIPs/structMap) and sends it to archival storage" - this does not match Archivematica's basic processing pipeline design, which generates AIPs and places them individually in archival storage. In reference to point 1, above, there would be no obvious mechanism for finding the UUID of the AIC and adding it to an AIP if the AIC has not already been placed in archival storage. "Over time, user can add new AIPs. Generate AIC structMap step can either replace structMap or update the old one" - it is not clear where or how this step would occur, since this would require searching for an AIC and its constituent AIPs in archival storage, then generating a new structMap from stored AIPs plus the AIP that is currently being created (and this process would need to be run separately for each new AIP). Updating the documentation AIP would be problematic if the documentation AIP doubled as the AIC. There is currently no mechanism for AIP versioning, and it is likely that "versioning" will actually mean creating replacement AIPs. In this case, updating the documentation would result in the creation of a new AIC with a new UUID, which would mean that all the AIC UUIDs in the child AIPs would become obsolete. Option 2 Description: An AIC consisting of a METS structMap and project/program-level (i.e. dataset) metadata and documentation; content AIPs consisting of data files and metadata about the data files. AIPs have information in the METS files (in the structMap?) linking them to the parent AIC. Workflow: To be determined - probably a dashboard tab with a gui to allow users to arrange existing AIPs into an AIC Pros: Don't have to duplicate program/project-level documentation in each AIP AIPs have a link up to the AIC, so if an AIP is orphaned the relationship to the AIC can easily be reconstructed If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed Cons: Workflow to create this structure may be complex No obvious mechanism for adding new AIPs over time Option 3 Description: An AIC with a unique identifier consisting of project/program-level (i.e. dataset) metadata and documentation only (no structMap); AIPs consisting of data files, metadata for those data files, and the same identifier as the AIC. The relationship between the AIC and AIPs in this scenario is inferred from the matching identifiers. Workflow: User creates an AIC consisting of project/program-level (i.e. dataset) metadata and documentation The AIC contains an identifier that distinguishes it from other AICs User creates AIPs consisting of data files and metadata for those data files User includes the AIC identifier in each AIP Over time, if needed the user can add more AIPs with the same identifier Pros: Don't have to duplicate program/project-level documentation in each AIP Simple workflow Minimal development requirements, just new metadata field for identifier added to transfer tab, corresponding entry in AIC/AIP METS files and ability to search by AIC identifier in archival storage tab If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed Easy to add more AIPs to the same AIC over time Cons: No structMap in the AIC means that there is no single source of information about how many AIPs are in the AIC Option 4 Description: No AIC; project/program-level metadata and documentation duplicated in all AIPs; links between AIPs belonging to one dataset inferred from metadata only Workflow: User creates any number of AIPs with complete copies of the project/program-leve (i.e. dataset) metadata and documentation in each AIP Pros: Minimal Archivematica development required, just ensuring that matching metadata elements are parsed to the AIP METS files or otherwise made available to ElasticSearch index Easy to add new AIPs over time Cons: User has to maintain copies of project/program-level metadata and documentation outside of Archivematica so they can be added to each AIP Updating the project/program-level metadata and documentation would require re-processing the AIPs Relationships between AIPs would have to be inferred from matching metadata elements alone; if an AIP were lost, there would be no list of AIPs belonging to the dataset which would reveal the loss