Dataset preservation

From Archivematica
Revision as of 16:16, 11 February 2020 by Sallain (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search


This page is no longer being maintained and may contain inaccurate information. Please see the Archivematica documentation for up-to-date information.

Workflow

  • Composition of AIPs: Large datasets may be divided into multiple transfers prior to ingest, so that one dataset ultimately consists of a number of AIPs. See Hierarchical AIC/AIP structure, below.
    • note: a related, follow-up Archivematica requirement is to break up large files (e.g. video) that exceed a configurable maximum file size into multiple AIPs, to which the hierarchical structure described below would also apply
  • Metadata ingest: Metadata will be created outside of Archivematica prior to ingest, and may be referenced from the dmdSec of the AIP METS file as an xlink reference. See Metadata, below.
  • Normalization: Some types of data files may require manual normalization: see https://projects.artefactual.com/issues/1499.


Metadata

METS and DDI/FGDC

  • DDI is Data Documentation Initiative, a metadata specification for the social and behavioral sciences; see http://www.ddialliance.org/.
  • FGDC is Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]; see http://www.fgdc.gov/metadata/csdgm/
  • DDI and FGDC are considered descriptive metadata (dmdSec) in METS. From http://www.loc.gov/standards/mets/METSOverview.v2.html: "Valid values for the MDTYPE element [in dmdSec] include...DDI (Data Documentation Initiative), FGDC (Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]."
    • In the Archivematica METS file, a DDI or FGDC file could be referenced from the dmdSec using mdRef, for example as follows: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="DDI" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>.


METS and other metadata standards


Hierarchical AIC/AIP structure

  • Because datasets can be large and heterogeneous, one "dataset" may be broken into multiple AIPs. In such cases, the multiple AIPs can be intellectually combined into one AIC, or Archival Information Collection, defined by the OAIS reference model as "[a]n Archival Information Package whose Content Information is an aggregation of other Archival Information Packages." (OAIS 1-9).
    • A basic AIC in Archivematica consists of 1) any number of related AIPs and 2) a METS file containing a fileSec and a logical structMap listing all the related AIPs.
    • In storage, a pointer.xml file gives storage and compression information for each AIC METS file and each AIP.


Workflow for preserving a dataset as an Archival Information Collection


Dataset workflow1.png


Metadata entry

Metadata entry for the AIC

This screenshot shows sample metadata entry for the AIC (an AIP consisting of program/project-level metadata and documentation). The screenshot also shows the Dublin Core values that would be generated from the data entry. Note that that Dublin Core values would also be copied to the AIC METS file generated at the end of the workflow.


AIC metadata.png


Metadata for AIPs

This screenshot shows sample metadata entry for an AIP (consisting of data files and associated metadata). The screenshot also shows the Dublin Core values that would be generated from the data entry. These values are saved in the AIP METS file but are not copied to the AIC METS file.


AIP metadata.png


AIC METS file

This shows the AIC METS file which is generated at the end of the workflow. Note the Dublin Core metadata copied from the METS file in the AIC (the AIP containing program/project-level metadata and documentation). 

<?xml version='1.0' encoding='ASCII'?>
<mets xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.loc.gov/METS/" xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd">
  <metsHdr CREATEDATE="2013-06-13T20:39:08"/>
  <dmdSec ID="dmdSec_1">
    <mdWrap MDTYPE="DC">
      <xmlData>
        <dublincore xmlns="http://purl.org/dc/terms/" xsi:schemaLocation="http://purl.org/dc/terms/ http://dublincore.org/schemas/xmls/qdc/2008/02/11/dcterms.xsd">
          <title>Global Warming and Arctic Marine Mammals (GWAMM)</title>
          <type>Archival Information Collection</type>
          <identifier>AIC#1774.28/D</identifier>
        </dublincore>
      </xmlData>
    </mdWrap>
  </dmdSec>
  <fileSec>
    <fileGrp>
      <file ID="GWAMM01-3b2be1ec-c70a-4270-9993-01c85d22c975"/>
      <file ID="GWAMM02-cfd9a713-6980-45cf-9db1-86f1cdcf3611"/>
      <file ID="GWAMM03-tgd9a814-4955-45cf-9db1-23f1gjkf3956"/> 
    </fileGrp>
  </fileSec>
  <structMap TYPE="logical">
    <div TYPE="Archival Information Collection" DMDID="dmdSec_1">
      <div LABEL="Global Warming and Arctic Marine Mammals (GWAMM)">
        <fptr FILEID="GWAMM01-3b2be1ec-c70a-4270-9993-01c85d22c975"/>
      </div>
      <div LABEL="Year two observations: Baffin Island">
        <fptr FILEID="GWAMM02-cfd9a713-6980-45cf-9db1-86f1cdcf3611"/>
      </div>
      <div LABEL="Beaufort Sea 2008-2009">
        <fptr FILEID="GWAMM03-tgd9a814-4955-45cf-9db1-23f1gjkf3956"/>
      </div>
    </div>     
  </structMap>
</mets>

Original screenshot

Creating an AIC structMap in the archival storage tab

Once all the AIPs have been created and stored, the user searches for them in the archival storage tab. Selecting the "Show AICs?" box at the top of the screen causes an AIC# column to appear in the search results. The user then clicks on a "Create/update AIC structMap" button below the search results table.


Archival storage AIC1.png


AICs and AIPs in archival storage

  • This diagram shows a storage area with standalone AIPs, an AIC METS file with child AIPs, and related pointer.xml files. Note that from a storage/data management standpoint, Archivematica considers the AIC METS file to be the AIC.


Archival storage area containing pointer files, AIC and AIPs


Sample pointer.xml file

<mets xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd"> 
  <metsHdr CREATEDATE="2013-06-13T20:39:08"/> 
  <amdSec ID="amdSec_1" > 
    <techMD ID="techMD_1" > 
      <mdWrap MDTYPE="PREMIS:OBJECT" > 
        <xmlData> 
          <object xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" 
            version="2.2" > 
            <objectIdentifier> 
              <objectIdentifierType>UUID</objectIdentifierType> 
              <objectIdentifierValue>3b2be1ec-c70a-4270-9993-O1c85d22c975</objectIdentifierValue> 
            </objectIdentifier> 
            <objectCharacteristics> 
              <compositionLevel>0</compositionLevel> 
              <fixity> 
                <messageDigestAlgorithm>sha256</messageDigestAlgorithm> 
                <messageDigest> 
                  c7eefbc60558cc127becb290c09d5e3cb90ce886504b79f33d94bcccOOb1728c 
                </messageDigest> 
              </fixity> 
              <size>4367852658</size> 
              <format> 
                <formatDesignation> 
                  <formatName>7Zip format</formatName> 
                  <formatVersion/> 
                </formatDesignation> 
                <formatRegistry> 
                  <formatRegistryName>PRONOM</formatRegistryName> 
                  <formatRegistryKey>fmt/484</formatRegistryKey> 
                </formatRegistry> 
              </format>
              <creatingApplication> 
                <creatingApplicationName>7-Zip</creatingApplicationName> 
                <creatingApplicationVersion>9.20</creatingApplicationVersion> 
                <dateCreatedByApplication>2013-06-13T20:39:08</dateCreatedByApplication> 
              </creatingApplication> 
            </objectCharacteristics> 
          </object> 
        </xmlData> 
      </mdWrap> 
    </techMD> 
  </amdSec> 
  <fileSec> 
    <fileGrp> 
      <file ID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975" ADMID="amdSec_1" >
        <FLocat xlini:href="filepath/GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975.7z" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/> 
      </file> 
    </fileGrp> 
  </fileSec> 
  <structMap TYPE="physical"> 
    <div TYPE="Archival Information Package"> 
      <fptr FILEID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975"/> 
    </div> 
  </structMap> 
</mets>

Pre-OCR: first half of image, second half of image

Previously-discussed options and analysis

Possible AIC/AIP designs

Option 1 (preferred)


AIP1G.png


Description: An AIC consisting of only a fileSec and structMap; AIPs consisting of data files and metadata for those data files; an AIP consisting of project/program-level (i.e. dataset) metadata and documentation.


Workflow:

  1. User creates X number of AIPs and puts them in archival storage
    • One of these AIPs consists only of metadata and documentation about the program/project as a whole
    • The AIPs must have one or more common metadata elements that allows them to be identified as being related
  2. User searches for AIPs in archival storage tab (using the common metadata element in the AIPs in the search query)
  3. Once search results are retrieved, user clicks "Create AIC" button
  4. AIC is created, containing only a METS structMap listing all AIPs
  5. Over time, user can add new AIPs and re-create the AIC at any time; the new AIC will either replace or update the old one
  6. Over time, if needed the user either updates the existing documentation AIP or adds new documentation AIPs (i.e. there can be more than one documentation AIP per dataset)


Pros:

  • Don't have to duplicate program/project-level documentation in each AIP
  • Simple workflow for creating AIC
  • Easy to add new AIPs
  • If program/project documentation needs updating, only one AIP has to be re-processed, or user can add new documentation AIP(s)


Cons:

  • There is only a one-way link between the AIC and child AIPs - i.e. the AIC has a structMap listing all child AIPs, but there is nothing in a child AIP to indicate that it belongs to a given AIC.


Sample AIC METS file


METS AIC AIP.png


Sample pointer.xml file


Pointer6G.png
Pointer7G.png


Option 1A (supplied by U of A)


Description:


UofAAIC.png


Workflow

  1. User selects type 'AIC' on Archivematica's Transfer page
  2. User provides input about the status of this AIC: Is this a new AIC
    • Yes: User needs to create a new AIC with a new system-generated or user generated AIC#
    • No: AIC already exists and the user just wants to add new AIPs or wants to update the AIC. User should be able to search this AIC through AIC# or using other search terms
  3. Once a new AIC has been created OR an existing one selected, a loop starts, during which new AIPs are prepared and added to this AIC. Each AIP contains a reference to the AIC through AIC#, as part of the system generated metadata. The loop ends when the user finishes adding AIPs
  4. User selects "Generate AIC structMap" option and it will generate a "structMap" for this collection. This structMap also has a reference to its corresponding AIC through AIC#. If possible, store it inside the AIC, otherwise as a seperate structMap AIP
  5. System prepares the whole package for preservation (AIC/AIPs/structMap) and sends it to archival storage
  6. Over time, user can add new AIPs. Generate AIC structMap step can either replace structMap or update the old one.
  7. Over time, the user can add/update the existing documentation in an AIC by updating the AIC's AIP


Pros

  • Don't have to duplicate program/project-level documentation in each AIP
  • AIPs and AIC are linked through AIC number


Cons

  • The process of creating an AIC first, then adding AIPs and having the system automatically add the AIC UUID to the AIP, would be complex and could not easily be built on existing code for generating AIP METS files.
  • "System prepares the whole package for preservation (AIC/AIPs/structMap) and sends it to archival storage" - this does not match Archivematica's basic processing pipeline design, which generates AIPs and places them individually in archival storage. In reference to point 1, above, there would be no obvious mechanism for finding the UUID of the AIC and adding it to an AIP if the AIC has not already been placed in archival storage.
  • "Over time, user can add new AIPs. Generate AIC structMap step can either replace structMap or update the old one" - it is not clear where or how this step would occur, since this would require searching for an AIC and its constituent AIPs in archival storage, then generating a new structMap from stored AIPs plus the AIP that is currently being created (and this process would need to be run separately for each new AIP).
  • Updating the documentation AIP would be problematic if the documentation AIP doubled as the AIC. There is currently no mechanism for AIP versioning, and it is likely that "versioning" will actually mean creating replacement AIPs. In this case, updating the documentation would result in the creation of a new AIC with a new UUID, which would mean that all the AIC UUIDs in the child AIPs would become obsolete.


Option 2


AIP2G.png


Description: An AIC consisting of a METS structMap and project/program-level (i.e. dataset) metadata and documentation; content AIPs consisting of data files and metadata about the data files. AIPs have information in the METS files (in the structMap?) linking them to the parent AIC.


Workflow: To be determined - probably a dashboard tab with a gui to allow users to arrange existing AIPs into an AIC


Pros:

  • Don't have to duplicate program/project-level documentation in each AIP
  • AIPs have a link up to the AIC, so if an AIP is orphaned the relationship to the AIC can easily be reconstructed
  • If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed


Cons:

  • Workflow to create this structure may be complex
  • No obvious mechanism for adding new AIPs over time


Option 3


AIP3G.png


Description: An AIC with a unique identifier consisting of project/program-level (i.e. dataset) metadata and documentation only (no structMap); AIPs consisting of data files, metadata for those data files, and the same identifier as the AIC. The relationship between the AIC and AIPs in this scenario is inferred from the matching identifiers.


Workflow:

  1. User creates an AIC consisting of project/program-level (i.e. dataset) metadata and documentation
    • The AIC contains an identifier that distinguishes it from other AICs
  2. User creates AIPs consisting of data files and metadata for those data files
    • User includes the AIC identifier in each AIP
  3. Over time, if needed the user can add more AIPs with the same identifier


Pros:

  • Don't have to duplicate program/project-level documentation in each AIP
  • Simple workflow
  • Minimal development requirements, just new metadata field for identifier added to transfer tab, corresponding entry in AIC/AIP METS files and ability to search by AIC identifier in archival storage tab
  • If program/project-level metadata and documentation needs to be updated, only the AIC needs to be re-processed
  • Easy to add more AIPs to the same AIC over time


Cons:

  • No structMap in the AIC means that there is no single source of information about how many AIPs are in the AIC


Option 4


AIP4G.png


Description: No AIC; project/program-level metadata and documentation duplicated in all AIPs; links between AIPs belonging to one dataset inferred from metadata only


Workflow: User creates any number of AIPs with complete copies of the project/program-leve (i.e. dataset) metadata and documentation in each AIP


Pros:

  • Minimal Archivematica development required, just ensuring that matching metadata elements are parsed to the AIP METS files or otherwise made available to ElasticSearch index
  • Easy to add new AIPs over time


Cons:

  • User has to maintain copies of project/program-level metadata and documentation outside of Archivematica so they can be added to each AIP
  • Updating the project/program-level metadata and documentation would require re-processing the AIPs
  • Relationships between AIPs would have to be inferred from matching metadata elements alone; if an AIP were lost, there would be no list of AIPs belonging to the dataset which would reveal the loss


Retrieved from "https://wiki.archivematica.org/index.php?title=Dataset_preservation&oldid=13370"