- Composition of AIPs: Large datasets may be divided into multiple transfers prior to ingest, so that one dataset ultimately consists of a number of AIPs. See Hierarchical AIC/AIP structure, below.
- note: a related, follow-up Archivematica requirement is to break up large files (e.g. video) that exceed a configurable maximum file size into multiple AIPs, to which the hierarchical structure described below would also apply
- Metadata ingest: Metadata will be created outside of Archivematica prior to ingest, and may be referenced from the dmdSec of the AIP METS file as an xlink reference. See Metadata, below.
- Normalization: Some types of data files may require manual normalization: see https://projects.artefactual.com/issues/1499.
METS and DDI/FGDC
- DDI is Data Documentation Initiative, a metadata specification for the social and behavioral sciences; see http://www.ddialliance.org/.
- FGDC is Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]; see http://www.fgdc.gov/metadata/csdgm/
- DDI and FGDC are considered descriptive metadata (dmdSec) in METS. From http://www.loc.gov/standards/mets/METSOverview.v2.html: "Valid values for the MDTYPE element [in dmdSec] include...DDI (Data Documentation Initiative), FGDC (Federal Geographic Data Committee Metadata Standard [FGDC-STD-001-1998]."
- In the Archivematica METS file, a DDI or FGDC file could be referenced from the dmdSec using mdRef, for example as follows: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="DDI" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>.
METS and other metadata standards
- Other metadata standards that could be used for ingested datasets include:
- North American Profile (NAP) of ISO 19119, for geospatial metadata: http://www.fgdc.gov/metadata/geospatial-metadata-standards
- SDMX for aggregate data: http://sdmx.org/?page_id=10
- EML, the Ecological Metadata Language: http://knb.ecoinformatics.org/software/eml/eml-2.1.1/index.html
- If these standards are used, the mdRef in the METS file would need to use OTHER as MDTYPE, for example: <mdRef LABEL="CCRI-CDN-Census1911V20110628.xml-73b93b28-be1b-433f-861e-03bc321dfe7e" xlink:href="metadata/CCRI-CDN-Census1911V20110628.xml" MDTYPE="OTHER" OTHERMDTYPE="SDMX" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/>
Hierarchical AIC/AIP structure
- Because datasets can be large and heterogeneous, one "dataset" may be broken into multiple AIPs. In such cases, the multiple AIPs can be intellectually combined into one AIC, or Archival Information Collection, defined by the OAIS reference model as "[a]n Archival Information Package whose Content Information is an aggregation of other Archival Information Packages." (OAIS 1-9).
- A basic AIC in Archivematica consists of 1) any number of related AIPs and 2) a METS file containing a fileSec and a logical structMap listing all the related AIPs.
- In storage, a pointer.xml file gives storage and compression information for each AIC METS file and each AIP.
Workflow for preserving a dataset as an Archival Information Collection
Metadata entry for the AIC
This screenshot shows sample metadata entry for the AIC (an AIP consisting of program/project-level metadata and documentation). The screenshot also shows the Dublin Core values that would be generated from the data entry. Note that that Dublin Core values would also be copied to the AIC METS file generated at the end of the workflow.
Metadata for AIPs
This screenshot shows sample metadata entry for an AIP (consisting of data files and associated metadata). The screenshot also shows the Dublin Core values that would be generated from the data entry. These values are saved in the AIP METS file but are not copied to the AIC METS file.
AIC METS file
This screenshot shows the AIC METS file which is generated at the end of the workflow. Note the Dublin Core metadata copied from the METS file in the AIC (the AIP containing program/project-level metadata and documentation).
Creating an AIC structMap in the archival storage tab
Once all the AIPs have been created and stored, the user searches for them in the archival storage tab. Selecting the "Show AICs?" box at the top of the screen causes an AIC# column to appear in the search results. The user then clicks on a "Create/update AIC structMap" button below the search results table.
AICs and AIPs in archival storage
- This diagram shows a storage area with standalone AIPs, an AIC METS file with child AIPs, and related pointer.xml files. Note that from a storage/data management standpoint, Archivematica considers the AIC METS file to be the AIC.
Sample pointer.xml file
<mets xsi:schemaLocation="http://www.loc.gov/METS/ http://www.loc.gov/standards/mets/version18/mets.xsd"> <metsHdr CREATEDATE="2013-06-13T20:39:08"/> <amdSec ID="amdSec_1" > <techMD ID="techMD_1" > <mdWrap MDTYPE="PREMIS:OBJECT" > <xmlData> <object xsi:type="file" xsi:schemaLocation="info:lc/xmlns/premis-v2 http://www.loc.gov/standards/premis/v2/premis-v2-2.xsd" version="2.2" > <objectIdentifier> <objectIdentifierType>UUID</objectIdentifierType> <objectIdentifierValue>3b2be1ec-c70a-4270-9993-O1c85d22c975</objectIdentifierValue> </objectIdentifier> <objectCharacteristics> <compositionLevel>0</compositionLevel> <fixity> <messageDigestAlgorithm>sha256</messageDigestAlgorithm> <messageDigest> c7eefbc60558cc127becb290c09d5e3cb90ce886504b79f33d94bcccOOb1728c </messageDigest> </fixity> <size>4367852658</size> <format> <formatDesignation> <formatName>7Zip format</formatName> <formatVersion/> </formatDesignation> <formatRegistry> <formatRegistryName>PRONOM</formatRegistryName> <formatRegistryKey>fmt/484</formatRegistryKey> </formatRegistry> </format> <creatingApplication> <creatingApplicationName>7-Zip</creatingApplicationName> <creatingApplicationVersion>9.20</creatingApplicationVersion> <dateCreatedByApplication>2013-06-13T20:39:08</dateCreatedByApplication> </creatingApplication> </objectCharacteristics> </object> </xmlData> </mdWrap> </techMD> </amdSec> <fileSec> <fileGrp> <file ID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975" ADMID="amdSec_1" > <FLocat xlini:href="filepath/GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975.7z" LOCTYPE="OTHER" OTHERLOCTYPE="SYSTEM"/> </file> </fileGrp> </fileSec> <structMap TYPE="physical"> <div TYPE="Archival Information Package"> <fptr FILEID="GWAMM01-3b2be1ec-c70a-4270-9993-O1c85d22c975"/> </div> </structMap> </mets>
Previously-discussed options and analysis
|Possible AIC/AIP designs|
Option 1 (preferred)
Description: An AIC consisting of only a fileSec and structMap; AIPs consisting of data files and metadata for those data files; an AIP consisting of project/program-level (i.e. dataset) metadata and documentation.
Sample AIC METS file
Sample pointer.xml file
Option 1A (supplied by U of A)
Description: An AIC consisting of a METS structMap and project/program-level (i.e. dataset) metadata and documentation; content AIPs consisting of data files and metadata about the data files. AIPs have information in the METS files (in the structMap?) linking them to the parent AIC.
Workflow: To be determined - probably a dashboard tab with a gui to allow users to arrange existing AIPs into an AIC
Description: An AIC with a unique identifier consisting of project/program-level (i.e. dataset) metadata and documentation only (no structMap); AIPs consisting of data files, metadata for those data files, and the same identifier as the AIC. The relationship between the AIC and AIPs in this scenario is inferred from the matching identifiers.
Description: No AIC; project/program-level metadata and documentation duplicated in all AIPs; links between AIPs belonging to one dataset inferred from metadata only
Workflow: User creates any number of AIPs with complete copies of the project/program-leve (i.e. dataset) metadata and documentation in each AIP