Ingest (0.5)

From Archivematica
Revision as of 12:46, 19 November 2009 by Evelyn McLellan (talk | contribs)
Jump to navigation Jump to search

Main Page > Documentation > Release 0.3 Documentation > Ingest (0.3)


AD1 Receive SIP

File:Archivematica AD1 ReceiveSIP v1.pdf

Expected Procedures

Case 1:Internal Producer

  • [1.1] --> [1.3] --> [1.4] --> [1.6] --> [1.8] --> [end]
    • Assumption - Submission agreements with internal producers will mandate that some type of checksum for validating the integrity of the SIP must be included in the SIP; therefore, this case should never have to invoke step 1.5 - Generate Checksum


Case 2: External Producer (may or may not include a checksum as part of the SIP)

  • [1.1] --> [1.2] --> [1.4] --> [1.6] --> [1.8] --> [end] (SIP includes checksum); or,
  • [1.1] --> [1.2] --> [1.5] --> [1.6] --> [1.8] --> [end] (SIP does not include checksum)

Exceptions

  • (for both cases) Submitted SIP includes integrity checksums and fails integrity check at [1.6]


Workflow diagram step Description UML diagram references
Producer places SIP in shared folder on host machine
  • The purpose of shared folders is to allow the Producer to drop SIPs into a folder on their host machine or network and have the SIPs automatically appear in a folder in Archivematica.
    • For instructions on setting up shared folders, please go to Virtual appliance instructions.
    • Neither the folder name nor any of the filenames should include spaces or special characters. Underscores are ok.
    • Make the shared folder in Archivematica /home/demoreceiveSIP/.
  • For testing purposes you can avoid setting up shared folders and simply use the test files found in /home/demo/testFiles/.
SIP appears in shared folder in Archivematica
  • SIP will appear in /home/demo/receiveSIP/.
  • If using files from /home/demo/testFiles/, copy a folder from /testFiles/ into /home/demo/receiveSIP/.
Archivist copies SIP from shared folder to SIP backup folder.
  • Copy the SIP in /home/demo/receiveSIP/ and paste it into /home/demo/receiveSIPbackup/. If anything goes wrong during the ingest process, this backup copy can be retrieved and processed.
Archivist moves SIP from shared folder into quarantine
  • Drag the SIP from /home/demo/receiveSIP/ to /home/demo/quarantine.
SIP is quarantined for 2 minutes
  • Select checksum report, right-click on report, select scripts > checkMD5.
  • Review report to ensure all checksums match (go to the bottom of the report; it should say "All files are OK!").
  • If the integrity check fails, go to 1.7 - Request resubmission of SIP. '(note - failure of the integrity check is regarded as an exception to the expected procedure)'
  • If the integrity check passes, go to 1.8 - Send confirmation of receipt to Producer
1.7 Request resubmission of SIP
  • Request resubmission of SIP from Producer
  • Go to 1.4 - Receive SIP from Producer
1.8 Send confirmation of receipt to Producer (UC-1.2)
  • Send e-mail to internal producer; send e-mail or other confirmation to external producer.
  • Save copy of confirmation to the accession documentation folder
  • Go to 2.1 - Quarantine SIP

AD2 Audit SIP

File:Archivematica AD2 AuditSIP v5.pdf

Expected Procedure

  • [2.1] --> [2.2] --> [2.4] --> [2.5] --> [2.8] --> [end]


Exceptions

  • malware detected at step 2.3
  • SIP not compliant at step 2.5


Step Implementation Notes
General procedures
2.1 Quarantine SIP
  • Leave SIP in current location (/home/demo/ingest/2009_0001). Wait for quarantine period to expire.
  • Go to: 2.2 - Check SIP for malware
2.2 Check SIP for malware

NOTE: There is no malware checking software included in the Archivimatica 0.3 release, so steps 2.2 and 2.3 are hypothetical only.

  • Check SIP for the presence of malware
  • Create malware check report, copy report to the Accession Documentation folder
  • If malware is detected, go to: 2.3 - Remove malware
  • If malware is not detected, go to: 2.4 - Audit SIP for compliance
  • Documentation should include:
    • list of software used to detect malware
      • related virus/malware definition used to perform the check
    • date and time of check
    • reports generated by the software identifying infected files, nature of the infection
  • "Because of the age of the digital components contained in the SIP, the Archive must ensure that malicious code check can recognize very old malicious code." Tufts/Yale Ingest Guide For University Electronic Records, B2.3 (http://dca.lib.tufts.edu/features/nhprc/reports/ingest/part_B-02-03.html)
2.3 Remove malware

Attempt to remove malware from the SIP

  • Document the tools and procedures used to remove the malware
  • Document success or failure of the malware removal for each infected file
  • Copy all malware removal report to the Accession Documentation folder

Go to: 2.4 - Audit SIP for compliance

Case: malware not removed

  • Create a plain text report (click on applications > accessories > text editor) describing the type(s) of malware, the efforts made to remove the malware and the reasons for failing to remove it. Save the report to /home/demo/accessions/2009_01.
  • Presumably any malware removal tools generate reports on success/failure. Depending on the software used, the detection occurring in step 2.2 and the removal occurring in this step may be documented in the same report
2.4 Audit SIP for compliance (UC-1.3; UC-4.6) Manually verify that the SIP conforms to the archives' data formatting and documentation standards and meets the specifications of the Submission Agreement. Do this by skimming the filenames and extensions to make sure that what was supposed to be in the SIP according to the Submission Agreement is actually there.

Create audit documentation

  • Create an audit documentation report as plain text file (click on applications > accessories > text editor)
    • If the SIP is wholly compliant, note this in the audit report; else
    • Document deficiencies in the SIP, identifying the nature of the deficiency (missing file extensions, unacceptable formats, unacceptable packaging, presence of unremoved malware, etc.) and the object(s) the deficiency pertains to (file or group of files, SIP packaging, etc.)
  • Save the SIP audit documentation to the Accession documentation folder (e.g., /home/demo/accessions/2009_0001/)

Go to: 2.5 - Assess SIP deficiencies

  • What else should we check for?
  • Maybe develop a checklist approach to reporting on deficiencies - for example: "contains unacceptable formats YES; records inadequately identified YES..."
2.5 - Assess SIP deficiencies

Based on the results of the audit performed in step 2.4 - Audit SIP for compliance, determine if the deficiencies identified, if any, warrant rejection of the SIP, or if the SIP can be accepted despite identified deficiencies. If the majority of objects conform to standards and Submission Agreement, the SIP may be considered acceptable. Note that the non-conforming objects can be deleted after appraisal (see AD3 step 3.9). Document the decision to accept or reject the SIP

If SIP can be accepted for ingest,

  • Document that SIP has been acepted for ingest
  • go to: 2.8 - Notify producer of SIP acceptance

If SIP can not be accepted for ingest, go to: 2.7 - Notify Producer of SIP rejection

2.6 Notify Producer of SIP rejection
  • Document that the SIP has been rejected, copy the report to the Accession Documentation folder
  • Send an e-mail notification, attaching a copy of the reports created in steps 2.4 or 2.5.
  • If the Producer appeals the SIP rejection:
    • Document receipt of the appeal
    • go to: 2.7 - Evaluate appeals
  • If the Producer does not appeal the SIP rejection: go to: 2.9 - Destroy SIP copies
2.7 Evaluate appeals

Evaluate the Producer's appeal of the SIP rejection

  • If the appeal is accepted:
    • Document acceptance of the appeal
    • Go to: 2.8 - Notify producer of SIP acceptance
  • If the appeal is rejected:
    • Document the rejection of the appeal
    • Notify the Producer of the appeal rejection
    • Go to 2.9 - Destroy SIP copies
2.8 Notify producer of SIP acceptance

Send Producer notification that the SIP has been accepted for ingest Document

Go to: 3.1 - Extract content information from SIP

2.9 Destroy SIP copies Delete SIP from /home/demo/dropoff and /home/demo/ingest.

End ingest

AD3 Accept SIP for Ingest

File:Archivematica AD3 AcceptSIPforIngest v4.pdf

Expected Procedure

[3.1] --> [3.2] --> [3.3] --> [3.4] --> [3.5] --> [3.6] --> [3.9] --> [3.10]

Exceptions

  • Producer notification required following [3.6]
  • Appraisal decision appealed following [3.7]


Step Implementation Notes
General procedures
3.1 Unpack SIP (UC-1.3)

Create folders for the SIP contents

  • Create the following 2 directories within the current folder (e.g., /home/demo/ingest/2009_01):
    • /content
    • /PDI
  • Unpack the SIP
    • If the SIP uses an archive file format such as .zip, .tar, etc.., extract the contents using the appropriate unpacking software.
    • Identify the content objects in the SIP and sort them into the /content directory
    • Identify the PDI and sort them into the /PDI directory

Go to: 3.2 - Modify/Provide additional PDI

3.2 Modify / provide additional PDI (UC-1.3)

Go to: 3.3 - Identify Formats

3.3 Identify format Use DROID and NLNZ Metadata extractor to identify objects in the SIP. Save the reports to /home/demo/ingest/2009_01/PDI/.

Go to: 3.4 - Validate formats

3.4 Validate format Use JHOVE to validate the objects in the SIP. Save the report to /home/demo/ingest/2009_01/PDI/.

Go to: 3.5 - Extract metadata

3.5 Extract metadata

Extract preservation metadata from content objects in the SIP

Go to: - 3.6 Audit submission and select for preservation

  • This step seems to have been accomplished in steps 3.4 and 3.6.
  • In archivematica, the metadata extraction activity takes place at the same time as format identification, as the NLNZ Metadata extractor tool also performs format identification.
  • What is the necessary metadata that must be extracted at this point?
3.6 Audit submission and select for preservation (UC-4.6)

Based on the results of steps 3.2, 3.3, and 3.4, apply Archives policies and determine which (if any) content objects in the SIP should not be included in the AIP Document which content objects will not be included and why.

If all SIP content is to be included in the AIP, go to: 3.10 - Accept selected SIP components for ingest


If some SIP content is to be excluded from the AIP, and submission agreement requires notifying Producer of appraisal decision, go to: 3.7 - Notify Producer of appraisal decision

  • Else, go to: 3.9 - Destroy unselected SIP components

Possible reasons for exclusion:

  • technical
    • insufficient preservation metadata
    • unrecognized format
    • unsupported format
    • invalid format
  • appraisal
    • duplicate content
    • does file level selection occur at this point (e.g. methodological sampling of records at the file level for selective retention?) or before the SIPs get to this point?
    • further to the above, this diagram only deals with one sip at a time. How do we manage appraisal decisions that must take into consideration the relationships among many SIPs?
3.7 Notify Producer about appraisal decision(s) Provide the Producer with copies of the appraisal report or other documentation as required by the submisssion agreement identifying SIP components to be destroyed.

If the Producer appeals the appraisal decision, go to: 3.8 - Evaluate appeals

Else, go to: 3.9 - Destroy unselected SIP components

3.8 Evaluate appeals
  • Receive Producer's appeals
  • Evaluate appeals
  • Based on the evaluation, make any necessary amendments to the appraisal decision
  • Notify the Producer of the outcome of the appeal process

Go to: 3.9 - Destroy unselected SIP components

3.9 - Destroy unselected SIP components
  • Destroy all SIP content objects identified for destruction in the appraisal decision (or amended appraisal decision as appropriate).
  • Document destruction
  • Note - it is possible that no components have been identified for destruction. In this case, move on to step 3.10
  • Do we need to destroy accompanying metadata as well?
3.10 Accept selected SIP components for ingest

AD4 Generate AIP

File:Archivematica AD4 GenerateAIP v2.pdf


Step Implementation Notes
4.1 Create AIP containers If 1 SIP = 1 AIP this step is not necessary. It is only necessary if the SIP is being divided into multiple AIPs.
4.2 Add Content Information to AIP See note for step 4.1, above
4.3 Transform Content Information
  • In /demo/ingest/2009_01, create a folder entitled normalized.
  • Using Xena, normalize the contents of each AIP.
    • Set destination folder for the normalization to /home/demo/ingest/2009_01/normalized/.
    • Set the destination folder for the Xena log file to /home/demo/ingest/2009_01/PDI/.
4.4 Add Transformed Content Information to AIP See step 4.3, above.
4.5 Add PDI to AIP Create a plain text report containing provenance and other PDI elements (including arrangement information) and save it to the AIP.
4.6 Generate Descriptive Information (UC-1.4)
  • Obtain available descriptive information from PDI added to AIP, Submission Agreement and/or records schedule, communications with donor, etc.
  • In Qubit, enter descriptive information at aggregate levels of description (fonds, series, file). Then create item-level descriptions for each object to be uploaded.
  • Upload digital objects from /home/demo/ingest/2009_01/content/.
    • In Qubit 1.0.7 batch upload of digital objects is not possible. The objects have to be uploaded 1 at a time.
  • Add additional descriptive information to item-level descriptions.
  • For instructions on using Qubit, go to the on-line user manual.

AD5 Transfer AIP to Archival Storage

File:Archivematica AD5 TransferAIPtoArchivalStorage v2.pdf


Step Implementation Notes
5.1 Request storage of AIP (UC-1.5)
5.2 Transfer AIP to Archival storage (UC-1.5)
  • Right click on folder /home/demo/ingest/2009_01.
  • Click Scripts > BagIt.
  • A bag will be created and stored automatically in /home/demo/mybags/.
  • Copy AIP to /home/ingest/archivalstorage/.
  • In archivematica 0.4 maybe configure BagIt script to drop the bag directly into /home/demo/ingest/archivalstorage/.
  • The BagIt script can also be edited manually if desired: open demo/.gnome2/nautilus-scripts/Bagit and change "mybags" to "archivalstorage".
5.3 Confirm receipt and storage of AIP (UC-1.5)
5.4 Add AIP storage location to descriptive information (UC-1.6) In the physical storage area in Qubit, add the storage location.
5.5 Add Descriptive Information to Data Management (UC-1.6) This was done in AD4, step 4.6. In OAIS, generating descriptive information and adding them to data management are two different steps; however, in Archivematica this is done in one step in Qubit, which is used to upload images to a web interface, generate derivatives for searching and browsing and record descriptive information.
5.6 Confirm update of Data Management
5.7 Destroy SIP and AIP copies Destroy /home/demo/ingest/2009_01 and /home/demo/mybags/2009_01.
Retrieved from "https://wiki.archivematica.org/index.php?title=Ingest_(0.5)&oldid=613"