Ingest (0.3)

Main Page > Documentation > Release 0.3 Documentation > Ingest (0.3)

AD1 Receive SIP

File:Archivematica AD1 ReceiveSIP v1.pdf

Expected Procedures

Case 1:Internal Producer

• [1.1] --> [1.3] --> [1.4] --> [1.6] --> [1.8] --> [end]
  • Assumption - Submission agreements with internal producers will mandate that some type of checksum for validating the integrity of the SIP must be included in the SIP; therefore, this case should never have to invoke step 1.5 - Generate Checksum

Case 2: External Producer (may or may not include a checksum as part of the SIP)

• [1.1] --> [1.2] --> [1.4] --> [1.6] --> [1.8] --> [end] (SIP includes checksum); or,
• [1.1] --> [1.2] --> [1.5] --> [1.6] --> [1.8] --> [end] (SIP does not include checksum)

Exceptions

• (for both cases) Submitted SIP includes integrity checksums and fails integrity check at [1.6]

Step	Implementation	Notes
1.1 Receive notification of intent to submit SIP from Producer (UC-1.1)	Receive e-mail notification from internal producer that scheduled records are ready for transfer; receive signed donor form or other agreement from external producer. Create an accession documentation folder. (E.g., /home/demo/accessions/2009_01/). Save notification and any related records to this folder. If producer is external, go to 1.2 - Assign offline storage location If producer is internal (City of Vancouver), go to 1.3 - Assign network storage location
1.2 [External producer] Assign offline storage location for SIP (UC-1.1)	Select a secure physical location in which to store the media (i.e. disks, hard drive or other). Go to 1.4 - Receive SIP from Producer
1.3 [Internal producer] Assign network storage location for SIP	Assign a network location where internal producers can transfer submissions Create a new folder: /home/demo/ingestdropoff/2009_01/. Go to 1.4 - Receive SIP from Producer	This can be made a shared folder, so that the Producer transfers the SIP to a dropoff folder on a network drive and the SIP appears automatically in /home/demo/ingestdropoff/. See Create a shared folder.
1.4 Receive SIP from Producer (UC-1.1)	Internal producer copies the SIP to /home/demo/ingest/2009_0001/; for external producer, archivist copies the SIP to that folder. If SIP includes checksum, go to 1.6 - Check integrity of transfered SIP If SIP does not include checksum, go to 1.5 - Generate Checksum	Does 1 accession = 1 SIP or multiple SIPs? And does 1 SIP = 1 AIP or multiple AIPs? See Discussion: what constitutes an AIP? For the purposes of testing I am determining that 1 accession = 1 SIP = 1 AIP. What is the purpose of the checksum at this stage? If the purpose is only to identify random file corruption that may have occured during the transfer, this may already be handled by CRC routines in the O/S and transfer protocols.
1.5 Generate checksum	Select all folders and files in the SIP, right-click and select scripts > makeMD5 and save report to the folder containing the SIP. Go to 1.8 - Send Confirmation of receipt to Producer
1.6 Check integrity of transferred SIP (UC-1.2)	Select checksum report, right-click on report, select scripts > checkMD5. Review report to ensure all checksums match (go to the bottom of the report; it should say "All files are OK!"). If the integrity check fails, go to 1.7 - Request resubmission of SIP. '(note - failure of the integrity check is regarded as an exception to the expected procedure)' If the integrity check passes, go to 1.8 - Send confirmation of receipt to Producer
1.7 Request resubmission of SIP	Request resubmission of SIP from Producer Go to 1.4 - Receive SIP from Producer
1.8 Send confirmation of receipt to Producer (UC-1.2)	Send e-mail to internal producer; send e-mail or other confirmation to external producer. Save copy of confirmation to the accession documentation folder Go to 2.1 - Quarantine SIP

AD2 Audit SIP

File:Archivematica AD2 AuditSIP v5.pdf

Expected Procedure

• [2.1] --> [2.2] --> [2.4] --> [2.5] --> [2.8] --> [end]

Exceptions

• malware detected at step 2.3
• SIP not compliant at step 2.5

Step	Implementation	Notes
General procedures
2.1 Quarantine SIP	Leave SIP in current location (/home/demo/ingest/2009_0001). Wait for quarantine period to expire. Go to: 2.2 - Check SIP for malware	In Archivematica 4: Force 0000 permissions on folder Cron to crawl(w/ clam) quarantine folder for malware A background CRC may be acceptable in lieu of an MD5 or other cryptographic hash to verify post-transfer integrity For information on how National Archives of Australia manages the quarantine process, see http://www.naa.gov.au/records-management/secure-and-store/e-preservation/at-NAA/process.aspx
2.2 Check SIP for malware	NOTE: There is no malware checking software included in the Archivimatica 0.3 release, so steps 2.2 and 2.3 are hypothetical only. Check SIP for the presence of malware Create malware check report, copy report to the Accession Documentation folder If malware is detected, go to: 2.3 - Remove malware If malware is not detected, go to: 2.4 - Audit SIP for compliance	Documentation should include: list of software used to detect malware related virus/malware definition used to perform the check date and time of check reports generated by the software identifying infected files, nature of the infection "Because of the age of the digital components contained in the SIP, the Archive must ensure that malicious code check can recognize very old malicious code." Tufts/Yale Ingest Guide For University Electronic Records, B2.3 (http://dca.lib.tufts.edu/features/nhprc/reports/ingest/part_B-02-03.html)
2.3 Remove malware	Attempt to remove malware from the SIP Document the tools and procedures used to remove the malware Document success or failure of the malware removal for each infected file Copy all malware removal report to the Accession Documentation folder Go to: 2.4 - Audit SIP for compliance	Case: malware not removed Create a plain text report (click on applications > accessories > text editor) describing the type(s) of malware, the efforts made to remove the malware and the reasons for failing to remove it. Save the report to /home/demo/accessions/2009_01. Presumably any malware removal tools generate reports on success/failure. Depending on the software used, the detection occurring in step 2.2 and the removal occurring in this step may be documented in the same report
2.4 Audit SIP for compliance (UC-1.3; UC-4.6)	Manually verify that the SIP conforms to the archives' data formatting and documentation standards and meets the specifications of the Submission Agreement. Do this by skimming the filenames and extensions to make sure that what was supposed to be in the SIP according to the Submission Agreement is actually there. Create audit documentation Create an audit documentation report as plain text file (click on applications > accessories > text editor) If the SIP is wholly compliant, note this in the audit report; else Document deficiencies in the SIP, identifying the nature of the deficiency (missing file extensions, unacceptable formats, unacceptable packaging, presence of unremoved malware, etc.) and the object(s) the deficiency pertains to (file or group of files, SIP packaging, etc.) Save the SIP audit documentation to the Accession documentation folder (e.g., /home/demo/accessions/2009_0001/) Go to: 2.5 - Assess SIP deficiencies	What else should we check for? Maybe develop a checklist approach to reporting on deficiencies - for example: "contains unacceptable formats YES; records inadequately identified YES..."
2.5 - Assess SIP deficiencies	Based on the results of the audit performed in step 2.4 - Audit SIP for compliance, determine if the deficiencies identified, if any, warrant rejection of the SIP, or if the SIP can be accepted despite identified deficiencies. If the majority of objects conform to standards and Submission Agreement, the SIP may be considered acceptable. Note that the non-conforming objects can be deleted after appraisal (see AD3 step 3.9). Document the decision to accept or reject the SIP If SIP can be accepted for ingest, Document that SIP has been acepted for ingest go to: 2.8 - Notify producer of SIP acceptance If SIP can not be accepted for ingest, go to: 2.7 - Notify Producer of SIP rejection
2.6 Notify Producer of SIP rejection	Document that the SIP has been rejected, copy the report to the Accession Documentation folder Send an e-mail notification, attaching a copy of the reports created in steps 2.4 or 2.5. If the Producer appeals the SIP rejection: Document receipt of the appeal go to: 2.7 - Evaluate appeals If the Producer does not appeal the SIP rejection: go to: 2.9 - Destroy SIP copies
2.7 Evaluate appeals	Evaluate the Producer's appeal of the SIP rejection If the appeal is accepted: Document acceptance of the appeal Go to: 2.8 - Notify producer of SIP acceptance If the appeal is rejected: Document the rejection of the appeal Notify the Producer of the appeal rejection Go to 2.9 - Destroy SIP copies
2.8 Notify producer of SIP acceptance	Send Producer notification that the SIP has been accepted for ingest Document Go to: 3.1 - Extract content information from SIP
2.9 Destroy SIP copies	Delete SIP from /home/demo/dropoff and /home/demo/ingest. End ingest

AD3 Accept SIP for Ingest

File:Archivematica AD3 AcceptSIPforIngest v4.pdf

Expected Procedure

[3.1] --> [3.2] --> [3.3] --> [3.4] --> [3.5] --> [3.6] --> [3.9] --> [3.10]

Exceptions

• Producer notification required following [3.6]
• Appraisal decision appealed following [3.7]

Step	Implementation	Notes
General procedures
3.1 Unpack SIP (UC-1.3)	Create folders for the SIP contents Create the following 2 directories within the current folder (e.g., /home/demo/ingest/2009_01): /content /PDI Unpack the SIP If the SIP uses an archive file format such as .zip, .tar, etc.., extract the contents using the appropriate unpacking software. Identify the content objects in the SIP and sort them into the /content directory Identify the PDI and sort them into the /PDI directory Go to: 3.2 - Modify/Provide additional PDI	In future versions of archivimatica, xml metadata from TRIM the export file will be extracted and stored in the database. However, the export file may also be placed into the PDI folder.
3.2 Modify / provide additional PDI (UC-1.3)	Go to: 3.3 - Identify Formats
3.3 Identify format	Use DROID and NLNZ Metadata extractor to identify objects in the SIP. Save the reports to /home/demo/ingest/2009_01/PDI/. Go to: 3.4 - Validate formats
3.4 Validate format	Use JHOVE to validate the objects in the SIP. Save the report to /home/demo/ingest/2009_01/PDI/. Go to: 3.5 - Extract metadata
3.5 Extract metadata	Extract preservation metadata from content objects in the SIP Go to: - 3.6 Audit submission and select for preservation	This step seems to have been accomplished in steps 3.4 and 3.6. In archivematica, the metadata extraction activity takes place at the same time as format identification, as the NLNZ Metadata extractor tool also performs format identification. What is the necessary metadata that must be extracted at this point?
3.6 Audit submission and select for preservation (UC-4.6)	Based on the results of steps 3.2, 3.3, and 3.4, apply Archives policies and determine which (if any) content objects in the SIP should not be included in the AIP Document which content objects will not be included and why. If all SIP content is to be included in the AIP, go to: 3.10 - Accept selected SIP components for ingest If some SIP content is to be excluded from the AIP, and submission agreement requires notifying Producer of appraisal decision, go to: 3.7 - Notify Producer of appraisal decision Else, go to: 3.9 - Destroy unselected SIP components	Possible reasons for exclusion: technical insufficient preservation metadata unrecognized format unsupported format invalid format appraisal duplicate content does file level selection occur at this point (e.g. methodological sampling of records at the file level for selective retention?) or before the SIPs get to this point? further to the above, this diagram only deals with one sip at a time. How do we manage appraisal decisions that must take into consideration the relationships among many SIPs?
3.7 Notify Producer about appraisal decision(s)	Provide the Producer with copies of the appraisal report or other documentation as required by the submisssion agreement identifying SIP components to be destroyed. If the Producer appeals the appraisal decision, go to: 3.8 - Evaluate appeals Else, go to: 3.9 - Destroy unselected SIP components
3.8 Evaluate appeals	Receive Producer's appeals Evaluate appeals Based on the evaluation, make any necessary amendments to the appraisal decision Notify the Producer of the outcome of the appeal process Go to: 3.9 - Destroy unselected SIP components
3.9 - Destroy unselected SIP components	Destroy all SIP content objects identified for destruction in the appraisal decision (or amended appraisal decision as appropriate). Document destruction	Note - it is possible that no components have been identified for destruction. In this case, move on to step 3.10 Do we need to destroy accompanying metadata as well?
3.10 Accept selected SIP components for ingest

AD4 Generate AIP

File:Archivematica AD4 GenerateAIP v2.pdf

Step	Implementation	Notes
4.1 Create AIP containers		If 1 SIP = 1 AIP this step is not necessary. It is only necessary if the SIP is being divided into multiple AIPs.
4.2 Add Content Information to AIP		See note for step 4.1, above
4.3 Transform Content Information	In /demo/ingest/2009_01, create a folder entitled normalized. Using Xena, normalize the contents of each AIP. Set destination folder for the normalization to /home/demo/ingest/2009_01/normalized/. Set the destination folder for the Xena log file to /home/demo/ingest/2009_01/PDI/.
4.4 Add Transformed Content Information to AIP	See step 4.3, above.
4.5 Add PDI to AIP	Create a plain text report containing provenance and other PDI elements (including arrangement information) and save it to the AIP.	See "Creating and storing PDI" in Discussion: what constitutes an AIP?
4.6 Generate Descriptive Information (UC-1.4)	Obtain available descriptive information from PDI added to AIP, Submission Agreement and/or records schedule, communications with donor, etc. In Qubit, enter descriptive information at aggregate levels of description (fonds, series, file). Then create item-level descriptions for each object to be uploaded. Upload digital objects from /home/demo/ingest/2009_01/content/. In Qubit 1.0.7 batch upload of digital objects is not possible. The objects have to be uploaded 1 at a time. Add additional descriptive information to item-level descriptions. For instructions on using Qubit, go to the on-line user manual.

AD5 Transfer AIP to Archival Storage

File:Archivematica AD5 TransferAIPtoArchivalStorage v2.pdf

Step	Implementation	Notes
5.1 Request storage of AIP (UC-1.5)
5.2 Transfer AIP to Archival storage (UC-1.5)	Right click on folder /home/demo/ingest/2009_01. Click Scripts > BagIt. A bag will be created and stored automatically in /home/demo/mybags/. Copy AIP to /home/ingest/archivalstorage/.	In archivematica 0.4 maybe configure BagIt script to drop the bag directly into /home/demo/ingest/archivalstorage/. The BagIt script can also be edited manually if desired: open demo/.gnome2/nautilus-scripts/Bagit and change "mybags" to "archivalstorage".
5.3 Confirm receipt and storage of AIP (UC-1.5)
5.4 Add AIP storage location to descriptive information (UC-1.6)	In the physical storage area in Qubit, add the storage location.
5.5 Add Descriptive Information to Data Management (UC-1.6)	This was done in AD4, step 4.6. In OAIS, generating descriptive information and adding them to data management are two different steps; however, in Archivematica this is done in one step in Qubit, which is used to upload images to a web interface, generate derivatives for searching and browsing and record descriptive information.
5.6 Confirm update of Data Management
5.7 Destroy SIP and AIP copies	Destroy /home/demo/ingest/2009_01 and /home/demo/mybags/2009_01.