Difference between revisions of "Ingest (0.5)"

From Archivematica
Jump to navigation Jump to search
 
(31 intermediate revisions by 2 users not shown)
Line 3: Line 3:
  
  
−
===AD1 Receive SIP===  
+
===Setting up shared folders===
−
[[File:Archivematica_AD1_ReceiveSIP_v1.pdf|Archivematica UML Activity diagram AD1 Receive SIP]]
+
In order to work through all of the steps in the tables below, you will need to set up two shared folders in Archivematica.
−
 
+
*The purpose of shared folders is to allow you to place digital objects into a folder on your host machine and have the objects automatically appear in a folder in Archivematica, and vice versa.
 +
*The two folders in Archivematica which need to be set up as shared folders are /home/demo/ingestSIP and /home/demo/storeAIP.
 +
**/home/demo/ingestSIP is used to ingest SIPs from the host machine into Archivematica.
 +
**/home/demo/storeAIP is used to drop AIPs into a folder in Archivematica and have them appear back in the host machine.
 +
*Recommended names for the folders on the home machine are sendSIP and archivalstorage.
 +
*For instructions on setting up shared folders, please go to [[Virtual appliance instructions#Import_files_into_virtual_appliance_.28optional.29|Virtual appliance instructions]].
 +
*For testing purposes you can avoid setting up shared folders and simply use the test files found in /home/demo/testFiles/. However, you will not be moving SIPs into Archivematica or moving stored AIPs out of it.
 +
<br />
  
 +
===Activity diagram 1 Receive SIP===
 +
[[Media:Archivematica_AD1_ReceiveSIP_v1.pdf|Archivematica UML Activity diagram AD1 Receive SIP]]
  
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
Line 13: Line 22:
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:55%"|'''Description'''
 
!style="width:55%"|'''Description'''
−
!style="width:25%"|'''UML diagram references'''
+
!style="width:25%"|'''Activity diagram references'''
−
|-
 
 
|-
 
|-
 
|Producer places SIP in shared folder on host machine
 
|Producer places SIP in shared folder on host machine
 
|
 
|
−
*The purpose of shared folders is to allow the Producer to drop SIPs into a folder on their host machine or network and have the SIPs automatically appear in a folder in Archivematica, and vice versa.
+
*Place a folder of digital files into the shared ingest folder on the host machine.
−
**For instructions on setting up shared folders, please go to [[Virtual appliance instructions#Import_files_into_virtual_appliance_.28optional.29|Virtual appliance instructions]].
+
*Note that the SIP does not need to be prepared in any way prior to ingest - i.e. you do not need to prepare it as a METS file or otherwise process the SIP. A simple folder with one or more files in it is fine.
−
**Neither the folder name nor any of the filenames should include spaces or special characters. Underscores are ok.
+
|
−
**Make the shared folder in Archivematica /home/demoreceiveSIP/.
 
−
*For testing purposes you can avoid setting up shared folders and simply use the test files found in /home/demo/testFiles/.
 
−
|1.3 assign network storage for SIP (UC-1.1)
 
 
|-
 
|-
 
|SIP appears in shared folder in Archivematica
 
|SIP appears in shared folder in Archivematica
 
|
 
|
−
*SIP will appear in /home/demo/receiveSIP/.
+
*SIP will appear in /home/demo/ingestSIP/.
−
*If using files from /home/demo/testFiles/, copy a folder from /testFiles/ into /home/demo/receiveSIP/.
+
*To navigate to this folder, click Places > Home folder.
 
|1.4 Receive SIP from Producer (UC-1.1)
 
|1.4 Receive SIP from Producer (UC-1.1)
 
|-
 
|-
−
|Archivist copies SIP from shared folder to SIP backup folder
+
|Archivist copies SIP from shared folder to SIP receipt folder
 
|
 
|
−
*Copy the SIP in /home/demo/receiveSIP/ and paste it into /home/demo/receiveSIPbackup/. If anything goes wrong during the ingest process, this backup copy can be retrieved and processed.
+
*Copy SIP from /home/demo/ingestSIP/ to /home/demo/receiveSIP/.
 +
*/home/demo/ingestSIP acts as a backup SIP copy. If anything goes wrong during the ingest process, this backup copy can be retrieved and processed.
 
|
 
|
 
|-
 
|-
−
|}
+
|}<br />
−
 
 
−
===AD2 Audit SIP===
 
−
[[File:Archivematica_AD2_AuditSIP_v5.pdf|Archivematica UML Activity diagram AD2 Audit SIP]]
 
−
 
 
  
 +
===Activity diagram 2 Audit SIP===
 +
[[Media:Archivematica_AD2_AuditSIP_v5.pdf|Archivematica UML Activity diagram AD2 Audit SIP]]
  
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
Line 48: Line 52:
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:55%"|'''Description'''
 
!style="width:55%"|'''Description'''
−
!style="width:25%"|'''UML diagram references'''
+
!style="width:25%"|'''Activity diagram references'''
 
|-
 
|-
−
|Archivist moves SIP from shared folder into quarantine
+
|Archivist moves SIP from SIP receipt folder into quarantine
 
|
 
|
−
*Drag the SIP from /home/demo/receiveSIP/ to /home/demo/quarantine.
+
*Drag the SIP from /home/demo/receiveSIP/ and drop it into /home/demo/quarantine.
 +
*Note that you must drag and drop, not copy and paste, in order to trigger the quarantine process.
 
|2.1 Quarantine SIP
 
|2.1 Quarantine SIP
 
|-
 
|-
Line 70: Line 75:
 
*Infected files will appear in /home/demo/possiblevirii/. If this occurs, do not take any further steps in the ingest process. Inform the Producer that infected files have been found. It is recommended at this point to delete all SIP copies and request that the Producer take steps to review the causes of the problem and eventually resubmit a malware-free SIP.
 
*Infected files will appear in /home/demo/possiblevirii/. If this occurs, do not take any further steps in the ingest process. Inform the Producer that infected files have been found. It is recommended at this point to delete all SIP copies and request that the Producer take steps to review the causes of the problem and eventually resubmit a malware-free SIP.
 
|
 
|
−
2.4 Audit SIP for compliance
+
2.4 Audit SIP for compliance<br />
−
2.5 Assess SIP defiencies
+
2.5 Assess SIP defiencies<br />
−
2.6 Notify Producer of SIP rejection
+
2.6 Notify Producer of SIP rejection<br />
 
2.8 Destroy SIP copies
 
2.8 Destroy SIP copies
 
|-
 
|-
Line 80: Line 85:
 
|2.4 Audit SIP for compliance
 
|2.4 Audit SIP for compliance
 
|-
 
|-
−
|}
+
|}<br />
−
 
 
−
===AD3 Accept SIP for Ingest===
 
−
[[File:Archivematica_AD3_AcceptSIPforIngest_v4.pdf|Archivematica UML Activity diagram AD3 Accept SIP for Ingest]]
 
−
 
 
  
 +
===Activity diagram 3 Accept SIP for Ingest===
 +
[[Media:Archivematica_AD3_AcceptSIPforIngest_v4.pdf|Archivematica UML Activity diagram AD3 Accept SIP for Ingest]]
  
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
Line 92: Line 95:
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:55%"|'''Description'''
 
!style="width:55%"|'''Description'''
−
!style="width:25%"|'''UML diagram references'''
+
!style="width:25%"|'''Activity diagram references'''
 
|-
 
|-
  
Line 100: Line 103:
 
*FITS incorporates format identification and validation tools such as DROID. JHOVE and the New Zealand Metadata Extractor, comparing the results of each tool and extracting a set of identification, validation and technical metadata. For more information on the FITS tool, see [http://code.google.com/p/fits/ http://code.google.com/p/fits/]
 
*FITS incorporates format identification and validation tools such as DROID. JHOVE and the New Zealand Metadata Extractor, comparing the results of each tool and extracting a set of identification, validation and technical metadata. For more information on the FITS tool, see [http://code.google.com/p/fits/ http://code.google.com/p/fits/]
 
|
 
|
−
3.3 Identify formats
+
3.3 Identify formats (UC-1.2, step 3)<br />
−
3.4 Validate formats
+
3.4 Validate formats (UC-1.2, step 3)<br />
−
3.5 Extract metadata
+
3.5 Extract metadata (UC-1.2, step 3)
 
|-
 
|-
 
|Identification/validation reports are sent to accessions folder
 
|Identification/validation reports are sent to accessions folder
Line 109: Line 112:
 
*Note that each report contains an MD5 checksum for the file.
 
*Note that each report contains an MD5 checksum for the file.
 
|
 
|
−
3.3 Identify formats
+
3.3 Identify formats (UC-1.2, step 3)<br />
−
3.4 Validate formats
+
3.4 Validate formats (UC-1.2, step 3)<br />
−
3.5 Extract metadata
+
3.5 Extract metadata (UC-1.2, step 3)
 
|-
 
|-
 
|Accession log is sent to accessions folder
 
|Accession log is sent to accessions folder
Line 120: Line 123:
 
|-
 
|-
  
−
|}
+
|}<br />
−
 
 
−
===AD4 Generate AIP===
 
−
[[File:Archivematica_AD4_GenerateAIP_v2.pdf|Archivematica UML Activity diagram AD4 Generate AIP]]
 
  
 +
===Activity diagram 4 Generate AIP===
 +
[[Media:Archivematica_AD4_GenerateAIP_v2.pdf|Archivematica UML Activity diagram AD4 Generate AIP]]
  
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
Line 131: Line 133:
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:55%"|'''Description'''
 
!style="width:55%"|'''Description'''
−
!style="width:25%"|'''UML diagram references'''
+
!style="width:25%"|'''Activity diagram references'''
 
|-
 
|-
 
|SIP is moved to AIP preparation folder
 
|SIP is moved to AIP preparation folder
Line 150: Line 152:
 
*Click OK to close pop-up window
 
*Click OK to close pop-up window
 
*Close Xena
 
*Close Xena
−
|4.3 Transform content information
+
|4.3 Transform content information (UC-1.3, step 9)
 
|-
 
|-
 
|Normalized files are saved to AIP preparation folder
 
|Normalized files are saved to AIP preparation folder
Line 156: Line 158:
 
*In the SIP, look for files with the extension .xena. These are normalized versions of the original files.
 
*In the SIP, look for files with the extension .xena. These are normalized versions of the original files.
 
*To view representations of normalized files, open the Xena Viewer from Archivematica's Linux desktop.
 
*To view representations of normalized files, open the Xena Viewer from Archivematica's Linux desktop.
−
|4.3 Transform content information
+
|4.3 Transform content information (UC-1.3, step 9)
 
|-
 
|-
 
|Normalization log is saved to accessions folder
 
|Normalization log is saved to accessions folder
 
|
 
|
 
*A log file showing all the actions taken by Xena will appear: /home/demo/accessionreports/xena_log.0.
 
*A log file showing all the actions taken by Xena will appear: /home/demo/accessionreports/xena_log.0.
−
|4.3 Transform content information
+
|
 
|-
 
|-
−
|Archivist moves PDI in accessions folder to SIP in AIP preparation folder
+
|Archivist moves PDI from accessions folder to SIP in AIP preparation folder
 
|
 
|
 
*In Archivematica, all the contents relating to the SIP in /home/demo/accessionreports/ is considered PDI (Preservation Description Information).
 
*In Archivematica, all the contents relating to the SIP in /home/demo/accessionreports/ is considered PDI (Preservation Description Information).
−
**Copy these contents to /home/demo/prepareAIP/[SIP]
+
**Cut these contents and paste them to /home/demo/prepareAIP/[SIP].
 
|4.5 Add PDI to AIP
 
|4.5 Add PDI to AIP
 
|-
 
|-
Line 172: Line 174:
 
|
 
|
 
*Drag the SIP from /home/demo/prepareAIP/ and drop it into /home/demo/generateAIP/.
 
*Drag the SIP from /home/demo/prepareAIP/ and drop it into /home/demo/generateAIP/.
 +
*Note that you must drag and drop, not copy and paste, in order to trigger the AIP generation process.
 
|
 
|
 
|-
 
|-
Line 177: Line 180:
 
|
 
|
 
*A script called BagIt will run in the background, converting the SIP into a single zipped file. In Archivematica, this zipped file, which also includes metadata generated by the bagging process, constitutes the AIP. For more information about BagIT, see [http://www.digitalpreservation.gov/library/challenge/data-transfer.html http://www.digitalpreservation.gov/library/challenge/data-transfer.html].
 
*A script called BagIt will run in the background, converting the SIP into a single zipped file. In Archivematica, this zipped file, which also includes metadata generated by the bagging process, constitutes the AIP. For more information about BagIT, see [http://www.digitalpreservation.gov/library/challenge/data-transfer.html http://www.digitalpreservation.gov/library/challenge/data-transfer.html].
−
|
+
|UC-1.3, step 10
 
|-
 
|-
 
|AIP is moved to AIP receipt folder
 
|AIP is moved to AIP receipt folder
Line 185: Line 188:
 
|
 
|
 
|-
 
|-
−
|}
+
|}<br />
−
 
 
−
===AD5 Transfer AIP to Archival Storage===
 
−
[[File:Archivematica_AD5_TransferAIPtoArchivalStorage_v2.pdf|Archivematica UML Activity diagram AD5 Transfer AIP to Archival Storage]]
 
  
 +
===Activity diagram 5 Transfer AIP to Archival Storage===
 +
[[Media:Archivematica_AD5_TransferAIPtoArchivalStorage_v3.pdf|Archivematica UML Activity diagram AD5 Transfer AIP to Archival Storage]]
  
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
 
{| border="1" cellpadding="10" cellspacing="0" width=90%
Line 196: Line 198:
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:20%"|'''Workflow diagram step'''
 
!style="width:55%"|'''Description'''
 
!style="width:55%"|'''Description'''
−
!style="width:25%"|'''UML diagram references'''
+
!style="width:25%"|'''Activity diagram references'''
 
|-
 
|-
−
|Archivist moves AIP to archival storage folder
+
|Archivist copies AIP to archival storage folder
 
|
 
|
−
*Drag the AIP from /home/demo/receiveAIP/ and drop it into /home/demo/storeAIP/.
+
*Copy the AIP from /home/demo/receiveAIP/ to /home/demo/storeAIP/.
−
*The folder /home/demo/storeAIP/ should be set up as a shared folder. See [[Archival Storage (0.5)]].
+
|5.2 Transfer AIP to archival storage (UC-1.5)
−
|5.2 Transfer AIP to archival storage
 
 
|-
 
|-
−
|Archivist deletes SIP from SIP backup folder
+
|}<br />
−
|
+
 
−
*Delete the SIP from /home/demo/receiveSIPbackup/.
+
Go to [[Archival Storage (0.5)]]
−
|5.7 Destroy SIP and AIP copies
 
−
|-
 
−
|}
 

Latest revision as of 16:21, 2 August 2012

Main Page > Documentation > Release 0.5 Documentation > Ingest (0.5)


Setting up shared folders[edit]

In order to work through all of the steps in the tables below, you will need to set up two shared folders in Archivematica.

  • The purpose of shared folders is to allow you to place digital objects into a folder on your host machine and have the objects automatically appear in a folder in Archivematica, and vice versa.
  • The two folders in Archivematica which need to be set up as shared folders are /home/demo/ingestSIP and /home/demo/storeAIP.
    • /home/demo/ingestSIP is used to ingest SIPs from the host machine into Archivematica.
    • /home/demo/storeAIP is used to drop AIPs into a folder in Archivematica and have them appear back in the host machine.
  • Recommended names for the folders on the home machine are sendSIP and archivalstorage.
  • For instructions on setting up shared folders, please go to Virtual appliance instructions.
  • For testing purposes you can avoid setting up shared folders and simply use the test files found in /home/demo/testFiles/. However, you will not be moving SIPs into Archivematica or moving stored AIPs out of it.


Activity diagram 1 Receive SIP[edit]

Archivematica UML Activity diagram AD1 Receive SIP

Workflow diagram step Description Activity diagram references
Producer places SIP in shared folder on host machine
  • Place a folder of digital files into the shared ingest folder on the host machine.
  • Note that the SIP does not need to be prepared in any way prior to ingest - i.e. you do not need to prepare it as a METS file or otherwise process the SIP. A simple folder with one or more files in it is fine.
SIP appears in shared folder in Archivematica
  • SIP will appear in /home/demo/ingestSIP/.
  • To navigate to this folder, click Places > Home folder.
 1.4 Receive SIP from Producer (UC-1.1)
Archivist copies SIP from shared folder to SIP receipt folder
  • Copy SIP from /home/demo/ingestSIP/ to /home/demo/receiveSIP/.
  • /home/demo/ingestSIP acts as a backup SIP copy. If anything goes wrong during the ingest process, this backup copy can be retrieved and processed.


Activity diagram 2 Audit SIP[edit]

Archivematica UML Activity diagram AD2 Audit SIP

Workflow diagram step Description Activity diagram references
Archivist moves SIP from SIP receipt folder into quarantine
  • Drag the SIP from /home/demo/receiveSIP/ and drop it into /home/demo/quarantine.
  • Note that you must drag and drop, not copy and paste, in order to trigger the quarantine process.
 2.1 Quarantine SIP
SIP is quarantined for 2 minutes
  • In a production system, SIPs would normally be quarantined for a set period of time (for example, four weeks), to allow anti-virus software to be updated with the latest virus profiles.
  • A lock should appear on the SIP folder in quarantine. The archivist will not be able to read or modify the files during this time.
 2.1 Quarantine SIP
SIP is scanned for malware
  • At the end of the quarantine period, ClamAV will automatically scan the files for viruses and other malware.
 2.2 Check SIP for malware
Infected files are sent to possiblevirii folder
  • Infected files will appear in /home/demo/possiblevirii/. If this occurs, do not take any further steps in the ingest process. Inform the Producer that infected files have been found. It is recommended at this point to delete all SIP copies and request that the Producer take steps to review the causes of the problem and eventually resubmit a malware-free SIP.

2.4 Audit SIP for compliance
2.5 Assess SIP defiencies
2.6 Notify Producer of SIP rejection
2.8 Destroy SIP copies

Virus-checker report is sent to accessions folder
  • A report on ClamAV's virus scan will appear automatically: home/demo/accessionreports/virus.log.
 2.4 Audit SIP for compliance


Activity diagram 3 Accept SIP for Ingest[edit]

Archivematica UML Activity diagram AD3 Accept SIP for Ingest

Workflow diagram step Description Activity diagram references
SIP contents are identified and validated using FITS
  • FITS (File Information Tool Set) is automatically launched once the quarantine period has ended and the files have been scanned for viruses.
  • FITS incorporates format identification and validation tools such as DROID. JHOVE and the New Zealand Metadata Extractor, comparing the results of each tool and extracting a set of identification, validation and technical metadata. For more information on the FITS tool, see http://code.google.com/p/fits/

3.3 Identify formats (UC-1.2, step 3)
3.4 Validate formats (UC-1.2, step 3)
3.5 Extract metadata (UC-1.2, step 3)

Identification/validation reports are sent to accessions folder
  • The FITS report will appear in /home/demo/accessionreports/. The report appears as a folder with a 10-digit number; inside the folder is a report for each file in the SIP.
  • Note that each report contains an MD5 checksum for the file.

3.3 Identify formats (UC-1.2, step 3)
3.4 Validate formats (UC-1.2, step 3)
3.5 Extract metadata (UC-1.2, step 3)

Accession log is sent to accessions folder
  • A report on the accession process will appear automatically: home/demo/accessionreports/accession.log.
  • For each file in the SIP, the accession log will state "Accession of /tmp/accession-[FITS folder number]/[SIP number]/filename] completed successfully."


Activity diagram 4 Generate AIP[edit]

Archivematica UML Activity diagram AD4 Generate AIP

Workflow diagram step Description Activity diagram references
SIP is moved to AIP preparation folder
  • At the end of the quarantine process, Archivematica automatically drops the SIP into /home/demo/prepareAIP
 4.2 Add content information to AIP
Archivist normalizes files
  • From Archivematica's Linux desktop, open Xena
  • Click Add Directory
  • Select /home/demo/prepareAIP/[SIP]/
  • In Tools > Xena 4.2.1 Preferences > Xena destination directory enter /home/demo/prepareAIP/[SIP].
  • In Tools > Xena 4.2.1 Preferences > Xena log file enter /home/demo/accessionreports/xena_log.
  • Click OK to close Xena 4.2.1 Preferences
  • Click Normalise
  • Wait for normalization process to be completed (a pop-up dialogue box will open indicating that the process has been completed).
  • Click OK to close pop-up window
  • Close Xena
 4.3 Transform content information (UC-1.3, step 9)
Normalized files are saved to AIP preparation folder
  • In the SIP, look for files with the extension .xena. These are normalized versions of the original files.
  • To view representations of normalized files, open the Xena Viewer from Archivematica's Linux desktop.
 4.3 Transform content information (UC-1.3, step 9)
Normalization log is saved to accessions folder
  • A log file showing all the actions taken by Xena will appear: /home/demo/accessionreports/xena_log.0.
Archivist moves PDI from accessions folder to SIP in AIP preparation folder
  • In Archivematica, all the contents relating to the SIP in /home/demo/accessionreports/ is considered PDI (Preservation Description Information).
    • Cut these contents and paste them to /home/demo/prepareAIP/[SIP].
 4.5 Add PDI to AIP
Archivist moves SIP to AIP generation folder
  • Drag the SIP from /home/demo/prepareAIP/ and drop it into /home/demo/generateAIP/.
  • Note that you must drag and drop, not copy and paste, in order to trigger the AIP generation process.
SIP content and PDI are zipped into AIP UC-1.3, step 10
AIP is moved to AIP receipt folder
  • The bagging process automatically moves the AIP to /home/demo/receiveAIP/.
    • To view the AIP, double-click it. When it opens in a separate window, double-click it again; this will allow you to view (but not modify or delete) the contents of the zipped bag.


Activity diagram 5 Transfer AIP to Archival Storage[edit]

Archivematica UML Activity diagram AD5 Transfer AIP to Archival Storage

Workflow diagram step Description Activity diagram references
Archivist copies AIP to archival storage folder
  • Copy the AIP from /home/demo/receiveAIP/ to /home/demo/storeAIP/.
 5.2 Transfer AIP to archival storage (UC-1.5)


Go to Archival Storage (0.5)

Retrieved from "https://wiki.archivematica.org/index.php?title=Ingest_(0.5)&oldid=6142"