Difference between revisions of "UM ingest"

From Archivematica
Jump to navigation Jump to search
Line 1: Line 1:
[[Main Page]] > [[Documentation]] > [[User Manual|User manual]] > [[User manual 0.8]] > Ingest
+
[[Main Page]] > [[Documentation]] > [[User Manual|User manual]] > [[User manual 0.9]] > Ingest
  
 
<div class="clearfix">
 
<div class="clearfix">
Line 9: Line 9:
 
Should you run into an error during ingest, please see [[UM error handling|Error handling]].
 
Should you run into an error during ingest, please see [[UM error handling|Error handling]].
  
== Create one or more SIP(s) ==
+
== Create SIP ==
  
#Process one or more transfers as described in [[UM transfer|Transfer]]. [[Image:ingest1g.png|600px|right|thumb|'''Figure 1'''  Select an option from the Create SIP(s) Actions drop-down menu]]
+
#Process transfers as described in [[UM transfer|Transfer]]. [[Image:ingest1g.png|600px|right|thumb|'''Figure 1'''  Select an option from the Create SIP(s) Actions drop-down menu]]
 
#While still viewing the transfer tab you should see a bell icon next to the transfer. The micro-service related to the bell is "Create SIP(s)".
 
#While still viewing the transfer tab you should see a bell icon next to the transfer. The micro-service related to the bell is "Create SIP(s)".
#Choose one of three options for creating a SIP (or reject the transfer if desired, to stop all processing and move the transfer to the rejected folder in the file browser) ('''figure 1'''). These options are described below.
 
  
</div>
+
#In the Action dropdown menu, choose "Create single SIP and continue processing" (or reject the transfer if desired, to stop all processing and move the transfer to the rejected folder in the file browser) (Note that Create SIP(s) manually is not currently functional. However, the ability to create one or more SIP(s) from one or more transfer(s) will return in Archivematica 1.0 with improved functionality in the web browser. See MOCKUPS PAGE LINK) ('''figure 1''').
  
<div class="clearfix">
 
  
</div>
+
#Click on the Ingest tab.
  
=== Option 1: Create single SIP and continue processing ===
+
#The single SIP will move through a number of micro-services. Wait until it reaches "Normalize" and a bell icon appears ('''figure 2''').
This will create a single SIP from the transfer and move it through the initial micro-services to the normalization step.[[Image:ingest3g.png|600px|right|thumb|'''Figure 2''' A SIP that has been run through ingest processing until the normalization step]]
 
  
#Once you have selected "Create single SIP and continue processing", click on the Ingest tab.
 
#The SIP will be running through a number of micro-services. Wait until it reaches "Normalize" and a bell icon appears ('''figure 2''').
 
 
#To add descriptive metadata, see Add metadata, below.
 
#To add descriptive metadata, see Add metadata, below.
 +
 +
#To add PREMIS rights information, see Add PREMIS, below.
 +
 
#For selecting a normalization option, see Normalization, below.
 
#For selecting a normalization option, see Normalization, below.
  
Line 35: Line 33:
 
</div>
 
</div>
  
=== Option 2: Create single SIP ===
 
This will create a single SIP from the transfer and allow the user to re-organize or delete digital objects if desired. This is useful if there has been no appraisal on the objects prior to processing in Archivematica. Note that the original order of the transfer is maintained in the transfer METS file, a copy of which is automatically added to the SIP.[[Image:singleSIPg.png|600px|right|thumb|'''Figure 3'''  In the file browser, navigate to your SIP in SIPsUnderConstruction]]
 
  
#Once you have selected "Create single SIP", click on the Ingest tab.
+
== Add metadata ==
#The SIP will appear in the dashboard with a bell icon next to it. In the file browser, open ''SIPsUnderConstruction'' using the shortcut in the left panel. Your SIP will be in this directory ('''figure 3''').
+
You must add metadata prior to the normalization step. Any metadata added after normalization will not be included in the AIP package. See [http://www.ica-atom.org/doc/Dublin_Core ICA-AtoM (Qubit) Dublin Core] for information about the Dublin Core elements available. See [http://ica-atom.org/doc/Glossary#Rights_record ICA-AtoM (Qubit) Rights Entity] for information about the rights elements available.
#Open the SIP, navigate to the ''objects'' directory and delete and re-organize the objects as desired.
+
 
#Return to the dashboard. Add descriptive metadata if desired (see Add metadata, below). In the Actions drop-down menu select "SIP creation complete" ('''figure 4''').
+
#Click on the magnifying glass icon. ('''figure 9''').[[Image:ingest2g.png|600px|right|thumb|'''Figure 9''' To add metadata, click on the magnifying glass icon]]
#Wait until the SIP reaches "Normalize" and a bell icon appears (see '''figure 2''', above).
+
#This will take you to the SIP detail panel. On the left-hand side, under metadata click Add ('''figure 10''').
#If you haven't already done so, you can add descriptive metadata at this point. See Add metadata, below.
+
#Add metadata as desired ('''figure 11''') and save it by clicking the Save button at the bottom of the screen. Note that in Archivematica 0.9 you can only add metadata at the SIP level. If you would like to add metadata to a digital object, you will need to do that once the object has been uploaded to your access system.
#For selecting a normalization option, see Normalization, below. [[Image:SIPCreationCompleteg.png|600px|right|thumb|'''Figure 4''' In the Actions drop-down menu, select "SIP creation complete"]]
+
#When you click Save, you will see the metadata entry in the list page ('''figure 12'''). To edit it further, click Edit on the right-hand side.
 +
#Return to the ingest tab to continue processing the SIP.[[Image:metadata1g.png|600px|right|thumb|'''Figure 10''' The SIP detail panel: click Add under Metadata to enter Dublin Core metadata for the SIP]][[Image:metadata2g.png|600px|right|thumb|'''Figure 11'''  Using the data entry form to add descriptive metadata]] [[Image:metadata3g.png|600px|right|thumb|'''Figure 12'''  After saving your metadata]]
  
 
</div>
 
</div>
Line 52: Line 49:
 
</div>
 
</div>
  
=== Option 3: Manual SIP creation ===
+
== Add PREMIS ==
[[Image:manual1g.png|600px|right|thumb|'''Figure 5'''  Right-click and select "Create a structured directory"]]
 
This option allows the user to break one transfer into more than one SIP, or to create one or more SIPs from more than one transfer. The permitted relationships between transfers and SIPs in Archivematica are completely flexible:
 
*One transfer --> one SIP
 
*One transfer --> multiple SIPs
 
*Multiple transfers --> one SIP
 
*Multiple transfers --> multiple SIPs</br>
 
The user can also re-organize and delete objects within the SIP(s). Note that the original order of the transfer is maintained in the transfer METS file, a copy of which is automatically added to each SIP. </br>
 
#Once you have selected "Manual SIP creation", open the file browser and navigate to SIPsUnderConstruction using the shortcut in the left panel.
 
#Hover your cursor over the empty right-hand panel, right-click and select "Create a structured directory" ('''figure 5''').
 
#Follow the steps to create the directory and repeat the process until you have the desired number of SIPs ('''figure 6''').
 
#Navigate to ''completedTransfers'' using the shortcut in the left-hand panel. Your completed transfer will be in that directory ('''figure 7''').
 
#Open the transfer and cut and paste digital objects from its objects directory into the objects directories of the SIPs created in steps 1-3. Note that you must cut and paste, not copy and paste. [[Image:manual2g.png|600px|right|thumb|'''Figure 6'''  Two SIPs created from a single transfer]]This is to prevent you from accidentally placing the same digital object from a transfer into more than one SIP.
 
#Return to the dashboard and open the Ingest tab.
 
#Your SIPs will appear in the dashboard with bell icons next to them ('''figure 8''').
 
Add descriptive metadata to the SIPs if desired (see Add metadata, below). In the Actions drop-down menu select "SIP creation complete". You can process the SIPs one at a time or at the same time if you prefer.
 
#Wait until the SIPs reach "Normalize" and a bell icon appears next to each SIP.
 
#If you haven't already done so, you can add descriptive metadata at this point. See Add metadata, below.
 
#For selecting a normalization option, see Normalization, below. [[Image:manual3g.png|600px|right|thumb|'''Figure 7'''  The completed transfer in the ''completedTransfers'' directory]] [[Image:manual4g.png|600px|right|thumb|'''Figure 8'''  Two SIPs appear in the dashboard]]
 
 
 
</div>
 
  
<div class="clearfix">
+
You must add PREMIS rights prior to the normalization step. Any rights added after normalization will not be included in the AIP package or the DIP upload. See [http://www.ica-atom.org/doc/Dublin_Core ICA-AtoM (Qubit) Dublin Core] for information about the PREMIS rights bases and elements available.
  
</div>
+
#Click on the magnifying glass icon. ('''figure 9''').[[Image:ingest2g.png|600px|right|thumb|'''Figure 9'''  To add metadata, click on the magnifying glass icon]]
 +
#This will take you to the SIP detail panel. On the left-hand side, under rights click Add ('''figure 10''').
 +
#Add rights as desired ('''figure 11''') and save it by clicking the Save button at the bottom of the screen. Note that in Archivematica 0.9 you can only add metadata at the SIP level. If you would like to add rights to a digital object, you will need to do that once the object has been uploaded to your access system. (Note that rights entries are two pages of content. To get to the second page to complete data entry, click Next) (figure...)
  
== Add metadata ==
+
#When you click Save, you will see the rights entry in the list page ('''figure 12'''). To edit it further, click Edit on the right-hand side.
You must add metadata prior to the normalization step. Any metadata added after normalization will not be included in the AIP package. See [http://www.ica-atom.org/doc/Dublin_Core ICA-AtoM (Qubit) Dublin Core] for information about the Dublin Core elements available. See [http://ica-atom.org/doc/Glossary#Rights_record ICA-AtoM (Qubit) Rights Entity] for information about the rights elements available.
 
  
#Click anywhere on the grey background of a SIP ('''figure 9''').[[Image:ingest2g.png|600px|right|thumb|'''Figure 9'''  To add metadata, click anywhere on the grey background of the SIP]]
 
#This will take you to the SIP detail panel. On the left-hand side, under metadata click Add ('''figure 10''').
 
#Add metadata as desired ('''figure 11''') and save it by clicking the Save button at the bottom of the screen. Note that in Archivematica 0.8 you can only add metadata at the SIP level. If you would like to add metadata to a digital object, you will need to do that once the object has been uploaded to your access system.
 
#When you click Save, you will see the metadata entry in the list page ('''figure 12'''). To edit it further, click Edit on the right-hand side.
 
 
#Return to the ingest tab to continue processing the SIP.[[Image:metadata1g.png|600px|right|thumb|'''Figure 10'''  The SIP detail panel: click Add under Metadata to enter Dublin Core metadata for the SIP]][[Image:metadata2g.png|600px|right|thumb|'''Figure 11'''  Using the data entry form to add descriptive metadata]] [[Image:metadata3g.png|600px|right|thumb|'''Figure 12'''  After saving your metadata]]
 
#Return to the ingest tab to continue processing the SIP.[[Image:metadata1g.png|600px|right|thumb|'''Figure 10'''  The SIP detail panel: click Add under Metadata to enter Dublin Core metadata for the SIP]][[Image:metadata2g.png|600px|right|thumb|'''Figure 11'''  Using the data entry form to add descriptive metadata]] [[Image:metadata3g.png|600px|right|thumb|'''Figure 12'''  After saving your metadata]]
  
Line 96: Line 70:
  
 
Normalizing is the process of converting ingested digital objects to preservation and/or access formats. Note that the original objects are always kept along with their normalized versions. For more information about Archivematica's preservation strategy, go to the [[UM preservation planning|Preservation Planning]] section of the manual.
 
Normalizing is the process of converting ingested digital objects to preservation and/or access formats. Note that the original objects are always kept along with their normalized versions. For more information about Archivematica's preservation strategy, go to the [[UM preservation planning|Preservation Planning]] section of the manual.
 +
 
[[Image:normalize1g.png|600px|right|thumb|'''Figure 13''' Selecting a normalization option]]
 
[[Image:normalize1g.png|600px|right|thumb|'''Figure 13''' Selecting a normalization option]]
 +
 
#At the normalization step, the SIP will appear in the dashboard with a bell icon next to it. Select one of the normalization options from the Actions drop-down menu ('''figure 13'''):
 
#At the normalization step, the SIP will appear in the dashboard with a bell icon next to it. Select one of the normalization options from the Actions drop-down menu ('''figure 13'''):
#*Preservation and access: creates preservation copies of the objects plus access copies which will be used to generate the DIP.
+
 
#*Access only: no preservation copies are created. Creates access copies which will be used to generate the DIP.
+
#*Normalize for preservation and access: creates preservation copies of the objects plus access copies which will be used to generate the DIP.
#*Preservation only: creates preservation copies. No access copies are created and no DIP will be generated.
+
#*Normalize for access: no preservation copies are created. Creates access copies which will be used to generate the DIP.
#*No normalization: no preservation copies are created. No access copies are created and no DIP will be generated.
+
#*Normalize for preservation: creates preservation copies. No access copies are created and no DIP will be generated.
 +
#*Do not normalize: no preservation copies are created. No access copies are created and no DIP will be generated.
 +
#*You may also Reject SIP at this stage.
 +
 
 
#Once normalization is completed you can review the results in the normalization report. Click on the report icon next to the Actions drop-down menu ('''figure 14''').
 
#Once normalization is completed you can review the results in the normalization report. Click on the report icon next to the Actions drop-down menu ('''figure 14''').
 +
 
#*The report shows what has been normalized and what is already in an acceptable preservation and access format ('''figure 15'''). For example, the file LAND2.BMP was normalized successfully to both a preservation and an access format; the file oakland03.jp2 was found to be in an acceptable preservation format already but it was necessary to generate an access copy for it.  
 
#*The report shows what has been normalized and what is already in an acceptable preservation and access format ('''figure 15'''). For example, the file LAND2.BMP was normalized successfully to both a preservation and an access format; the file oakland03.jp2 was found to be in an acceptable preservation format already but it was necessary to generate an access copy for it.  
 +
 +
#You may review the normalization results in a new tab by clicking on Review. If your browser has the correct plug-ins to open a file, you may open it in another tab by clicking on it. If you click on a file and your browser cannot open it, it will download locally so you can view it using the appropriate software.
 +
 
#Approve normalization in the Actions drop-down menu to continue processing the SIP.[[Image:normalize2g.png|600px|right|thumb|'''Figure 14'''  Click on the report icon to open the normalization report]][[Image:normalize3g.png|600px|right|thumb|'''Figure 15'''  The normalization report shows what was normalized and what was already in an acceptable preservation or access format]]
 
#Approve normalization in the Actions drop-down menu to continue processing the SIP.[[Image:normalize2g.png|600px|right|thumb|'''Figure 14'''  Click on the report icon to open the normalization report]][[Image:normalize3g.png|600px|right|thumb|'''Figure 15'''  The normalization report shows what was normalized and what was already in an acceptable preservation or access format]]
 +
[[Image:normalize3g.png|600px|right|thumb|'''Figure 15'''  Review normalization results]]
 +
  
 
</div>
 
</div>
Line 113: Line 98:
  
 
== Store AIP ==
 
== Store AIP ==
#After normalization is approved, the SIP is run through a number of micro-services, including processing of the submission documentation, generation of the METS file, generation of the DIP and packaging of the AIP.
+
 
#When these micro-services are complete, the user has two approval steps remaining in the dashboard: upload DIP and store AIP.[[Image:storeAIP1g.png|600px|right|thumb|'''Figure 16'''  Moving the AIP into archival storage.]]
+
#After normalization is approved, the SIP is run through a number of micro-services, including processing of the submission documentation, generation of the METS file, indexing, generation of the DIP and packaging of the AIP.
#If desired, open the file browser, navigate to the ''StoreAIP'' directory using the shortcuts in the left-hand panel, and review the contents of the AIP. More information on Archivematica's AIP structure and the METS/PREMIS file is available on the Archivematica wiki: see [[AIP structure]] and [[METS]].
+
 
#Select "Store AIP" to move the AIP into archival storage ('''figure 16''').
+
#When these micro-services are complete, the user can upload DIP and store AIP.[[Image:storeAIP1g.png|600px|right|thumb|'''Figure 16'''  Moving the AIP into archival storage.]]
 +
 
 +
#If desired, review the contents of the AIP in another tab by clicking on Review. More information on Archivematica's AIP structure and the METS/PREMIS file is available on the Archivematica wiki: see [[AIP structure]] and [[METS]]. You can download the AIP at this stage, as well, by clicking on it.
 +
 
 +
#From the Action dropdown menu, select "Store AIP" to move the AIP into archival storage ('''figure 16'''). You can store an AIP in any number of preconfigured directories. For instructions to configure AIP storage locations, see [[Administration Manual|Administrataion manual - AIP storage locations]]
 +
 
 +
#From the Action drodown menu, select the AIP storage location from a pre-configured set of options.
 +
 
 
#For information on viewing and managing stored AIPs go to [[UM archival storage|Archival storage]]
 
#For information on viewing and managing stored AIPs go to [[UM archival storage|Archival storage]]
  

Revision as of 12:49, 23 August 2012

Main Page > Documentation > User manual > User manual 0.9 > Ingest

General description

During ingest, digital objects are packaged into SIPs and run through several micro-services, including normalization, packaging into an AIP and generation of a DIP.

Should you run into an error during ingest, please see Error handling.

Create SIP

  1. Process transfers as described in Transfer.
    Figure 1 Select an option from the Create SIP(s) Actions drop-down menu
  2. While still viewing the transfer tab you should see a bell icon next to the transfer. The micro-service related to the bell is "Create SIP(s)".
  1. In the Action dropdown menu, choose "Create single SIP and continue processing" (or reject the transfer if desired, to stop all processing and move the transfer to the rejected folder in the file browser) (Note that Create SIP(s) manually is not currently functional. However, the ability to create one or more SIP(s) from one or more transfer(s) will return in Archivematica 1.0 with improved functionality in the web browser. See MOCKUPS PAGE LINK) (figure 1).


  1. Click on the Ingest tab.
  1. The single SIP will move through a number of micro-services. Wait until it reaches "Normalize" and a bell icon appears (figure 2).
  1. To add descriptive metadata, see Add metadata, below.
  1. To add PREMIS rights information, see Add PREMIS, below.
  1. For selecting a normalization option, see Normalization, below.


Add metadata

You must add metadata prior to the normalization step. Any metadata added after normalization will not be included in the AIP package. See ICA-AtoM (Qubit) Dublin Core for information about the Dublin Core elements available. See ICA-AtoM (Qubit) Rights Entity for information about the rights elements available.

  1. Click on the magnifying glass icon. (figure 9).
    Figure 9 To add metadata, click on the magnifying glass icon
  2. This will take you to the SIP detail panel. On the left-hand side, under metadata click Add (figure 10).
  3. Add metadata as desired (figure 11) and save it by clicking the Save button at the bottom of the screen. Note that in Archivematica 0.9 you can only add metadata at the SIP level. If you would like to add metadata to a digital object, you will need to do that once the object has been uploaded to your access system.
  4. When you click Save, you will see the metadata entry in the list page (figure 12). To edit it further, click Edit on the right-hand side.
  5. Return to the ingest tab to continue processing the SIP.
    Figure 10 The SIP detail panel: click Add under Metadata to enter Dublin Core metadata for the SIP
    Figure 11 Using the data entry form to add descriptive metadata
    Figure 12 After saving your metadata

Add PREMIS

You must add PREMIS rights prior to the normalization step. Any rights added after normalization will not be included in the AIP package or the DIP upload. See ICA-AtoM (Qubit) Dublin Core for information about the PREMIS rights bases and elements available.

  1. Click on the magnifying glass icon. (figure 9).
    Figure 9 To add metadata, click on the magnifying glass icon
  2. This will take you to the SIP detail panel. On the left-hand side, under rights click Add (figure 10).
  3. Add rights as desired (figure 11) and save it by clicking the Save button at the bottom of the screen. Note that in Archivematica 0.9 you can only add metadata at the SIP level. If you would like to add rights to a digital object, you will need to do that once the object has been uploaded to your access system. (Note that rights entries are two pages of content. To get to the second page to complete data entry, click Next) (figure...)
  1. When you click Save, you will see the rights entry in the list page (figure 12). To edit it further, click Edit on the right-hand side.
  1. Return to the ingest tab to continue processing the SIP.
    Figure 10 The SIP detail panel: click Add under Metadata to enter Dublin Core metadata for the SIP
    Figure 11 Using the data entry form to add descriptive metadata
    Figure 12 After saving your metadata

Normalize

Normalizing is the process of converting ingested digital objects to preservation and/or access formats. Note that the original objects are always kept along with their normalized versions. For more information about Archivematica's preservation strategy, go to the Preservation Planning section of the manual.

Figure 13 Selecting a normalization option
  1. At the normalization step, the SIP will appear in the dashboard with a bell icon next to it. Select one of the normalization options from the Actions drop-down menu (figure 13):
    • Normalize for preservation and access: creates preservation copies of the objects plus access copies which will be used to generate the DIP.
    • Normalize for access: no preservation copies are created. Creates access copies which will be used to generate the DIP.
    • Normalize for preservation: creates preservation copies. No access copies are created and no DIP will be generated.
    • Do not normalize: no preservation copies are created. No access copies are created and no DIP will be generated.
    • You may also Reject SIP at this stage.
  1. Once normalization is completed you can review the results in the normalization report. Click on the report icon next to the Actions drop-down menu (figure 14).
    • The report shows what has been normalized and what is already in an acceptable preservation and access format (figure 15). For example, the file LAND2.BMP was normalized successfully to both a preservation and an access format; the file oakland03.jp2 was found to be in an acceptable preservation format already but it was necessary to generate an access copy for it.
  1. You may review the normalization results in a new tab by clicking on Review. If your browser has the correct plug-ins to open a file, you may open it in another tab by clicking on it. If you click on a file and your browser cannot open it, it will download locally so you can view it using the appropriate software.
  1. Approve normalization in the Actions drop-down menu to continue processing the SIP.
    Figure 14 Click on the report icon to open the normalization report
    Figure 15 The normalization report shows what was normalized and what was already in an acceptable preservation or access format
Figure 15 Review normalization results


Store AIP

  1. After normalization is approved, the SIP is run through a number of micro-services, including processing of the submission documentation, generation of the METS file, indexing, generation of the DIP and packaging of the AIP.
  1. When these micro-services are complete, the user can upload DIP and store AIP.
    Figure 16 Moving the AIP into archival storage.
  1. If desired, review the contents of the AIP in another tab by clicking on Review. More information on Archivematica's AIP structure and the METS/PREMIS file is available on the Archivematica wiki: see AIP structure and METS. You can download the AIP at this stage, as well, by clicking on it.
  1. From the Action dropdown menu, select "Store AIP" to move the AIP into archival storage (figure 16). You can store an AIP in any number of preconfigured directories. For instructions to configure AIP storage locations, see Administrataion manual - AIP storage locations
  1. From the Action drodown menu, select the AIP storage location from a pre-configured set of options.
  1. For information on viewing and managing stored AIPs go to Archival storage

Upload DIP

For information on uploading the DIP, go to Access.