Improvements/Disk Image Preservation - Revision history

Sallain at 21:55, 11 February 2020

2020-02-11T21:55:13Z

Kelly: /* Second iteration */

2017-05-11T17:46:53Z

Second iteration

Kelly: /* Second iteration */

2017-05-04T18:08:43Z

Second iteration

Kelly: /* Second/third iterations */

2017-05-04T18:07:27Z

Second/third iterations

Sromkey: /* Second iteration */

2017-04-24T16:57:42Z

Second iteration

Sromkey: /* Second iteration */

2017-04-19T22:57:05Z

Second iteration

Sromkey at 22:46, 19 April 2017

2017-04-19T22:46:07Z

Hbecker: Add development documentation category

2017-03-10T19:57:13Z

Add development documentation category

Sromkey at 20:00, 27 January 2017

2017-01-27T20:00:50Z

Sromkey at 18:08, 24 January 2017

2017-01-24T18:08:53Z

Show changes

← Older revision		Revision as of 21:55, 11 February 2020
Line 1:		Line 1:
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information. </div> <p>
		+
	== Synopsis ==		== Synopsis ==

← Older revision		Revision as of 17:46, 11 May 2017
Line 182:		Line 182:

	[1] Script from CCA Disk Image Processor can be run outside the GUI: https://github.com/timothyryanwalsh/cca-diskimageprocessor/blob/master/diskimageanalyzer.py		[1] Script from CCA Disk Image Processor can be run outside the GUI: https://github.com/timothyryanwalsh/cca-diskimageprocessor/blob/master/diskimageanalyzer.py
		+
		+	'''Results of work as of May 9, 2017'''
		+
		+	The HFS disk image extraction and characterization functionality should work on http://am16hfs.archivematica.org/transfer/ and it should also work on a new system that is running Archivematica at branch dev/issue-10818-hfs-disk-images.
		+
		+	To install such a system using Archivematica's deploy-pub Vagrant repo, modify the vars-singlenode-1.6.yml file so that ``archivematica_src_am_version`` is set to ``"dev/issue-10818-hfs-disk-images."`` and provision::
		+
		+	$ vagrant provision
		+
		+	In order to get the new HFS disk image functionality to work in Archivematica, the following tools and their dependencies must be manually installed:
		+
		+	- hfs2dfxml
		+	- hfsexplorer
		+
		+	The following instructions were used on Ubuntu 14.04.
		+
		+	To install hfsutils::
		+
		+	$ sudo apt-get update
		+	$ sudo apt-get install hfsutils
		+
		+	Install python-magic. WARNING: it's important that you do NOT install this python-magic: https://github.com/ahupp/python-magic. Instead, you must install the python-magic (see https://github.com/threatstack/libmagic) that Ubuntu installs when you call the following::
		+
		+	$ sudo apt-get install python-magic
		+
		+	If your install has separate virtual environments for each Archivematica component, then the MCPClient virtualenv needs to have magic installed also::
		+
		+	$ sudo ln -s /usr/lib/python2.7/dist-packages/magic.py /usr/share/python/archivematica-mcp-client/lib/python2.7/site-packages/magic.py
		+
		+	Install (Holly Becker's patch of) the hfs2dfxml source in your home directory
		+	on the machine where Archivematica is installed::
		+
		+	$ pwd
		+	/home/vagrant
		+	$ mkdir bin
		+	$ cd bin
		+	$ git clone https://github.com/Hwesta/hfs2dfxml
		+	$ cd hfs2dfxml/hfs2dfxml
		+	$ git checkout -t origin/patch-1
		+	$ git clone https://github.com/simsong/dfxml/
		+
		+	Install hfsexplorer:
		+
		+	- Go to http://www.catacombae.org/hfsexplorer/
		+	- Download the ZIP at link: Download application as ZIP file (cross-platform)
		+	- Extract ZIP into directory accessible by Archivematica (suggested: extract
		+	into /home/vagrant/hfsexplorer)
		+	- Make sure that the hfsexplorer directory is in the directory that extraction
		+	command expects it to be, i.e., /usr/local/hfsexplorer/bin/unhfs.sh
		+
		+	Also make sure to modify the default processing config so that Siegfried is the
		+	file identification command during transfer and packages are not deleted after
		+	extraction: this allows the disk image to be characterized after its contents
		+	have been extracted.

	'''Results of work as of May 3, 2017'''		'''Results of work as of May 3, 2017'''

@@ Line 202: / Line 202: @@
 . Ran a transfer on artefactual/227_026/ and it was identified as an HFS disk image (by hfs2dfxml) and it was extracted using unhfs.
-Things to Note:
+'''Things to Note:'''
 a. The METS file will list "No Matching Format" for the format identification event for the disk image. This is because the FormatVersion that Siegfried/blkid is assigning to the HFS disk image has no pronom id. This could be "fixed" by assigning a fake pronom id to that FormatVersion, as seems to have been done in the past in Archivematica::
@@ Line 214: / Line 214: @@
 b. Still left to do is the creation of ansible tasks to install the tools and their dependencies: unhfs, HFSutils, hfs2dfxml
 == Third iteration ==

@@ Line 170: / Line 170: @@
 Explore reporting possibilities either in Archivematica or using other reporting tools.
-== Second/third iterations ==
+== Second iteration ==
 The goal of the second iteration is to get around the need to identify by extension. This will allow the disk image itself and its contents to be properly identified using a normal identification tool. For the purposes of this project we'll do it using Siegfried.
@@ Line 180: / Line 180: @@
 * integrate the script in to the Siegfried File identification script running within Archivematica. This could be developed to run anytime Siegfried is run as the file identification tool or as a special configuration using the Disk Image transfer type.
 * this will produce a special format (e.g., "img-hfs" or similar) which will need to be added to the FPR in order to prompt the desired tools for further mirco-services.
 The goal of a third iteration would be to prevent the need to use a special format, and instead have extract packages and characterization tools chosen appropriately based on file system information rather than format. Two possibilities for enacting FPR rules based on file system type rather than on file format:

@@ Line 170: / Line 170: @@
 Explore reporting possibilities either in Archivematica or using other reporting tools.
-== Second iteration ==
+== Second/third iterations ==
 The goal of the second iteration is to get around the need to identify by extension. This will allow the disk image itself and its contents to be properly identified using a normal identification tool. For the purposes of this project we'll do it using Siegfried.
@@ Line 176: / Line 176: @@
 Steps to the second iteration:
-- test analyzer script from [https://github.com/timothyryanwalsh/cca-diskimageprocessor CCA Disk Image Processor] on disk image samples
+* test analyzer script from [https://github.com/timothyryanwalsh/cca-diskimageprocessor CCA Disk Image Processor][1] on disk image samples
-- verify that for our samples, the output csv file has identified the file system
+* verify that for our samples, the output csv file has identified the file system
-- integrate the script in to the Siegfried File identification script running within Archivematica
+* integrate the script in to the Siegfried File identification script running within Archivematica. This could be developed to run anytime Siegfried is run as the file identification tool or as a special configuration using the Disk Image transfer type.
 == Tools/resources ==

← Older revision		Revision as of 22:57, 19 April 2017
Line 171:		Line 171:

	== Second iteration ==		== Second iteration ==
		+
		+	The goal of the second iteration is to get around the need to identify by extension. This will allow the disk image itself and its contents to be properly identified using a normal identification tool. For the purposes of this project we'll do it using Siegfried.
		+
		+	Steps to the second iteration:
		+
		+	- test analyzer script from [https://github.com/timothyryanwalsh/cca-diskimageprocessor CCA Disk Image Processor] on disk image samples
		+	- verify that for our samples, the output csv file has identified the file system
		+	- integrate the script in to the Siegfried File identification script running within Archivematica

	== Tools/resources ==		== Tools/resources ==

@@ Line 22: / Line 22: @@
 Develop a pre-ingest script which would identify the file system and store this metadata in such a way that it can be passed to Archivematica. This could be made part of an [https://github.com/artefactual/automation-tools automation tool] script.
 ===File identification script [1 support tickets]===
 Currently, the file identification scripts run in Archivematica will identify the type of disk image but not the file system of the disk image. This script will use the data from the pre-ingest script to identify both the disk image type and the file system.
 ===Characterize [1 support ticket]===
 Write meaningful characterization about the size and file type of the disk images so that statistics can be gathered from the AIPs. Currently, when fiwalk is run as the characterization tool for a disk image, dfxml is written in premis:objectCharacteristicsExtension.
 ===File extraction [1 support ticket]===
 Implement tools such as [https://www.mars.org/home/rob/proj/hfs/ HFS Utilities] that will allow files from hfs disk images to be extracted.
 ===Deployment of development above for testing [1 support ticket]===
@@ Line 40: / Line 169: @@
 Explore reporting possibilities either in Archivematica or using other reporting tools.
 == Tools/resources ==

← Older revision		Revision as of 19:57, 10 March 2017
Line 460:		Line 460:

	[[Digital_forensics_image_ingest\|Original requirements for forensic image ingest]]		[[Digital_forensics_image_ingest\|Original requirements for forensic image ingest]]
		+
		+	[[Category:Development documentation]]

@@ Line 45: / Line 45: @@
 * [https://www.mars.org/home/rob/proj/hfs/ HFS Utilities]
 * [https://github.com/cul-it/hfs2dfxml hfs2dfxml] - Utility to parse hfsutils output and produce DFXML for HFS-formatted disk images
 == Example dfxml ==