Digital forensics image ingest

Main Page > Development > Development documentation > Digital forensics image ingest

Related issues: #5265 NOTE: Wherever possible, use BitCurator packages for forensics tools.

Forensics image transfer type

• Archivematica transfer type: forensic image
  • One or more images make up a transfer
  • Repository makes image using outside imaging software prior to ingest
  • Some metadata from ingest process will be included, first from FTK Imager, but later from other tools like Guymager (see metadata requirements below)
• Image types to base development on (more analysis needed): raw sector images =dd, bin, ISO; E01, AFF, AD1; ISO images with CUE files that contain track information; STREAM images (Kryoflux STREAM: This is a representation of non-decoded raw magnetic flux transitions acquired using a Kryoflux.) (these formats are sponsored, support for the formats listed here is desirable in future releases)

Forensics image transfer workflow

Detail

• User images external media outside the Archivematica workflow
• User uploads image(s) into the Archivematica transfer tab of the dashboard by browsing to the appropriate transfer source directory and selecting a directory containing their image(s)
• User enters transfer name and accession number
• User selects MD entry template for entering MD about the imaging process
  • User enters MD (see MD requirements below)
  • User saves MD and starts transfer processes
• User selects Start transfer to begin Archivematica transfer processing
• Fiwalk with Fido or BitCurator fiwalk package completes the Characterize and extract metadata micro-service
• Archivematica runs Bulk Extractor tool(Examine contents micro-service) and indexes output (this is to allow for reporting and visualization in the transfer backlog search for SIP creation and/or the AIP advanced search to allow for minimal description)
• Transfer micro-services complete
• At Create SIP from Transfer micro-service, user selects one of two options:
  • If the user is an archivist/curator ready to process the image through to storage and/or access, choose Create single SIP and continue processing
  • If the user is uploading multiple images as part of one accession, for processing by an archivist/curator later, choose Send to backlog
    • In the second scenario, once all images from an accession are in the backlog, user alerts archivist/curator that the accession is ready for further processing
  • Archivist searches for the accession in the transfer backlog, selects the appropriate transfers, and selects Create SIP
• In ingest tab, user approves SIP creation
• In ingest tab, prior to normalization, there is a decision point at Extract packages micro-service - User selects from drop-down: Extract objects from image, Do not extract objects from image, Reject
  • If user chooses not to extract objects, then skip micro-service decision about tool output to base normalization on, choose normalization for preservation only or no normalization, and continue standard micro-services to store AIP.
  • If user chooses to extract objects, Archivematica runs FITS on the extracted contents. The user continues standard workflow, choosing any of the normalization options (including manual normalization) and continues processing to storage and/or access.

• EXCEPTIONS:
  • In the case of AD1 images, user should be able to choose to extract the objects from the AD1 image before transfer. Archivematica should recognize an AD1 image and issue an alert/warning. If the user chooses to proceed with the transfer/ingest then the AD1 file just gets stored without any normalization or metadata extraction.

Metadata requirements

When the user selects Forensic image transfer type, each image uploaded as part of the transfer will include a metadata form icon that, if selected, will open a form in another browser tab. There, the user will enter some or all of the MD indicated below in the Template for manual data entry list.

• Template for manual data entry
  • accession number - recorded in transfer upload in dashboard
  • media number - manual
  • label text - manual (long text field)
  • media manufacture - manual
  • serial number - manual
  • media format - manual, could be controlled value list
  • media density - manual, could be controlled value list
  • source filesystem
  • notes about the imaging process
  • imaging interface - manual, could be controlled value list
  • examiner - AUTOPOPULATED based on Archivematica user (PREMIS agent)
  • image format - manual, could be controlled value list
  • imaging software - manual, could be controlled value list
  • notes about the imaging process - manual (long text field)

• Import from imaging tool FTK or fiwalk/sleuthkit
  • imaging date (FTK or other imaging tool output)
  • imaging success - Yes, Yes with errors (FTK or other imaging tool output)
  • image fixity (FTK or other imaging tool output
  • source filesystem (fiwalk)
  • accession data about extent (fiwalk)

element	description	DACS (2013)	ISAD(G)	EAD	PREMIS 2.2
media number	repository specific alphanumeric designation assigned to individual physical media/carrier	2.1.3 local identifier - At the highest level of a multilevel description or in a single level description, provide a unique identifier for the materials being described in accordance with the institution’s administrative control system. Optionally, devise unique identifiers at lower levels of a multilevel description.	3.1.1 - Reference codes	<unitid>
label text	textual transcription	7.1.2 Record, as needed, information not accommodated by any of the defined elements of description.	3.6.1 Note	<odd>, <note>
media manufacturer		7.1.4 - If the materials being described are in electronic form, give details of any migration or logical reformatting since its transfer to archival custody. Indicate the location of any relevant documentation. Information regarding digitization is provided in the Existence and Location of Copies Element (6.2).	3.6.1 Note	<odd>, <note>
serial number	when applicable to external media	7.1.4 or 7.1.6 If appropriate at the file or item level of description, make a note of any important numbers borne by the unit being described.	3.6.1 Note	<odd>, <note>
media format	a controlled value list (e.g. 3.5" floppy, 5.25" floppy, CD-R, etc)	7.1.4	3.6.1 Note	<odd>, <note>
media density	a controlled value list (e.g. single density, double density, quad density, high density)	7.1.4	3.6.1 Note	<odd>, <note>
source filesystem	a controlled file list(e.g. HFS, FAT, etc.) with the ability to add terms	7.1.4	3.6.1 Note	<odd>, <note>
notes about the imaging process	textual field to describe more detail about the imaging process	7.1.4	3.6.1 Note	<odd>, <note>
imaging interface	a controlled value list (e.g. Catweasel, Firewire, USB, IDE, etc.)				2.2 - Event: Image capture
examiner	the person doing the imaging				2.2 - Agent
imaging date	date of imaging				2.2 - Event: Image capture
imaging success	ex Yes/Yes, with errors				2.2 - Event: Image capture
image format	a controlled value list (e.g. AFF3, dd/secort image, AD1, etc.) with the ability to add terms				2.2 - Object
imaging software	a controlled value list (e.g. FTK imager 3.1.0.1514, Kryoflux, DTC 2.00 beta 9, etc.) with the ability to add terms				2.2 - Agent
image fixity	type(s) and value(s) from FTK csv output				2.2 - Object

Forensic image transfer tools

fiwalk

• Characterize and extract metadata micro-service
• Use Mark Matienzo's github version which includes FIDO for format identification since fiwalk's format identification is libmagic (unsatisfactory for our purposes)

Sample fiwalk xml output:

<?xml version='1.0' encoding='ISO-8859-1'?>
<fiwalk xmloutputversion='0.2'>
  <metadata 
  xmlns='http://example.org/myapp/' 
  xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' 
  xmlns:dc='http://purl.org/dc/elements/1.1/'>
    <dc:type>Disk Image</dc:type>
  </metadata>
  <creator>
    <program>fiwalk</program>
    <version>0.5.7</version>
    <os>Darwin</os>
    <library name="tsk" version="3.0.1"></library>
    <library name="afflib" version="3.5.2"></library>
    <command_line>fiwalk -x /dev/disk2</command_line>
  </creator>
  <source>
    <imagefile>/dev/disk2</imagefile>
  </source>
<!-- fs start: 512 -->
  <volume offset='512'>
    <Partition_Offset>512</Partition_Offset>
    <block_size>512</block_size>
    <ftype>2</ftype>
    <ftype_str>fat12</ftype_str>
    <block_count>5062</block_count>
    <first_block>0</first_block>
    <last_block>5061</last_block>
    <fileobject>
      <filename>README.txt</filename>
      <id>2</id>
      <filesize>43</filesize>
      <partition>1</partition>
      <alloc>1</alloc>
      <used>1</used>
      <inode>6</inode>
      <type>1</type>
      <mode>511</mode>
      <nlink>1</nlink>
      <uid>0</uid>
      <gid>0</gid>
      <mtime>1258916904</mtime>
      <atime>1258876800</atime>
      <crtime>1258916900</crtime>
      <byte_runs>
       <run file_offset='0' fs_offset='37376' img_offset='37888' len='43'/>
      </byte_runs>
      <hashdigest type='md5'>2bbe5c3b554b14ff710a0a2e77ce8c4d</hashdigest>
      <hashdigest type='sha1'>b3ccdbe2db1c568e817c25bf516e3bf976a1dea6</hashdigest>
    </fileobject>
  </volume>
<!-- end of volume -->
<!-- clock: 0 -->
  <runstats>
    <user_seconds>0</user_seconds>
    <system_seconds>0</system_seconds>
    <maxrss>1814528</maxrss>
    <reclaims>546</reclaims>
    <faults>1</faults>
    <swaps>0</swaps>
    <inputs>56</inputs>
    <outputs>0</outputs>
    <stop_time>Sun Nov 22 11:08:36 2009</stop_time>
  </runstats>
</fiwalk>