Large datasets - Revision history

Sallain at 21:55, 11 February 2020

2020-02-11T21:55:42Z

Evelyn at 20:10, 8 February 2013

2013-02-08T20:10:49Z

Evelyn at 20:09, 8 February 2013

2013-02-08T20:09:12Z

Evelyn at 20:08, 8 February 2013

2013-02-08T20:08:53Z

Evelyn at 20:07, 8 February 2013

2013-02-08T20:07:46Z

Evelyn: Created page with "- For instance, in the case of big data which Chuck just mentioned, an AIP bag should be open until its size reaches xxMB and then this bag closes and another bag opens up to ..."

2013-02-08T19:55:38Z

Created page with "- For instance, in the case of big data which Chuck just mentioned, an AIP bag should be open until its size reaches xxMB and then this bag closes and another bag opens up to ..."

New page

- For instance, in the case of big data which Chuck just mentioned, an AIP bag should be open until its size reaches xxMB and then this bag closes and another bag opens up to be filled with the remaining smaller files in the dataset.
- When a single huge file has to be preserved, the critical task is how to split this file into multiple smaller units. Take an example of a movie file which could be more than 1GB, on the YouTube one can access a number of smaller clips (part 1 of 17 and so on) to watch the whole movie. In a similar way, a built-in splitter is needed during the ingest process which creates sibling AIPs for a huge file.

[[Category:Development documentation]]

← Older revision		Revision as of 21:55, 11 February 2020
Line 1:		Line 1:
	[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > Large datasets		[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > Large datasets
		+
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information. </div> <p>

	What happens when a body of materials to be ingested consists of thousands of files (eg a large social science research dataset), or when one file is extremely large (eg an HD video file)?		What happens when a body of materials to be ingested consists of thousands of files (eg a large social science research dataset), or when one file is extremely large (eg an HD video file)?

@@ Line 5: / Line 5: @@
 **The dataset could be broken into a parent AIP which acts as an Archival Information Collection, consisting entirely of a METS structMap listing all its child AIPs; each child AIP would have a link back to the parent AIP in its own structMap.
 *The large single file could be broken into multiple segments, each in its own AIP. Video files could be delivered to end users in these segments, the way large video files are delivered on Youtube, for example.
-*Other types of large files might have to be merged back into one for delivery to a user.
+**Other types of large files might have to be merged back into one for delivery to a user.
 [[Category:Development documentation]]

← Older revision		Revision as of 20:09, 8 February 2013
Line 1:		Line 1:
−	[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > ~~Email preservation~~	+	[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > Large datasets

	What happens when a body of materials to be ingested consists of thousands of files (eg a large social science research dataset), or when one file is extremely large (eg an HD video file)?		What happens when a body of materials to be ingested consists of thousands of files (eg a large social science research dataset), or when one file is extremely large (eg an HD video file)?

@@ Line 1: / Line 1: @@
 What happens when a body of materials to be ingested consists of thousands of files (eg a large social science research dataset), or when one file is extremely large (eg an HD video file)?
 *The large number of files could be broken up and distributed across multiple AIPs, with relationships between them expressed in the METS structMaps.
@@ Line 4: / Line 6: @@
 *The large single file could be broken into multiple segments, each in its own AIP. Video files could be delivered to end users in these segments, the way large video files are delivered on Youtube, for example.
 *Other types of large files might have to be merged back into one for delivery to a user.
 [[Category:Development documentation]]