Email preservation - Revision history

Hbecker: Move to feature requirements category

2017-03-23T19:48:57Z

Move to feature requirements category

Evelyn at 00:02, 30 May 2012

2012-05-30T00:02:10Z

Evelyn at 00:00, 30 May 2012

2012-05-30T00:00:17Z

Evelyn: /* Tools */

2012-03-09T00:06:01Z

Tools

Evelyn: /* Tools */

2012-03-08T23:55:59Z

Tools

Evelyn at 23:53, 8 March 2012

2012-03-08T23:53:18Z

Evelyn McLellan: /* Maildir */

2012-03-08T20:20:29Z

Maildir

Evelyn at 18:35, 8 March 2012

2012-03-08T18:35:54Z

Evelyn McLellan at 00:44, 2 March 2012

2012-03-02T00:44:49Z

Evelyn McLellan at 00:36, 2 March 2012

2012-03-02T00:36:19Z

@@ Line 1: / Line 1: @@
 [[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Email preservation
 ==Functional Requirements==
@@ Line 92: / Line 94: @@
 *Each Maildir subdirectory is normalized to an mbox file for access purposes
 *Question: how will we link the mbox file back to the directory in the AIP? May not have to do this for 0.9.

@@ Line 17: / Line 17: @@
 **Access format should allow simple, intuitive navigation between messages, attachments, email boxes, contacts, calendars
 **Access format should allow navigation to normalized access copies of attachments
 ==Formats==
@@ Line 29: / Line 30: @@
 [http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&id=1018 PRONOM]:
 *Defined in RFC5322 the Internet Message Format provides a syntax for text messages that are sent among computer users, within the framework of "electronic mail". Consisting of header fields and an optional body, encoded using ASCII characters. The Internet Message Format (*.eml) is a format used by numerous email systems. The format does not prevent the use of alternative formats for storage of electronic mail on those systems.
 ===MBOX===
@@ Line 34: / Line 36: @@
 From [http://www.significantproperties.org.uk/email-testingreport.html Inspect: Significant Properties Testing Report: Electronic Mail] Gareth Knight, 30 March 2009:
 *"The mbox family refers to four related, but only semi-compatible formats for the storage of one or more email messages and attachments. The four formats - mboxo, mboxrd, mboxcl, and mboxcl2 – originate from different versions of Unix. Each mbox file represents a set of email messages that are ordered sequentially and grouped into a ‘folder ‘. Email messages are stored in their source format, e.g. plain text may be stored as ASCII or Unicode, binary data is stored as Base64-encoded text. The format is well supported by a number of email applications and, thanks to its text-based composition can be processed, rendered and converted by a wide range of text processing software."
 ===Maildir===
@@ Line 43: / Line 46: @@
 Notes:
 *Archivematica sanitizes filenames to remove characters that are prohibited in various operating systems. In the case of maildir, a filename such as ''1330052116_1.7424.dell-desktop,U=153,FMD5=f72fc74533f0b9f432010a357af35516:2,'' would be changed to ''1330052116_1.7424.dell-desktop_U_153_FMD5_f72fc74533f0b9f432010a357af35516_2_''. If we do ingest and preserver maildir files we may need to skip the name cleanup micro-service if email systems depend on the format of the filename to recognize maildir files.
 ==Tools==
@@ Line 48: / Line 52: @@
 *[http://offlineimap.org/ OfflineImap] connects to IMAP accounts and saves the contents locally as maildir backups.
 *[https://gist.github.com/1709069 md2mb.py] is a python script that converts maildir to mbox format.
 ===Muse===
@@ Line 61: / Line 66: @@
 **Poorly designed user interface.
 **Poor user documentation.
 ==Email preservation research==
@@ Line 73: / Line 79: @@
 [http://www.ifs.tuwien.ac.at/dp/ipres2010/schedule.html 2010. Reshaping the Repository: The Challenge of Email Archiving], Goethals, A. and Gogel, W. In 7th International Conference on Preservation of Digital Objects], (iPRES2010). Vienna, Austria.
 *"We have preliminarily chosen to use the CERP/EMCAP schema, because we think it strikes the right balance between fully supporting the complexities of email headers and structure with a welcome lack of manipulation of the message bodies and attachments. Unlike most of the other schemas, it uses generic <Header> elements to store the names and values of the message headers. The advantage of this approach is that it can accommodate unanticipated headers, for example custom headers added by client systems, or those that will be added to future revisions of the email RFCs. It can support multiple message bodies per email, including HTML, and pointers to externally-stored attachments. They also have a separate schema for wrapping base64-encoded attachments, however we will likely decode attachments and store them in their original formats. While the CERP/EMCAP schema is designed to contain all the email messages for an account, we anticipate that it will work equally well at storing a single email message, which is how we intend to use it." (p.3)
 ==Archivematica 0.9 maildir ingest requirements==

@@ Line 1: / Line 1: @@
 [[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Email preservation
-==Requirements==
+==Functional Requirements==
 These are requirements for email preservation in Archivematica:
@@ Line 74: / Line 74: @@
 *"We have preliminarily chosen to use the CERP/EMCAP schema, because we think it strikes the right balance between fully supporting the complexities of email headers and structure with a welcome lack of manipulation of the message bodies and attachments. Unlike most of the other schemas, it uses generic <Header> elements to store the names and values of the message headers. The advantage of this approach is that it can accommodate unanticipated headers, for example custom headers added by client systems, or those that will be added to future revisions of the email RFCs. It can support multiple message bodies per email, including HTML, and pointers to externally-stored attachments. They also have a separate schema for wrapping base64-encoded attachments, however we will likely decode attachments and store them in their original formats. While the CERP/EMCAP schema is designed to contain all the email messages for an account, we anticipate that it will work equally well at storing a single email message, which is how we intend to use it." (p.3)

@@ Line 47: / Line 47: @@
 *[http://www.five-ten-sg.com/libpst/rn01re01.html Readpst], which can be downloaded from http://www.five-ten-sg.com/libpst/packages/, converts PST (MS Outlook Personal Folders) files to mbox and other formats.
 *[http://offlineimap.org/ OfflineImap] connects to IMAP accounts and saves the contents locally as maildir backups.
 ==Email preservation research==

@@ Line 46: / Line 46: @@
 ==Tools==
 *[http://www.five-ten-sg.com/libpst/rn01re01.html Readpst], which can be downloaded from http://www.five-ten-sg.com/libpst/packages/, converts PST (MS Outlook Personal Folders) files to mbox and other formats.
 ==Email preservation research==

← Older revision		Revision as of 00:44, 2 March 2012
Line 36:		Line 36:

	===Maildir===		===Maildir===
		+
		+	From [http://www.significantproperties.org.uk/email-testingreport.html Inspect: Significant Properties Testing Report: Electronic Mail] Gareth Knight, 30 March 2009:
		+	*"Maildir is an organizational structure for the storage of one or more emails on a file system. Each email is stored as a distinct file in one of three sub-directories: the ‘tmp ‘ sub-directory temporarily stores emails during processing; ‘new ‘ contains newly delivered emails; and ‘cur ‘ contains emails that have been processed by the client ‘s mail-reader software. The storage of each email as a distinct file in the file structure is cited as workaround to file locking issues that affect compound formats, such as mbox that update the mail data file that the user is accessing. However, the filename convention used for the storage of emails may cause incompatibility in implementations of Maildir for Unix-compatible and Microsoft Windows operating systems. The colon character is an illegal character in Microsoft Windows. However, there is no standard on the alternative character that may be used in the environment."

	==Email preservation research==		==Email preservation research==

← Older revision		Revision as of 00:36, 2 March 2012
Line 32:		Line 32:
	===MBOX===		===MBOX===

		+	From [http://www.significantproperties.org.uk/email-testingreport.html Inspect: Significant Properties Testing Report: Electronic Mail] Gareth Knight, 30 March 2009:
		+	*"The mbox family refers to four related, but only semi-compatible formats for the storage of one or more email messages and attachments. The four formats - mboxo, mboxrd, mboxcl, and mboxcl2 – originate from different versions of Unix. Each mbox file represents a set of email messages that are ordered sequentially and grouped into a ‘folder ‘. Email messages are stored in their source format, e.g. plain text may be stored as ASCII or Unicode, binary data is stored as Base64-encoded text. The format is well supported by a number of email applications and, thanks to its text-based composition can be processed, rendered and converted by a wide range of text processing software."

	===Maildir===		===Maildir===