Archivematica - User contributions [en]

Virtual appliance instructions

2011-04-21T16:55:30Z

Misty De Meo: Sun --> Oracle; product name has changed

[[Main Page]] > Virtual appliance instructions

Archivematica is distributed as a [http://en.wikipedia.org/wiki/Virtual_appliance virtual appliance] which integrates a number of software tools into one common virtual machine environment.

This allows it to be run on almost any workstation or server hardware without compromising the host machine's operating system or application software. At the same time, the virtual appliance is able to interact with any number of networked and/or external storage devices to allow for the flexible implementation of an archival storage and backup strategy.

*See the [[software]] page for a full list of the tools used in each system release.
*See the [[documentation]] page for instructions on how to use the virtual appliance to accomplish the system [[requirements]].
*See http://digitalcommons.uconn.edu/libr_pubs/34/ for instructions on installing Archivematica v0.7 on a custom-sized xubuntu v10.04.2 virtual machine hosted in VirtualBox v4.0.4, written by Michael J. Bennett. See also the related discussion on the [https://groups.google.com/group/archivematica/browse_thread/thread/570559472f3afea4?hl=en Archivematica discussion list]

=Minimum hardware requirements=

*Processor: Intel core 2 or AMD Opteron
*Memory: 512MB for the virtual appliance ('guest') operating system, i.e: if the 'host' operating system has 2GB available, 512MB needs to allocated to the 'guest'. Depending on the operating system, machines with less than 1.5GB total memory will likely have trouble running Archivematica. Note that the default allocation setting in Archivematica is 512MB; however, the more that is allocated the better the system will run. The setting can be changed once Archivematica is running.
*Hard Drive space: a minimum of 3GB to test the system on a small scale (i.e. use the available test files or import a small set of test files); 12GB or more for larger implementations

=Instructions for using the VM image=

==Install Oracle Virtual Box==

*Archivematica uses the [http://en.wikipedia.org/wiki/Open_Virtualization_Format Open Virtualization Format] and has been tested with the free and open-source [http://www.sun.com/software/products/virtualbox/index.jsp Oracle Virtual Box ] virtualization platform.
*There are VirtualBox versions available for every major operating system.
**Download and install Oracle VirtualBox: [http://dlc.sun.com/virtualbox/vboxdownload.html http://dlc.sun.com/virtualbox/vboxdownload.html]. Note that if you are installing VirtualBox on Windows you will have to click through a number of warnings that you are attempting to install non-verified software.

==Download Archivematica ==

*[http://archivematica.org/download Download] the latest version of the Archivematica appliance.
*Unzip the Archivematica file. This should result in the following two files appearing in an Archivematica folder:
**Archivematica-0.x.ovf
**disk0.vmdk

==Start Archivematica virtual appliance==

*Open the Oracle VirtualBox virtual machine.
*Click File > Import Appliance
*Click Choose
*Select ''Archivematica-0.x.ovf''
*Click Open
*Click Next
*Click Import
*Read and agree to the Software License Agreement (GPL2)
*The virtual box will open with Archivematica 0.x listed on the left-hand side. Select Archivematica 0.x and click Start (the green arrow in the menu).
*The image should launch, showing you a Linux Ubuntu desktop and some of the digital preservation applications which are included in Archivematica 0.x
**You may have to log in using the user name ''demo'' and the password ''demo''.
**If your mouse pointer does not appear to be working in the virtual machine, click the letter f while holding down the right-hand control button. Do the same thing to switch back to using your mouse pointer outside the virtual box.
*Congratulations, you have a running copy of Archivematica! See [[Documentation]] for instructions on how to use the software.

==Fixing archivematica sources.list==
The following commands entered in terminal will re-add the ubuntu repositories (makes repository software installable)
<pre>
sudo -s
mv /etc/apt/sources.list /etc/apt/sources.list.backup
wget http://archivematica.org/downloads/sources.list
mv ./sources.list /etc/apt/sources.list
aptitude update
</pre>

==Import files into virtual appliance (optional)==
===Using SFTP===

* While in virtualbox right click the archivematica virtualmachine and click settings
* Click the 'Network' tab
* 'Adapter 1' should be set to NAT by default(this allows you to get to the internet), click on 'Adapter 2' tab
* Click enable adapter and set attached to 'host only adapter'
* Power up the archivematica virtualmachine
* Once in the xubuntu interface goto Applications > Accessories > Terminal
* Type the following in terminal (the password is demo) - this will take a minute and requires internet
<pre>sudo aptitude install ssh </pre>
* type ifconfig in terminal you should see a IP address like '192.168.56.101' (likely eth1 interface)
<pre>
$ ifconfig
eth1 Link encap:Ethernet HWaddr fe:54:00:9d:92:64
inet addr:192.168.56.101 Bcast:192.168.56.255 Mask:255.255.255.0
inet6 addr: fe80::1c6b:7bff:fe07:ddb6/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24 errors:0 dropped:0 overruns:0 frame:0
TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1400 (1.4 KB) TX bytes:5815 (5.8 KB)
</pre>
* From here your machine should be connectable via SFTP. Download a SFTP client, a popular opensource option is FileZilla, which works on Linux and Windows. If using OSX cyber duck is reported to be a decent opensource SFTP client.

* the connection information should be as follows
<pre>
username: demo
password: demo
IP/Hostname: 192.168.56.101 < results of ifconfig likely '192.168.56.*'
port: 22
destination folder: /home/demo/ < if this is not set you will have to navigate to /home/demo directory
</pre>

==Turn off virtual machine==
*At the end of your session, turn off the VirtualBox virtual machine by going to Machine > Close > and selecting one of the three shutdown options. It is best to choose either "Save the machine state" or "Send the shutdown signal". This is from the VirtualBox help manual:

<blockquote>3.4.3. Saving the state of the machine</blockquote>

<blockquote>When you click on the "Close" button of your virtual machine window (at the top right of the window, just like you would close any other window on your system) (or press the Host key together with "Q"), VirtualBox asks you whether you want to "save" or "power off" the VM.</blockquote>

<blockquote>The difference between these three options is crucial. They mean:</blockquote>

<blockquote>Save the machine state: With this option, VirtualBox "freezes" the virtual machine by completely saving its state to your local disk. When you later resume the VM (by again clicking the "Start" button in the VirtualBox main window), you will find that the VM continues exactly where it was left off. All your programs will still be open, and your computer resumes operation.</blockquote>

<blockquote>Saving the state of a virtual machine is thus in some ways similar to suspending a laptop computer (e.g. by closing its lid).</blockquote>

<blockquote>Send the shutdown signal. This will send an ACPI shutdown signal to the virtual machine, which has the same effect as if you had pressed the power button on a real computer. So long as a fairly modern operating system is installed and running in the VM, this should trigger a proper shutdown mechanism in the VM.</blockquote>

<blockquote>Power off the machine: With this option, VirtualBox also stops running the virtual machine, but without saving its state.</blockquote>

<blockquote>This is equivalent to pulling the power plug on a real computer without shutting it down properly. If you start the machine again after powering it off, your operating system will have to reboot completely and may begin a lengthy check of its (virtual) system disks.</blockquote>

<blockquote>As a result, this should not normally be done, since it can potentially cause data loss or an inconsistent state of the guest system on disk.</blockquote>

<blockquote>The "Discard" button in the main VirtualBox window discards a virtual machine's saved state. This has the same effect as powering it off, and the same warnings apply.</blockquote>

JPEG2000

2011-04-07T19:32:19Z

Misty De Meo: Preservation Issues

User talk:Sdyppoypgo

2011-03-25T14:34:19Z

Misty De Meo: Eliminate spam

JPEG2000

2011-03-23T19:03:17Z

Misty De Meo:

[[Main Page]] > [[Documentation]] > [[Media type preservation plans]] > JPEG2000

===File extension(s)===
.jp2

===Format registry information===
None. Closest is [http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&id=794 PRONOM (JPX - JPEG 2000 Extended)]

===Significant characteristics===
See [[Significant characteristics of raster images]]

===Preservation action plan===
See [[Raster images]]

===Preservation issues===
*JPEG2000 image compression ranges from completely lossless to very lossy.
*Florida digital Archive has "high confidence level" in ability to preserve losslessly compressed JP2 files: [http://www.fcla.edu/digitalArchive/formatInfo.htm FDA Recommended Data File Formats].

== Encoding JPEG2000-MXF video ==

===Method one: OpenDCP===

*FFmpeg
*OpenJPEG 1.4 ([http://www.openjpeg.org/])
*OpenDCP ([http://code.google.com/p/opendcp/])

This workflow is designed around digital cinema package (DCP) production.
Most of the open-source tools are based around DCP since they're being
designed by indie filmmakers.

OpenDCP is a utility for creating DCPs. The [http://www.cinecert.com/asdcplib/ AS-DCP library] and the [http://sourceforge.net/projects/mxflib/ mxflib library] it's based on are possibilities
for someone with more programming skills than me to build an
archives-targeted tool.

The disadvantage to this route is that DCPs use separate files for audio
and video, accompanied by an XML file describing the pair, which might
make playback less convenient than a single AV MXF. The available tools
are also targeted more to film framerates, and I'm not sure if it's
possible to obtain a video FPS like 29.97 using unmodified tools.

Step 1: Decoding source video

The source video should be decoded into a series of single frames before
being encoded into JPEG2000. This can be done using Ffmpeg.

<code>ffmpeg -i invideo -vcodec tiff f%10d.tif</code>

This will produce one file for every frame in the video. Why do we do
this? Well, the various Motion JPEG2000 formats are essentially a series
of single JPEG2000 files; each frame will be encoded to JPEG2000 before
multiplexing into a video.

The audio should be decoded to uncompressed WAV. This is a simple process
so I won't go into much detail here. The command should be a variant of

<code>ffmpeg -i invideo outfile.wav</code>

Step 2: Create JP2 frames

Next, these frames should be compressed into JPEG2000. Note that this will
'''not''' be done using OpenJPEG's support for Motion JPEG2000! Motion
JPEG2000 is a specific container for single JPEG2000 frames, while MXF
wraps its frames in a different way. Technically, JPEG2000/MXF
is considered a separate format.

<code>image_to_j2k -ImgDir directory -OutFor j2c</code>

This will create JP2 frames using the default lossless compression frames,
but you may wish to customize this. Frames will be saved as .j2c files in
the same directory as the TIFF frames.

Step 3: Create MXF file

<code>opendcp_mxf -i image_directory -r framerate -o video.mxf

opendcp_mxf -i audio.wav -o audio.mxf

opendcp_xml --reel video.mxf audio.mxf</code>

This will produce two MXFs, one containing video and one containing audio.
The third command creates an accompanying XML file, mainly targeted at
digital projectors.

===Method two: GStreamer===

GStreamer can encode to and from JPEG2000 MXF. By default it uses JasPer
as the JP2 encoder, which has some limitations. I believe there may be a
project underway to replace JasPer with OpenJPEG. GStreamer's commandline
utility also has very inconvenient syntax; it's designed primarily to be
used as a library to be called by custom programs. The gst-launch utility
is mainly meant for debug purposes. For production would be best to write
a custom archives-oriented converter based on Gstream rather than use the
gst-launch utility.

The advantage to GStreamer is that it provides all of the required
functionality, and can go straight from a source video to producing an
output.

On the other hand, GStreamer currently uses JasPer for JP2 encoding. It
may be better to use it in conjunction with an external JP2 encoder.

Requirements:

*GStreamer ([http://gstreamer.freedesktop.org])
*gst-plugins-bad

The GStreamer debug utility is based around a chain of smaller tools
chained together. Here's a sample command which takes a video-only file,
in QuickTime format with PNG video, and turns it into a JPEG2000 MXF:

<code>gst-launch-0.10 filesrc location=video.mov ! qtdemux ! pngdec !
ffmpegcolorspace ! videoscale ! jp2kenc ! mxfmux ! filesink
location=video.mxf</code>

This goes through a series of steps to get from your source video to the
encoded video. This example does:

*filesrc location=video.mov : Tells gst-launch where to find the source video
*qtdemux : Demultiplexes the video. This means that it "opens" the container and extracts the tracks inside it, making them available to the other GStreamer tools
*pngdec : Decodes PNG compressed video into uncompressed video
*ffmpegcolorspace : Interprets the colour space in the video. Necessary to be able to pass it to other tools
*videoscale : Negotiates the size of the video. When used with no arguments, it just ensures the next tool knows what the size is
*jp2kenc : Encodes the video stream to JPEG 2000. You can specify the settings here if you want
*mxfmux : Combines (multiplexes) the streams being processed into an MXF container
*filesink location=video.mxf : Saves the video to a "filesink", e.g. a location on the hard drive

The exact commands will depend on the source video. You can also do a
combined audio/video file, but the commands for that are kind of complex
and, to be honest, I haven't quite gotten my head around it yet.

You can also decode and convert videos using GStreamer. Here's a sample
command to convert the previous video to MPEG-4 with default settings:

<code>gst-launch-0.10 filesrc location=video.mxf ! mxfdemux name=d d. !
jp2kdec ! ffmpegcolorspace ! videoscale ! ffenc_mpeg4 ! filesink
location=video.mp4</code>

===Method three: GStreamer + OpenJPEG===

Requirements:

*GStreamer
*gstreamer-plugins-bad
*FFmpeg
*OpenJPEG

As I mentioned before, GStreamer's JP2 encoder isn't ideal right now.
However, you can easily encode a video with another tool and wrap it into
MXF using GStreamer.

First, extract your video to frames with ffmpeg as described above, then
compress them to JP2 frames using the following command:

<code>image_to_j2k -ImgDir directory -OutFor j2k</code>

You should then wrap them into an MJ2 video using the following command:

<code>wrap_j2k_in_mj2 directory video.mj2</code>

This will produce a single MJ2 file using the frames you extracted. By
default the framerate will be 25fps, which is probably wrong - consult the
OpenJPEG documentation to find out how to set the framerate.

You can then open this video file using GStreamer and rewrap the frames
into an MXF using a command like the following:

<code>gst-launch-0.10 filesrc location=video.mj2 ! mj2demux ! mxfmux ! filesink
location=video.mxf</code>

This will produce an MXF with the contents as your video - no
recompression or loss of quality. Hooray!

===Areas for future work===

GStreamer seems ripe for someone to develop a custom, archives-oriented video conversion program. Any volunteers?

==More information==
[http://www.digitalpreservation.gov/formats/fdd/fdd000143.shtml Library of Congress Sustainability of Digital Formats: JPEG 2000 Part 1 (Core) jp2 File Format]

JPEG2000

2011-03-23T19:01:59Z

Misty De Meo: Reparent "more information"

[[Main Page]] > [[Documentation]] > [[Media type preservation plans]] > JPEG2000

===File extension(s)===
.jp2

===Format registry information===
None. Closest is [http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&id=794 PRONOM (JPX - JPEG 2000 Extended)]

===Significant characteristics===
See [[Significant characteristics of raster images]]

===Preservation action plan===
See [[Raster images]]

===Preservation issues===
*JPEG2000 image compression ranges from completely lossless to very lossy.
*Florida digital Archive has "high confidence level" in ability to preserve losslessly compressed JP2 files: [http://www.fcla.edu/digitalArchive/formatInfo.htm FDA Recommended Data File Formats].

== Encoding JPEG2000-MXF video ==

===Method one: OpenDCP===

*FFmpeg
*OpenJPEG 1.4 ([http://www.openjpeg.org/])
*OpenDCP ([http://code.google.com/p/opendcp/])

This workflow is designed around digital cinema package (DCP) production.
Most of the open-source tools are based around DCP since they're being
designed by indie filmmakers.

OpenDCP is a utility for creating DCPs. The [http://www.cinecert.com/asdcplib/ AS-DCP library] and the [http://sourceforge.net/projects/mxflib/ mxflib library] it's based on are possibilities
for someone with more programming skills than me to build an
archives-targeted tool.

The disadvantage to this route is that DCPs use separate files for audio
and video, accompanied by an XML file describing the pair, which might
make playback less convenient than a single AV MXF. The available tools
are also targeted more to film framerates, and I'm not sure if it's
possible to obtain a video FPS like 29.97 using unmodified tools.

Step 1: Decoding source video

The source video should be decoded into a series of single frames before
being encoded into JPEG2000. This can be done using Ffmpeg.

<code>ffmpeg -i invideo -vcodec tiff f%10d.tif</code>

This will produce one file for every frame in the video. Why do we do
this? Well, the various Motion JPEG2000 formats are essentially a series
of single JPEG2000 files; each frame will be encoded to JPEG2000 before
multiplexing into a video.

The audio should be decoded to uncompressed WAV. This is a simple process
so I won't go into much detail here. The command should be a variant of

<code>ffmpeg -i invideo outfile.wav</code>

Step 2: Create JP2 frames

Next, these frames should be compressed into JPEG2000. Note that this will
'''not''' be done using OpenJPEG's support for Motion JPEG2000! Motion
JPEG2000 is a specific container for single JPEG2000 frames, while MXF
wraps its frames in a different way. Technically, JPEG2000/MXF
is considered a separate format.

<code>image_to_j2k -ImgDir directory -OutFor j2c</code>

This will create JP2 frames using the default lossless compression frames,
but you may wish to customize this. Frames will be saved as .j2c files in
the same directory as the TIFF frames.

Step 3: Create MXF file

<code>opendcp_mxf -i image_directory -r framerate -o video.mxf

opendcp_mxf -i audio.wav -o audio.mxf

opendcp_xml --reel video.mxf audio.mxf</code>

This will produce two MXFs, one containing video and one containing audio.
The third command creates an accompanying XML file, mainly targeted at
digital projectors.

===Method two: GStreamer===

GStreamer can encode to and from JPEG2000 MXF. By default it uses JasPer
as the JP2 encoder, which has some limitations. I believe there may be a
project underway to replace JasPer with OpenJPEG. GStreamer's commandline
utility also has very inconvenient syntax; it's designed primarily to be
used as a library to be called by custom programs. The gst-launch utility
is mainly meant for debug purposes. For production would be best to write
a custom archives-oriented converter based on Gstream rather than use the
gst-launch utility.

The advantage to GStreamer is that it provides all of the required
functionality, and can go straight from a source video to producing an
output.

On the other hand, GStreamer currently uses JasPer for JP2 encoding. It
may be better to use it in conjunction with an external JP2 encoder.

Requirements:

GStreamer ([http://gstreamer.freedesktop.org])
gst-plugins-bad

The GStreamer debug utility is based around a chain of smaller tools
chained together. Here's a sample command which takes a video-only file,
in QuickTime format with PNG video, and turns it into a JPEG2000 MXF:

<code>gst-launch-0.10 filesrc location=video.mov ! qtdemux ! pngdec !
ffmpegcolorspace ! videoscale ! jp2kenc ! mxfmux ! filesink
location=video.mxf</code>

This goes through a series of steps to get from your source video to the
encoded video. This example does:

*filesrc location=video.mov : Tells gst-launch where to find the source
video
*qtdemux : Demultiplexes the video. This means that it "opens" the
container and extracts the tracks inside it, making them available to the
other GStreamer tools
*pngdec : Decodes PNG compressed video into uncompressed video
*ffmpegcolorspace : Interprets the colour space in the video. Necessary to
be able to pass it to other tools
*videoscale : Negotiates the size of the video. When used with no
arguments, it just ensures the next tool knows what the size is
*jp2kenc : Encodes the video stream to JPEG 2000. You can specify the
settings here if you want
*mxfmux : Combines (multiplexes) the streams being processed into an MXF
container
*filesink location=video.mxf : Saves the video to a "filesink", e.g. a
location on the hard drive

The exact commands will depend on the source video. You can also do a
combined audio/video file, but the commands for that are kind of complex
and, to be honest, I haven't quite gotten my head around it yet.

You can also decode and convert videos using GStreamer. Here's a sample
command to convert the previous video to MPEG-4 with default settings:

<code>gst-launch-0.10 filesrc location=video.mxf ! mxfdemux name=d d. !
jp2kdec ! ffmpegcolorspace ! videoscale ! ffenc_mpeg4 ! filesink
location=video.mp4</code>

===Method three: GStreamer + OpenJPEG===

Requirements:

GStreamer
gstreamer-plugins-bad
FFmpeg
OpenJPEG

As I mentioned before, GStreamer's JP2 encoder isn't ideal right now.
However, you can easily encode a video with another tool and wrap it into
MXF using GStreamer.

First, extract your video to frames with ffmpeg as described above, then
compress them to JP2 frames using the following command:

<code>image_to_j2k -ImgDir directory -OutFor j2k</code>

You should then wrap them into an MJ2 video using the following command:

<code>wrap_j2k_in_mj2 directory video.mj2</code>

This will produce a single MJ2 file using the frames you extracted. By
default the framerate will be 25fps, which is probably wrong - consult the
OpenJPEG documentation to find out how to set the framerate.

You can then open this video file using GStreamer and rewrap the frames
into an MXF using a command like the following:

<code>gst-launch-0.10 filesrc location=video.mj2 ! mj2demux ! mxfmux ! filesink
location=video.mxf</code>

This will produce an MXF with the contents as your video - no
recompression or loss of quality. Hooray!

===Areas for future work===

GStreamer seems ripe for someone to develop a custom, archives-oriented video conversion program. Any volunteers?

==More information==
[http://www.digitalpreservation.gov/formats/fdd/fdd000143.shtml Library of Congress Sustainability of Digital Formats: JPEG 2000 Part 1 (Core) jp2 File Format]

JPEG2000

2011-03-23T18:34:40Z

Misty De Meo: JP2-MXF video

[[Main Page]] > [[Documentation]] > [[Media type preservation plans]] > JPEG2000

===File extension(s)===
.jp2

===Format registry information===
None. Closest is [http://www.nationalarchives.gov.uk/PRONOM/Format/proFormatSearch.aspx?status=detailReport&id=794 PRONOM (JPX - JPEG 2000 Extended)]

===Significant characteristics===
See [[Significant characteristics of raster images]]

===Preservation action plan===
See [[Raster images]]

===Preservation issues===
*JPEG2000 image compression ranges from completely lossless to very lossy.
*Florida digital Archive has "high confidence level" in ability to preserve losslessly compressed JP2 files: [http://www.fcla.edu/digitalArchive/formatInfo.htm FDA Recommended Data File Formats].

== Encoding JPEG2000-MXF video ==

===Method one: OpenDCP===

*FFmpeg
*OpenJPEG 1.4 ([http://www.openjpeg.org/])
*OpenDCP ([http://code.google.com/p/opendcp/])

This workflow is designed around digital cinema package (DCP) production.
Most of the open-source tools are based around DCP since they're being
designed by indie filmmakers.

OpenDCP is a utility for creating DCPs. The [http://www.cinecert.com/asdcplib/ AS-DCP library] and the [http://sourceforge.net/projects/mxflib/ mxflib library] it's based on are possibilities
for someone with more programming skills than me to build an
archives-targeted tool.

The disadvantage to this route is that DCPs use separate files for audio
and video, accompanied by an XML file describing the pair, which might
make playback less convenient than a single AV MXF. The available tools
are also targeted more to film framerates, and I'm not sure if it's
possible to obtain a video FPS like 29.97 using unmodified tools.

Step 1: Decoding source video

The source video should be decoded into a series of single frames before
being encoded into JPEG2000. This can be done using Ffmpeg.

<code>ffmpeg -i invideo -vcodec tiff f%10d.tif</code>

This will produce one file for every frame in the video. Why do we do
this? Well, the various Motion JPEG2000 formats are essentially a series
of single JPEG2000 files; each frame will be encoded to JPEG2000 before
multiplexing into a video.

The audio should be decoded to uncompressed WAV. This is a simple process
so I won't go into much detail here. The command should be a variant of

<code>ffmpeg -i invideo outfile.wav</code>

Step 2: Create JP2 frames

Next, these frames should be compressed into JPEG2000. Note that this will
'''not''' be done using OpenJPEG's support for Motion JPEG2000! Motion
JPEG2000 is a specific container for single JPEG2000 frames, while MXF
wraps its frames in a different way. Technically, JPEG2000/MXF
is considered a separate format.

<code>image_to_j2k -ImgDir directory -OutFor j2c</code>

This will create JP2 frames using the default lossless compression frames,
but you may wish to customize this. Frames will be saved as .j2c files in
the same directory as the TIFF frames.

Step 3: Create MXF file

<code>opendcp_mxf -i image_directory -r framerate -o video.mxf

opendcp_mxf -i audio.wav -o audio.mxf

opendcp_xml --reel video.mxf audio.mxf</code>

This will produce two MXFs, one containing video and one containing audio.
The third command creates an accompanying XML file, mainly targeted at
digital projectors.

===Method two: GStreamer===

GStreamer can encode to and from JPEG2000 MXF. By default it uses JasPer
as the JP2 encoder, which has some limitations. I believe there may be a
project underway to replace JasPer with OpenJPEG. GStreamer's commandline
utility also has very inconvenient syntax; it's designed primarily to be
used as a library to be called by custom programs. The gst-launch utility
is mainly meant for debug purposes. For production would be best to write
a custom archives-oriented converter based on Gstream rather than use the
gst-launch utility.

The advantage to GStreamer is that it provides all of the required
functionality, and can go straight from a source video to producing an
output.

On the other hand, GStreamer currently uses JasPer for JP2 encoding. It
may be better to use it in conjunction with an external JP2 encoder.

Requirements:

GStreamer ([http://gstreamer.freedesktop.org])
gst-plugins-bad

The GStreamer debug utility is based around a chain of smaller tools
chained together. Here's a sample command which takes a video-only file,
in QuickTime format with PNG video, and turns it into a JPEG2000 MXF:

<code>gst-launch-0.10 filesrc location=video.mov ! qtdemux ! pngdec !
ffmpegcolorspace ! videoscale ! jp2kenc ! mxfmux ! filesink
location=video.mxf</code>

This goes through a series of steps to get from your source video to the
encoded video. This example does:

*filesrc location=video.mov : Tells gst-launch where to find the source
video
*qtdemux : Demultiplexes the video. This means that it "opens" the
container and extracts the tracks inside it, making them available to the
other GStreamer tools
*pngdec : Decodes PNG compressed video into uncompressed video
*ffmpegcolorspace : Interprets the colour space in the video. Necessary to
be able to pass it to other tools
*videoscale : Negotiates the size of the video. When used with no
arguments, it just ensures the next tool knows what the size is
*jp2kenc : Encodes the video stream to JPEG 2000. You can specify the
settings here if you want
*mxfmux : Combines (multiplexes) the streams being processed into an MXF
container
*filesink location=video.mxf : Saves the video to a "filesink", e.g. a
location on the hard drive

The exact commands will depend on the source video. You can also do a
combined audio/video file, but the commands for that are kind of complex
and, to be honest, I haven't quite gotten my head around it yet.

You can also decode and convert videos using GStreamer. Here's a sample
command to convert the previous video to MPEG-4 with default settings:

<code>gst-launch-0.10 filesrc location=video.mxf ! mxfdemux name=d d. !
jp2kdec ! ffmpegcolorspace ! videoscale ! ffenc_mpeg4 ! filesink
location=video.mp4</code>

===Method three: GStreamer + OpenJPEG===

Requirements:

GStreamer
gstreamer-plugins-bad
FFmpeg
OpenJPEG

As I mentioned before, GStreamer's JP2 encoder isn't ideal right now.
However, you can easily encode a video with another tool and wrap it into
MXF using GStreamer.

First, extract your video to frames with ffmpeg as described above, then
compress them to JP2 frames using the following command:

<code>image_to_j2k -ImgDir directory -OutFor j2k</code>

You should then wrap them into an MJ2 video using the following command:

<code>wrap_j2k_in_mj2 directory video.mj2</code>

This will produce a single MJ2 file using the frames you extracted. By
default the framerate will be 25fps, which is probably wrong - consult the
OpenJPEG documentation to find out how to set the framerate.

You can then open this video file using GStreamer and rewrap the frames
into an MXF using a command like the following:

<code>gst-launch-0.10 filesrc location=video.mj2 ! mj2demux ! mxfmux ! filesink
location=video.mxf</code>

This will produce an MXF with the contents as your video - no
recompression or loss of quality. Hooray!

===Areas for future work===

GStreamer seems ripe for someone to develop a custom, archives-oriented video conversion program. Any volunteers?

===More information===
[http://www.digitalpreservation.gov/formats/fdd/fdd000143.shtml Library of Congress Sustainability of Digital Formats: JPEG 2000 Part 1 (Core) jp2 File Format]

Talk:Virtual appliance instructions

2011-03-18T18:00:38Z

Misty De Meo:

I am curious why 2.5.2 Using Shared Folders was removed from the virtual appliance instructions. I wanted to try Archivematica out on local copies of some of our files, and I needed these instructions. I found them in the Google cache and they worked just fine. I would suggest putting them back! Thanks. [[User:ChristieP|ChristieP]] 10:51, 18 March 2011 (PDT)

:Many users have been having trouble using shared folders in VirtualBox with the latest version of Archivematica, so the devs have temporarily removed the instructions. I was able to get shared folders working in VMware Fusion, so I'm planning to post a set of VMware instructions. [[User:Misty De Meo|Misty De Meo]] 11:00, 18 March 2011 (PDT)

Transcoder

2011-03-10T14:51:02Z

Misty De Meo: /* Known Issues */ Changed expected fix date

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|/etc/transcoder/archivematicaFormatPolicies/
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|/usr/lib/transcoder/transcoderScripts/
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|/usr/bin/convert
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|/usr/bin/ffmpeg
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|/usr/bin/ffmpeg2theora
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|/usr/bin/unoconv
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh
|-
|DublinCore
|File name for Dublin Core metadata
|dublincore.xml
|-
|MD5FileName
|File name containing SIP MD5 checksum
|MD5checksum.txt
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|FileUUIDs.log
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies directory contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<pre>
<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>
</pre>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>''' contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

===Known Issues===

*Transcode will not normalize between codecs when using the same container. Transcoder does not normalize files which are already in an access/preservation format. Because it currently uses the file extension to identify file formats, this means that transcode erroneously conflates video containers with video codecs, and won't normalize from one file extension to the same file extension. This is expected to be fixed in Archivematica 0.8. See [http://code.google.com/p/archivematica/issues/detail?id=468 issue 468]

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T20:46:32Z

Misty De Meo: /* archivematicaFormatPolicies */ added known issues

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|/etc/transcoder/archivematicaFormatPolicies/
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|/usr/lib/transcoder/transcoderScripts/
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|/usr/bin/convert
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|/usr/bin/ffmpeg
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|/usr/bin/ffmpeg2theora
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|/usr/bin/unoconv
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh
|-
|DublinCore
|File name for Dublin Core metadata
|dublincore.xml
|-
|MD5FileName
|File name containing SIP MD5 checksum
|MD5checksum.txt
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|FileUUIDs.log
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies directory contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<pre>
<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>
</pre>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>''' contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

===Known Issues===

*Transcode will not normalize between codecs when using the same container. Transcoder does not normalize files which are already in an access/preservation format. Because it currently uses the file extension to identify file formats, this means that transcode erroneously conflates video containers with video codecs, and won't normalize from one file extension to the same file extension. This is expected to be fixed in Archivematica 0.7.1. See [http://code.google.com/p/archivematica/issues/detail?id=468 issue 468]

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T17:53:42Z

Misty De Meo: /* archivematicaFormatPolicies */ Fix bolding

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|/etc/transcoder/archivematicaFormatPolicies/
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|/usr/lib/transcoder/transcoderScripts/
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|/usr/bin/convert
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|/usr/bin/ffmpeg
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|/usr/bin/ffmpeg2theora
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|/usr/bin/unoconv
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh
|-
|DublinCore
|File name for Dublin Core metadata
|dublincore.xml
|-
|MD5FileName
|File name containing SIP MD5 checksum
|MD5checksum.txt
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|FileUUIDs.log
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies directory contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>''' contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T17:52:50Z

Misty De Meo: /* transcoderConfig.conf */ Those italics were kind of ugly

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|/etc/transcoder/archivematicaFormatPolicies/
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|/usr/lib/transcoder/transcoderScripts/
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|/usr/bin/convert
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|/usr/bin/ffmpeg
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|/usr/bin/ffmpeg2theora
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|/usr/bin/unoconv
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh
|-
|DublinCore
|File name for Dublin Core metadata
|dublincore.xml
|-
|MD5FileName
|File name containing SIP MD5 checksum
|MD5checksum.txt
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|FileUUIDs.log
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies directory contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T17:45:12Z

Misty De Meo: /* archivematicaFormatPolicies */ Folder --> Directory

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|''/etc/transcoder/archivematicaFormatPolicies/''
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|''/usr/lib/transcoder/transcoderScripts/''
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|''/usr/bin/convert''
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|''/usr/bin/ffmpeg''
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|''/usr/bin/ffmpeg2theora''
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|''/usr/bin/unoconv''
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|''/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh''
|-
|DublinCore
|File name for Dublin Core metadata
|''dublincore.xml''
|-
|MD5FileName
|File name containing SIP MD5 checksum
|''MD5checksum.txt''
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|''FileUUIDs.log''
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies directory contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T17:43:39Z

Misty De Meo: /* archivematicaFormatPolicies */

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|''/etc/transcoder/archivematicaFormatPolicies/''
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|''/usr/lib/transcoder/transcoderScripts/''
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|''/usr/bin/convert''
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|''/usr/bin/ffmpeg''
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|''/usr/bin/ffmpeg2theora''
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|''/usr/bin/unoconv''
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|''/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh''
|-
|DublinCore
|File name for Dublin Core metadata
|''dublincore.xml''
|-
|MD5FileName
|File name containing SIP MD5 checksum
|''MD5checksum.txt''
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|''FileUUIDs.log''
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies folder contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content, e.g.: <inherit></inherit>

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T17:42:48Z

Misty De Meo: /* Configuration */ added file policy XML documentation

Transcode: convert (language or information) from one form of coded representation to another.[ source: [http://oxforddictionaries.com/view/entry/m_en_gb0876960#m_en_gb0876960 Oxford English Dictionary] ]

=Overview=
The transcoder is developed by artefactual, for the purpose of normalization and generating access copies in the archivematica system. In earlier versions it was called normalizer. It will try to identify the file type by the file extension, or other metadata, and look for matching configured actions for those identified. It will then perform those actions, and exit with a zero status if it believes those actions have been completed successfully.

=Development=
Presently to manage the complexity of automating the link between file identification and actions, a database based implementation of the transcoder is being built to replace the current xml one.

=Configuration=
Configuration files are located in the /etc/transcoder/ directory.

==transcoderConfig.conf==

transcoderConfig.conf is the primary transcoder configuration file. It is a bash script which defines the variables used in the various file format policy XML files; it primarily contains paths to conversion tools and standard file names.

Variables are stored as standard bash shell script variables. Variables can be added or edited using any text editor; any new variables added become available for use in format policy XML files. They use the format:

variableName="variable contents"

Default variables:

{|
|'''Variable'''
|'''Description'''
|'''Default value'''
|-
|formatPoliciesPath
|Directory containing format policy XML files
|''/etc/transcoder/archivematicaFormatPolicies/''
|-
|transcoderScriptsDir
|Directory containing transcoder normalization scripts
|''/usr/lib/transcoder/transcoderScripts/''
|-
|convertPath
|Path to ImageMagick for image conversion. Requires a space at the end.
|''/usr/bin/convert''
|-
|ffmpegPath
|Path to ffmpeg for audio and video. Requires a space at the end.
|''/usr/bin/ffmpeg''
|-
|theoraPath
|Path to ffmpeg2theora script to create Ogg Theora and Vorbis files. Currently unused. Requires a space at the end.
|''/usr/bin/ffmpeg2theora''
|-
|unoconvPath
|Path to unoconv binary for converting document files. Currently unused. Requires a space at the end.
|''/usr/bin/unoconv''
|-
|unoconvAlternatePath
|Path to unoconv launcher script for converting document files. Requires a space at the end.
|''/usr/lib/transcoder/transcoderScripts/unoconvAlternative.sh''
|-
|DublinCore
|File name for Dublin Core metadata
|''dublincore.xml''
|-
|MD5FileName
|File name containing SIP MD5 checksum
|''MD5checksum.txt''
|-
|fileUUIDHumanReadable
|Log file containing unique IDs for items within a SIP
|''FileUUIDs.log''
|}

==archivematicaFormatPolicies==

The /etc/transcoder/archivematicaFormatPolicies folder contains XML files which control how Archivematica performs normalization. Transcoder reads the file extension of a file and selects the matching XML file to determine how to perform normalization. Note that, because normalization is based on file extension, objects with an incorrect file extension or no extension will usually fail to normalize - this is [http://code.google.com/p/archivematica/issues/detail?id=156 scheduled] to change in Archivematica 0.7.2.

The following sample configuration file illustrates the syntax:

<source lang="xml">
<formatPolicy>
<inherit></inherit>
<accessFormat>MP3</accessFormat>
<preservationFormat>WAV</preservationFormat>
<accessConversionCommand>%ffmpegPath% -i %fileFullName% -ab 192000 %accessFileDirectory%%fileTitle%.%accessFormat%</accessConversionCommand>
<preservationConversionCommand>%ffmpegPath% -i %fileFullName% %preservationFileDirectory%%fileTitle%.%preservationFormat%</preservationConversionCommand>
<preservationConversionCommand>%xenaPath%</preservationConversionCommand>
</formatPolicy>
</source>

Every format policy XML encloses its contents in the '''<formatPolicy></formatPolicy>''' tag. All content intended to be read by transcoder for normalization should be between the opening and close tags.

* '''<inherit>''' should should be used when a format shares the same normalization processes as another format. For instance, all audio files (MP3, WMA, etc.) share a common access format and preservation format and are converted using the same command. For instance, the XML files for MP3 and WMA contain only an <inherit> tag pointing to the "parent" AUDIO document that contains the commands for all audio types. If <inherit> is used, all other tags should be empty.
* '''<accessFormat>''' defines the access format defined for the specified object type. It should be expressed using the file format for that type, in capital letters. For instance, '''MP3''' is the accessFormat for audio files.
* '''<preservationFormat>''' defines the preservation format for the specified object type. It uses the same syntax as <accessFormat>
* '''<accessConversionCommand>''' contains the commandline options for launching the tool which creates an access copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.
* '''<preservationConversionCommand>contains the commandline options for launching the tool which creates a preservation copy from the specified object type. This can be customized or replaced. Any variables in use are defined by normalizationConfig.conf and should be customized there.

Note that all tags should be present, regardless if they are being used. For instance, if the format in question does not inherit another type, the inherit opening and closing tags should still be present without enclosing any content: <inherit></inherit>

TODO: learn why there can be two <preservationConversionCommand>s and update documentation

[[Category:Development documentation]]

Transcoder

2011-03-08T16:38:04Z

Misty De Meo: /* transcoderConfig.conf */ Added additional editing documentation

Transcoder

2011-03-08T16:32:51Z

Misty De Meo: Added table documenting contents of transcoderConfig.conf file