https://wiki.archivematica.org/api.php?action=feedcontributions&feedformat=atom&user=JosephArchivematica - User contributions [en]2026-06-03T07:00:08ZUser contributionsMediaWiki 1.35.4https://wiki.archivematica.org/index.php?title=Digital_forensics_image_ingest&diff=8589Digital forensics image ingest2013-06-25T20:52:03Z<p>Joseph: /* Detail */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Digital forensics image ingest<br />
[[Category:Development documentation]]<br />
<br />
<br />
Related issues: #5265<br />
<br />
<br />
== Forensics image transfer type ==<br />
<br />
* Archivematica transfer type: forensic image<br />
** One or more images make up a transfer<br />
** Repository makes image using outside imaging software prior to ingest<br />
** Some metadata from ingest process will be included, first from FTK, but later from other tools like Guymager (see metadata requirements below)<br />
* Forensic image types accepted: dd (Raw), ISO, AD1, BIN (these formats are sponsored, support for the formats listed [http://www.forensicswiki.org/wiki/Forensic_file_formats here] is desirable in future releases)<br />
<br />
== Forensics image transfer workflow ==<br />
[[File:ArchivematicaForensicImageIngest.png|900px|thumb|center|]]<br />
[[File:ArchivematicaForensicImageIngest(2).png|700px|thumb|center|]]<br />
====Detail====<br />
* User images external media outside the Archivematica workflow<br />
* User uploads image(s) into the Archivematica transfer tab of the dashboard by browsing to the appropriate transfer source directory and selecting a directory containing their image(s)<br />
* User enters transfer name and accession number<br />
* User selects MD entry template for entering MD about the imaging process <br />
** User enters MD (see MD requirements below)<br />
** User saves MD and starts transfer processes<br />
* User selects Start transfer to begin Archivematica transfer processing<br />
* Fiwalk with Fido completes the Characterize and extract metadata micro-service<br />
* Archivematica runs [http://www.forensicswiki.org/wiki/Bulk_extractor Bulk Extractor] tool('''Examine contents micro-service''') and indexes output (this is to allow for reporting and visualization in the transfer backlog search for SIP creation and/or the AIP advanced search to allow for minimal description)<br />
* Transfer micro-services complete<br />
* At Create SIP from Transfer micro-service, user selects one of two options:<br />
** If the user is an archivist/curator ready to process the image through to storage and/or access, choose Create single SIP and continue processing<br />
** If the user is uploading multiple images as part of one accession, for processing by an archivist/curator later, choose Send to backlog<br />
*** In the second scenario, once all images from an accession are in the backlog, user alerts archivist/curator that the accession is ready for further processing<br />
** Archivist searches for the accession in the transfer backlog, selects the appropriate transfers, and selects Create SIP <br />
* In ingest tab, user approves SIP creation<br />
* In ingest tab, prior to normalization, there is a decision point at '''Extract packages micro-service''' - User selects from drop-down: Extract objects from image, Do not extract objects from image, Reject<br />
** If user chooses not to extract objects, then skip micro-service decision about tool output to base normalization on, choose normalization for preservation only or no normalization, and continue standard micro-services to store AIP.<br />
** If user chooses to extract objects, Archivematica runs FITS on the extracted contents. The user continues standard workflow, choosing any of the normalization options (including manual normalization) and continues processing to storage and/or access.<br />
<br />
==Metadata requirements==<br />
When the user selects Forensic image transfer type, each image uploaded as part of the transfer will include a metadata form icon that, if selected, will open a form in another browser tab. There, the user will enter some or all of the MD indicated below in the Template for manual data entry list. <br />
<br />
* Template for manual data entry<br />
**accession number - recorded in transfer upload in dashboard<br />
**media number - manual <br />
**label text - manual (long text field)<br />
**media manufacture - manual<br />
**serial number - manual<br />
**media format - manual, could be controlled value list<br />
**media density - manual, could be controlled value list<br />
**source filesystem<br />
**notes about the imaging process<br />
**imaging interface - manual, could be controlled value list<br />
**examiner - manual, could be from Archivematica registered users<br />
**image format - manual, could be controlled value list<br />
**imaging software - manual, could be controlled value list<br />
**notes about the imaging process - manual (long text field)<br />
<br />
* Import from imaging tool FTK or fiwalk/sleuthkit<br />
**imaging date (FTK or other imaging tool output)<br />
**imaging success - Yes, Yes with errors (FTK or other imaging tool output)<br />
**image fixity (FTK or other imaging tool output<br />
**source filesystem (fiwalk)<br />
**accession data about extent (fiwalk)<br />
<br />
<br />
{| border="1" cellpadding="10" cellspacing="0" width="100%"<br />
|-<br />
!'''element'''<br />
!'''description'''<br />
!'''DACS (2013)'''<br />
!'''ISAD(G)'''<br />
!'''EAD'''<br />
!'''PREMIS 2.2'''<br />
|-<br />
|media number<br />
|repository specific alphanumeric designation assigned to individual physical media/carrier<br />
|2.1.3 local identifier - At the highest level of a multilevel description or in a single level description,<br />
provide a unique identifier for the materials being described in accordance with the<br />
institution’s administrative control system. Optionally, devise unique identifiers at lower<br />
levels of a multilevel description.<br />
|3.1.1 - Reference codes<br />
|<unitid> <br />
|<br />
|-<br />
|label text<br />
|textual transcription<br />
|7.1.2 Record, as needed, information not accommodated by any of the defined elements of description. <br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|media manufacturer<br />
|<br />
|7.1.4 - If the materials being described are in electronic form, give details of any migration or logical reformatting since its transfer to archival custody. Indicate the location of any relevant documentation. Information regarding digitization is provided in the Existence and Location of Copies Element (6.2).<br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|serial number<br />
|when applicable to external media<br />
|7.1.4 or 7.1.6 If appropriate at the file or item level of description, make a note of any important numbers borne by the unit being described. <br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|media format<br />
|a controlled value list (e.g. 3.5" floppy, 5.25" floppy, CD-R, etc)<br />
|7.1.4 <br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|media density<br />
|a controlled value list (e.g. single density, double density, quad density, high density)<br />
|7.1.4 <br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|source filesystem<br />
|a controlled file list(e.g. HFS, FAT, etc.) with the ability to add terms<br />
|7.1.4<br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|notes about the imaging process<br />
|textual field to describe more detail about the imaging process<br />
|7.1.4<br />
|3.6.1 Note<br />
|<odd>, <note><br />
|<br />
|-<br />
|imaging interface<br />
|a controlled value list (e.g. Catweasel, Firewire, USB, IDE, etc.)<br />
|<br />
|<br />
|<br />
|2.2 - Event: Image capture<br />
|-<br />
|examiner<br />
|the person doing the imaging<br />
|<br />
|<br />
|<br />
|2.2 - Agent<br />
|-<br />
|imaging date<br />
|date of imaging<br />
|<br />
|<br />
|<br />
|2.2 - Event: Image capture<br />
|-<br />
|imaging success<br />
|ex Yes/Yes, with errors<br />
|<br />
|<br />
|<br />
|2.2 - Event: Image capture<br />
|-<br />
|image format<br />
|a controlled value list (e.g. AFF3, dd/secort image, AD1, etc.) with the ability to add terms<br />
|<br />
|<br />
|<br />
|2.2 - Object<br />
|-<br />
|imaging software<br />
|a controlled value list (e.g. FTK imager 3.1.0.1514, Kryoflux, DTC 2.00 beta 9, etc.) with the ability to add terms<br />
|<br />
|<br />
|<br />
|2.2 - Agent<br />
|-<br />
|image fixity<br />
|type(s) and value(s) from FTK csv output<br />
|<br />
|<br />
|<br />
|2.2 - Object<br />
|-<br />
|}<br />
<br />
</br><br />
<br />
==Forensic image transfer tools ==<br />
<br />
===fiwalk===<br />
<br />
* Characterize and extract metadata micro-service<br />
* Use Mark Matienzo's github version which includes FIDO for format identification since fiwalk's format identification is libmagic (unsatisfactory for our purposes)<br />
<br />
Sample fiwalk xml output:<br />
<br />
<pre><br />
<br />
<?xml version='1.0' encoding='ISO-8859-1'?><br />
<fiwalk xmloutputversion='0.2'><br />
<metadata <br />
xmlns='http://example.org/myapp/' <br />
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' <br />
xmlns:dc='http://purl.org/dc/elements/1.1/'><br />
<dc:type>Disk Image</dc:type><br />
</metadata><br />
<creator><br />
<program>fiwalk</program><br />
<version>0.5.7</version><br />
<os>Darwin</os><br />
<library name="tsk" version="3.0.1"></library><br />
<library name="afflib" version="3.5.2"></library><br />
<command_line>fiwalk -x /dev/disk2</command_line><br />
</creator><br />
<source><br />
<imagefile>/dev/disk2</imagefile><br />
</source><br />
<!-- fs start: 512 --><br />
<volume offset='512'><br />
<Partition_Offset>512</Partition_Offset><br />
<block_size>512</block_size><br />
<ftype>2</ftype><br />
<ftype_str>fat12</ftype_str><br />
<block_count>5062</block_count><br />
<first_block>0</first_block><br />
<last_block>5061</last_block><br />
<fileobject><br />
<filename>README.txt</filename><br />
<id>2</id><br />
<filesize>43</filesize><br />
<partition>1</partition><br />
<alloc>1</alloc><br />
<used>1</used><br />
<inode>6</inode><br />
<type>1</type><br />
<mode>511</mode><br />
<nlink>1</nlink><br />
<uid>0</uid><br />
<gid>0</gid><br />
<mtime>1258916904</mtime><br />
<atime>1258876800</atime><br />
<crtime>1258916900</crtime><br />
<byte_runs><br />
<run file_offset='0' fs_offset='37376' img_offset='37888' len='43'/><br />
</byte_runs><br />
<hashdigest type='md5'>2bbe5c3b554b14ff710a0a2e77ce8c4d</hashdigest><br />
<hashdigest type='sha1'>b3ccdbe2db1c568e817c25bf516e3bf976a1dea6</hashdigest><br />
</fileobject><br />
</volume><br />
<!-- end of volume --><br />
<!-- clock: 0 --><br />
<runstats><br />
<user_seconds>0</user_seconds><br />
<system_seconds>0</system_seconds><br />
<maxrss>1814528</maxrss><br />
<reclaims>546</reclaims><br />
<faults>1</faults><br />
<swaps>0</swaps><br />
<inputs>56</inputs><br />
<outputs>0</outputs><br />
<stop_time>Sun Nov 22 11:08:36 2009</stop_time><br />
</runstats><br />
</fiwalk><br />
</pre><br />
===Bulk Extractor===</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8376Adding Format Identification Tools2013-05-29T22:28:27Z<p>Joseph: </p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
<br />
The basic steps this section is looking to implement are:<br />
* Make the fileID tool a selectable choice<br />
* Specify that normalization will be using that fileIDType during normalization<br />
* Identify files using that tool, with valid archivematica fileIDs (format IDs)<br />
* Use those format IDs and associated commands to normalize files.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a peice of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.<br />
<br />
For FIDO:<br />
<pre><br />
INSERT INTO `MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`, `replaces`, `lastModified`) <br />
VALUES ('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`, `replaces`, `lastModified`) <br />
VALUES ('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`, `createdTime`, `updatedTime`) <br />
VALUES ('f130c16d-d419-4063-8c8b-2e4c3ad138bb','normalizationFileIdentificationToolIdentifierTypes','FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'',NULL,'2012-10-23 19:41:23','0000-00-00 00:00:00');<br />
INSERT INTO `MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`, `replaces`, `lastModified`) <br />
VALUES ('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully',NULL,'2012-10-23 21:39:43');<br />
</pre><br />
<br />
==Run tool==<br />
The next step is to run the tool on the objects. This requires both definition of the workflow, and a task to support it.<br />
<br />
===workflow===<br />
FIDO workflow link:<br />
<pre><br />
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';<br />
<br />
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';<br />
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';<br />
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';<br />
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';<br />
SET @defaultNextChainLink = @YLink;<br />
SET @NextMicroServiceChainLink = @YLink;<br />
<br />
INSERT INTO StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)<br />
VALUES (@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');<br />
INSERT INTO TasksConfigs (pk, taskType, taskTypePKReference, description)<br />
VALUES<br />
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');<br />
INSERT INTO MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)<br />
VALUES (@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);<br />
INSERT INTO MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)<br />
VALUES (@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);<br />
SET @NextMicroServiceChainLink = @MicroServiceChainLink;<br />
</pre><br />
<br />
===Identify files script===<br />
The script must be created to identify the file and put an entry(s) in the FilesIdentifiedIDs table.<br />
For Fido this script is [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/lib/clientScripts/archivematicaFido.py archivematicaFido.py]<br />
<br />
This script must be added to the client as a supported module.<br />
<pre><br />
archivematicaFido_v0.0 = %clientScriptsDirectory%archivematicaFido.py <br />
</pre><br />
[https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]<br />
<br />
==Return==<br />
The last link, to return to the processing workflow already exists, and must be defined as the next chain to process.<br />
The link is 83484326-7be7-4f9f-b252-94553cd42370<br />
<br />
=Add FPR rules=<br />
Populating the local FPR rules can be done by creating a temporary script, and using a set of sample files with extensions. Use the FileIDs for the extensions to populate values for already in preservation/access status, and related commands.<br />
<br />
Fido example: [https://github.com/artefactual/archivematica/commit/ea02f7d6ffc420d5675fcd3bf261c63870881a65]<br />
==FileIDTypes==<br />
Remember an entry for FileIDTypes is required.<br />
<br />
<pre><br />
INSERT INTO `FileIDTypes` (`pk`, `description`, `replaces`, `lastModified`, `enabled`) VALUES ('afdbee13-eec5-4182-8c6c-f5638ee290f3','FileIDByFIDO',NULL,'2013-03-12 01:54:24',1);<br />
</pre><br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=8375Development environment2013-05-29T21:58:59Z<p>Joseph: /* Update */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete AtoM setup<br />
*** The database should already be created.<br />
*** http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Configuring and Using Archivematica=<br />
These manuals contain information about the use and configuration of Archivematica:<br />
<br/>https://www.archivematica.org/wiki/Administrator_manual<br />
<br/>https://www.archivematica.org/wiki/User_Manual<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, and reset your dev install, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update AtoM and restart its atom-worker service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
** <pre>sudo apt-get install postfix</pre><br />
** Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=8372Development environment2013-05-29T20:59:02Z<p>Joseph: /* Configuring and Using Archivematica */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete AtoM setup<br />
*** The database should already be created.<br />
*** http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Configuring and Using Archivematica=<br />
These manuals contain information about the use and configuration of Archivematica:<br />
<br/>https://www.archivematica.org/wiki/Administrator_manual<br />
<br/>https://www.archivematica.org/wiki/User_Manual<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, and reset your dev install, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update AtoM (branch 1.x) and restart its atom-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
** <pre>sudo apt-get install postfix</pre><br />
** Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=8371Development environment2013-05-29T20:58:43Z<p>Joseph: /* Update */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete AtoM setup<br />
*** The database should already be created.<br />
*** http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Configuring and Using Archivematica=<br />
These manuals contain information about the use and configuration of Archivematica:<br />
https://www.archivematica.org/wiki/Administrator_manual<br />
https://www.archivematica.org/wiki/User_Manual<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, and reset your dev install, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update AtoM (branch 1.x) and restart its atom-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
** <pre>sudo apt-get install postfix</pre><br />
** Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=8370Development environment2013-05-29T20:57:52Z<p>Joseph: /* Run */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete AtoM setup<br />
*** The database should already be created.<br />
*** http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Configuring and Using Archivematica=<br />
These manuals contain information about the use and configuration of Archivematica:<br />
https://www.archivematica.org/wiki/Administrator_manual<br />
https://www.archivematica.org/wiki/User_Manual<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update AtoM (branch 1.x) and restart its atom-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
** <pre>sudo apt-get install postfix</pre><br />
** Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8355Adding Format Identification Tools2013-05-24T23:14:52Z<p>Joseph: /* Set selection */</p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a peice of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.<br />
<br />
For FIDO:<br />
<pre><br />
INSERT INTO `MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`, `replaces`, `lastModified`) <br />
VALUES ('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`, `replaces`, `lastModified`) <br />
VALUES ('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`, `createdTime`, `updatedTime`) <br />
VALUES ('f130c16d-d419-4063-8c8b-2e4c3ad138bb','normalizationFileIdentificationToolIdentifierTypes','FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'',NULL,'2012-10-23 19:41:23','0000-00-00 00:00:00');<br />
INSERT INTO `MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`, `replaces`, `lastModified`) <br />
VALUES ('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully',NULL,'2012-10-23 21:39:43');<br />
</pre><br />
<br />
==Run tool==<br />
The next step is to run the tool on the objects. This requires both definition of the workflow, and a task to support it.<br />
<br />
===workflow===<br />
FIDO workflow link:<br />
<pre><br />
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';<br />
<br />
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';<br />
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';<br />
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';<br />
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';<br />
SET @defaultNextChainLink = @YLink;<br />
SET @NextMicroServiceChainLink = @YLink;<br />
<br />
INSERT INTO StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)<br />
VALUES (@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');<br />
INSERT INTO TasksConfigs (pk, taskType, taskTypePKReference, description)<br />
VALUES<br />
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');<br />
INSERT INTO MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)<br />
VALUES (@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);<br />
INSERT INTO MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)<br />
VALUES (@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);<br />
SET @NextMicroServiceChainLink = @MicroServiceChainLink;<br />
</pre><br />
<br />
===Identify files script===<br />
The script must be created to identify the file and put an entry(s) in the FilesIdentifiedIDs table.<br />
For Fido this script is [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/lib/clientScripts/archivematicaFido.py archivematicaFido.py]<br />
<br />
This script must be added to the client as a supported module.<br />
<pre><br />
archivematicaFido_v0.0 = %clientScriptsDirectory%archivematicaFido.py <br />
</pre><br />
[https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]<br />
<br />
==Return==<br />
The last link, to return to the processing workflow already exists, and must be defined as the next chain to process.<br />
The link is 83484326-7be7-4f9f-b252-94553cd42370<br />
<br />
=Add FPR rules=<br />
Populating the local FPR rules can be done by creating a temporary script, and using a set of sample files with extensions. Use the FileIDs for the extensions to populate values for already in preservation/access status, and related commands.<br />
<br />
Fido example: [https://github.com/artefactual/archivematica/commit/ea02f7d6ffc420d5675fcd3bf261c63870881a65]<br />
==FileIDTypes==<br />
Remember an entry for FileIDTypes is required.<br />
<br />
<pre><br />
INSERT INTO `FileIDTypes` (`pk`, `description`, `replaces`, `lastModified`, `enabled`) VALUES ('afdbee13-eec5-4182-8c6c-f5638ee290f3','FileIDByFIDO',NULL,'2013-03-12 01:54:24',1);<br />
</pre><br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8354Adding Format Identification Tools2013-05-24T23:12:27Z<p>Joseph: /* Add FPR rules */</p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a peice of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.<br />
<br />
For FIDO:<br />
<pre><br />
INSERT INTO `MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`, `replaces`, `lastModified`) VALUES ('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`, `replaces`, `lastModified`) VALUES ('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`, `createdTime`, `updatedTime`) VALUES ('f130c16d-d419-4063-8c8b-2e4c3ad138bb','normalizationFileIdentificationToolIdentifierTypes','FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'',NULL,'2012-10-23 19:41:23','0000-00-00 00:00:00');<br />
INSERT INTO `MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`, `replaces`, `lastModified`) VALUES ('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully',NULL,'2012-10-23 21:39:43');<br />
</pre><br />
<br />
==Run tool==<br />
The next step is to run the tool on the objects. This requires both definition of the workflow, and a task to support it.<br />
<br />
===workflow===<br />
FIDO workflow link:<br />
<pre><br />
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';<br />
<br />
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';<br />
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';<br />
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';<br />
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';<br />
SET @defaultNextChainLink = @YLink;<br />
SET @NextMicroServiceChainLink = @YLink;<br />
<br />
INSERT INTO StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)<br />
VALUES (@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');<br />
INSERT INTO TasksConfigs (pk, taskType, taskTypePKReference, description)<br />
VALUES<br />
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');<br />
INSERT INTO MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)<br />
VALUES (@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);<br />
INSERT INTO MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)<br />
VALUES (@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);<br />
SET @NextMicroServiceChainLink = @MicroServiceChainLink;<br />
</pre><br />
<br />
===Identify files script===<br />
The script must be created to identify the file and put an entry(s) in the FilesIdentifiedIDs table.<br />
For Fido this script is [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/lib/clientScripts/archivematicaFido.py archivematicaFido.py]<br />
<br />
This script must be added to the client as a supported module.<br />
<pre><br />
archivematicaFido_v0.0 = %clientScriptsDirectory%archivematicaFido.py <br />
</pre><br />
[https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]<br />
<br />
==Return==<br />
The last link, to return to the processing workflow already exists, and must be defined as the next chain to process.<br />
The link is 83484326-7be7-4f9f-b252-94553cd42370<br />
<br />
=Add FPR rules=<br />
Populating the local FPR rules can be done by creating a temporary script, and using a set of sample files with extensions. Use the FileIDs for the extensions to populate values for already in preservation/access status, and related commands.<br />
<br />
Fido example: [https://github.com/artefactual/archivematica/commit/ea02f7d6ffc420d5675fcd3bf261c63870881a65]<br />
==FileIDTypes==<br />
Remember an entry for FileIDTypes is required.<br />
<br />
<pre><br />
INSERT INTO `FileIDTypes` (`pk`, `description`, `replaces`, `lastModified`, `enabled`) VALUES ('afdbee13-eec5-4182-8c6c-f5638ee290f3','FileIDByFIDO',NULL,'2013-03-12 01:54:24',1);<br />
</pre><br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8353Adding Format Identification Tools2013-05-24T20:52:29Z<p>Joseph: </p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a peice of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.<br />
<br />
For FIDO:<br />
<pre><br />
INSERT INTO `MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`, `replaces`, `lastModified`) VALUES ('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`, `replaces`, `lastModified`) VALUES ('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`, `createdTime`, `updatedTime`) VALUES ('f130c16d-d419-4063-8c8b-2e4c3ad138bb','normalizationFileIdentificationToolIdentifierTypes','FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'',NULL,'2012-10-23 19:41:23','0000-00-00 00:00:00');<br />
INSERT INTO `MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`, `replaces`, `lastModified`) VALUES ('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully',NULL,'2012-10-23 21:39:43');<br />
</pre><br />
<br />
==Run tool==<br />
The next step is to run the tool on the objects. This requires both definition of the workflow, and a task to support it.<br />
<br />
===workflow===<br />
FIDO workflow link:<br />
<pre><br />
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';<br />
<br />
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';<br />
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';<br />
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';<br />
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';<br />
SET @defaultNextChainLink = @YLink;<br />
SET @NextMicroServiceChainLink = @YLink;<br />
<br />
INSERT INTO StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)<br />
VALUES (@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');<br />
INSERT INTO TasksConfigs (pk, taskType, taskTypePKReference, description)<br />
VALUES<br />
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');<br />
INSERT INTO MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)<br />
VALUES (@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);<br />
INSERT INTO MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)<br />
VALUES (@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);<br />
SET @NextMicroServiceChainLink = @MicroServiceChainLink;<br />
</pre><br />
<br />
===Identify files script===<br />
The script must be created to identify the file and put an entry(s) in the FilesIdentifiedIDs table.<br />
For Fido this script is [https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/lib/clientScripts/archivematicaFido.py archivematicaFido.py]<br />
<br />
This script must be added to the client as a supported module.<br />
<pre><br />
archivematicaFido_v0.0 = %clientScriptsDirectory%archivematicaFido.py <br />
</pre><br />
[https://github.com/artefactual/archivematica/blob/642a3df29707e014f6c9ee5b7ea64785454c76cc/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]<br />
<br />
==Return==<br />
The last link, to return to the processing workflow already exists, and must be defined as the next chain to process.<br />
The link is 83484326-7be7-4f9f-b252-94553cd42370<br />
<br />
=Add FPR rules=<br />
==FileIDTypes==<br />
<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8352Adding Format Identification Tools2013-05-24T20:42:46Z<p>Joseph: </p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
The first step in the workflow is to set the selection as the tool to use during normalization. This is done by making an insert into the unit's variables table for the variable normalizationFileIdentificationToolIdentifierTypes. The value set is a peice of a SQL query used in [https://github.com/artefactual/archivematica/blob/master/src/MCPServer/lib/linkTaskManagerSplitOnFileIdAndruleset.py linkTaskManagerSplitOnFileIdAndruleset.py] to restrict the fileIDs used to the desired type.<br />
<br />
For FIDO:<br />
<pre><br />
INSERT INTO `MicroServiceChainLinks` (`pk`, `currentTask`, `defaultNextChainLink`, `defaultPlaySound`, `microserviceGroup`, `reloadFileList`, `defaultExitMessage`, `replaces`, `lastModified`) VALUES ('982229bd-73b8-432e-a1d9-2d9d15d7287d','1e516ea6-6814-4292-9ea9-552ebfaa0d23','4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Normalize',1,'Failed',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigs` (`pk`, `taskType`, `taskTypePKReference`, `description`, `replaces`, `lastModified`) VALUES ('1e516ea6-6814-4292-9ea9-552ebfaa0d23','6f0b612c-867f-4dfd-8e43-5b35b7f882d7','f130c16d-d419-4063-8c8b-2e4c3ad138bb','Set SIP to normalize with FIDO file identification.',NULL,'2012-10-23 19:41:23');<br />
INSERT INTO `TasksConfigsSetUnitVariable` (`pk`, `variable`, `variableValue`, `microServiceChainLink`, `createdTime`, `updatedTime`) VALUES ('f130c16d-d419-4063-8c8b-2e4c3ad138bb','normalizationFileIdentificationToolIdentifierTypes','FileIDTypes.pk = \'afdbee13-eec5-4182-8c6c-f5638ee290f3\'',NULL,'2012-10-23 19:41:23','0000-00-00 00:00:00');<br />
INSERT INTO `MicroServiceChainLinksExitCodes` (`pk`, `microServiceChainLink`, `exitCode`, `nextMicroServiceChainLink`, `playSound`, `exitMessage`, `replaces`, `lastModified`) VALUES ('82c97f8d-087d-4636-9dd9-59bbc04e6520','982229bd-73b8-432e-a1d9-2d9d15d7287d',0,'4c4281a1-43cd-4c6e-b1dc-573bd1a23c43',NULL,'Completed successfully',NULL,'2012-10-23 21:39:43');<br />
</pre><br />
<br />
==Run tool==<br />
The next step is to run the tool on the objects.<br />
<br />
<pre><br />
SET @YLink = '83484326-7be7-4f9f-b252-94553cd42370';<br />
<br />
SET @TasksConfigPKReference = '46883944-8561-44d0-ac50-e1c3fd9aeb59';<br />
SET @TasksConfig = '7f786b5c-c003-4ef1-97c2-c2269a04e89a';<br />
SET @MicroServiceChainLink = '4c4281a1-43cd-4c6e-b1dc-573bd1a23c43';<br />
SET @MicroServiceChainLinksExitCodes = 'd7653bbd-cd71-473d-b09e-fdd5b36a1d65';<br />
SET @defaultNextChainLink = @YLink;<br />
SET @NextMicroServiceChainLink = @YLink;<br />
<br />
INSERT INTO StandardTasksConfigs (pk, filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)<br />
VALUES (@TasksConfigPKReference, NULL, NULL, 'objects/', FALSE, NULL, NULL, 'archivematicaFido_v0.0', '--fileUUID "%fileUUID%" --SIPUUID "%SIPUUID%" --filePath "%relativeLocation%" --eventIdentifierUUID "%taskUUID%" --date "%date%" --fileGrpUse "%fileGrpUse%"');<br />
INSERT INTO TasksConfigs (pk, taskType, taskTypePKReference, description)<br />
VALUES<br />
(@TasksConfig, 'a6b1c323-7d36-428e-846a-e7e819423577', @TasksConfigPKReference, 'Identify file formats with FIDO');<br />
INSERT INTO MicroServiceChainLinks (pk, microserviceGroup, currentTask, defaultNextChainLink)<br />
VALUES (@MicroServiceChainLink, @microserviceGroup, @TasksConfig, @defaultNextChainLink);<br />
INSERT INTO MicroServiceChainLinksExitCodes (pk, microServiceChainLink, exitCode, nextMicroServiceChainLink)<br />
VALUES (@MicroServiceChainLinksExitCodes, @MicroServiceChainLink, 0, @NextMicroServiceChainLink);<br />
SET @NextMicroServiceChainLink = @MicroServiceChainLink;<br />
</pre><br />
<br />
==Return==<br />
<br />
<br />
=Add FPR rules=<br />
==FileIDTypes==<br />
<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8351Adding Format Identification Tools2013-05-24T20:14:55Z<p>Joseph: </p>
<hr />
<div><br />
These are developer facing instructions for adding a new tool to base normalization on.<br />
=Add tool=<br />
To include a new tool in the archivematica packages, it should be a dependancy package itself.<br />
<br />
=Add workflow=<br />
The choices for file identification tool choices are made at link f4dea20e-f3fe-4a37-b20f-0e70a7bc960e.<br />
<br />
Additional choices can be added by adding entries.<br />
<pre><br />
SELECT <br />
chainAvailable, startingLink, description <br />
FROM <br />
MicroServiceChainChoice <br />
JOIN MicroServiceChains ON chainAvailable = MicroServiceChains.pk <br />
WHERE <br />
choiceAvailableAtLink = 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e';<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| chainAvailable | startingLink | description |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
| 229e34d9-3768-4b78-97b7-6cd4a2f07868 | b549130c-943b-4791-b1f6-93b837990138 | extension (default) |<br />
| c44e0251-1c69-482d-a679-669b70d09fb1 | 56b42318-3eb3-466c-8a0d-7ac272136a96 | FITS - DROID |<br />
| 1d8836cf-ac02-437c-9283-4ddb7b018810 | 37f2e794-6485-4524-a384-37b3209916ed | FITS - ffident |<br />
| d607f083-7c86-49a2-bc36-06a03db28a80 | 766b23ad-65ed-46a3-aa2e-b9bdaf3386d0 | FITS - JHOVE |<br />
| 586006d1-f3af-4b5f-9f1a-c893244fa7a9 | d7a0e33d-aa3c-435f-a6ef-8e39f2e7e3a0 | FITS - summary |<br />
| 50f47870-3932-4a88-879d-d021a24758ad | f87f13d2-8aae-45c9-bc8a-e5c32a37654e | FITS - file utility |<br />
| c76624a8-6f85-43cf-8ea7-0663502c712f | 982229bd-73b8-432e-a1d9-2d9d15d7287d | FIDO |<br />
+--------------------------------------+--------------------------------------+---------------------+<br />
</pre><br />
<br />
FIDO was added by:<br />
<pre><br />
INSERT INTO MicroServiceChains (pk, startingLink, description)<br />
VALUES ('c76624a8-6f85-43cf-8ea7-0663502c712f', '982229bd-73b8-432e-a1d9-2d9d15d7287d', 'FIDO');<br />
INSERT INTO MicroServiceChainChoice (pk, choiceAvailableAtLink, chainAvailable)<br />
VALUES ('e95b8f27-ea52-4247-bdf0-615273bc5fca', 'f4dea20e-f3fe-4a37-b20f-0e70a7bc960e', 'c76624a8-6f85-43cf-8ea7-0663502c712f');<br />
</pre><br />
<br />
<br />
==Set selection==<br />
fido 982229bd<br />
==Run tool==<br />
ID files 4c4281a1<br />
<br />
=Add FPR rules=<br />
<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Adding_Format_Identification_Tools&diff=8350Adding Format Identification Tools2013-05-24T19:02:55Z<p>Joseph: Created page with "=Add tool= =Add workflow= f4dea ==Set selection== fido 982229bd ==Run tool== ID files 4c4281a1 =Add FPR rules= Category:Development documentation"</p>
<hr />
<div>=Add tool=<br />
<br />
=Add workflow=<br />
f4dea<br />
==Set selection==<br />
fido 982229bd<br />
==Run tool==<br />
ID files 4c4281a1<br />
<br />
=Add FPR rules=<br />
<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Archivematica_0.10_Micro-services&diff=8168Archivematica 0.10 Micro-services2013-05-01T21:35:52Z<p>Joseph: /* Transfer */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 0.10 Micro-services<br />
<br />
A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 0.9 dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).<br />
<br />
</br><br />
<br />
[[Image:NormPresAccess-10.png|600px|center|thumb|Arhivematica dashboard showing a micro-service and its jobs]]<br />
<br />
</br><br />
<br />
The table below shows micro-services and jobs in Archivematica 0.10. Note that this is is only a list of micro-services; detailed user instructions are available in the [[User manual 0.10|user manual]].<br />
</br><br />
=Transfer=<br />
{| border="1" cellpadding="10" cellspacing="0" width=90%<br />
|-<br />
|- style="background-color:#cccccc;"<br />
!style="width:30%"|'''Micro-service'''<br />
!style="width:70%"|'''Description'''<br />
|-<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve Transfer'''<div class="mw-collapsible-content"><br />
Set file permissions<br />
</div><br />
</div><br />
| This is the approval step that moves the transfer into the Archivematica processing pipeline.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content"><br />
<br/>Set file permissions<br />
<br/>Move to processing directory<br />
<br/>Set transfer type<br />
<br/>Remove hidden files and directories<br />
<br/>Attempt restructure for compliance<br />
<br/>Verify transfer compliance<br />
</div><br />
</div><br />
| Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''. <br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename with transfer UUID'''<div class="mw-collapsible-content"><br />
<br/>Rename with transfer UUID<br />
</div><br />
</div><br />
| Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default Transfer processingMCP.xml'''<div class="mw-collapsible-content"><br />
<br/>Include default Transfer processingMCP.xml<br />
</div><br />
</div><br />
| Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Assign file UUIDs and checksums'''<div class="mw-collapsible-content"><br />
<br/>Assign file UUIDs to objects<br />
<br/>Assign checksums and file sizes to objects<br />
</div><br />
</div><br />
| Assigns a unique universal identifier and sha-256 checksum to each file in the ''/objects/'' directory.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer checksums'''<div class="mw-collapsible-content"><br />
<br/>Verify metadata directory checksums<br />
</div><br />
</div><br />
| Checks any checksum files that were placed in the ''/metadata/'' folder of the transfer prior to moving the transfer into Archivematica.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate METS.xml document'''<div class="mw-collapsible-content"><br />
<br/>Generate METS.xml document<br />
</div><br />
</div><br />
| Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the ''/objects/'' directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Extract packages'''<div class="mw-collapsible-content"><br />
<br/>Extract packages<br />
<br/>Extract attachments<br />
</div><br />
</div><br />
| Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content"><br />
<br/>Scan for viruses<br />
</div><br />
</div><br />
| Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in ''/sharedDirectoryStructure/failed/'' and all processing on the transfer is stopped.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Quarantine'''<div class="mw-collapsible-content"><br />
<br/>Workflow decision - send transfer to quarantine<br />
<br/>Move to quarantine<br />
<br/>Remove from quarantine<br />
</div><br />
</div><br />
| Quarantine's the transfer for a set duration, to allow virus definitions to update, before virus scan.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content"><br />
<br/>Sanitize object's file and directory names<br />
<br/>Sanitize Transfer name<br />
</div><br />
</div><br />
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content"><br />
<br/>Characterize and extract metadata<br />
<br/>Identify Files ByExtension<br />
<br/>Load labels from metadata/file_labels.csv<br />
</div><br />
</div><br />
| Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Also identifies file extensions, which are used for selecting normalization paths.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content"><br />
<br/>Index transfer contents<br />
<br/>Set file permissions<br />
<br/>Move to completedTransfers directory<br />
</div><br />
</div><br />
| Indexes transfer contents, then marks the transfer as complete.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content"><br />
<br/>Check transfer directory for objects<br />
<br/>Create SIP(s)<br />
<br/>Create SIP from transfer objects<br />
</div><br />
</div><br />
| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest).<br />
|-<br />
|}<br />
</br><br />
<br />
=Ingest=<br />
{| border="1" cellpadding="10" cellspacing="0" width=90%<br />
|-<br />
|- style="background-color:#cccccc;"<br />
!style="width:30%"|'''Micro-service'''<br />
!style="width:70%"|'''Description'''<br />
|-<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify SIP compliance'''<div class="mw-collapsible-content"><br />
<br/>Set file permissions<br />
<br/>Move to processing directory<br />
<br/>Verify SIP compliance<br />
</div><br />
</div><br />
| Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content"><br />
<br/>Rename SIP directory with SIP UUID<br />
</div><br />
</div><br />
| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content"><br />
<br/>Include default SIP processingMCP.xml<br />
</div><br />
</div><br />
| Copies the processing config file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content"><br />
<br/>Remove cache files<br />
</div><br />
</div><br />
| Removes any Thumbs.db files.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content"><br />
<br/>Identify manually normalized files<br />
<br/>Check for Service directory<br />
<br/>Check for Access directory<br />
<br/>Find options to normalize as<br />
<br/>Create DIP directory<br />
<br/>Normalize<br />
<br/>Normalize access<br />
<br/>Normalize preservation<br />
<br/>Remove files without linking information (failed normalization artifacts etc.)<br />
<br/>Create thumbnails directory<br />
<br/>Normalize thumbnails<br />
<br/>Verify checksums generated on ingest<br />
<br/>Approve normalization<br />
<br/>Check for manual normalized files<br />
<br/>Move access files to DIP<br />
<br/>Assign UUIDs to manually normalized preservation files<br />
<br/>Assign checksums to manually normalized preservation files<br />
<br/>Run FITS on manually normalized preservation files<br />
<br/>Relate manually normalized preservation files to the original files<br />
</div><br />
</div><br />
| Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content"><br />
<br/>Check for submission documentation<br />
<br/>Copy transfer submission documentation<br />
<br/>Move to processing directory<br />
<br/>Set file permissions<br />
<br/>Assign file UUIDs to submission documentation<br />
<br/>Assign checksums and file sizes to submissionDocumentation<br />
<br/>Extract packages in submission documentation<br />
<br/>Sanitize file and directory names in submission documentation<br />
<br/>Scan for viruses in submission documentation<br />
<br/>Characterize and extract metadata on submission documentation<br />
<br/>Identify Files ByExtension<br />
<br/>Normalize submission documentation to preservation format<br />
</div><br />
</div><br />
| Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content"><br />
<br/>Copy thumbnails to DIP directory<br />
<br/>Copy METS to DIP directory<br />
<br/>Set file permissions<br />
<br/>Generate DIP <br />
</div><br />
</div><br />
| Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP'''<div class="mw-collapsible-content"><br />
<br/>Upload DIP<br />
</div><br />
</div><br />
| Allows the user to choose to upload the DIP to either ICA-AtoM or CONTENTdm.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to ICA-AtoM'''<div class="mw-collapsible-content"><br />
<br/>Upload DIP <br />
<br/>Move to the uploadedDIPs directory<br />
</div><br />
</div><br />
| The user uploads the DIP to a selected description in ICA-AtoM.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to CONTENTdm'''<div class="mw-collapsible-content"><br />
<br/>Restructure DIP for CONTENTdm upload<br />
<br/>Select upload type (Project Client or direct upload)<br />
<br/>Select target CONTENTdm server<br />
<br/>Get list of collections on server<br />
<br/>Select destination collection<br />
<br/>Upload DIP to contentDM<br />
<br/>Move to the uploadedDIPs directory<br />
</div><br />
</div><br />
| The user uploads the DIP to a selected description in CONTENTdm.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content"><br />
<br/>Copy transfers metadata and logs<br />
<br/>Remove files without linking information (failed normalization artifacts etc.)<br />
<br/>Verify checksums generated on ingest<br />
<br/>Generate METS.xml document<br />
<br/>Index AIP contents<br />
<br/>Prepare AIP <br />
<br/>Select compression algorithm<br />
<br/>Select compression level<br />
<br/>Compress AIP <br />
<br/>Set bag file permissions<br />
<br/>Removed bagged files<br />
</div><br />
</div><br />
| Creates an AIP in Bagit format. Indexes the AIP, then losslessly compresses it.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content"><br />
<br/>Move to the store AIP approval directory<br />
<br/>Store AIP <br />
<br/>Store AIP location<br />
<br/>Store the AIP<br />
</div><br />
</div><br />
| Moves the AIP to ''/sharedDirectoryStructure/www/AIPsStore/'' or another specified directory. Once the AIP has been stored, a copy of it is extracted from storage to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.<br />
|- <br />
|}</div>Josephhttps://wiki.archivematica.org/index.php?title=Archivematica_0.10_Micro-services&diff=8161Archivematica 0.10 Micro-services2013-05-01T21:21:53Z<p>Joseph: /* Ingest */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > [[Technical Architecture]] > [[Micro-services]] > Archivematica 0.10 Micro-services<br />
<br />
A micro-service may consist of a number of discrete tasks, or jobs. In the Archivematica 0.9 dashboard, micro-services are always shown, while jobs may be viewed by expanding the micro-service (i.e. by clicking on the grey background behind the micro-service name).<br />
<br />
</br><br />
<br />
[[Image:NormPresAccess-10.png|600px|center|thumb|Arhivematica dashboard showing a micro-service and its jobs]]<br />
<br />
</br><br />
<br />
The table below shows micro-services and jobs in Archivematica 0.10. Note that this is is only a list of micro-services; detailed user instructions are available in the [[User manual 0.10|user manual]].<br />
</br><br />
=Transfer=<br />
{| border="1" cellpadding="10" cellspacing="0" width=90%<br />
|-<br />
|- style="background-color:#cccccc;"<br />
!style="width:30%"|'''Micro-service'''<br />
!style="width:70%"|'''Description'''<br />
|-<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Approve Transfer'''<div class="mw-collapsible-content"><br />
Set file permissions<br />
</div><br />
</div><br />
| This is the approval step that moves the transfer into the Archivematica processing pipeline.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer compliance'''<div class="mw-collapsible-content"><br />
<br/>Set file permissions<br />
<br/>Move to processing directory<br />
<br/>Set transfer type<br />
<br/>Remove hidden files and directories<br />
<br/>Attempt restructure for compliance<br />
<br/>Verify transfer compliance<br />
</div><br />
</div><br />
| Moves the transfer to a processing directory based on selected transfer type (standard, zipped bag, unzipped bag, DSPace export or maildir). Verifies that the transfer conforms to the folder structure required for processing in Archivematica and restructures if required. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''. <br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename with transfer UUID'''<div class="mw-collapsible-content"><br />
<br/>Rename with transfer UUID<br />
</div><br />
</div><br />
| Directly associates the transfer with its metadata by appending the transfer UUID to the transfer directory name.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default Transfer processingMCP.xml'''<div class="mw-collapsible-content"><br />
<br/>Include default Transfer processingMCP.xml<br />
</div><br />
</div><br />
| Adds a file named processingMCP.xml to the root of the transfer. This is a configurable xml file to pre-configure processing decisions. It can configure workflow options such as creating transfer backups, quarantining the transfer and selecting a SIP creation option.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Assign file UUIDs and checksums'''<div class="mw-collapsible-content"><br />
<br/>Assign file UUIDs to objects<br />
<br/>Assign checksums and file sizes to objects<br />
</div><br />
</div><br />
| Assigns a unique universal identifier and sha-256 checksum to each file in the ''/objects/'' directory.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify transfer checksums'''<div class="mw-collapsible-content"><br />
<br/>Verify metadata directory checksums<br />
</div><br />
</div><br />
| Checks any checksum files that were placed in the ''/metadata/'' folder of the transfer prior to moving the transfer into Archivematica.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Generate METS.xml document'''<div class="mw-collapsible-content"><br />
<br/>Generate METS.xml document<br />
</div><br />
</div><br />
| Generates a basic METS file with a fileSec and structMap to record the presence of all objects in the ''/objects/'' directory and their locations in any subdirectories. Designed to capture the original order of the transfer in the event the user chooses subsequently to delete, rename or move files or break the transfer into multiple SIPs. A copy of the METS file is automatically added to any SIP generated from the transfer.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Extract packages'''<div class="mw-collapsible-content"><br />
<br/>Extract packages<br />
<br/>Extract attachments<br />
</div><br />
</div><br />
| Extracts objects from any zipped files or other packages. Extracts attachments from maildir transfers.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Scan for viruses'''<div class="mw-collapsible-content"><br />
<br/>Scan for viruses<br />
</div><br />
</div><br />
| Uses ClamAV to scan for viruses and other malware. If a virus is found, the transfer is automatically placed in ''/sharedDirectoryStructure/failed/'' and all processing on the transfer is stopped.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Clean up names'''<div class="mw-collapsible-content"><br />
<br/>Sanitize object's file and directory names<br />
<br/>Sanitize Transfer name<br />
</div><br />
</div><br />
| Some file systems do not support unicode or other special characters in filenames. This micro-service removes prohibited characters and replaces them with dashes. Original filenames are preserved in the PREMIS metadata.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Characterize and extract metadata'''<div class="mw-collapsible-content"><br />
<br/>Characterize and extract metadata<br />
<br/>Identify Files ByExtension<br />
<br/>Load labels from metadata/file_labels.csv<br />
</div><br />
</div><br />
| Identifies and validates formats and extracts object metadata using the File Information Tool Set (FITS). Also identifies file extensions, which are used for selecting normalization paths.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Complete transfer'''<div class="mw-collapsible-content"><br />
<br/>Index transfer contents<br />
<br/>Set file permissions<br />
<br/>Move to completedTransfers directory<br />
</div><br />
</div><br />
| Indexes transfer contents, then marks the transfer as complete.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Create SIP from Transfer'''<div class="mw-collapsible-content"><br />
<br/>Check transfer directory for objects<br />
<br/>Create SIP(s)<br />
<br/>Create SIP from transfer objects<br />
</div><br />
</div><br />
| This is the approval step that moves the transfer to the SIP packaging micro-services (Ingest).<br />
|-<br />
|}<br />
</br><br />
<br />
=Ingest=<br />
{| border="1" cellpadding="10" cellspacing="0" width=90%<br />
|-<br />
|- style="background-color:#cccccc;"<br />
!style="width:30%"|'''Micro-service'''<br />
!style="width:70%"|'''Description'''<br />
|-<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Verify SIP compliance'''<div class="mw-collapsible-content"><br />
<br/>Set file permissions<br />
<br/>Move to processing directory<br />
<br/>Verify SIP compliance<br />
</div><br />
</div><br />
| Verifies that the SIP conforms to the folder structure required for processing in Archivematica. The structure is as follows: ''/logs/'', ''/metadata/'', ''/metadata/submissionDocumentation/'', ''/objects/''.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Rename SIP directory with SIP UUID'''<div class="mw-collapsible-content"><br />
<br/>Rename SIP directory with SIP UUID<br />
</div><br />
</div><br />
| Directly associates the SIP with its metadata by appending the SIP UUID to the SIP directory name.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Include default SIP processingMCP.xml'''<div class="mw-collapsible-content"><br />
<br/>Include default SIP processingMCP.xml<br />
</div><br />
</div><br />
| Copies the processing config file added to the transfer in '''Include default Transfer processingMCP.xml''', above, to the SIP.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Remove cache files'''<div class="mw-collapsible-content"><br />
<br/>Remove cache files<br />
</div><br />
</div><br />
| Removes any Thumbs.db files.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Normalize'''<div class="mw-collapsible-content"><br />
<br/>Identify manually normalized files<br />
<br/>Check for Service directory<br />
<br/>Check for Access directory<br />
<br/>Find options to normalize as<br />
<br/>Create DIP directory<br />
<br/>Normalize<br />
<br/>Normalize access<br />
<br/>Normalize preservation<br />
<br/>Remove files without linking information (failed normalization artifacts etc.)<br />
<br/>Create thumbnails directory<br />
<br/>Normalize thumbnails<br />
<br/>Verify checksums generated on ingest<br />
<br/>Approve normalization<br />
<br/>Check for manual normalized files<br />
<br/>Move access files to DIP<br />
<br/>Assign UUIDs to manually normalized preservation files<br />
<br/>Assign checksums to manually normalized preservation files<br />
<br/>Run FITS on manually normalized preservation files<br />
<br/>Relate manually normalized preservation files to the original files<br />
</div><br />
</div><br />
| Determines which normalization options are available for the SIP and presents them to the user as choices. Normalizes (i.e. generates preservation and/or access copies) based on selection. Thumbnail files are also generated during this micro-service.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Process submission documentation'''<div class="mw-collapsible-content"><br />
<br/>Check for submission documentation<br />
<br/>Copy transfer submission documentation<br />
<br/>Move to processing directory<br />
<br/>Set file permissions<br />
<br/>Assign file UUIDs to submission documentation<br />
<br/>Assign checksums and file sizes to submissionDocumentation<br />
<br/>Extract packages in submission documentation<br />
<br/>Sanitize file and directory names in submission documentation<br />
<br/>Scan for viruses in submission documentation<br />
<br/>Characterize and extract metadata on submission documentation<br />
<br/>Identify Files ByExtension<br />
<br/>Normalize submission documentation to preservation format<br />
</div><br />
</div><br />
| Processes any submission documentation included in the SIP and adds it to the ''/objects/'' directory.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare DIP'''<div class="mw-collapsible-content"><br />
<br/>Copy thumbnails to DIP directory<br />
<br/>Copy METS to DIP directory<br />
<br/>Set file permissions<br />
<br/>Generate DIP <br />
</div><br />
</div><br />
| Creates a DIP containing access copies of the objects, thumbnails and a copy of the METS file.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP'''<div class="mw-collapsible-content"><br />
<br/>Upload DIP<br />
</div><br />
</div><br />
| Allows the user to choose to upload the DIP to either ICA-AtoM or CONTENTdm.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to ICA-AtoM'''<div class="mw-collapsible-content"><br />
<br/>Upload DIP <br />
<br/>Move to the uploadedDIPs directory<br />
</div><br />
</div><br />
| The user uploads the DIP to a selected description in ICA-AtoM.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Upload DIP to CONTENTdm'''<div class="mw-collapsible-content"><br />
<br/>Restructure DIP for CONTENTdm upload<br />
<br/>Select upload type (Project Client or direct upload)<br />
<br/>Select target CONTENTdm server<br />
<br/>Get list of collections on server<br />
<br/>Select destination collection<br />
<br/>Upload DIP to contentDM<br />
<br/>Move to the uploadedDIPs directory<br />
</div><br />
</div><br />
| The user uploads the DIP to a selected description in CONTENTdm.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Prepare AIP'''<div class="mw-collapsible-content"><br />
<br/>Copy transfers metadata and logs<br />
<br/>Remove files without linking information (failed normalization artifacts etc.)<br />
<br/>Verify checksums generated on ingest<br />
<br/>Generate METS.xml document<br />
<br/>Index AIP contents<br />
<br/>Prepare AIP <br />
<br/>Select compression algorithm<br />
<br/>Select compression level<br />
<br/>Compress AIP <br />
<br/>Set bag file permissions<br />
<br/>Removed bagged files<br />
</div><br />
</div><br />
| Creates an AIP in Bagit format. Indexes the AIP, then losslessly compresses it.<br />
|- <br />
| <div class="toccolours mw-collapsible mw-collapsed">'''Store AIP'''<div class="mw-collapsible-content"><br />
<br/>Move to the store AIP approval directory<br />
<br/>Store AIP <br />
<br/>Store AIP location<br />
<br/>Store the AIP<br />
</div><br />
</div><br />
| Moves the AIP to ''/sharedDirectoryStructure/www/AIPsStore/'' or another specified directory. Once the AIP has been stored, a copy of it is extracted from storage to a local temp directory, where it is subjected to standard BagIt checks: verifyvalid, checkpayloadoxum, verifycomplete, verifypayloadmanifests, verifytagmanifests.<br />
|- <br />
|}</div>Josephhttps://wiki.archivematica.org/index.php?title=MCP_Basic_Configuration&diff=8075MCP Basic Configuration2013-04-30T22:09:28Z<p>Joseph: </p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > [[MCP]] > Editing MCP Basic Configuration<br />
<br />
<div class="status"><br />
<div><br />
Design<br />
<div class="description"><br />
This page proposes a new feature and reviews design options<br />
</div><br />
</div><div class="active"><br />
Development<br />
<div class="description"><br />
This page describes a feature that's in development<br />
</div><br />
</div><div><br />
Documentation<br />
<div class="description"><br />
This page documents an implemented feature<br />
</div><br />
</div><br />
</div><br />
<br />
==Server==<br />
===Configurations===<br />
====Server ConfigFile====<br />
/etc/archivematica/MCPServer/ServerConfig.conf<br />
<br />
====Workflow Configurations====<br />
Please see [[Creating_Custom_Workflows]]<br />
<br />
===MCP Server Section===<br />
<br />
====File Names====<br />
These shouldn't be changed.<br />
#File Names<br />
checksumsNoExtention="checksum"<br />
<br />
<br />
====Gearman Server====<br />
MCPArchivematicaServer = localhost:4730<br />
GearmanServerWorker = localhost:4730<br />
<br />
Note: to set the gearman process to listen on more interfaces than localhost, set it in '/etc/default/gearman-job-server'. I think you can set this to 1.1.1.1 for all interfaces.<br />
<br />
<br />
====Other====<br />
watchDirectoryPath: Directory containing the directories watched by the MCP.<br />
watchDirectoryPath="/var/archivematica/sharedDirectory/watchedDirectories/"<br />
<br />
sharedDirectory: directory shared on the network with MCP clients. Containing the processing directory.<br />
sharedDirectory="/var/archivematica/sharedDirectory/"<br />
<br />
processingDirectory: Directory SIPs reside in while they are actively being processed.<br />
processingDirectory="${sharedDirectory}.currentlyProcessing/"<br />
<br />
AIPsStore: Directory to store the AIPs when finished processing. This can be a mounted directory on an external storage device.<br />
AIPsStore="${sharedDirectory}AIPsStore/"<br />
<br />
The database access settings for the shared database:<br />
/etc/archivematica/archivematicaCommon/dbsettings<br />
<pre><br />
[client]<br />
user=demo<br />
password="demo"<br />
host=localhost<br />
</pre><br />
<br />
==Client==<br />
===Supported Modules===<br />
/etc/archivematica/archivematicaClient/clientConfig.conf<br />
<br>[http://code.google.com/p/archivematica/source/browse/trunk/src/MCPClient/etc/archivematicaClientModules Supported modules]<br />
<br/>/etc/archivematica/MCPClient/archivematicaClientModules<br />
<br/>It's highly recommended only the MCP server host machine client support the upload and store tasks.<br />
<br/>Change the in theLoadSupportedCommandsSpecial in clientConfig.conf on other machines<br />
<pre><br />
LoadSupportedCommandsSpecial = False<br />
</pre><br />
<br />
===Client ConfigFile===<br />
/etc/archivematica/MCPClient/clientConfig.conf<br />
Server to connect to. Specify hostname/IP and port.<br />
MCPArchivematicaServer = localhost:4730<br />
Where the shared directory on the MCP Server is mounted:<br />
sharedDirectoryMounted="/var/archivematica/sharedDirectory/"<br />
The maximum number of tasks the MCP is allowed to run on this host.<br />
* numberOfTasks is no longer used. The client will detect the number of cores on the machine, and use that number of threads. This can be overridden by changing the number to something other than 0. <br />
<pre>numberOfTasks=0</pre><br />
<br>The location of the client modules. These specify which modules this host supports.<br />
archivematicaClientModules="/etc/archivematica/MCPClient/archivematicaClientModules"<br />
The location of the client scripts.<br />
clientScriptsDirectory="/usr/lib/archivematica/MCPClient/clientScripts/"<br />
The database access settings for the shared database:<br />
/etc/archivematica/archivematicaCommon/dbsettings<br />
<pre><br />
[client]<br />
user=demo<br />
password="demo"<br />
host=localhost<br />
</pre><br />
The supportedCommandsSpecial listed in the archivematicaClientModules should only be run on one machine. This is a mechanism to limit/control processing to a single instance. This should be enabled on one, and only one client in a distributed processing system.<br />
<pre>LoadSupportedCommandsSpecial = True</pre><br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Administrator_manual_0.10&diff=8042Administrator manual 0.102013-04-30T20:15:27Z<p>Joseph: /* Processing configuration */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Administrator Manual<br />
<br />
This manual covers administrator-specific instructions for Archivematica. It will also provide help for using forms in the Administration tab of the Archivematica dashboard and the administrator capabilities in the Format Policy Registry (FPR), which you will find in the Preservation planning tab of the dashboard.<br />
<br />
For end-user instructions, please see the [[User_manual_0.10|user manual]].<br />
<br />
= Installation =<br />
* [[Installation|Instructions for installing the latest build of Archivematica on your server]]<br />
<br />
= Upgrading =<br />
* Currently, Archivematica does not support upgrading from one version to the next. A re-install is required.<br />
<br />
= Dashboard administration tab =<br />
<br />
The Archivematica administration pages, under the Administration tab of the dashboard, allows you to configure application components and manage users.<br />
<br />
== Transfer source directories ==<br />
<br />
Unlike previous versions, Archivematica v0.9 allows you to start transfers using the web interface rather than the operating system's file browser. Source files for transfers, however, can't be uploaded using the web interface: they must exist on volumes accessible to the Archivematica server.<br />
<br />
When starting a transfer you're required to select one or more directories of files to add to the transfer. To speed up the process of selecting directories, Archivematica allows you to specify "source directories". A source directory is a directory in which files and directories likely to be added to a transfer are present.<br />
<br />
To add a source directory, while on the transfer source directories page of the Administration tab in the dashboard, simply click the folder icon to expand the starting directory and navigate the interface until you find a directory you'd like to select as a source directory. Once you've found a suitable source directory, click the "Add" button to the right of the directory and it will be added.<br />
<br />
To remove a source directory, simply click the "Remove" button to the right of it in the source directory path list.<br />
<br />
== AIP storage directories ==<br />
<br />
== AtoM DIP upload ==<br />
<br />
Archivematica can upload DIPs directly to an [https://www.ica-atom.org/ AtoM] website so the contents can be accessed online. The AtoM DIP upload configuration page is where you specify the details of the AtoM installation you'd like the DIPs uploaded to (and, if using Rsync to transfer the DIP files, Rsync transfer details).<br />
<br />
The parameters that you'll most likely want to set are <code>url</code>, <code>email</code>, and <code>password</code>. These parameters, respectively, specify the destination AtoM website's URL, the email address used to log in to the website, and the password used to log in to the website.<br />
<br />
AtoM DIP upload can also use [http://en.wikipedia.org/wiki/Rsync Rsync] as a transfer mechanism. Rsync is an open source utility for efficiently transferring files. The <code>rsync-target</code> parameter is used to specify an Rsync-style target host/directory pairing, "foobar.com:~/dips/" for example. The <code>rsync-command</code> parameter is used to specify rsync connection options, "ssh -p 22222 -l user" for example. If you are using the rsync option, please see AtoM server configuration below.<br />
<br />
To set any parameters for AtoM DIP upload change the values, preserving the existing format they're specified in, in the "Command arguments" field then click "Save".<br />
<br />
Note that in AtoM, the sword plugin (Admin --> qtSwordPlugin) and job scheduling (Admin --> Settings --> Job scheduling) must both be enabled in order for AtoM to receive uploaded DIPs.<br />
<br />
=== AtoM server configuration ===<br />
<br />
This server configuration step is necessary to allow Archivematica to log in to the AtoM server without passwords, and only when the user is deploying the rsync option described above in the AtoM DIP upload section. <br />
<br />
To enable sending DIPs from Archivematica to the AtoM server:<br />
<br />
Generate SSH keys for the Archivematica user. Leave the passphrase field blank.<br />
<pre><br />
$ sudo -i -u archivematica<br />
$ cd ~<br />
$ ssh-keygen<br />
</pre><br />
<br />
Copy the contents of <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> somewhere handy, you will need it later.<br />
<br />
Now, it's time to configure the AtoM server so Archivematica can send the DIPs using SSH/rsync. For that purpose, you will create a user called <code>archivematica</code> and we are going to assign that user a restricted shell with access only to rsync:<br />
<br />
<pre><br />
$ sudo apt-get install rssh<br />
$ sudo useradd -d /home/archivematica -m -s /usr/bin/rssh archivematica<br />
$ sudo passswd -l archivematica<br />
$ sudo vim /etc/rssh.conf // Make sure that allowrsync is uncommented!<br />
</pre><br />
<br />
Add the SSH key that we generated before:<br />
<br />
<pre><br />
$ sudo mkdir /home/archivematica/.ssh<br />
$ chmod 700 /home/archivematica/.ssh/<br />
$ sudo vim /home/archivematica/.ssh/authorized_keys // Paste here the contents of id_dsa.pub<br />
$ chown -R archivematica:archivematica /home/archivematica<br />
</pre><br />
<br />
In Archivematica, make sure that you update the <code>--rsync-target</code> accordingly.<br /><br />
These are the parameters that we are passing to the upload-qubit microservice.<br /><br />
Go to the Administration > Upload DIP page in the dashboard.<br />
<br />
Generic parameters:<br />
<br />
<pre><br />
--url="http://atom-hostname/index.php" \<br />
--email="demo@example.com" \<br />
--password="demo" \<br />
--uuid="%SIPUUID%" \<br />
--rsync-target="archivematica@atom-hostname:/tmp" \<br />
--debug<br />
</pre><br />
<br />
== CONTENTdm DIP upload ==<br />
<br />
Archivematica can also upload DIPs to [http://www.contentdm.org/ CONTENTdm] websites. Multiple CONTENTdm destinations may be configured.<br />
For each possible CONTENTdm DIP upload destination, you'll specify a brief description and configuration parameters appropriate for the destination. Possible paramters include <code>%ContentdmServer%</code>, <code>%ContentdmUser%</code>, and <code>%ContentdmGroup%</code>.<br />
<br />
When changing parameters for a CONTENTdm DIP upload destination simply change the values, preserving the existing format they're specified in. To add an upload destination fill in the form at the bottom of the page with the appropriate values. When you've completed your changes click the "Save" button.<br />
<br />
== Processing configuration ==<br />
<br />
When processing a SIP or transfer, you may want to automate some of the workflow choices. Choices can be preconfigured by putting a 'processingMCP.xml' file into the root directory of a SIP/transfer. The processing configuration administration page allows you to configure the default 'processingMCP.xml' that's added to a SIP or Transfer if it doesn't alreay contain one. <br />
<br />
If a SIP or transfer is submitted with a 'processingMCP.xml' file, processing decisions will be made with the included file.<br />
<br />
The XML file format is:<br />
<pre><processingMCP><br />
<preconfiguredChoices><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - create transfer backup</appliesTo><br />
<goToChain>Do not backup transfer</goToChain><br />
</preconfiguredChoice><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - send transfer to quarantine</appliesTo><br />
<goToChain>Skip quarantine</goToChain><br />
</preconfiguredChoice><br />
<preconfiguredChoice><br />
<appliesTo>Remove from quarantine</appliesTo><br />
<goToChain>Unquarantine</goToChain><br />
<delay unitCtime="yes">50</delay><br />
</preconfiguredChoice><br />
</preconfiguredChoices><br />
</processingMCP><br />
<br />
</pre><br />
<br />
Where appliesTo is the name of the job presented in the dashboard, and goToChain is the desired selection. Note: these are case sensitive.<br />
<br />
<br />
The default processingMCP.xml file is located at '/var/archivematica/sharedDirectory/sharedMicroServiceTasksConfigs/processingMCPConfigs/defaultProcessingMCP.xml'<br />
Again, it can be modified by making selections under the processing configuration section of the administration tab in the dashboard.<br />
Information regarding the processing choices is available in the [[User_Manual | user manual]].<br />
<br />
== PREMIS agent ==<br />
<br />
The PREMIS agent name and code can be set via the administration interface.<br />
<br />
<br />
== Rest API ==<br />
<br />
In addition to automation using the processingMCP.xml file, Archivematica includes a REST API for automating transfer approval. Using this API, you can create a custom script that copies a transfer to the appropriate directory then uses the <code>curl</code> command, or some other means, to let Archivematica know that the copy is complete.<br />
<br />
=== API keys ===<br />
<br />
Use of the REST API requires the use of API keys. An API key is associated with a specific user. To generate an API key for a user:<br />
<br />
# Browse to <code>/administration/accounts/list/</code><br />
# Click the "Edit" button for the user you'd like to generate an API key for<br />
# Click the "Regenerate API key" checkbox<br />
# Click "Save"<br />
<br />
After generating an API key, you can click the "Edit" button for the user and you should see the API key.<br />
<br />
=== IP whitelist ===<br />
<br />
In addition to creating API keys, you'll need to add the IP of any computer making REST requests to the REST API whitelist. The IP whitelist can be edited in the administration interface at <code>/administration/api/</code>.<br />
<br />
=== Approving a transfer ===<br />
<br />
The REST API can be used to approve a transfer. The transfer must first be copied into the appropriate watch directory. To determine the location of the appropriate watch directory, first figure out where the shared directory is from the <code>sharedDirectory</code> value of <code>/etc/archivematica/MCPServer/serverConfig.conf</code>. Within that directory is a subdirectory <code>activeTransfers</code>. In this subdirectory are watch directories for the various transfer types.<br />
<br />
When using the REST API to approve a transfer, if a transfer type isn't specified, the transfer will be deemed a standard transfer.<br />
<br />
'''HTTP Method:''' POST<br />
<br />
'''URL:''' <code>/api/transfer/approve</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>directory</code>: directory name of the transfer<br />
<br />
<code>type</code> (optional): transfer type [standard|dspace|unzipped bag|zipped bag]<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl --data "username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6&type=standard&directory=MyTransfer" http://127.0.0.1/api/transfer/approve<br />
<br />
Example result:<br />
<br />
{"message": "Approval successful."}<br />
<br />
=== Listing unapproved transfers ===<br />
<br />
The REST API can be used to get a list of unapproved transfers. Each transfer's directory name and type is returned.<br />
<br />
'''Method:''' <code>GET</code><br />
<br />
'''URL:''' <code>/api/transfer/unapproved</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl "http://127.0.0.1/api/transfer/unapproved?username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6"<br />
<br />
Example result:<br />
<br />
{<br />
"message": "Fetched unapproved transfers successfully.",<br />
"results": [{<br />
"directory": "MyTransfer",<br />
"type": "standard"<br />
}<br />
]<br />
}<br />
== Users ==<br />
<br />
The dashboard provides a simple cookie-based user authentication system using the [https://docs.djangoproject.com/en/1.4/topics/auth/ Django authentication framework]. Access to the dashboard is limited only to logged-in users and a login page will be shown when the user is not recognized. If the application can't find any user in the database, the user creation page will be shown instead, allowing the creation of an administrator account.<br />
<br />
Users can be also created, modified and deleted from the Administration tab. Here you can manage which users have access to Archivematica and what level of access they have. Standard users are able to access all sections of the interterface except for the administration section.<br />
<br />
You can add a new user to the system by clicking the "Add new" button on the user administration page. These users won't have administrator capability. By adding a user you provide a way to access Archivematica using a username/password combination. Should you need to change a user's username or password, you can do so by clicking the "Edit" button, corresponding to the user, on the administration page. Should you need to revoke a user's access, you can click the corresponding "Delete" button.<br />
<br />
=== CLI creation of administrative users ===<br />
<br />
If you need an additional administrator user one can be created via the command-line after navigating to the <code>src/dashboard/src</code> directory in the source tree.<br />
<br />
python manage.py createsuperuser --settings='settings.common'<br />
<br />
=== CLI password resetting ===<br />
<br />
If you've forgotten the password for your administrator user, or any other user, you can change it via the command-line.<br />
<br />
python manage.py changepassword <username> --settings='settings.common'<br />
<br />
===Security===<br />
<br />
Archivematica uses [http://en.wikipedia.org/wiki/PBKDF2 PBKDF2] as the default algorithm to store passwords. This should be sufficient for most users: it's quite secure, requiring massive amounts of computing time to break. However, other algorithms could be used as the following document explains: [https://docs.djangoproject.com/en/1.4/topics/auth/#how-django-stores-passwords How Django stores passwords].<br />
<br />
Our plan is to extend this functionality in the future adding groups and granular permissions support: see issue 922 to read more about it.<br />
<br />
= Dashboard preservation planning tab =<br />
<br />
== Format Policy Registry (FPR) ==<br />
<br />
= Customization and automation =<br />
* Workflow processing decisions can be made in the processingMCP.xml file. [https://www.archivematica.org/wiki/Administrator_manual_0.9#Processing_configuration See here.]<br />
* Workflows are currently created at the development level. <br />
*: Some resources avialable<br />
*:* [[MCP_Basic_Configuration]]<br />
*:* [[MCP]]<br />
*:* [[Creating_Custom_Workflows]]<br />
*:* [[Development]]<br />
* Normalization commands can be viewed in the presveration planning tab.<br />
* Normalization paths and commands are currently editedable under the preservation planning tab in the dashboard.<br />
<br />
= Elasticsearch =<br />
<br />
Archivematica has the capability of indexing data about files contained in AIPs and this data can be [[Elasticsearch Development|accessed programatically]] for various applications.<br />
<br />
If, for whatever reason, you need to delete an ElasticSearch index please see [[ElasticSearch Administration]].<br />
<br />
If, for whatever reason, you need to delete an Elasticsearch index programmatically, this can be done with pyes using the following code.<br />
<br />
<pre><br />
import sys<br />
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")<br />
from pyes import *<br />
conn = ES('127.0.0.1:9200')<br />
<br />
try:<br />
conn.delete_index('aips')<br />
except:<br />
print "Error deleting index or index already deleted."<br />
</pre><br />
<br />
= [[Data backup]] =<br />
* How to protect your data from disaster.<br />
<br />
= Security =<br />
* It is recommended to change the default passwords.<br />
** The ICA-ATOM/ContentDM passwords can be set in the administration tab in the dashboard.<br />
** The database connection credentials can be set in the /etc/archivematica/archivematicaCommon/dbsettings. These will need to a mysql user with permissions to the MCP database.<br />
:* Database<br />
::* The database user<br />
::* /etc/archivematica/archivematicaCommon/dbsettings<br />
::* The dashboard config file?<br />
:* Content DM<br />
:* ICA-AtoM<br />
* Archivematica does not presently support secured sockets. See [https://projects.artefactual.com/issues/1645 1645], [https://projects.artefactual.com/issues/ 1345] Because of this, it's highly recommended to run Archivematica on a secure, isolated network.<br />
<br />
= Questions =<br />
<br />
== Frequently asked questions ==<br />
* [[AM_FAQ|Solutions to common questions]]<br />
<br />
== Discussion group ==<br />
* [[http://groups.google.com/group/archivematica?hl=en Discussion group] For questions not on the FAQ]</div>Josephhttps://wiki.archivematica.org/index.php?title=Administrator_manual_0.10&diff=8008Administrator manual 0.102013-04-29T23:27:53Z<p>Joseph: /* Processing configuration */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Administrator Manual<br />
<br />
This manual covers administrator-specific instructions for Archivematica. It will also provide help for using forms in the Administration tab of the Archivematica dashboard and the administrator capabilities in the Format Policy Registry (FPR), which you will find in the Preservation planning tab of the dashboard.<br />
<br />
For end-user instructions, please see the [[User_manual_0.10|user manual]].<br />
<br />
= Installation =<br />
* [[Installation|Instructions for installing the latest build of Archivematica on your server]]<br />
<br />
= Upgrading =<br />
* Currently, Archivematica does not support upgrading from one version to the next. A re-install is required.<br />
<br />
= Dashboard administration tab =<br />
<br />
The Archivematica administration pages, under the Administration tab of the dashboard, allows you to configure application components and manage users.<br />
<br />
== Transfer source directories ==<br />
<br />
Unlike previous versions, Archivematica v0.9 allows you to start transfers using the web interface rather than the operating system's file browser. Source files for transfers, however, can't be uploaded using the web interface: they must exist on volumes accessible to the Archivematica server.<br />
<br />
When starting a transfer you're required to select one or more directories of files to add to the transfer. To speed up the process of selecting directories, Archivematica allows you to specify "source directories". A source directory is a directory in which files and directories likely to be added to a transfer are present.<br />
<br />
To add a source directory, while on the transfer source directories page of the Administration tab in the dashboard, simply click the folder icon to expand the starting directory and navigate the interface until you find a directory you'd like to select as a source directory. Once you've found a suitable source directory, click the "Add" button to the right of the directory and it will be added.<br />
<br />
To remove a source directory, simply click the "Remove" button to the right of it in the source directory path list.<br />
<br />
== AIP storage directories ==<br />
<br />
== AtoM DIP upload ==<br />
<br />
Archivematica can upload DIPs directly to an [https://www.ica-atom.org/ AtoM] website so the contents can be accessed online. The AtoM DIP upload configuration page is where you specify the details of the AtoM installation you'd like the DIPs uploaded to (and, if using Rsync to transfer the DIP files, Rsync transfer details).<br />
<br />
The parameters that you'll most likely want to set are <code>url</code>, <code>email</code>, and <code>password</code>. These parameters, respectively, specify the destination AtoM website's URL, the email address used to log in to the website, and the password used to log in to the website.<br />
<br />
AtoM DIP upload can also use [http://en.wikipedia.org/wiki/Rsync Rsync] as a transfer mechanism. Rsync is an open source utility for efficiently transferring files. The <code>rsync-target</code> parameter is used to specify an Rsync-style target host/directory pairing, "foobar.com:~/dips/" for example. The <code>rsync-command</code> parameter is used to specify rsync connection options, "ssh -p 22222 -l user" for example. If you are using the rsync option, please see AtoM server configuration below.<br />
<br />
To set any parameters for AtoM DIP upload change the values, preserving the existing format they're specified in, in the "Command arguments" field then click "Save".<br />
<br />
Note that in AtoM, the sword plugin (Admin --> qtSwordPlugin) and job scheduling (Admin --> Settings --> Job scheduling) must both be enabled in order for AtoM to receive uploaded DIPs.<br />
<br />
=== AtoM server configuration ===<br />
<br />
This server configuration step is necessary to allow Archivematica to log in to the AtoM server without passwords, and only when the user is deploying the rsync option described above in the AtoM DIP upload section. <br />
<br />
To enable sending DIPs from Archivematica to the AtoM server:<br />
<br />
Generate SSH keys for the Archivematica user. Leave the passphrase field blank.<br />
<pre><br />
$ sudo -i -u archivematica<br />
$ cd ~<br />
$ ssh-keygen<br />
</pre><br />
<br />
Copy the contents of <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> somewhere handy, you will need it later.<br />
<br />
Now, it's time to configure the AtoM server so Archivematica can send the DIPs using SSH/rsync. For that purpose, you will create a user called <code>archivematica</code> and we are going to assign that user a restricted shell with access only to rsync:<br />
<br />
<pre><br />
$ sudo apt-get install rssh<br />
$ sudo useradd -d /home/archivematica -m -s /usr/bin/rssh archivematica<br />
$ sudo passswd -l archivematica<br />
$ sudo vim /etc/rssh.conf // Make sure that allowrsync is uncommented!<br />
</pre><br />
<br />
Add the SSH key that we generated before:<br />
<br />
<pre><br />
$ sudo mkdir /home/archivematica/.ssh<br />
$ chmod 700 /home/archivematica/.ssh/<br />
$ sudo vim /home/archivematica/.ssh/authorized_keys // Paste here the contents of id_dsa.pub<br />
$ chown -R archivematica:archivematica /home/archivematica<br />
</pre><br />
<br />
In Archivematica, make sure that you update the <code>--rsync-target</code> accordingly.<br /><br />
These are the parameters that we are passing to the upload-qubit microservice.<br /><br />
Go to the Administration > Upload DIP page in the dashboard.<br />
<br />
Generic parameters:<br />
<br />
<pre><br />
--url="http://atom-hostname/index.php" \<br />
--email="demo@example.com" \<br />
--password="demo" \<br />
--uuid="%SIPUUID%" \<br />
--rsync-target="archivematica@atom-hostname:/tmp" \<br />
--debug<br />
</pre><br />
<br />
== CONTENTdm DIP upload ==<br />
<br />
Archivematica can also upload DIPs to [http://www.contentdm.org/ CONTENTdm] websites. Multiple CONTENTdm destinations may be configured.<br />
For each possible CONTENTdm DIP upload destination, you'll specify a brief description and configuration parameters appropriate for the destination. Possible paramters include <code>%ContentdmServer%</code>, <code>%ContentdmUser%</code>, and <code>%ContentdmGroup%</code>.<br />
<br />
When changing parameters for a CONTENTdm DIP upload destination simply change the values, preserving the existing format they're specified in. To add an upload destination fill in the form at the bottom of the page with the appropriate values. When you've completed your changes click the "Save" button.<br />
<br />
== Processing configuration ==<br />
<br />
When processing a SIP or transfer, you may want to automate some of the workflow choices. Choices can be preconfigured by putting a 'processingMCP.xml' file into the root directory of a SIP/transfer. The processing configuration administration page allows you to configure the default 'processingMCP.xml' that's added to a SIP or Transfer if it doesn't alreay contain one. <br />
<br />
If a SIP or transfer is submitted with a 'processingMCP.xml' file, processing decisions will be made with the included file.<br />
<br />
The XML file format is:<br />
<pre><processingMCP><br />
<preconfiguredChoices><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - create transfer backup</appliesTo><br />
<goToChain>Do not backup transfer</goToChain><br />
</preconfiguredChoice><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - send transfer to quarantine</appliesTo><br />
<goToChain>Skip quarantine</goToChain><br />
</preconfiguredChoice><br />
<preconfiguredChoice><br />
<appliesTo>Remove from quarantine</appliesTo><br />
<goToChain>Unquarantine</goToChain><br />
<delay unitCtime="yes">50</delay><br />
</preconfiguredChoice><br />
</preconfiguredChoices><br />
</processingMCP><br />
<br />
</pre><br />
<br />
Where appliesTo is the name of the job presented in the dashboard, and goToChain is the desired selection. Note: these are case sensitive.<br />
<br />
<br />
The default processingMCP.xml file is located at '/var/archivematica/sharedDirectory/sharedMicroServiceTasksConfigs/processingMCPConfigs/defaultProcessingMCP.xml'<br />
Again, it can be modified by making selections under the processing configuration section of the administration tab in the dashboard.<br />
<br />
== PREMIS agent ==<br />
<br />
The PREMIS agent name and code can be set via the administration interface.<br />
<br />
<br />
== Rest API ==<br />
<br />
In addition to automation using the processingMCP.xml file, Archivematica includes a REST API for automating transfer approval. Using this API, you can create a custom script that copies a transfer to the appropriate directory then uses the <code>curl</code> command, or some other means, to let Archivematica know that the copy is complete.<br />
<br />
=== API keys ===<br />
<br />
Use of the REST API requires the use of API keys. An API key is associated with a specific user. To generate an API key for a user:<br />
<br />
# Browse to <code>/administration/accounts/list/</code><br />
# Click the "Edit" button for the user you'd like to generate an API key for<br />
# Click the "Regenerate API key" checkbox<br />
# Click "Save"<br />
<br />
After generating an API key, you can click the "Edit" button for the user and you should see the API key.<br />
<br />
=== IP whitelist ===<br />
<br />
In addition to creating API keys, you'll need to add the IP of any computer making REST requests to the REST API whitelist. The IP whitelist can be edited in the administration interface at <code>/administration/api/</code>.<br />
<br />
=== Approving a transfer ===<br />
<br />
The REST API can be used to approve a transfer. The transfer must first be copied into the appropriate watch directory. To determine the location of the appropriate watch directory, first figure out where the shared directory is from the <code>sharedDirectory</code> value of <code>/etc/archivematica/MCPServer/serverConfig.conf</code>. Within that directory is a subdirectory <code>activeTransfers</code>. In this subdirectory are watch directories for the various transfer types.<br />
<br />
When using the REST API to approve a transfer, if a transfer type isn't specified, the transfer will be deemed a standard transfer.<br />
<br />
'''HTTP Method:''' POST<br />
<br />
'''URL:''' <code>/api/transfer/approve</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>directory</code>: directory name of the transfer<br />
<br />
<code>type</code> (optional): transfer type [standard|dspace|unzipped bag|zipped bag]<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl --data "username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6&type=standard&directory=MyTransfer" http://127.0.0.1/api/transfer/approve<br />
<br />
Example result:<br />
<br />
{"message": "Approval successful."}<br />
<br />
=== Listing unapproved transfers ===<br />
<br />
The REST API can be used to get a list of unapproved transfers. Each transfer's directory name and type is returned.<br />
<br />
'''Method:''' <code>GET</code><br />
<br />
'''URL:''' <code>/api/transfer/unapproved</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl "http://127.0.0.1/api/transfer/unapproved?username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6"<br />
<br />
Example result:<br />
<br />
{<br />
"message": "Fetched unapproved transfers successfully.",<br />
"results": [{<br />
"directory": "MyTransfer",<br />
"type": "standard"<br />
}<br />
]<br />
}<br />
== Users ==<br />
<br />
The dashboard provides a simple cookie-based user authentication system using the [https://docs.djangoproject.com/en/1.4/topics/auth/ Django authentication framework]. Access to the dashboard is limited only to logged-in users and a login page will be shown when the user is not recognized. If the application can't find any user in the database, the user creation page will be shown instead, allowing the creation of an administrator account.<br />
<br />
Users can be also created, modified and deleted from the Administration tab. Here you can manage which users have access to Archivematica and what level of access they have. Standard users are able to access all sections of the interterface except for the administration section.<br />
<br />
You can add a new user to the system by clicking the "Add new" button on the user administration page. These users won't have administrator capability. By adding a user you provide a way to access Archivematica using a username/password combination. Should you need to change a user's username or password, you can do so by clicking the "Edit" button, corresponding to the user, on the administration page. Should you need to revoke a user's access, you can click the corresponding "Delete" button.<br />
<br />
=== CLI creation of administrative users ===<br />
<br />
If you need an additional administrator user one can be created via the command-line after navigating to the <code>src/dashboard/src</code> directory in the source tree.<br />
<br />
python manage.py createsuperuser --settings='settings.common'<br />
<br />
=== CLI password resetting ===<br />
<br />
If you've forgotten the password for your administrator user, or any other user, you can change it via the command-line.<br />
<br />
python manage.py changepassword <username> --settings='settings.common'<br />
<br />
===Security===<br />
<br />
Archivematica uses [http://en.wikipedia.org/wiki/PBKDF2 PBKDF2] as the default algorithm to store passwords. This should be sufficient for most users: it's quite secure, requiring massive amounts of computing time to break. However, other algorithms could be used as the following document explains: [https://docs.djangoproject.com/en/1.4/topics/auth/#how-django-stores-passwords How Django stores passwords].<br />
<br />
Our plan is to extend this functionality in the future adding groups and granular permissions support: see issue 922 to read more about it.<br />
<br />
= Dashboard preservation planning tab =<br />
<br />
== Format Policy Registry (FPR) ==<br />
<br />
= Customization and automation =<br />
* Workflow processing decisions can be made in the processingMCP.xml file. [https://www.archivematica.org/wiki/Administrator_manual_0.9#Processing_configuration See here.]<br />
* Workflows are currently created at the development level. <br />
*: Some resources avialable<br />
*:* [[MCP_Basic_Configuration]]<br />
*:* [[MCP]]<br />
*:* [[Creating_Custom_Workflows]]<br />
*:* [[Development]]<br />
* Normalization commands can be viewed in the presveration planning tab.<br />
* Normalization paths and commands are currently editedable under the preservation planning tab in the dashboard.<br />
<br />
= Elasticsearch =<br />
<br />
Archivematica has the capability of indexing data about files contained in AIPs and this data can be [[Elasticsearch Development|accessed programatically]] for various applications.<br />
<br />
If, for whatever reason, you need to delete an ElasticSearch index please see [[ElasticSearch Administration]].<br />
<br />
If, for whatever reason, you need to delete an Elasticsearch index programmatically, this can be done with pyes using the following code.<br />
<br />
<pre><br />
import sys<br />
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")<br />
from pyes import *<br />
conn = ES('127.0.0.1:9200')<br />
<br />
try:<br />
conn.delete_index('aips')<br />
except:<br />
print "Error deleting index or index already deleted."<br />
</pre><br />
<br />
= [[Data backup]] =<br />
* How to protect your data from disaster.<br />
<br />
= Security =<br />
* It is recommended to change the default passwords.<br />
** The ICA-ATOM/ContentDM passwords can be set in the administration tab in the dashboard.<br />
** The database connection credentials can be set in the /etc/archivematica/archivematicaCommon/dbsettings. These will need to a mysql user with permissions to the MCP database.<br />
:* Database<br />
::* The database user<br />
::* /etc/archivematica/archivematicaCommon/dbsettings<br />
::* The dashboard config file?<br />
:* Content DM<br />
:* ICA-AtoM<br />
* Archivematica does not presently support secured sockets. See [https://projects.artefactual.com/issues/1645 1645], [https://projects.artefactual.com/issues/ 1345] Because of this, it's highly recommended to run Archivematica on a secure, isolated network.<br />
<br />
= Questions =<br />
<br />
== Frequently asked questions ==<br />
* [[AM_FAQ|Solutions to common questions]]<br />
<br />
== Discussion group ==<br />
* [[http://groups.google.com/group/archivematica?hl=en Discussion group] For questions not on the FAQ]</div>Josephhttps://wiki.archivematica.org/index.php?title=Administrator_manual_0.10&diff=7989Administrator manual 0.102013-04-29T22:36:24Z<p>Joseph: /* Security */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Administrator Manual<br />
<br />
This manual covers administrator-specific instructions for Archivematica. It will also provide help for using forms in the Administration tab of the Archivematica dashboard and the administrator capabilities in the Format Policy Registry (FPR), which you will find in the Preservation planning tab of the dashboard.<br />
<br />
For end-user instructions, please see the [[User_manual_0.10|user manual]].<br />
<br />
= Installation =<br />
* [[Installation|Instructions for installing the latest build of Archivematica on your server]]<br />
<br />
= Upgrading =<br />
* Currently, Archivematica does not support upgrading from one version to the next. A re-install is required.<br />
<br />
= Dashboard administration tab =<br />
<br />
The Archivematica administration pages, under the Administration tab of the dashboard, allows you to configure application components and manage users.<br />
<br />
== Transfer source directories ==<br />
<br />
Unlike previous versions, Archivematica v0.9 allows you to start transfers using the web interface rather than the operating system's file browser. Source files for transfers, however, can't be uploaded using the web interface: they must exist on volumes accessible to the Archivematica server.<br />
<br />
When starting a transfer you're required to select one or more directories of files to add to the transfer. To speed up the process of selecting directories, Archivematica allows you to specify "source directories". A source directory is a directory in which files and directories likely to be added to a transfer are present.<br />
<br />
To add a source directory, while on the transfer source directories page of the Administration tab in the dashboard, simply click the folder icon to expand the starting directory and navigate the interface until you find a directory you'd like to select as a source directory. Once you've found a suitable source directory, click the "Add" button to the right of the directory and it will be added.<br />
<br />
To remove a source directory, simply click the "Remove" button to the right of it in the source directory path list.<br />
<br />
== AIP storage directories ==<br />
<br />
== AtoM DIP upload ==<br />
<br />
Archivematica can upload DIPs directly to an [https://www.ica-atom.org/ AtoM] website so the contents can be accessed online. The AtoM DIP upload configuration page is where you specify the details of the AtoM installation you'd like the DIPs uploaded to (and, if using Rsync to transfer the DIP files, Rsync transfer details).<br />
<br />
The parameters that you'll most likely want to set are <code>url</code>, <code>email</code>, and <code>password</code>. These parameters, respectively, specify the destination AtoM website's URL, the email address used to log in to the website, and the password used to log in to the website.<br />
<br />
AtoM DIP upload can also use [http://en.wikipedia.org/wiki/Rsync Rsync] as a transfer mechanism. Rsync is an open source utility for efficiently transferring files. The <code>rsync-target</code> parameter is used to specify an Rsync-style target host/directory pairing, "foobar.com:~/dips/" for example. The <code>rsync-command</code> parameter is used to specify rsync connection options, "ssh -p 22222 -l user" for example. If you are using the rsync option, please see AtoM server configuration below.<br />
<br />
To set any parameters for AtoM DIP upload change the values, preserving the existing format they're specified in, in the "Command arguments" field then click "Save".<br />
<br />
Note that in AtoM, the sword plugin (Admin --> qtSwordPlugin) and job scheduling (Admin --> Settings --> Job scheduling) must both be enabled in order for AtoM to receive uploaded DIPs.<br />
<br />
=== AtoM server configuration ===<br />
<br />
This server configuration step is necessary to allow Archivematica to log in to the AtoM server without passwords, and only when the user is deploying the rsync option described above in the AtoM DIP upload section. <br />
<br />
To enable sending DIPs from Archivematica to the AtoM server:<br />
<br />
Generate SSH keys for the Archivematica user. Leave the passphrase field blank.<br />
<pre><br />
$ sudo -i -u archivematica<br />
$ cd ~<br />
$ ssh-keygen<br />
</pre><br />
<br />
Copy the contents of <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> somewhere handy, you will need it later.<br />
<br />
Now, it's time to configure the AtoM server so Archivematica can send the DIPs using SSH/rsync. For that purpose, you will create a user called <code>archivematica</code> and we are going to assign that user a restricted shell with access only to rsync:<br />
<br />
<pre><br />
$ sudo apt-get install rssh<br />
$ sudo useradd -d /home/archivematica -m -s /usr/bin/rssh archivematica<br />
$ sudo passswd -l archivematica<br />
$ sudo vim /etc/rssh.conf // Make sure that allowrsync is uncommented!<br />
</pre><br />
<br />
Add the SSH key that we generated before:<br />
<br />
<pre><br />
$ sudo mkdir /home/archivematica/.ssh<br />
$ chmod 700 /home/archivematica/.ssh/<br />
$ sudo vim /home/archivematica/.ssh/authorized_keys // Paste here the contents of id_dsa.pub<br />
$ chown -R archivematica:archivematica /home/archivematica<br />
</pre><br />
<br />
In Archivematica, make sure that you update the <code>--rsync-target</code> accordingly.<br /><br />
These are the parameters that we are passing to the upload-qubit microservice.<br /><br />
Go to the Administration > Upload DIP page in the dashboard.<br />
<br />
Generic parameters:<br />
<br />
<pre><br />
--url="http://atom-hostname/index.php" \<br />
--email="demo@example.com" \<br />
--password="demo" \<br />
--uuid="%SIPUUID%" \<br />
--rsync-target="archivematica@atom-hostname:/tmp" \<br />
--debug<br />
</pre><br />
<br />
== CONTENTdm DIP upload ==<br />
<br />
Archivematica can also upload DIPs to [http://www.contentdm.org/ CONTENTdm] websites. Multiple CONTENTdm destinations may be configured.<br />
For each possible CONTENTdm DIP upload destination, you'll specify a brief description and configuration parameters appropriate for the destination. Possible paramters include <code>%ContentdmServer%</code>, <code>%ContentdmUser%</code>, and <code>%ContentdmGroup%</code>.<br />
<br />
When changing parameters for a CONTENTdm DIP upload destination simply change the values, preserving the existing format they're specified in. To add an upload destination fill in the form at the bottom of the page with the appropriate values. When you've completed your changes click the "Save" button.<br />
<br />
== Processing configuration ==<br />
<br />
When processing a SIP or transfer, you may want to automate some of the workflow choices. Choices can be preconfigured by putting a 'processingMCP.xml' file into the root directory of a SIP/transfer. The processing configuration administration page allows you to configure the default 'processingMCP.xml' that's added to a SIP or Transfer if it doesn't alreay contain one. If a SIP or transfer is submitted with a 'processingMCP.xml' file, however, processing decisions will be made with the included file.<br />
<br />
Pre-configured choices made here are based off the choices that appear in the dashboard.<br />
<br />
The format is:<br />
<pre><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - send transfer to quarantine</appliesTo><br />
<goToChain>Skip quarantine</goToChain><br />
</preconfiguredChoice><br />
</pre><br />
<br />
Where appliesTo is the name of the job presented in the dashboard, and goToChain is the desired selection. Note: these are case sensitive.<br />
<br />
== PREMIS agent ==<br />
<br />
The PREMIS agent name and code can be set via the administration interface.<br />
<br />
<br />
== Rest API ==<br />
<br />
In addition to automation using the processingMCP.xml file, Archivematica includes a REST API for automating transfer approval. Using this API, you can create a custom script that copies a transfer to the appropriate directory then uses the <code>curl</code> command, or some other means, to let Archivematica know that the copy is complete.<br />
<br />
=== API keys ===<br />
<br />
Use of the REST API requires the use of API keys. An API key is associated with a specific user. To generate an API key for a user:<br />
<br />
# Browse to <code>/administration/accounts/list/</code><br />
# Click the "Edit" button for the user you'd like to generate an API key for<br />
# Click the "Regenerate API key" checkbox<br />
# Click "Save"<br />
<br />
After generating an API key, you can click the "Edit" button for the user and you should see the API key.<br />
<br />
=== IP whitelist ===<br />
<br />
In addition to creating API keys, you'll need to add the IP of any computer making REST requests to the REST API whitelist. The IP whitelist can be edited in the administration interface at <code>/administration/api/</code>.<br />
<br />
=== Approving a transfer ===<br />
<br />
The REST API can be used to approve a transfer. The transfer must first be copied into the appropriate watch directory. To determine the location of the appropriate watch directory, first figure out where the shared directory is from the <code>sharedDirectory</code> value of <code>/etc/archivematica/MCPServer/serverConfig.conf</code>. Within that directory is a subdirectory <code>activeTransfers</code>. In this subdirectory are watch directories for the various transfer types.<br />
<br />
When using the REST API to approve a transfer, if a transfer type isn't specified, the transfer will be deemed a standard transfer.<br />
<br />
'''HTTP Method:''' POST<br />
<br />
'''URL:''' <code>/api/transfer/approve</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>directory</code>: directory name of the transfer<br />
<br />
<code>type</code> (optional): transfer type [standard|dspace|unzipped bag|zipped bag]<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl --data "username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6&type=standard&directory=MyTransfer" http://127.0.0.1/api/transfer/approve<br />
<br />
Example result:<br />
<br />
{"message": "Approval successful."}<br />
<br />
=== Listing unapproved transfers ===<br />
<br />
The REST API can be used to get a list of unapproved transfers. Each transfer's directory name and type is returned.<br />
<br />
'''Method:''' <code>GET</code><br />
<br />
'''URL:''' <code>/api/transfer/unapproved</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl "http://127.0.0.1/api/transfer/unapproved?username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6"<br />
<br />
Example result:<br />
<br />
{<br />
"message": "Fetched unapproved transfers successfully.",<br />
"results": [{<br />
"directory": "MyTransfer",<br />
"type": "standard"<br />
}<br />
]<br />
}<br />
== Users ==<br />
<br />
The dashboard provides a simple cookie-based user authentication system using the [https://docs.djangoproject.com/en/1.4/topics/auth/ Django authentication framework]. Access to the dashboard is limited only to logged-in users and a login page will be shown when the user is not recognized. If the application can't find any user in the database, the user creation page will be shown instead, allowing the creation of an administrator account.<br />
<br />
Users can be also created, modified and deleted from the Administration tab. Here you can manage which users have access to Archivematica and what level of access they have. Standard users are able to access all sections of the interterface except for the administration section.<br />
<br />
You can add a new user to the system by clicking the "Add new" button on the user administration page. These users won't have administrator capability. By adding a user you provide a way to access Archivematica using a username/password combination. Should you need to change a user's username or password, you can do so by clicking the "Edit" button, corresponding to the user, on the administration page. Should you need to revoke a user's access, you can click the corresponding "Delete" button.<br />
<br />
=== CLI creation of administrative users ===<br />
<br />
If you need an additional administrator user one can be created via the command-line after navigating to the <code>src/dashboard/src</code> directory in the source tree.<br />
<br />
python manage.py createsuperuser --settings='settings.common'<br />
<br />
=== CLI password resetting ===<br />
<br />
If you've forgotten the password for your administrator user, or any other user, you can change it via the command-line.<br />
<br />
python manage.py changepassword <username> --settings='settings.common'<br />
<br />
===Security===<br />
<br />
Archivematica uses [http://en.wikipedia.org/wiki/PBKDF2 PBKDF2] as the default algorithm to store passwords. This should be sufficient for most users: it's quite secure, requiring massive amounts of computing time to break. However, other algorithms could be used as the following document explains: [https://docs.djangoproject.com/en/1.4/topics/auth/#how-django-stores-passwords How Django stores passwords].<br />
<br />
Our plan is to extend this functionality in the future adding groups and granular permissions support: see issue 922 to read more about it.<br />
<br />
= Dashboard preservation planning tab =<br />
<br />
== Format Policy Registry (FPR) ==<br />
<br />
= Customization and automation =<br />
* Workflow processing decisions can be made in the processingMCP.xml file. [https://www.archivematica.org/wiki/Administrator_manual_0.9#Processing_configuration See here.]<br />
* Workflows are currently created at the development level. <br />
*: Some resources avialable<br />
*:* [[MCP_Basic_Configuration]]<br />
*:* [[MCP]]<br />
*:* [[Creating_Custom_Workflows]]<br />
*:* [[Development]]<br />
* Normalization commands can be viewed in the presveration planning tab.<br />
* Normalization paths and commands are currently editedable under the preservation planning tab in the dashboard.<br />
<br />
= Elasticsearch =<br />
<br />
Archivematica has the capability of indexing data about files contained in AIPs and this data can be [[Elasticsearch Development|accessed programatically]] for various applications.<br />
<br />
If, for whatever reason, you need to delete an ElasticSearch index please see [[ElasticSearch Administration]].<br />
<br />
If, for whatever reason, you need to delete an Elasticsearch index programmatically, this can be done with pyes using the following code.<br />
<br />
<pre><br />
import sys<br />
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")<br />
from pyes import *<br />
conn = ES('127.0.0.1:9200')<br />
<br />
try:<br />
conn.delete_index('aips')<br />
except:<br />
print "Error deleting index or index already deleted."<br />
</pre><br />
<br />
= [[Data backup]] =<br />
* How to protect your data from disaster.<br />
<br />
= Security =<br />
* It is recommended to change the default passwords.<br />
** The ICA-ATOM/ContentDM passwords can be set in the administration tab in the dashboard.<br />
** The database connection credentials can be set in the /etc/archivematica/archivematicaCommon/dbsettings. These will need to a mysql user with permissions to the MCP database.<br />
:* Database<br />
::* The database user<br />
::* /etc/archivematica/archivematicaCommon/dbsettings<br />
::* The dashboard config file?<br />
:* Content DM<br />
:* ICA-AtoM<br />
* Archivematica does not presently support secured sockets. See [https://projects.artefactual.com/issues/1645 1645], [https://projects.artefactual.com/issues/ 1345] Because of this, it's highly recommended to run Archivematica on a secure, isolated network.<br />
<br />
= Questions =<br />
<br />
== Frequently asked questions ==<br />
* [[AM_FAQ|Solutions to common questions]]<br />
<br />
== Discussion group ==<br />
* [[http://groups.google.com/group/archivematica?hl=en Discussion group] For questions not on the FAQ]</div>Josephhttps://wiki.archivematica.org/index.php?title=Administrator_manual_0.10&diff=7985Administrator manual 0.102013-04-29T22:32:08Z<p>Joseph: /* Customization and automation */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Administrator Manual<br />
<br />
This manual covers administrator-specific instructions for Archivematica. It will also provide help for using forms in the Administration tab of the Archivematica dashboard and the administrator capabilities in the Format Policy Registry (FPR), which you will find in the Preservation planning tab of the dashboard.<br />
<br />
For end-user instructions, please see the [[User_manual_0.10|user manual]].<br />
<br />
= Installation =<br />
* [[Installation|Instructions for installing the latest build of Archivematica on your server]]<br />
<br />
= Upgrading =<br />
* Currently, Archivematica does not support upgrading from one version to the next. A re-install is required.<br />
<br />
= Dashboard administration tab =<br />
<br />
The Archivematica administration pages, under the Administration tab of the dashboard, allows you to configure application components and manage users.<br />
<br />
== Transfer source directories ==<br />
<br />
Unlike previous versions, Archivematica v0.9 allows you to start transfers using the web interface rather than the operating system's file browser. Source files for transfers, however, can't be uploaded using the web interface: they must exist on volumes accessible to the Archivematica server.<br />
<br />
When starting a transfer you're required to select one or more directories of files to add to the transfer. To speed up the process of selecting directories, Archivematica allows you to specify "source directories". A source directory is a directory in which files and directories likely to be added to a transfer are present.<br />
<br />
To add a source directory, while on the transfer source directories page of the Administration tab in the dashboard, simply click the folder icon to expand the starting directory and navigate the interface until you find a directory you'd like to select as a source directory. Once you've found a suitable source directory, click the "Add" button to the right of the directory and it will be added.<br />
<br />
To remove a source directory, simply click the "Remove" button to the right of it in the source directory path list.<br />
<br />
== AIP storage directories ==<br />
<br />
== AtoM DIP upload ==<br />
<br />
Archivematica can upload DIPs directly to an [https://www.ica-atom.org/ AtoM] website so the contents can be accessed online. The AtoM DIP upload configuration page is where you specify the details of the AtoM installation you'd like the DIPs uploaded to (and, if using Rsync to transfer the DIP files, Rsync transfer details).<br />
<br />
The parameters that you'll most likely want to set are <code>url</code>, <code>email</code>, and <code>password</code>. These parameters, respectively, specify the destination AtoM website's URL, the email address used to log in to the website, and the password used to log in to the website.<br />
<br />
AtoM DIP upload can also use [http://en.wikipedia.org/wiki/Rsync Rsync] as a transfer mechanism. Rsync is an open source utility for efficiently transferring files. The <code>rsync-target</code> parameter is used to specify an Rsync-style target host/directory pairing, "foobar.com:~/dips/" for example. The <code>rsync-command</code> parameter is used to specify rsync connection options, "ssh -p 22222 -l user" for example. If you are using the rsync option, please see AtoM server configuration below.<br />
<br />
To set any parameters for AtoM DIP upload change the values, preserving the existing format they're specified in, in the "Command arguments" field then click "Save".<br />
<br />
Note that in AtoM, the sword plugin (Admin --> qtSwordPlugin) and job scheduling (Admin --> Settings --> Job scheduling) must both be enabled in order for AtoM to receive uploaded DIPs.<br />
<br />
=== AtoM server configuration ===<br />
<br />
This server configuration step is necessary to allow Archivematica to log in to the AtoM server without passwords, and only when the user is deploying the rsync option described above in the AtoM DIP upload section. <br />
<br />
To enable sending DIPs from Archivematica to the AtoM server:<br />
<br />
Generate SSH keys for the Archivematica user. Leave the passphrase field blank.<br />
<pre><br />
$ sudo -i -u archivematica<br />
$ cd ~<br />
$ ssh-keygen<br />
</pre><br />
<br />
Copy the contents of <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> somewhere handy, you will need it later.<br />
<br />
Now, it's time to configure the AtoM server so Archivematica can send the DIPs using SSH/rsync. For that purpose, you will create a user called <code>archivematica</code> and we are going to assign that user a restricted shell with access only to rsync:<br />
<br />
<pre><br />
$ sudo apt-get install rssh<br />
$ sudo useradd -d /home/archivematica -m -s /usr/bin/rssh archivematica<br />
$ sudo passswd -l archivematica<br />
$ sudo vim /etc/rssh.conf // Make sure that allowrsync is uncommented!<br />
</pre><br />
<br />
Add the SSH key that we generated before:<br />
<br />
<pre><br />
$ sudo mkdir /home/archivematica/.ssh<br />
$ chmod 700 /home/archivematica/.ssh/<br />
$ sudo vim /home/archivematica/.ssh/authorized_keys // Paste here the contents of id_dsa.pub<br />
$ chown -R archivematica:archivematica /home/archivematica<br />
</pre><br />
<br />
In Archivematica, make sure that you update the <code>--rsync-target</code> accordingly.<br /><br />
These are the parameters that we are passing to the upload-qubit microservice.<br /><br />
Go to the Administration > Upload DIP page in the dashboard.<br />
<br />
Generic parameters:<br />
<br />
<pre><br />
--url="http://atom-hostname/index.php" \<br />
--email="demo@example.com" \<br />
--password="demo" \<br />
--uuid="%SIPUUID%" \<br />
--rsync-target="archivematica@atom-hostname:/tmp" \<br />
--debug<br />
</pre><br />
<br />
== CONTENTdm DIP upload ==<br />
<br />
Archivematica can also upload DIPs to [http://www.contentdm.org/ CONTENTdm] websites. Multiple CONTENTdm destinations may be configured.<br />
For each possible CONTENTdm DIP upload destination, you'll specify a brief description and configuration parameters appropriate for the destination. Possible paramters include <code>%ContentdmServer%</code>, <code>%ContentdmUser%</code>, and <code>%ContentdmGroup%</code>.<br />
<br />
When changing parameters for a CONTENTdm DIP upload destination simply change the values, preserving the existing format they're specified in. To add an upload destination fill in the form at the bottom of the page with the appropriate values. When you've completed your changes click the "Save" button.<br />
<br />
== Processing configuration ==<br />
<br />
When processing a SIP or transfer, you may want to automate some of the workflow choices. Choices can be preconfigured by putting a 'processingMCP.xml' file into the root directory of a SIP/transfer. The processing configuration administration page allows you to configure the default 'processingMCP.xml' that's added to a SIP or Transfer if it doesn't alreay contain one. If a SIP or transfer is submitted with a 'processingMCP.xml' file, however, processing decisions will be made with the included file.<br />
<br />
Pre-configured choices made here are based off the choices that appear in the dashboard.<br />
<br />
The format is:<br />
<pre><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - send transfer to quarantine</appliesTo><br />
<goToChain>Skip quarantine</goToChain><br />
</preconfiguredChoice><br />
</pre><br />
<br />
Where appliesTo is the name of the job presented in the dashboard, and goToChain is the desired selection. Note: these are case sensitive.<br />
<br />
== PREMIS agent ==<br />
<br />
The PREMIS agent name and code can be set via the administration interface.<br />
<br />
<br />
== Rest API ==<br />
<br />
In addition to automation using the processingMCP.xml file, Archivematica includes a REST API for automating transfer approval. Using this API, you can create a custom script that copies a transfer to the appropriate directory then uses the <code>curl</code> command, or some other means, to let Archivematica know that the copy is complete.<br />
<br />
=== API keys ===<br />
<br />
Use of the REST API requires the use of API keys. An API key is associated with a specific user. To generate an API key for a user:<br />
<br />
# Browse to <code>/administration/accounts/list/</code><br />
# Click the "Edit" button for the user you'd like to generate an API key for<br />
# Click the "Regenerate API key" checkbox<br />
# Click "Save"<br />
<br />
After generating an API key, you can click the "Edit" button for the user and you should see the API key.<br />
<br />
=== IP whitelist ===<br />
<br />
In addition to creating API keys, you'll need to add the IP of any computer making REST requests to the REST API whitelist. The IP whitelist can be edited in the administration interface at <code>/administration/api/</code>.<br />
<br />
=== Approving a transfer ===<br />
<br />
The REST API can be used to approve a transfer. The transfer must first be copied into the appropriate watch directory. To determine the location of the appropriate watch directory, first figure out where the shared directory is from the <code>sharedDirectory</code> value of <code>/etc/archivematica/MCPServer/serverConfig.conf</code>. Within that directory is a subdirectory <code>activeTransfers</code>. In this subdirectory are watch directories for the various transfer types.<br />
<br />
When using the REST API to approve a transfer, if a transfer type isn't specified, the transfer will be deemed a standard transfer.<br />
<br />
'''HTTP Method:''' POST<br />
<br />
'''URL:''' <code>/api/transfer/approve</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>directory</code>: directory name of the transfer<br />
<br />
<code>type</code> (optional): transfer type [standard|dspace|unzipped bag|zipped bag]<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl --data "username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6&type=standard&directory=MyTransfer" http://127.0.0.1/api/transfer/approve<br />
<br />
Example result:<br />
<br />
{"message": "Approval successful."}<br />
<br />
=== Listing unapproved transfers ===<br />
<br />
The REST API can be used to get a list of unapproved transfers. Each transfer's directory name and type is returned.<br />
<br />
'''Method:''' <code>GET</code><br />
<br />
'''URL:''' <code>/api/transfer/unapproved</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl "http://127.0.0.1/api/transfer/unapproved?username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6"<br />
<br />
Example result:<br />
<br />
{<br />
"message": "Fetched unapproved transfers successfully.",<br />
"results": [{<br />
"directory": "MyTransfer",<br />
"type": "standard"<br />
}<br />
]<br />
}<br />
== Users ==<br />
<br />
The dashboard provides a simple cookie-based user authentication system using the [https://docs.djangoproject.com/en/1.4/topics/auth/ Django authentication framework]. Access to the dashboard is limited only to logged-in users and a login page will be shown when the user is not recognized. If the application can't find any user in the database, the user creation page will be shown instead, allowing the creation of an administrator account.<br />
<br />
Users can be also created, modified and deleted from the Administration tab. Here you can manage which users have access to Archivematica and what level of access they have. Standard users are able to access all sections of the interterface except for the administration section.<br />
<br />
You can add a new user to the system by clicking the "Add new" button on the user administration page. These users won't have administrator capability. By adding a user you provide a way to access Archivematica using a username/password combination. Should you need to change a user's username or password, you can do so by clicking the "Edit" button, corresponding to the user, on the administration page. Should you need to revoke a user's access, you can click the corresponding "Delete" button.<br />
<br />
=== CLI creation of administrative users ===<br />
<br />
If you need an additional administrator user one can be created via the command-line after navigating to the <code>src/dashboard/src</code> directory in the source tree.<br />
<br />
python manage.py createsuperuser --settings='settings.common'<br />
<br />
=== CLI password resetting ===<br />
<br />
If you've forgotten the password for your administrator user, or any other user, you can change it via the command-line.<br />
<br />
python manage.py changepassword <username> --settings='settings.common'<br />
<br />
===Security===<br />
<br />
Archivematica uses [http://en.wikipedia.org/wiki/PBKDF2 PBKDF2] as the default algorithm to store passwords. This should be sufficient for most users: it's quite secure, requiring massive amounts of computing time to break. However, other algorithms could be used as the following document explains: [https://docs.djangoproject.com/en/1.4/topics/auth/#how-django-stores-passwords How Django stores passwords].<br />
<br />
Our plan is to extend this functionality in the future adding groups and granular permissions support: see issue 922 to read more about it.<br />
<br />
= Dashboard preservation planning tab =<br />
<br />
== Format Policy Registry (FPR) ==<br />
<br />
= Customization and automation =<br />
* Workflow processing decisions can be made in the processingMCP.xml file. [https://www.archivematica.org/wiki/Administrator_manual_0.9#Processing_configuration See here.]<br />
* Workflows are currently created at the development level. <br />
*: Some resources avialable<br />
*:* [[MCP_Basic_Configuration]]<br />
*:* [[MCP]]<br />
*:* [[Creating_Custom_Workflows]]<br />
*:* [[Development]]<br />
* Normalization commands can be viewed in the presveration planning tab.<br />
* Normalization paths and commands are currently editedable under the preservation planning tab in the dashboard.<br />
<br />
= Elasticsearch =<br />
<br />
Archivematica has the capability of indexing data about files contained in AIPs and this data can be [[Elasticsearch Development|accessed programatically]] for various applications.<br />
<br />
If, for whatever reason, you need to delete an ElasticSearch index please see [[ElasticSearch Administration]].<br />
<br />
If, for whatever reason, you need to delete an Elasticsearch index programmatically, this can be done with pyes using the following code.<br />
<br />
<pre><br />
import sys<br />
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")<br />
from pyes import *<br />
conn = ES('127.0.0.1:9200')<br />
<br />
try:<br />
conn.delete_index('aips')<br />
except:<br />
print "Error deleting index or index already deleted."<br />
</pre><br />
<br />
= [[Data backup]] =<br />
* How to protect your data from disaster.<br />
<br />
= Security =<br />
* It is recommended to change the default passwords.<br />
** The ICA-ATOM/ContentDM passwords can be set in the administration tab in the dashboard.<br />
** The database connection credentials can be set in the /etc/archivematica/archivematicaCommon/dbsettings. These will need to a mysql user with permissions to the MCP database.<br />
:* Database<br />
::* The database user<br />
::* /etc/archivematica/archivematicaCommon/dbsettings<br />
::* The dashboard config file?<br />
:* Content DM<br />
:* ICA-AtoM<br />
* Archivematica does not presently support secured sockets. See Issue 990, Issue 690 Because of this, it's highly recommended to run Archivematica on a secure, isolated network.<br />
<br />
= Questions =<br />
<br />
== Frequently asked questions ==<br />
* [[AM_FAQ|Solutions to common questions]]<br />
<br />
== Discussion group ==<br />
* [[http://groups.google.com/group/archivematica?hl=en Discussion group] For questions not on the FAQ]</div>Josephhttps://wiki.archivematica.org/index.php?title=Administrator_manual_0.10&diff=7977Administrator manual 0.102013-04-29T22:04:12Z<p>Joseph: /* Dashboard administration tab */</p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Administrator Manual<br />
<br />
This manual covers administrator-specific instructions for Archivematica. It will also provide help for using forms in the Administration tab of the Archivematica dashboard and the administrator capabilities in the Format Policy Registry (FPR), which you will find in the Preservation planning tab of the dashboard.<br />
<br />
For end-user instructions, please see the [[User_manual_0.10|user manual]].<br />
<br />
= Installation =<br />
* [[Installation|Instructions for installing the latest build of Archivematica on your server]]<br />
<br />
= Upgrading =<br />
* Currently, Archivematica does not support upgrading from one version to the next. A re-install is required.<br />
<br />
= Dashboard administration tab =<br />
<br />
The Archivematica administration pages, under the Administration tab of the dashboard, allows you to configure application components and manage users.<br />
<br />
== Transfer source directories ==<br />
<br />
Unlike previous versions, Archivematica v0.9 allows you to start transfers using the web interface rather than the operating system's file browser. Source files for transfers, however, can't be uploaded using the web interface: they must exist on volumes accessible to the Archivematica server.<br />
<br />
When starting a transfer you're required to select one or more directories of files to add to the transfer. To speed up the process of selecting directories, Archivematica allows you to specify "source directories". A source directory is a directory in which files and directories likely to be added to a transfer are present.<br />
<br />
To add a source directory, while on the transfer source directories page of the Administration tab in the dashboard, simply click the folder icon to expand the starting directory and navigate the interface until you find a directory you'd like to select as a source directory. Once you've found a suitable source directory, click the "Add" button to the right of the directory and it will be added.<br />
<br />
To remove a source directory, simply click the "Remove" button to the right of it in the source directory path list.<br />
<br />
== AIP storage directories ==<br />
<br />
== AtoM DIP upload ==<br />
<br />
Archivematica can upload DIPs directly to an [https://www.ica-atom.org/ AtoM] website so the contents can be accessed online. The AtoM DIP upload configuration page is where you specify the details of the AtoM installation you'd like the DIPs uploaded to (and, if using Rsync to transfer the DIP files, Rsync transfer details).<br />
<br />
The parameters that you'll most likely want to set are <code>url</code>, <code>email</code>, and <code>password</code>. These parameters, respectively, specify the destination AtoM website's URL, the email address used to log in to the website, and the password used to log in to the website.<br />
<br />
AtoM DIP upload can also use [http://en.wikipedia.org/wiki/Rsync Rsync] as a transfer mechanism. Rsync is an open source utility for efficiently transferring files. The <code>rsync-target</code> parameter is used to specify an Rsync-style target host/directory pairing, "foobar.com:~/dips/" for example. The <code>rsync-command</code> parameter is used to specify rsync connection options, "ssh -p 22222 -l user" for example. If you are using the rsync option, please see AtoM server configuration below.<br />
<br />
To set any parameters for AtoM DIP upload change the values, preserving the existing format they're specified in, in the "Command arguments" field then click "Save".<br />
<br />
Note that in AtoM, the sword plugin (Admin --> qtSwordPlugin) and job scheduling (Admin --> Settings --> Job scheduling) must both be enabled in order for AtoM to receive uploaded DIPs.<br />
<br />
=== AtoM server configuration ===<br />
<br />
This server configuration step is necessary to allow Archivematica to log in to the AtoM server without passwords, and only when the user is deploying the rsync option described above in the AtoM DIP upload section. <br />
<br />
To enable sending DIPs from Archivematica to the AtoM server:<br />
<br />
Generate SSH keys for the Archivematica user. Leave the passphrase field blank.<br />
<pre><br />
$ sudo -i -u archivematica<br />
$ cd ~<br />
$ ssh-keygen<br />
</pre><br />
<br />
Copy the contents of <code>/var/lib/archivematica/.ssh/id_rsa.pub</code> somewhere handy, you will need it later.<br />
<br />
Now, it's time to configure the AtoM server so Archivematica can send the DIPs using SSH/rsync. For that purpose, you will create a user called <code>archivematica</code> and we are going to assign that user a restricted shell with access only to rsync:<br />
<br />
<pre><br />
$ sudo apt-get install rssh<br />
$ sudo useradd -d /home/archivematica -m -s /usr/bin/rssh archivematica<br />
$ sudo passswd -l archivematica<br />
$ sudo vim /etc/rssh.conf // Make sure that allowrsync is uncommented!<br />
</pre><br />
<br />
Add the SSH key that we generated before:<br />
<br />
<pre><br />
$ sudo mkdir /home/archivematica/.ssh<br />
$ chmod 700 /home/archivematica/.ssh/<br />
$ sudo vim /home/archivematica/.ssh/authorized_keys // Paste here the contents of id_dsa.pub<br />
$ chown -R archivematica:archivematica /home/archivematica<br />
</pre><br />
<br />
In Archivematica, make sure that you update the <code>--rsync-target</code> accordingly.<br /><br />
These are the parameters that we are passing to the upload-qubit microservice.<br /><br />
Go to the Administration > Upload DIP page in the dashboard.<br />
<br />
Generic parameters:<br />
<br />
<pre><br />
--url="http://atom-hostname/index.php" \<br />
--email="demo@example.com" \<br />
--password="demo" \<br />
--uuid="%SIPUUID%" \<br />
--rsync-target="archivematica@atom-hostname:/tmp" \<br />
--debug<br />
</pre><br />
<br />
== CONTENTdm DIP upload ==<br />
<br />
Archivematica can also upload DIPs to [http://www.contentdm.org/ CONTENTdm] websites. Multiple CONTENTdm destinations may be configured.<br />
For each possible CONTENTdm DIP upload destination, you'll specify a brief description and configuration parameters appropriate for the destination. Possible paramters include <code>%ContentdmServer%</code>, <code>%ContentdmUser%</code>, and <code>%ContentdmGroup%</code>.<br />
<br />
When changing parameters for a CONTENTdm DIP upload destination simply change the values, preserving the existing format they're specified in. To add an upload destination fill in the form at the bottom of the page with the appropriate values. When you've completed your changes click the "Save" button.<br />
<br />
== Processing configuration ==<br />
<br />
When processing a SIP or transfer, you may want to automate some of the workflow choices. Choices can be preconfigured by putting a 'processingMCP.xml' file into the root directory of a SIP/transfer. The processing configuration administration page allows you to configure the default 'processingMCP.xml' that's added to a SIP or Transfer if it doesn't alreay contain one. If a SIP or transfer is submitted with a 'processingMCP.xml' file, however, processing decisions will be made with the included file.<br />
<br />
Pre-configured choices made here are based off the choices that appear in the dashboard.<br />
<br />
The format is:<br />
<pre><br />
<preconfiguredChoice><br />
<appliesTo>Workflow decision - send transfer to quarantine</appliesTo><br />
<goToChain>Skip quarantine</goToChain><br />
</preconfiguredChoice><br />
</pre><br />
<br />
Where appliesTo is the name of the job presented in the dashboard, and goToChain is the desired selection. Note: these are case sensitive.<br />
<br />
== PREMIS agent ==<br />
<br />
The PREMIS agent name and code can be set via the administration interface.<br />
<br />
<br />
== Rest API ==<br />
<br />
In addition to automation using the processingMCP.xml file, Archivematica includes a REST API for automating transfer approval. Using this API, you can create a custom script that copies a transfer to the appropriate directory then uses the <code>curl</code> command, or some other means, to let Archivematica know that the copy is complete.<br />
<br />
=== API keys ===<br />
<br />
Use of the REST API requires the use of API keys. An API key is associated with a specific user. To generate an API key for a user:<br />
<br />
# Browse to <code>/administration/accounts/list/</code><br />
# Click the "Edit" button for the user you'd like to generate an API key for<br />
# Click the "Regenerate API key" checkbox<br />
# Click "Save"<br />
<br />
After generating an API key, you can click the "Edit" button for the user and you should see the API key.<br />
<br />
=== IP whitelist ===<br />
<br />
In addition to creating API keys, you'll need to add the IP of any computer making REST requests to the REST API whitelist. The IP whitelist can be edited in the administration interface at <code>/administration/api/</code>.<br />
<br />
=== Approving a transfer ===<br />
<br />
The REST API can be used to approve a transfer. The transfer must first be copied into the appropriate watch directory. To determine the location of the appropriate watch directory, first figure out where the shared directory is from the <code>sharedDirectory</code> value of <code>/etc/archivematica/MCPServer/serverConfig.conf</code>. Within that directory is a subdirectory <code>activeTransfers</code>. In this subdirectory are watch directories for the various transfer types.<br />
<br />
When using the REST API to approve a transfer, if a transfer type isn't specified, the transfer will be deemed a standard transfer.<br />
<br />
'''HTTP Method:''' POST<br />
<br />
'''URL:''' <code>/api/transfer/approve</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>directory</code>: directory name of the transfer<br />
<br />
<code>type</code> (optional): transfer type [standard|dspace|unzipped bag|zipped bag]<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl --data "username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6&type=standard&directory=MyTransfer" http://127.0.0.1/api/transfer/approve<br />
<br />
Example result:<br />
<br />
{"message": "Approval successful."}<br />
<br />
=== Listing unapproved transfers ===<br />
<br />
The REST API can be used to get a list of unapproved transfers. Each transfer's directory name and type is returned.<br />
<br />
'''Method:''' <code>GET</code><br />
<br />
'''URL:''' <code>/api/transfer/unapproved</code><br />
<br />
'''Parameters:'''<br />
<br />
<code>api_key</code>: an API key<br />
<br />
<code>username</code>: the username associated with the API key<br />
<br />
Example curl command:<br />
<br />
curl "http://127.0.0.1/api/transfer/unapproved?username=rick&api_key=f12d6b323872b3cef0b71be64eddd52f87b851a6"<br />
<br />
Example result:<br />
<br />
{<br />
"message": "Fetched unapproved transfers successfully.",<br />
"results": [{<br />
"directory": "MyTransfer",<br />
"type": "standard"<br />
}<br />
]<br />
}<br />
== Users ==<br />
<br />
The dashboard provides a simple cookie-based user authentication system using the [https://docs.djangoproject.com/en/1.4/topics/auth/ Django authentication framework]. Access to the dashboard is limited only to logged-in users and a login page will be shown when the user is not recognized. If the application can't find any user in the database, the user creation page will be shown instead, allowing the creation of an administrator account.<br />
<br />
Users can be also created, modified and deleted from the Administration tab. Here you can manage which users have access to Archivematica and what level of access they have. Standard users are able to access all sections of the interterface except for the administration section.<br />
<br />
You can add a new user to the system by clicking the "Add new" button on the user administration page. These users won't have administrator capability. By adding a user you provide a way to access Archivematica using a username/password combination. Should you need to change a user's username or password, you can do so by clicking the "Edit" button, corresponding to the user, on the administration page. Should you need to revoke a user's access, you can click the corresponding "Delete" button.<br />
<br />
=== CLI creation of administrative users ===<br />
<br />
If you need an additional administrator user one can be created via the command-line after navigating to the <code>src/dashboard/src</code> directory in the source tree.<br />
<br />
python manage.py createsuperuser --settings='settings.common'<br />
<br />
=== CLI password resetting ===<br />
<br />
If you've forgotten the password for your administrator user, or any other user, you can change it via the command-line.<br />
<br />
python manage.py changepassword <username> --settings='settings.common'<br />
<br />
===Security===<br />
<br />
Archivematica uses [http://en.wikipedia.org/wiki/PBKDF2 PBKDF2] as the default algorithm to store passwords. This should be sufficient for most users: it's quite secure, requiring massive amounts of computing time to break. However, other algorithms could be used as the following document explains: [https://docs.djangoproject.com/en/1.4/topics/auth/#how-django-stores-passwords How Django stores passwords].<br />
<br />
Our plan is to extend this functionality in the future adding groups and granular permissions support: see issue 922 to read more about it.<br />
<br />
= Dashboard preservation planning tab =<br />
<br />
== Format Policy Registry (FPR) ==<br />
<br />
= Customization and automation =<br />
* Workflow processing decisions can be made in the processingMCP.xml file. [https://www.archivematica.org/wiki/Administrator_manual_0.9#Processing_configuration See here.]<br />
* Workflows are currently created at the development level. <br />
*: Some resources avialable<br />
*:* [[MCP_Basic_Configuration]]<br />
*:* [[MCP]]<br />
*:* [[Creating_Custom_Workflows]]<br />
*:* [[Development]]<br />
* Normalization commands can be viewed in the presveration planning tab.<br />
* Normalization paths and commands are currently edited/created at the development level. [[Transcoder|See transcoder for more details.]]<br />
<br />
<br />
<br />
= Elasticsearch =<br />
<br />
Archivematica has the capability of indexing data about files contained in AIPs and this data can be [[Elasticsearch Development|accessed programatically]] for various applications.<br />
<br />
If, for whatever reason, you need to delete an ElasticSearch index please see [[ElasticSearch Administration]].<br />
<br />
If, for whatever reason, you need to delete an Elasticsearch index programmatically, this can be done with pyes using the following code.<br />
<br />
<pre><br />
import sys<br />
sys.path.append("/home/demo/archivematica/src/archivematicaCommon/lib/externals")<br />
from pyes import *<br />
conn = ES('127.0.0.1:9200')<br />
<br />
try:<br />
conn.delete_index('aips')<br />
except:<br />
print "Error deleting index or index already deleted."<br />
</pre><br />
<br />
= [[Data backup]] =<br />
* How to protect your data from disaster.<br />
<br />
= Security =<br />
* It is recommended to change the default passwords.<br />
** The ICA-ATOM/ContentDM passwords can be set in the administration tab in the dashboard.<br />
** The database connection credentials can be set in the /etc/archivematica/archivematicaCommon/dbsettings. These will need to a mysql user with permissions to the MCP database.<br />
:* Database<br />
::* The database user<br />
::* /etc/archivematica/archivematicaCommon/dbsettings<br />
::* The dashboard config file?<br />
:* Content DM<br />
:* ICA-AtoM<br />
* Archivematica does not presently support secured sockets. See Issue 990, Issue 690 Because of this, it's highly recommended to run Archivematica on a secure, isolated network.<br />
<br />
= Questions =<br />
<br />
== Frequently asked questions ==<br />
* [[AM_FAQ|Solutions to common questions]]<br />
<br />
== Discussion group ==<br />
* [[http://groups.google.com/group/archivematica?hl=en Discussion group] For questions not on the FAQ]</div>Josephhttps://wiki.archivematica.org/index.php?title=File:Archivematica_generated_workflow_diagram.svg&diff=7958File:Archivematica generated workflow diagram.svg2013-04-29T21:08:07Z<p>Joseph: Joseph uploaded a new version of &quot;File:Archivematica generated workflow diagram.svg&quot;</p>
<hr />
<div>This file was generated during 1.0 development using script:<br />
https://github.com/artefactual/archivematica/blob/joseph/src/MCPgraphing/lib/graphLinks.py</div>Josephhttps://wiki.archivematica.org/index.php?title=Development&diff=7858Development2013-04-24T20:56:01Z<p>Joseph: </p>
<hr />
<div>[[Main Page]] > Development<br />
<br />
<br />
This page lists the resources available for project contributors:<br />
<br />
==Project Communication==<br />
* [http://groups.google.ca/group/archivematica Discussion list]<br />
*[http://code.google.com/p/archivematica/issues/list Issues list] ([http://groups.google.com/group/archivematica-issues auto-updates])<br />
* [[Chat room]]<br />
* [[:Category:meetings|Weekly project meeting]]<br />
* Archivematica.org wiki: [[Special:UserLogin|create an account]] to correct and add content on this wiki<br />
<br />
==Developer Resources==<br />
* [https://github.com/artefactual/archivematica Code repository]<br />
* [[Development environment]]<br />
* [[Contribute code]]<br />
**[[Patches]]<br />
**[[Contributor Agreement]]<br />
**[[License]]<br />
**[[Trademark]]<br />
* [[:Category:Development documentation|Development documentation]]<br />
* [[Development_roadmap:_Archivematica_1.0|Development Roadmap]]<br />
* [[Creating Custom Workflows]]<br />
* [[Development_Glossary]]<br />
<br />
==Project Management==<br />
Archivematica software development, release management, and community support is managed by [http://artefactual.com Artefactual Systems] in collaboration with its contract clients and a growing network of Archivematica users and service partners.<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7833MCPServer/TaskTypes2013-04-24T20:28:50Z<p>Joseph: </p>
<hr />
<div><br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/magic_links]]<br />
* [[MCP/TaskTypes/unit_variables]]<br />
* [[MCP/TaskTypes/processing_choices]]<br />
* [[MCP/TaskTypes/Normalization]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7807MCPServer/TaskTypes2013-04-23T21:43:10Z<p>Joseph: </p>
<hr />
<div><br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/magic_links]]<br />
* [[MCP/TaskTypes/unit_variables]]<br />
* [[MCP/TaskTypes/processing_choices]]<br />
<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7806MCPServer/TaskTypes2013-04-23T21:39:10Z<p>Joseph: </p>
<hr />
<div><br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/magic_links]]<br />
<br />
* [[MCP/TaskTypes/unit_variables]]<br />
<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
<br />
<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
<br />
<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7804MCPServer/TaskTypes2013-04-23T21:33:57Z<p>Joseph: </p>
<hr />
<div><br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/magic_links]]<br />
<br />
* [[MCP/TaskTypes/linkTaskManagerUnitVariableLinkPull]]<br />
* [[MCP/TaskTypes/linkTaskManagerSetUnitVariable]]<br />
<br />
<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7803MCPServer/TaskTypes2013-04-23T21:24:24Z<p>Joseph: </p>
<hr />
<div>The Archivematica MCP task types are as follows:<br />
* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/linkTaskManagerSetUnitVariable]]<br />
* [[MCP/TaskTypes/magic_links]]<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/linkTaskManagerUnitVariableLinkPull]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7799MCPServer/TaskTypes2013-04-23T20:57:20Z<p>Joseph: </p>
<hr />
<div>The Archivematica MCP task types are as follows:<br />
* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
* [[MCP/TaskTypes/assign_magic_link]]<br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/linkTaskManagerSetUnitVariable]]<br />
* [[MCP/TaskTypes/goto_magic_link]]<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/linkTaskManagerUnitVariableLinkPull]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7798MCPServer/TaskTypes2013-04-23T20:55:55Z<p>Joseph: </p>
<hr />
<div>* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
* [[MCP/TaskTypes/assign_magic_link]]<br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/linkTaskManagerSetUnitVariable]]<br />
* [[MCP/TaskTypes/goto_magic_link]]<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/linkTaskManagerUnitVariableLinkPull]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer/TaskTypes&diff=7797MCPServer/TaskTypes2013-04-23T20:55:30Z<p>Joseph: Created page with "{{Special:PrefixIndex/MCP/TaskTypes}} * MCP/TaskTypes/Get_user_choice_from_microservice_generated_list * MCP/TaskTypes/assign_magic_link * [[MCP/TaskTypes/one_instanc..."</p>
<hr />
<div>{{Special:PrefixIndex/MCP/TaskTypes}}<br />
<br />
* [[MCP/TaskTypes/Get_user_choice_from_microservice_generated_list]]<br />
* [[MCP/TaskTypes/assign_magic_link]]<br />
* [[MCP/TaskTypes/one_instance]]<br />
* [[MCP/TaskTypes/Split_Job_into_many_links_based_on_file_ID]]<br />
* [[MCP/TaskTypes/Transcoder_task_type]]<br />
* [[MCP/TaskTypes/get_user_choice_to_proceed_with]]<br />
* [[MCP/TaskTypes/linkTaskManagerSetUnitVariable]]<br />
* [[MCP/TaskTypes/goto_magic_link]]<br />
* [[MCP/TaskTypes/Split_creating_Jobs_for_each_file]]<br />
* [[MCP/TaskTypes/get_replacement_dic_from_user_choice]]<br />
* [[MCP/TaskTypes/Get_microservice_generated_list_in_stdOut]]<br />
* [[MCP/TaskTypes/for_each_file]]<br />
* [[MCP/TaskTypes/linkTaskManagerUnitVariableLinkPull]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer&diff=7796MCPServer2013-04-23T20:49:57Z<p>Joseph: /* Tasks */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCP<br />
<br />
<br />
<div class="status"><br />
<div><br />
Design<br />
<div class="description"><br />
This page proposes a new feature and reviews design options<br />
</div><br />
</div><div class="active"><br />
Development<br />
<div class="description"><br />
This page describes a feature that's in development<br />
</div><br />
</div><div><br />
Documentation<br />
<div class="description"><br />
This page documents an implemented feature<br />
</div><br />
</div><br />
</div><br />
<br />
==Overview==<br />
The MCP is the core of the Archivematica system. It controls the various [[micro-services]] in the Archivematica system. Configuration and processing information are held in the database. The user monitors and controls the MCP via the [[ dashboard ]]. The MCP maintains a log of all completed work.<br />
<br />
The MCP uses the [http://gearman.org gearman]. The MCP Clients are relatively "dumb". They are gearman worker implementations, that inform the gearman server what tasks they can perform, and wait for the server to assign them a task.<br />
<br />
The Archivematica system relies on client and server having access to the same directory, to process the commands. On a distributed system, this is done through the shared directory.<br />
<br />
Basic configuration can be seen here [[MCP Basic Configuration]] <-- deprecated<br />
<br />
<br />
<br />
==Server And Database==<br />
The MCP has watched directories, which are linked to Job Chains. Each Job Chain is designed to carry out a function. The function is broken down into managable peices, which are called Job Chain Links. Each of these links performs a task. Like previous versions of the MCP, these tasks may be configured to run once, or once for each file in a directory. <br />
<br />
One major fundamental change is that the MCP is no longer as linear as it once was. Decision points allow the user to select the next Microservice chain to process, based on what is available at that point. This allows for the creation of alternative, yet similar workflows to co-exist in the Archivematica-MCP system.<br />
<br />
===Job Chains===<br />
<br />
===Job Chain Links===<br />
====Decision Point====<br />
====Regular Job====<br />
<br />
===Task Types===<br />
[[MCP/TaskTypes]]<br />
<br />
===mcp Modules===<br />
The mcp Modules are configured in the database, with the following schema.<br />
<br />
This may be a little out of date. Note, was generated using mysql workbench (sudo apt-get install mysql-workbench).<br />
<br />
[[File:MCP_configuration_database_schema.png]]<br />
<br />
==Client==<br />
Clients connect to the specified gearman server and provides a list of modules they support. When the MCP informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCP.<br />
<br />
===Client on Windows===<br />
There has been some consideration of getting an MCP client to run in the Microsoft Windows environment. This would be advantageous for [[normalizing in a windows environment]]. Some testing has been done to this end. See issue 372.<br />
<br />
==Debugging==<br />
Debugging the MCP can be a difficult task.<br />
Logs can be large, and are placed in the /tmp/ directory, so they are automatically removed upon reboot.<br />
<br />
===Parsing Logs===<br />
Here are some commands to help parse logs:<br />
<pre>grep "DEBUG type=\"archivematicaMCP\"" -v /tmp/archivematicaMCPServer* -h > /tmp/archivematicaOutput.txt </pre><br />
Removes the periodic debug message prints.<br />
<pre>grep "Traceback (most recent call last):" /tmp/archivematicaOutput.txt -n</pre><br />
<pre>grep -i EXCEPTION /tmp/archivematicaMCPServer-* -n</pre><br />
-n will prepend the line number<br />
<pre>sed -n '302092,+50'p /tmp/archivematicaMCPServer-*</pre><br />
prints 50 lines from the file, including line number 302092. This is useful to look at sections of the log that have exceptions, which can be found with the command above.<br />
<br />
===debugging tools===<br />
In extreme cases, you can setup your dev enviroment, so you log in as the archivematica user, and use eclipse with pyDev in debug mode, to run the MCP.<br />
<br />
===what clients are connected===<br />
<pre>python -c '<br />
import gearman<br />
admin = gearman.admin_client.GearmanAdminClient(host_list=["127.0.0.1"])<br />
for client in admin. get_workers():<br />
if client["client_id"] != "-": #exclude server task connections<br />
print client["client_id"], client["ip"]<br />
<br />
for stat in admin.get_status():<br />
if stat["running"] != 0 or stat["queued"] != 0:<br />
print stat<br />
' <br />
</pre><br />
<br />
<br />
===Waching activity===<br />
<pre>tail /tmp/archivematicaMCP* -f</pre><br />
<pre>watch mysql -u root MCP --execute "\"SELECT * FROM Tasks WHERE endTime = 0;\""</pre><br />
<br />
===Turning on printing all sql queries===<br />
sudo nano /usr/lib/archivematica/archivematicaCommon/databaseInterface.py<br />
:http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/archivematicaCommon/lib/databaseInterface.py<br />
:edit lines 34 and 73<br />
:"printSQL = False" -> printSQL = True<br />
:" print printSQL" -> " print sql"<br />
<br />
This will cause archivematica to print ALL of it's queries issues to the database.<br />
<br />
=Change Log=<br />
==0.8==<br />
* Switched to database configuration.<br />
* Allows for alternative workflows (ie. don't create DIP)<br />
* Start, MCP server will try to match any existing directories in the watched directories, to a processing directory/SIP.<br />
<br />
==0.7.1==<br />
* Work was done on microservices to make the system more stable. <br />
* A config to set the underlying protocol max length was added.<br />
<br />
==0.7==<br />
* Work was done on microservices to make the system more stable. <br />
<br />
==0.6.2==<br />
* MCP was released.<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCP_Basic_Configuration&diff=7795MCP Basic Configuration2013-04-22T21:50:53Z<p>Joseph: /* Workflow Configurations */</p>
<hr />
<div><div class="status"><br />
<div><br />
Design<br />
<div class="description"><br />
This page proposes a new feature and reviews design options<br />
</div><br />
</div><div class="active"><br />
Development<br />
<div class="description"><br />
This page describes a feature that's in development<br />
</div><br />
</div><div><br />
Documentation<br />
<div class="description"><br />
This page documents an implemented feature<br />
</div><br />
</div><br />
</div><br />
<br />
==Server==<br />
===Configurations===<br />
====Server ConfigFile====<br />
/etc/archivematica/MCPServer/ServerConfig.conf<br />
<br />
====Workflow Configurations====<br />
Please see [[Creating_Custom_Workflows]]<br />
<br />
===MCP Server Section===<br />
<br />
====File Names====<br />
These shouldn't be changed.<br />
#File Names<br />
checksumsNoExtention="checksum"<br />
<br />
<br />
====Gearman Server====<br />
MCPArchivematicaServer = localhost:4730<br />
GearmanServerWorker = localhost:4730<br />
<br />
Note: to set the gearman process to listen on more interfaces than localhost, set it in '/etc/default/gearman-job-server'. I think you can set this to 1.1.1.1 for all interfaces.<br />
<br />
<br />
====Other====<br />
watchDirectoryPath: Directory containing the directories watched by the MCP.<br />
watchDirectoryPath="/var/archivematica/sharedDirectory/watchedDirectories/"<br />
<br />
sharedDirectory: directory shared on the network with MCP clients. Containing the processing directory.<br />
sharedDirectory="/var/archivematica/sharedDirectory/"<br />
<br />
processingDirectory: Directory SIPs reside in while they are actively being processed.<br />
processingDirectory="${sharedDirectory}.currentlyProcessing/"<br />
<br />
AIPsStore: Directory to store the AIPs when finished processing. This can be a mounted directory on an external storage device.<br />
AIPsStore="${sharedDirectory}AIPsStore/"<br />
<br />
The database access settings for the shared database:<br />
/etc/archivematica/archivematicaCommon/dbsettings<br />
<pre><br />
[client]<br />
user=demo<br />
password="demo"<br />
host=localhost<br />
</pre><br />
<br />
==Client==<br />
===Supported Modules===<br />
/etc/archivematica/archivematicaClient/clientConfig.conf<br />
<br>[http://code.google.com/p/archivematica/source/browse/trunk/src/MCPClient/etc/archivematicaClientModules Supported modules]<br />
<br/>/etc/archivematica/MCPClient/archivematicaClientModules<br />
<br/>It's highly recommended only the MCP server host machine client support the upload and store tasks.<br />
<br/>Change the in theLoadSupportedCommandsSpecial in clientConfig.conf on other machines<br />
<pre><br />
LoadSupportedCommandsSpecial = False<br />
</pre><br />
<br />
===Client ConfigFile===<br />
/etc/archivematica/MCPClient/clientConfig.conf<br />
Server to connect to. Specify hostname/IP and port.<br />
MCPArchivematicaServer = localhost:4730<br />
Where the shared directory on the MCP Server is mounted:<br />
sharedDirectoryMounted="/var/archivematica/sharedDirectory/"<br />
The maximum number of tasks the MCP is allowed to run on this host.<br />
* numberOfTasks is no longer used. The client will detect the number of cores on the machine, and use that number of threads. This can be overridden by changing the number to something other than 0. <br />
<pre>numberOfTasks=0</pre><br />
<br>The location of the client modules. These specify which modules this host supports.<br />
archivematicaClientModules="/etc/archivematica/MCPClient/archivematicaClientModules"<br />
The location of the client scripts.<br />
clientScriptsDirectory="/usr/lib/archivematica/MCPClient/clientScripts/"<br />
The database access settings for the shared database:<br />
/etc/archivematica/archivematicaCommon/dbsettings<br />
<pre><br />
[client]<br />
user=demo<br />
password="demo"<br />
host=localhost<br />
</pre><br />
The supportedCommandsSpecial listed in the archivematicaClientModules should only be run on one machine. This is a mechanism to limit/control processing to a single instance. This should be enabled on one, and only one client in a distributed processing system.<br />
<pre>LoadSupportedCommandsSpecial = True</pre><br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Transfer_backlog_requirements&diff=7781Transfer backlog requirements2013-04-04T19:41:25Z<p>Joseph: /* Accession metadata */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Transfer backlog requirements<br />
<br />
Release 1.0<br />
<br />
== Transfer Backlog Management ==<br />
* Related issues: Issue 951, Issue 1220, Issue 1141, Issue 1225, Issue 1257<br />
<br />
= Requirements for transfer backlog search =<br />
<br />
* Add ability to search transfer backlog and send one or more transfers to Ingest<br />
* Add ability to download and/or view files/transfers (via right click)<br />
* Search the following fields: Any field, transfer name, file name, accession number, PUID, Mimetype, Date - Ingest<br />
<br />
= Mockup of transfer backlog search =<br />
<br />
<br />
[[File:1.0_TransferBacklogSearch.png|680px|thumb|center|]]<br />
<br />
[[File:1.0_TransBacklogSearchResults.png|680px|thumb|center|]]<br />
<br />
= Transfer Workflow = <br />
<br />
* Administration - allow MCP access to media or storage where transfer is located<br />
* Assign accession number to transfer<br />
* Remove transfer backup from workflow - no long processing configuration option<br />
* Add Send transfer to backlog microservice<br />
* Add Search transfer backlog tab from Ingest in Dashboard<br />
* Add ability to download and/or view transfers and files from Search tab<br />
* Add ability to send transfers from backlog search to Ingest/Create SIP (checkboxes, send button)<br />
* see workflow diagrams below <br />
<br />
'''0.9 Transfer workflow'''<br />
<br />
* grey steps are automated, white are manual<br />
<br />
[[File:transferWorkflow0.9.png|680px|thumb|center|]]<br />
[[File:transferWorkflow0.9pt2.png|680px|thumb|center|]]<br />
<br />
[[Media:transferWorkflow0.9.pdf|transferWorkflow0.9.pdf]]<br />
<br />
= Administration Tab in Dashboard = <br />
<br />
* Assign permission and access to the MCPServer to copy from transfer media (hard drives, optical media, USB, etc.) or network location. <br />
* Assign transfer backlog locations (configuration is done outside of AM)<br />
* Assign source directories<br />
* Define transfer types<br />
* Assign report locations (post-1.0)<br />
* Set AIP storage location<br />
* Set DIP upload location<br />
<br />
= Accession metadata =<br />
<br />
* PREMIS Event = Assign accession number<br />
<event><br />
<eventIdentifier><br />
<eventIdentifierType>UUID</eventIdentifierType><br />
<eventIdentifierValue>35cbe00d-d661-4174-b11a-e203f5608008</eventIdentifierValue><br />
</eventIdentifier><br />
<eventType>registration</eventType><br />
<eventDateTime>2012-03-14</eventDateTime><br />
<eventDetail>accession#2012-029</eventDetail><br />
<eventOutcomeInformation><br />
<eventOutcome></eventOutcome><br />
<eventOutcomeDetail><br />
<eventOutcomeDetailNote></eventOutcomeDetailNote><br />
</eventOutcomeDetail><br />
</eventOutcomeInformation><br />
<linkingAgentIdentifier><br />
<linkingAgentIdentifierType>archivist</linkingAgentIdentifierType><br />
<linkingAgentIdentifierValue>Courtney Mumma</linkingAgentIdentifierValue><br />
</linkingAgentIdentifier><br />
</event><br />
<br />
* Manually input metadata in template on dashboard (See [[File_Browser_Requirements]]) : accession number<br />
* Agent is the archivist logged in at the time doing the accession (post-1.0, for 1.0 this will still be repository)<br />
* Event name is "Assign accession number" (to be added to PREMIS events master list should we decide to implement)<br />
* UUID<br />
<br />
* Also see Issue 787 on the Archivematica issues list<br />
<br />
= Microservices Completed Before Move to Backlog =<br />
<br />
* All transfer microservices<br />
* Indexing: See [[Transfer_and_SIP_creation#Transfer_indexing_requirements_0.9_and_beyond]]<br />
<br />
= Handling of Submission Documentation =<br />
<br />
* [http://sites.tufts.edu/dca/about-us/research-initiatives/taper-tufts-accessioning-program-for-electronic-records/| TAPER]?<br />
* Normalized with objects in AIP (0.8)<br />
* Upload submission documentation with transfer in transfer tab - Issue 1255<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Database_Data_Management&diff=7718Database Data Management2013-03-14T00:17:09Z<p>Joseph: updated release names 1.0 -> 0.10 and 1.1 -> 1.0</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Database Data Management<br />
[[Category:Development documentation]]<br />
<br />
=For the 0.10 Release=<br />
The data for the transcoder will need to be maintained by the FPR. The table structure will be maintained by the package.<br />
<br />
All pk INT's will be replaced will UUIDs. Issue 1223.<br />
This will allow for local customizations to be distinguished from the original data provided in archivematica.<br />
<br />
All tables will have a replaces UUID column added. Issue 1224 .<br />
This column will be populated when a provided rule is replaced with an updated/custom one. <br />
As part of the update, the system will need to identify items that have been replaced, and update the foreign keys pointing at it to the updated rule. Warning: not all foreign keys are explicit in the database. Some are a "combined foriegn key", like Tasks Configs: Task type will inform the code what table to look at, and taskTypePKReference will specify the fk. <br />
<br />
==Development==<br />
To separate data for the transcoder, we'll be using a series of mysql dumps. Issue 1225. Once the FPR data is seperate, the developer can work on a means of populating that into their database, with the columns they require, and getting them into the database through an API. <br />
<br />
<br />
==Deployed Data management==<br />
Every change to the DB for workflow/FPR should be assigned a new UUID, so that it is not confused or overwritten on update. For tracking purposes the replaces column should be populated accordingly.<br />
<br />
==Release==<br />
A custom script for updating from the 0.9 to .10 will be developed to track existing SIPs/Tranfers/Files in the system, and not lose any data associated with them. This should be a msyql script to be run by the tool Austin has found for managing DB's across releases. Issue 1226.<br />
<br />
<br />
=For the 1.0 Release=<br />
Development changes for the db will occur through a sql script, applied against the previous release. Issue 1227.<br />
<br />
==Development==<br />
The .10 schema will remain in the code. Rebuilding the database will consist of the following steps:<br />
# Build the database from the .10 schema<br />
# Apply update patch sql script file<br />
# Run update script, that will replace the updated rows where specified.<br />
<br />
The .10 schema will remain untouched from the .10 release. <br />
<br/>The apply patch will be edited to make changes for the 1.0 release.<br />
<br/>The update script will be developed for this release. <br />
<br />
All dev changes to the db will occur in the patch or from the FPR.<br />
<br />
==Release==<br />
Updating from .10 to 1.0 can run the patch. This requires no data loss.<br />
<br />
New installs will run a script generated as part of the release process. It will be a dump of the .10 with the 1.0 patch applied. Future releases will use this as the base for development in the next release.<br />
<br />
==Deployed Data management==<br />
Same as .10<br />
<br />
==How this works with local customizations==<br />
The local customization made in .10 will not be overwritten by the ones made in the update, because all the fk's will already have been updated, so there would be nothing to replace, and point at the new rule.<br />
<br />
=For the future Releases=<br />
Each release keeps the base sql dump from the previous release. <br />
A patch file is generated against this dump for the release.<br />
We could look at generating the patch using http://dev.mysql.com/doc/refman/5.0/en/binary-log.html</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=7716Development environment2013-03-08T21:34:29Z<p>Joseph: /* Troubleshooting */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete ATOM setup<br />
*** The database should already be created.<br />
*** http://localhost/ica-atom http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Run=<br />
*Start Archivematica MCP Server & Client<br />
**Open a terminal and type in the following commands:<br />
***<pre>sudo stop openoffice-service; sudo start openoffice-service</pre><br />
***<pre>sudo stop archivematica-mcp-server; sudo start archivematica-mcp-server</pre><br />
***<pre>sudo stop archivematica-mcp-client; sudo start archivematica-mcp-client</pre><br />
<br />
*To start processing a SIP:<br />
**Open Thunar file manager<br />
**if using sample SIPs, 'export' the SIP directory<br />
***<pre>svn export sampledata ~/sampledata</pre><br />
***OR run the dev-helper script, and hit yes on the option to export the sample data.<br />
**'cut' the SIP directory and paste into the receiveSIP directory<br />
**Open the Firefox browser and type 'localhost' as the address to view SIP processing progress via the Dashboard<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update Qubit (trunk) and restart its qubit-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
** <pre>sudo apt-get install postfix</pre><br />
** Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=7715Development environment2013-03-08T21:33:35Z<p>Joseph: /* Troubleshooting */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete ATOM setup<br />
*** The database should already be created.<br />
*** http://localhost/ica-atom http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Run=<br />
*Start Archivematica MCP Server & Client<br />
**Open a terminal and type in the following commands:<br />
***<pre>sudo stop openoffice-service; sudo start openoffice-service</pre><br />
***<pre>sudo stop archivematica-mcp-server; sudo start archivematica-mcp-server</pre><br />
***<pre>sudo stop archivematica-mcp-client; sudo start archivematica-mcp-client</pre><br />
<br />
*To start processing a SIP:<br />
**Open Thunar file manager<br />
**if using sample SIPs, 'export' the SIP directory<br />
***<pre>svn export sampledata ~/sampledata</pre><br />
***OR run the dev-helper script, and hit yes on the option to export the sample data.<br />
**'cut' the SIP directory and paste into the receiveSIP directory<br />
**Open the Firefox browser and type 'localhost' as the address to view SIP processing progress via the Dashboard<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update Qubit (trunk) and restart its qubit-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
<pre>sudo apt-get install postfix</pre><br />
* Restart the dev-helper "Would you like to update/install package requirements?"<br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=7714Development environment2013-03-08T21:28:33Z<p>Joseph: /* Troubleshooting */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete ATOM setup<br />
*** The database should already be created.<br />
*** http://localhost/ica-atom http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Run=<br />
*Start Archivematica MCP Server & Client<br />
**Open a terminal and type in the following commands:<br />
***<pre>sudo stop openoffice-service; sudo start openoffice-service</pre><br />
***<pre>sudo stop archivematica-mcp-server; sudo start archivematica-mcp-server</pre><br />
***<pre>sudo stop archivematica-mcp-client; sudo start archivematica-mcp-client</pre><br />
<br />
*To start processing a SIP:<br />
**Open Thunar file manager<br />
**if using sample SIPs, 'export' the SIP directory<br />
***<pre>svn export sampledata ~/sampledata</pre><br />
***OR run the dev-helper script, and hit yes on the option to export the sample data.<br />
**'cut' the SIP directory and paste into the receiveSIP directory<br />
**Open the Firefox browser and type 'localhost' as the address to view SIP processing progress via the Dashboard<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update Qubit (trunk) and restart its qubit-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If it stalls during update "Would you like to update/install package requirements?" (y/N)<br />
** Stop the script with CTRL + C<br />
** Try installing the item it failed on on the command line. Ie.<br />
<pre>sudo apt-get install postfix</pre><br />
<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=7713Development environment2013-03-08T20:40:40Z<p>Joseph: /* Update */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete ATOM setup<br />
*** The database should already be created.<br />
*** http://localhost/ica-atom http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Run=<br />
*Start Archivematica MCP Server & Client<br />
**Open a terminal and type in the following commands:<br />
***<pre>sudo stop openoffice-service; sudo start openoffice-service</pre><br />
***<pre>sudo stop archivematica-mcp-server; sudo start archivematica-mcp-server</pre><br />
***<pre>sudo stop archivematica-mcp-client; sudo start archivematica-mcp-client</pre><br />
<br />
*To start processing a SIP:<br />
**Open Thunar file manager<br />
**if using sample SIPs, 'export' the SIP directory<br />
***<pre>svn export sampledata ~/sampledata</pre><br />
***OR run the dev-helper script, and hit yes on the option to export the sample data.<br />
**'cut' the SIP directory and paste into the receiveSIP directory<br />
**Open the Firefox browser and type 'localhost' as the address to view SIP processing progress via the Dashboard<br />
<br />
=Update=<br />
<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*Check to see if you have any local changes that need to be stashed<br />
<pre>git diff</pre><br />
*If there are local changes please stash or commit them. (Or you won't be able to update).<br />
<pre>git stash</pre><br />
Update...<br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update Qubit (trunk) and restart its qubit-sword service?" (y/N) y</pre><br />
If you stashed changes, re-apply them with<br />
<pre>git stash pop</pre><br />
<br />
=Troubleshooting=<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Development_environment&diff=7712Development environment2013-03-08T20:36:53Z<p>Joseph: /* Update */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Development Environment<br />
<br />
This page explains how you can configure and use a standard Linux system as an Archivematica development environment.<br />
The Archivematica development environment is available for developers that want the ability to customize or enhance their own Archivematica installation and/or [[contribute code]] back to the Archivematica project.<br />
<br />
=Setup=<br />
*Install ubuntu 12.04<br />
*Use git to checkout Archivematica code<br />
**If you have commit access, use the private repository<br />
**If you have read-only access, use the public [https://github.com/artefactual/archivematica GitHub repository]<br />
<pre>git clone https://github.com/artefactual/archivematica.git</pre><br />
<br />
<br />
*Run the install and helper scripts. <br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-installer</pre><br />
** Answer Y to all prompts.<br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** Restart the machine.<br />
** <pre>cd archivematica</pre><br />
** <pre>./dev-helper</pre><br />
** Answer Y to all prompts.<br />
** You may need to restart the services (see below for instructions)<br />
** Complete ATOM setup<br />
*** The database should already be created.<br />
*** http://localhost/ica-atom http://localhost/atom<br />
*** https://www.qubit-toolkit.org/wiki/Installation#Open_Qubit.2C_ICA-AtoM.2C_or_DCB_in_your_web_browser<br />
** Open the [dashboard http://localhost]<br />
<br />
=Run=<br />
*Start Archivematica MCP Server & Client<br />
**Open a terminal and type in the following commands:<br />
***<pre>sudo stop openoffice-service; sudo start openoffice-service</pre><br />
***<pre>sudo stop archivematica-mcp-server; sudo start archivematica-mcp-server</pre><br />
***<pre>sudo stop archivematica-mcp-client; sudo start archivematica-mcp-client</pre><br />
<br />
*To start processing a SIP:<br />
**Open Thunar file manager<br />
**if using sample SIPs, 'export' the SIP directory<br />
***<pre>svn export sampledata ~/sampledata</pre><br />
***OR run the dev-helper script, and hit yes on the option to export the sample data.<br />
**'cut' the SIP directory and paste into the receiveSIP directory<br />
**Open the Firefox browser and type 'localhost' as the address to view SIP processing progress via the Dashboard<br />
<br />
=Update=<br />
To pull down the latest code commits from the repository, navigate to the directory where Archivematica has been cloned:<br />
*Change Directory to the archivematica git directory.<br />
<pre>cd ~/archivematica/</pre><br />
*<pre>./dev-helper</pre><br />
*<pre>"Would you like to git pull?" (y/N) y</pre><br />
*<pre>"Would you like to recreate the databases?" (y/N) y</pre><br />
**Note: this command will remove any processing sips/transfers.<br />
*<pre>"Would you like to Include mock FPR data?" (y/N) y </pre><br />
*<pre>"Would you like to restart archivematica services?" (y/N) y</pre><br />
*<pre>"Would you like to re-create gui-scripts editor file?" (y/N) y</pre><br />
*<pre>"Would you like to update Qubit (trunk) and restart its qubit-sword service?" (y/N) y</pre><br />
<br />
=Troubleshooting=<br />
* If you get this error error starting service:<br />
** <pre>sudo start archivematica-mcp-server</pre><br />
** <pre>start: Unknown job: archivematica-mcp-server</pre><br />
** Then reboot the machine.<br />
<br />
*If a SIP processing fails, it will move it to the 'failed' directory which is located:<br />
**<pre>/var/archivematica/sharedDirectory/watchedDirectories/failed</pre><br />
<br />
*if ArchivematicaServer freezes <br />
**<pre>sudo restart archivematica-mcp-server</pre><br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*if ArchivematicaClient freezes (in terminal kill command)<br />
**<pre>sudo restart archivematica-mcp-client</pre><br />
<br />
*Updates to the Dashboard may require an Apache webserver restart:<br />
**<pre>sudo /etc/init.d/apache2 restart</pre><br />
<br />
*If you find a problem running the Dashboard and you want to get a detailed error log to report us, please switch it to debug mode following [http://archivematica.org/wiki/index.php?title=Dashboard#Debug_mode these instructions].<br />
<br />
*MCP is currently logging to the /tmp/directory<br />
** /tmp/archivematicaMCPClient-HOST.log<br />
** /tmp/archivematicaMCPServer-HOST-DATE.log<br />
** /tmp/archivematicaMCPServerPID<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Meeting_201300206&diff=7516Meeting 2013002062013-02-06T19:39:35Z<p>Joseph: </p>
<hr />
<div>Archivematica meeting February 06, 2013<br />
<br />
==dev==<br />
* peterVG release numbering change 1.0 -> 0.10-beta; 1.1 -> 1.0<br />
* courtneyM went through roadmap and evaluated related issue status, sent email to archivematica group<br />
* berwin22 added gitproperties to help with versioning. It can replace $Id: $ with the sha5 of the commit.<br />
* Evelyn changed https://projects.artefactual.com/issues/1576 to "won't fix" as it's not longer required by the institution.<br />
* mcantelon: Mike C spent time looking into browser memory use monitoring and has been working on an implementation of dashboard grip paging that won't gobble memory.<br />
* epmclellan added new issue, hoping to get it into .10: https://projects.artefactual.com/issues/4605<br />
* epmclellan create data entry template to allow user to editprocessing config settings<br />
* berwin22 archivematica file identification now also includes tool version for most tools, including droid.<br />
* mcantelon to have meeting to discuss his .10 schedule.<br />
<br />
==docs==<br />
* David to schedule a meeting about handling client data.<br />
* Courtney to ALL: please include breadcrumbs when creating a new wiki page.<br />
<br />
== deployment ==<br />
* berwin22 to get an estimate on our release schedule.<br />
* Austin UBC ran into some issues with linking and is waiting to hear from IT.<br />
<br />
== Testing ==<br />
* Austin doing maildir testing - scalability testing<br />
* Justin using collectd (http://collectd.org/) on ARTi's scalability testing.<br />
<br />
<br />
[[Category: Meetings]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Meeting_201300206&diff=7515Meeting 2013002062013-02-06T19:38:06Z<p>Joseph: Created page with "==dev== * peterVG release numbering change 1.0 -> 0.10-beta; 1.1 -> 1.0 * courtneyM went through roadmap and evaluated related issue status, sent email to archivematica group..."</p>
<hr />
<div>==dev==<br />
* peterVG release numbering change 1.0 -> 0.10-beta; 1.1 -> 1.0<br />
* courtneyM went through roadmap and evaluated related issue status, sent email to archivematica group<br />
* berwin22 added gitproperties to help with versioning. It can replace $Id: $ with the sha5 of the commit.<br />
* Evelyn changed https://projects.artefactual.com/issues/1576 to "won't fix" as it's not longer required by the institution.<br />
* mcantelon: Mike C spent time looking into browser memory use monitoring and has been working on an implementation of dashboard grip paging that won't gobble memory.<br />
* epmclellan added new issue, hoping to get it into .10: https://projects.artefactual.com/issues/4605<br />
* epmclellan create data entry template to allow user to editprocessing config settings<br />
* berwin22 archivematica file identification now also includes tool version for most tools, including droid.<br />
* mcantelon to have meeting to discuss his .10 schedule.<br />
<br />
==docs==<br />
* David to schedule a meeting about handling client data.<br />
* Courtney to ALL: please include breadcrumbs when creating a new wiki page.<br />
<br />
== deployment ==<br />
* berwin22 to get an estimate on our release schedule.<br />
* Austin UBC ran into some issues with linking and is waiting to hear from IT.<br />
<br />
== Testing ==<br />
* Austin doing maildir testing - scalability testing<br />
* Justin using collectd (http://collectd.org/) on ARTi's scalability testing.</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer&diff=7240MCPServer2012-11-30T23:15:56Z<p>Joseph: /* what clients are connected */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCP<br />
<br />
<br />
<div class="status"><br />
<div><br />
Design<br />
<div class="description"><br />
This page proposes a new feature and reviews design options<br />
</div><br />
</div><div class="active"><br />
Development<br />
<div class="description"><br />
This page describes a feature that's in development<br />
</div><br />
</div><div><br />
Documentation<br />
<div class="description"><br />
This page documents an implemented feature<br />
</div><br />
</div><br />
</div><br />
<br />
==Overview==<br />
The MCP is the core of the Archivematica system. It controls the various [[micro-services]] in the Archivematica system. Configuration and processing information are held in the database. The user monitors and controls the MCP via the [[ dashboard ]]. The MCP maintains a log of all completed work.<br />
<br />
The MCP uses the [http://gearman.org gearman]. The MCP Clients are relatively "dumb". They are gearman worker implementations, that inform the gearman server what tasks they can perform, and wait for the server to assign them a task.<br />
<br />
The Archivematica system relies on client and server having access to the same directory, to process the commands. On a distributed system, this is done through the shared directory.<br />
<br />
Basic configuration can be seen here [[MCP Basic Configuration]] <-- deprecated<br />
<br />
<br />
<br />
==Server And Database==<br />
The MCP has watched directories, which are linked to Job Chains. Each Job Chain is designed to carry out a function. The function is broken down into managable peices, which are called Job Chain Links. Each of these links performs a task. Like previous versions of the MCP, these tasks may be configured to run once, or once for each file in a directory. <br />
<br />
One major fundamental change is that the MCP is no longer as linear as it once was. Decision points allow the user to select the next Microservice chain to process, based on what is available at that point. This allows for the creation of alternative, yet similar workflows to co-exist in the Archivematica-MCP system.<br />
<br />
===Job Chains===<br />
<br />
===Job Chain Links===<br />
====Decision Point====<br />
====Regular Job====<br />
<br />
===Tasks===<br />
<br />
===mcp Modules===<br />
The mcp Modules are configured in the database, with the following schema.<br />
<br />
This may be a little out of date. Note, was generated using mysql workbench (sudo apt-get install mysql-workbench).<br />
<br />
[[File:MCP_configuration_database_schema.png]]<br />
<br />
==Client==<br />
Clients connect to the specified gearman server and provides a list of modules they support. When the MCP informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCP.<br />
<br />
===Client on Windows===<br />
There has been some consideration of getting an MCP client to run in the Microsoft Windows environment. This would be advantageous for [[normalizing in a windows environment]]. Some testing has been done to this end. See issue 372.<br />
<br />
==Debugging==<br />
Debugging the MCP can be a difficult task.<br />
Logs can be large, and are placed in the /tmp/ directory, so they are automatically removed upon reboot.<br />
<br />
===Parsing Logs===<br />
Here are some commands to help parse logs:<br />
<pre>grep "DEBUG type=\"archivematicaMCP\"" -v /tmp/archivematicaMCPServer* -h > /tmp/archivematicaOutput.txt </pre><br />
Removes the periodic debug message prints.<br />
<pre>grep "Traceback (most recent call last):" /tmp/archivematicaOutput.txt -n</pre><br />
<pre>grep -i EXCEPTION /tmp/archivematicaMCPServer-* -n</pre><br />
-n will prepend the line number<br />
<pre>sed -n '302092,+50'p /tmp/archivematicaMCPServer-*</pre><br />
prints 50 lines from the file, including line number 302092. This is useful to look at sections of the log that have exceptions, which can be found with the command above.<br />
<br />
===debugging tools===<br />
In extreme cases, you can setup your dev enviroment, so you log in as the archivematica user, and use eclipse with pyDev in debug mode, to run the MCP.<br />
<br />
===what clients are connected===<br />
<pre>python -c '<br />
import gearman<br />
admin = gearman.admin_client.GearmanAdminClient(host_list=["127.0.0.1"])<br />
for client in admin. get_workers():<br />
if client["client_id"] != "-": #exclude server task connections<br />
print client["client_id"], client["ip"]<br />
<br />
for stat in admin.get_status():<br />
if stat["running"] != 0 or stat["queued"] != 0:<br />
print stat<br />
' <br />
</pre><br />
<br />
<br />
===Waching activity===<br />
<pre>tail /tmp/archivematicaMCP* -f</pre><br />
<pre>watch mysql -u root MCP --execute "\"SELECT * FROM Tasks WHERE endTime = 0;\""</pre><br />
<br />
===Turning on printing all sql queries===<br />
sudo nano /usr/lib/archivematica/archivematicaCommon/databaseInterface.py<br />
:http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/archivematicaCommon/lib/databaseInterface.py<br />
:edit lines 34 and 73<br />
:"printSQL = False" -> printSQL = True<br />
:" print printSQL" -> " print sql"<br />
<br />
This will cause archivematica to print ALL of it's queries issues to the database.<br />
<br />
=Change Log=<br />
==0.8==<br />
* Switched to database configuration.<br />
* Allows for alternative workflows (ie. don't create DIP)<br />
* Start, MCP server will try to match any existing directories in the watched directories, to a processing directory/SIP.<br />
<br />
==0.7.1==<br />
* Work was done on microservices to make the system more stable. <br />
* A config to set the underlying protocol max length was added.<br />
<br />
==0.7==<br />
* Work was done on microservices to make the system more stable. <br />
<br />
==0.6.2==<br />
* MCP was released.<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=MCPServer&diff=7239MCPServer2012-11-30T23:13:05Z<p>Joseph: /* Debugging */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCP<br />
<br />
<br />
<div class="status"><br />
<div><br />
Design<br />
<div class="description"><br />
This page proposes a new feature and reviews design options<br />
</div><br />
</div><div class="active"><br />
Development<br />
<div class="description"><br />
This page describes a feature that's in development<br />
</div><br />
</div><div><br />
Documentation<br />
<div class="description"><br />
This page documents an implemented feature<br />
</div><br />
</div><br />
</div><br />
<br />
==Overview==<br />
The MCP is the core of the Archivematica system. It controls the various [[micro-services]] in the Archivematica system. Configuration and processing information are held in the database. The user monitors and controls the MCP via the [[ dashboard ]]. The MCP maintains a log of all completed work.<br />
<br />
The MCP uses the [http://gearman.org gearman]. The MCP Clients are relatively "dumb". They are gearman worker implementations, that inform the gearman server what tasks they can perform, and wait for the server to assign them a task.<br />
<br />
The Archivematica system relies on client and server having access to the same directory, to process the commands. On a distributed system, this is done through the shared directory.<br />
<br />
Basic configuration can be seen here [[MCP Basic Configuration]] <-- deprecated<br />
<br />
<br />
<br />
==Server And Database==<br />
The MCP has watched directories, which are linked to Job Chains. Each Job Chain is designed to carry out a function. The function is broken down into managable peices, which are called Job Chain Links. Each of these links performs a task. Like previous versions of the MCP, these tasks may be configured to run once, or once for each file in a directory. <br />
<br />
One major fundamental change is that the MCP is no longer as linear as it once was. Decision points allow the user to select the next Microservice chain to process, based on what is available at that point. This allows for the creation of alternative, yet similar workflows to co-exist in the Archivematica-MCP system.<br />
<br />
===Job Chains===<br />
<br />
===Job Chain Links===<br />
====Decision Point====<br />
====Regular Job====<br />
<br />
===Tasks===<br />
<br />
===mcp Modules===<br />
The mcp Modules are configured in the database, with the following schema.<br />
<br />
This may be a little out of date. Note, was generated using mysql workbench (sudo apt-get install mysql-workbench).<br />
<br />
[[File:MCP_configuration_database_schema.png]]<br />
<br />
==Client==<br />
Clients connect to the specified gearman server and provides a list of modules they support. When the MCP informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCP.<br />
<br />
===Client on Windows===<br />
There has been some consideration of getting an MCP client to run in the Microsoft Windows environment. This would be advantageous for [[normalizing in a windows environment]]. Some testing has been done to this end. See issue 372.<br />
<br />
==Debugging==<br />
Debugging the MCP can be a difficult task.<br />
Logs can be large, and are placed in the /tmp/ directory, so they are automatically removed upon reboot.<br />
<br />
===Parsing Logs===<br />
Here are some commands to help parse logs:<br />
<pre>grep "DEBUG type=\"archivematicaMCP\"" -v /tmp/archivematicaMCPServer* -h > /tmp/archivematicaOutput.txt </pre><br />
Removes the periodic debug message prints.<br />
<pre>grep "Traceback (most recent call last):" /tmp/archivematicaOutput.txt -n</pre><br />
<pre>grep -i EXCEPTION /tmp/archivematicaMCPServer-* -n</pre><br />
-n will prepend the line number<br />
<pre>sed -n '302092,+50'p /tmp/archivematicaMCPServer-*</pre><br />
prints 50 lines from the file, including line number 302092. This is useful to look at sections of the log that have exceptions, which can be found with the command above.<br />
<br />
===debugging tools===<br />
In extreme cases, you can setup your dev enviroment, so you log in as the archivematica user, and use eclipse with pyDev in debug mode, to run the MCP.<br />
<br />
===what clients are connected===<br />
<pre>python -c '<br />
import gearman<br />
admin = gearman.admin_client.GearmanAdminClient(host_list=["127.0.0.1"])<br />
for client in admin. get_workers():<br />
if client["client_id"] != "-": #exclude server task connections<br />
print client["client_id"], client["ip"]<br />
<br />
for stat in admin.get_status():<br />
<br />
===Waching activity===<br />
<pre>tail /tmp/archivematicaMCP* -f</pre><br />
<pre>watch mysql -u root MCP --execute "\"SELECT * FROM Tasks WHERE endTime = 0;\""</pre><br />
if stat["running"] != 0 or stat["queued"] != 0:<br />
print stat<br />
<br />
' <br />
</pre><br />
<br />
===Turning on printing all sql queries===<br />
sudo nano /usr/lib/archivematica/archivematicaCommon/databaseInterface.py<br />
:http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/archivematicaCommon/lib/databaseInterface.py<br />
:edit lines 34 and 73<br />
:"printSQL = False" -> printSQL = True<br />
:" print printSQL" -> " print sql"<br />
<br />
This will cause archivematica to print ALL of it's queries issues to the database.<br />
<br />
=Change Log=<br />
==0.8==<br />
* Switched to database configuration.<br />
* Allows for alternative workflows (ie. don't create DIP)<br />
* Start, MCP server will try to match any existing directories in the watched directories, to a processing directory/SIP.<br />
<br />
==0.7.1==<br />
* Work was done on microservices to make the system more stable. <br />
* A config to set the underlying protocol max length was added.<br />
<br />
==0.7==<br />
* Work was done on microservices to make the system more stable. <br />
<br />
==0.6.2==<br />
* MCP was released.<br />
<br />
<br />
[[Category:Development documentation]]</div>Josephhttps://wiki.archivematica.org/index.php?title=Install-0.9-beta&diff=7219Install-0.9-beta2012-11-28T19:37:21Z<p>Joseph: /* Advanced */</p>
<hr />
<div>[[Main Page]] > [[Software]] > [[Install]] > Release 0.9-beta<br />
<br />
There are multiple ways to install and use Archivematica 0.9-beta. Once it is installed, consult the [[User_Manual]] for user instructions.<br />
<br />
= Virtual appliance =<br />
<br />
1. What is a [[wikipedia:Virtual appliance|virtual appliance]]?<br />
<br />
2. Download your preferred virtual appliance format:<br />
<br />
* VirtualBox, VMware: [http://archivematica.org/downloads/archivematica-0.9-beta.vmdk.tbz archivematica-0.9-beta.vmdk.tbz] (1.8G)<br />
** European mirror: [http://europe.artefactual.com/archivematica/archivematica-0.9-beta.vmdk.tbz archivematica-0.9-beta.vmdk.tbz] (1.8G)<br />
** [http://www.archivematica.org/download/archivematica-0.9-beta.vmdk.torrent Torrent file]<br />
<br />
To run a checksum on the .tbz file, select one of the following: ([http://archivematica.org/downloads/sha256.list sha256], [http://archivematica.org/downloads/md5sum.list md5sum]).<br />
<br />
3. If you don't already have a virtual machine player, we recommend VirtualBox. See these [[Virtual appliance instructions]] on how to load your Archivematica virtual appliance using VirtualBox.<br />
<br />
= Advanced =<br />
<br />
This options requires experience in linux<br />
* Install from Packages on ubuntu<br />
** Packages allow a user to quickly deploy a multinode environment, [http://archivematica.org/wiki/index.php?title=Multi_Node_Install How to install packages]<br />
<br />
__NOTOC__</div>Josephhttps://wiki.archivematica.org/index.php?title=Contribute_code&diff=7205Contribute code2012-11-27T00:58:14Z<p>Joseph: /* Patches */</p>
<hr />
<div>[[Main Page]] > [[Development]] > Contribute code<br />
<br />
<br />
==Patches==<br />
If you find a bug in this project or would like to make an enhancement, please be encouraged to contribute a patch by following [[Pull requests|these instructions]].<br />
<br />
==Commit access==<br />
Anyone can contribute code patches to this project. Project collaborators and regular patch contributors will be given access to commit directly to the git code repository.<br />
<br />
==Contributor's Agreement==<br />
In order to accept any patches or code commits, contributors must first sign the [[Contributor Agreement|Archivematica Contributor's Agreement]].<br />
<br />
==Standards==<br />
<br />
=== Code Style Guide For Archivematica ===<br />
This coding convention is adopted from:<br />
<br/>http://www.python.org/dev/peps/pep-0008/<br />
<br/>authors: Guido van Rossum <guido at python.org>, Barry Warsaw <barry at python.org><br />
<br />
*Tabs are forbidden.<br />
*4 spaces per indent.<br />
*cammelCase all variables, starting with lower case.<br />
*Max 79 characters per line.<br />
*Each import is on it's own line (no comma separated imports)<br />
*No spaces next to parenthesis '('<br />
*No space before comma, space after comma.<br />
*Assignments and comparators are given a single space separator 'x = 1'<br />
<br />
=== File Structure ===<br />
The file structure in Archivematica will comply with the Filesystem Hierarchy Standard (FHS). <br />
<br/>More information on this standard is available at: <br />
<br/>http://www.pathname.com/fhs/</div>Josephhttps://wiki.archivematica.org/index.php?title=File:Archivematica_generated_workflow_diagram.svg&diff=7192File:Archivematica generated workflow diagram.svg2012-11-22T01:34:18Z<p>Joseph: This file was generated during 1.0 development using script:
https://github.com/artefactual/archivematica/blob/joseph/src/MCPgraphing/lib/graphLinks.py</p>
<hr />
<div>This file was generated during 1.0 development using script:<br />
https://github.com/artefactual/archivematica/blob/joseph/src/MCPgraphing/lib/graphLinks.py</div>Josephhttps://wiki.archivematica.org/index.php?title=Database_Data_Management&diff=6946Database Data Management2012-10-10T20:41:45Z<p>Joseph: /* For the future Releases */</p>
<hr />
<div>[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > Database Data Management<br />
[[Category:Development documentation]]<br />
<br />
=For the 1.0 Release=<br />
The data for the transcoder will need to be maintained by the FPR. The table structure will be maintained by the package.<br />
<br />
All pk INT's will be replaced will UUIDs. Issue 1223.<br />
This will allow for local customizations to be distinguished from the original data provided in archivematica.<br />
<br />
All tables will have a replaces UUID column added. Issue 1224 .<br />
This column will be populated when a provided rule is replaced with an updated/custom one. <br />
As part of the update, the system will need to identify items that have been replaced, and update the foreign keys pointing at it to the updated rule. Warning: not all foreign keys are explicit in the database. Some are a "combined foriegn key", like Tasks Configs: Task type will inform the code what table to look at, and taskTypePKReference will specify the fk. <br />
<br />
==Development==<br />
To separate data for the transcoder, we'll be using a series of mysql dumps. Issue 1225. Once the FPR data is seperate, the developer can work on a means of populating that into their database, with the columns they require, and getting them into the database through an API. <br />
<br />
<br />
==Deployed Data management==<br />
Every change to the DB for workflow/FPR should be assigned a new UUID, so that it is not confused or overwritten on update. For tracking purposes the replaces column should be populated accordingly.<br />
<br />
==Release==<br />
A custom script for updating from the 0.9 to 1.0 will be developed to track existing SIPs/Tranfers/Files in the system, and not lose any data associated with them. This should be a msyql script to be run by the tool Austin has found for managing DB's across releases. Issue 1226.<br />
<br />
<br />
=For the 1.1 Release=<br />
Development changes for the db will occur through a sql script, applied against the previous release. Issue 1227.<br />
<br />
==Development==<br />
The 1.0 schema will remain in the code. Rebuilding the database will consist of the following steps:<br />
# Build the database from the 1.0 schema<br />
# Apply update patch sql script file<br />
# Run update script, that will replace the updated rows where specified.<br />
<br />
The 1.0 schema will remain untouched from the 1.0 release. <br />
<br/>The apply patch will be edited to make changes for the 1.1 release.<br />
<br/>The update script will be developed for this release. <br />
<br />
All dev changes to the db will occur in the patch or from the FPR.<br />
<br />
==Release==<br />
Updating from 1.0 to 1.1 can run the patch. This requires no data loss.<br />
<br />
New installs will run a script generated as part of the release process. It will be a dump of the 1.0 with the 1.1 patch applied. Future releases will use this as the base for development in the next release.<br />
<br />
==Deployed Data management==<br />
Same as 1.0<br />
<br />
==How this works with local customizations==<br />
The local customization made in 1.0 will not be overwritten by the ones made in the update, because all the fk's will already have been updated, so there would be nothing to replace, and point at the new rule.<br />
<br />
=For the future Releases=<br />
Each release keeps the base sql dump from the previous release. <br />
A patch file is generated against this dump for the release.<br />
We could look at generating the patch using http://dev.mysql.com/doc/refman/5.0/en/binary-log.html</div>Josephhttps://wiki.archivematica.org/index.php?title=Screencasts&diff=6928Screencasts2012-10-10T18:35:46Z<p>Joseph: </p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Screencasts<br />
<br />
<br />
* [http://youtu.be/GWmNfuO1ofw Archivematica 0.9-beta demonstration] (Aug 29, 2012)<br />
* [http://youtu.be/KqOFZeGmdK0 Archivematica 0.9-beta Bagit] (Oct 10, 2012)<br />
* [http://youtu.be/BSfQ4bVGiDY Archivematica 0.9-beta Premis] (Oct 10, 2012)<br />
* [http://www.youtube.com/watch?v=w7ZS9I1NLtU Archivematica 0.8-alpha demotration]<br />
* [http://http://youtu.be/dFPtDA4nAPY Archivematica 0.7.1-alpha demonstration] (Jun 23, 2011)<br />
* [http://www.youtube.com/watch?v=Py-LtfxWQhY Archivematica 0.7-alpha demonstration] (Feb 18, 2011)</div>Josephhttps://wiki.archivematica.org/index.php?title=Screencasts&diff=6927Screencasts2012-10-10T18:34:15Z<p>Joseph: </p>
<hr />
<div>[[Main Page]] > [[Documentation]] > Screencasts<br />
<br />
<br />
* [http://youtu.be/GWmNfuO1ofw Archivematica 0.9-beta demonstration] (Aug 29, 2012)<br />
* [http://youtu.be/KqOFZeGmdK0 Archivematica 0.9-Bagit] (Oct 10, 2012)<br />
* [http://youtu.be/BSfQ4bVGiDY Archivematica 0.9-Premis] (Oct 10, 2012)<br />
* [http://http://youtu.be/dFPtDA4nAPY Archivematica 0.7.1-alpha demonstration] (Jun 23, 2011)<br />
* [http://www.youtube.com/watch?v=Py-LtfxWQhY Archivematica 0.7-alpha demonstration] (Feb 18, 2011)</div>Josephhttps://wiki.archivematica.org/index.php?title=Meeting_20121010&diff=6919Meeting 201210102012-10-10T17:56:08Z<p>Joseph: /* chat log */</p>
<hr />
<div>Artefactual Systems, Internal Archivematica Dev Mtg, 2012-10-03<br />
<br />
= Development =<br />
<br />
* Joseph worked on Issue 1232: Parse csv metadata in transfer to directories or files within the transfer.<br />
* Joseph starting to work on parsing FITS for file identification insert values into database.<br />
* Courtney mocking up the UI for normalization via FITS today<br />
* Mike C fixed issues with non-standard characters breaking the transfer component picker (thanks berwin22 for help!)<br />
* Mike C added the ability to change user passwords with the dashboard<br />
<br />
= Deployment =<br />
<br />
<br />
<br />
= Testing =<br />
* Joseph updated Evelyn's efir to git<br />
* Courtney needed to create key for authentication<br />
<br />
= Documentation =<br />
* Courtney created screencasts for bags and premis<br />
* Courtney added a brief UM page for Bags https://www.archivematica.org/wiki/UM_Bags<br />
* Evelyn has been analyzing and documenting ingest requirements for TRIM exports https://www.archivematica.org/wiki/TRIM_exports<br />
<br />
=chat log=<br />
<br />
<pre><br />
(10:32:56 AM) djjuhasz: Archivematica!<br />
(10:33:10 AM) epmclellan: hello!<br />
(10:33:15 AM) courtney: yo<br />
(10:33:32 AM) berwin22: I can take notes<br />
(10:33:41 AM) epmclellan: thanks berwin22<br />
(10:33:49 AM) berwin22: dev<br />
(10:33:51 AM) berwin22: I've worked on Issue 1232: Parse csv metadata in transfer to directories or files within the transfer<br />
(10:33:51 AM) berwin22: starting to work on parsing FITS for file identification insert values into database.<br />
(10:34:08 AM) courtney: i'm mocking up that UI today<br />
(10:34:20 AM) artii [~yaaic@out-bc-221.wireless.telus.com] entered the room.<br />
(10:34:27 AM) epmclellan: I've been working closely with Mark Jordan on CONTENTdm DIP upload requirements<br />
(10:34:36 AM) epmclellan: with bulk metadata import<br />
(10:34:45 AM) epmclellan: he has everything he needs now<br />
(10:34:54 AM) JessicaB: great!<br />
(10:35:08 AM) courtney: will this be a ubc patch?<br />
(10:35:53 AM) epmclellan: probably part of the repo, and we'll install a dev version of Archivematica there for testing<br />
(10:35:56 AM) epmclellan: er...right?<br />
(10:36:08 AM) epmclellan: berwin22: ^<br />
(10:36:29 AM) berwin22: I don't think I've seen any git commits from mark... I recall sevein setting him up?<br />
(10:36:40 AM) epmclellan: he's got access, he just hasn't done the dev yet<br />
(10:36:53 AM) djjuhasz: Show me the dev!<br />
(10:36:57 AM) djjuhasz: :P<br />
(10:36:59 AM) mcantelon: Haha<br />
(10:37:04 AM) courtney: ok<br />
(10:37:13 AM) mcantelon: Mike C worked on reorganizing dashboard views/URLs/templates into separate modules<br />
(10:37:21 AM) berwin22: it probably makes the most sense to make ubc a dev install... I wonder if we should make them a git branch?<br />
(10:37:40 AM) epmclellan: it depends on how busted Archivematica is by then<br />
(10:38:14 AM) mcantelon: Mike C fixed issues with non-standard characters breaking the transfer component picker (thanks berwin22 for help!)<br />
(10:38:25 AM) mcantelon: Mike C added the ability to change user passwords with the dashboard<br />
(10:39:15 AM) epmclellan: finished dev?<br />
(10:39:17 AM) fiver: Question: i remember hearing at one of the demos that 1.0 may have plans for incorporating Chain of Preservation model... is this work begun/still a plan for 1.0, or later, or not? <br />
(10:39:43 AM) courtney: fiver: we've already done a gap analysis against COP<br />
(10:39:44 AM) epmclellan: fiver: we'll be doing an analysis of our metadata bs chain of preservation model<br />
(10:39:49 AM) epmclellan: but that won't involve actual dev<br />
(10:39:57 AM) courtney: fiver: that's how we have all the stopping points for appraisal/arrangement<br />
(10:40:01 AM) epmclellan: courtney: I think he's talking about our work with Joe Tennis<br />
(10:40:27 AM) courtney: still, fiver, you can read the Archivaria article if you're interested in the work we did with COP so far<br />
(10:40:31 AM) berwin22: np mcantelon: what do you think of python + non ascii?<br />
(10:40:58 AM) ***mcantelon stares into the darkness and weeps<br />
(10:41:06 AM) fiver: Wheres the article? would love to learn more about how these kind of models get turned into a functional workflow through AM, so if you need any further help lemme know<br />
(10:41:20 AM) fiver: oh, hahah, sorry, archivaria<br />
(10:41:26 AM) fiver: read archivematica<br />
(10:42:03 AM) epmclellan: deployment?<br />
(10:42:19 AM) courtney: fiver: i'll send link<br />
(10:42:37 AM) djjuhasz: That's okay fiver, Courtney was working for the mysterious "Artactual" yesterday <br />
(10:42:37 AM) fiver: courtney: sure! i can also find it...<br />
(10:42:45 AM) berwin22: anything for deployment?<br />
(10:42:51 AM) fiver: hahaha<br />
(10:42:58 AM) fiver: competitors<br />
(10:43:02 AM) berwin22: testing?<br />
(10:43:14 AM) courtney: i can't test until i can update<br />
(10:43:20 AM) epmclellan: berwin22 installed Archivematica directly on my desktop<br />
(10:43:29 AM) epmclellan: after upgrading it to Ubuntu 12.04<br />
(10:43:30 AM) courtney: i have ssh key (public), but no idea what to do <br />
(10:43:35 AM) epmclellan: so I'm able to test now<br />
(10:43:51 AM) courtney: djjuhasz: what? when was i working for Artactual?<br />
(10:44:10 AM) epmclellan: courtney: for about 20 minutes yesterday, until peter fixed a typo<br />
(10:44:21 AM) courtney: oh... missed that<br />
(10:44:21 AM) fiver: re: testing, not sure if i am set up to assist with this yet. have only used AM through the VirtualBox so far...<br />
(10:44:24 AM) berwin22: courtney: sevein will need to add the key to the git server, then we can set you up<br />
(10:44:28 AM) JessicaB: i think you get paid for that time.<br />
(10:44:55 AM) courtney: i emailed sevein, but i think he's off the grid until the 14th or so<br />
(10:45:02 AM) JessicaB: yep<br />
(10:45:04 AM) fiver: good for him!<br />
(10:45:09 AM) epmclellan: berwin22's an expert, he can set you up<br />
(10:45:23 AM) epmclellan: oh, once you get the key, I mean<br />
(10:46:05 AM) epmclellan: docs?<br />
(10:46:17 AM) berwin22: we'll set you up read only for the time being then, then switch you over later<br />
(10:46:28 AM) mcantelon: artii: Seems like Elasticsearch isn't starting consistently... wonder if having something like Monit watch over it would be worth doing?<br />
(10:46:28 AM) djjuhasz: courtney: I think I can set you up on the git server<br />
(10:46:46 AM) djjuhasz: courtney: can you please send your public key via PM or email?<br />
(10:47:13 AM) ARTi [~austin@24-207-101-202.eastlink.ca] entered the room.<br />
(10:47:14 AM) courtney: it is on the internal wiki, dj<br />
(10:47:29 AM) mcantelon: ARTi: ^<br />
(10:47:47 AM) courtney: docs - two new screencasts today: bags and premis<br />
(10:48:01 AM) courtney: added a brief UM page for Bags<br />
(10:48:07 AM) epmclellan: I've been analyzing and documenting ingest requirements for TRIM exports<br />
(10:48:07 AM) ARTi: mcantelon: Ill look at monit<br />
(10:48:09 AM) epmclellan: https://www.archivematica.org/wiki/TRIM_exports<br />
(10:48:20 AM) courtney: berwin22: we can include a link to the screencast there: https://www.archivematica.org/wiki/UM_Bags<br />
(10:48:34 AM) mcantelon: ARTi: Sweet... the ES server seems a bit fragile or something.<br />
(10:49:03 AM) courtney: ES is a delicate flower<br />
(10:49:03 AM) ARTi: yeah.. seems very fragile.<br />
(10:49:36 AM) ARTi: I guess its our only option? too bad its java :]<br />
(10:49:38 AM) berwin22: k<br />
(10:50:07 AM) courtney: docs... i also added a bit more about searching the AIP store on the Archival Storage page of the UM<br />
(10:50:10 AM) ARTi: think its the way that its packaged thats causing issues?<br />
(10:50:14 AM) ARTi: mcantelon: ^<br />
(10:50:17 AM) courtney: https://www.archivematica.org/wiki/UM_archival_storage<br />
(10:51:28 AM) mcantelon: ARTi: Unfortunately I'm not sure if it's crashing or if it doesn't start up, occasionally, in the first place.<br />
(10:51:56 AM) courtney: great twitter feedback from Derangement and Description's Rebecca Goldman about the quality of our documentation<br />
</pre><br />
[[Category:meetings]]</div>Joseph