Difference between revisions of "Creating Custom Workflows 0.8 alpha"

From Archivematica
Jump to navigation Jump to search
 
 
(22 intermediate revisions by one other user not shown)
Line 1: Line 1:
 
[[Main Page]] > [[Development]] > [[Creating Custom Workflows]]
 
[[Main Page]] > [[Development]] > [[Creating Custom Workflows]]
 +
 +
<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information. </div> <p>
 +
 +
=Read First=
 +
This page is used to described editing workflows in the archivematica system.
 +
 +
[http://archivematica.org/wiki/index.php?title=Contribute_code We strongly encourage code contribution]. It requires use of the [[Development_environment]]
 +
 +
==Overview==
 +
The MCP operates on a set of 'MicroService Chains' defined in the MCP database. Thes chains have a starting link, and default next link. The MCP will continue to process along these chains until it reaches a next chain link of Null. Note these chains can have branches, which will process a different set of commands.
 +
 +
The archivematica workflow is now broken into two parts. The first is the transfer area. This section of the workflow is to prepare objects for SIP processing in archivematica. Transfer processing handles the format the objects were received in, where SIP processing is a very narrow workflow, designed to create archivematica DIPs and AIPs.
 +
==key components==
 +
 +
=== MicroServiceChainLinks Fields===
 +
# currentTask
 +
#: FK TasksConfigs pk
 +
#: The task that operates at this stage in the chain.
 +
# defaultNextChainLink
 +
#: If the exit code of this Job is not defined in the MicroServiceChainLinksExitCodes, goto this next chain.
 +
#:
 +
#:The following are really only used for advanced/not implemented features.
 +
# defaultPlaySound
 +
# microserviceGroup
 +
# reloadFileList
 +
# defaultExitMessage
 +
 +
===TasksConfigs Fields===
 +
# taskType
 +
#: The task type is highly important.
 +
#: select * from TaskTypes;
 +
#: +----+---------------------------------+
 +
#: | pk | description                    |
 +
#: +----+---------------------------------+
 +
#: |  0 | one instance                    |
 +
#: |  1 | for each file                  |
 +
#: |  2 | get user choice to proceed with |
 +
#: |  3 | assign magic link              |
 +
#: |  4 | goto magic link                |
 +
#: +----+---------------------------------+
 +
#:
 +
# taskTypePKReference
 +
#: Used in combination with the taskType. The type allows the MCP code to map to a table, and the pkReference knows which entry in that table to look at.
 +
# description
 +
#: a text description to appear in the dashboard
 +
 +
===StandardTasksConfigs Fields===
 +
# File search filters
 +
#: filterFileEnd - useful for looking for extensions.
 +
#: filterFileStart
 +
#: filterSubDir - sub directory to operate on within the unit location
 +
# requiresOutputLock
 +
#: boolean. used when logging to files. If a number of tasks are writing to the same file it's used.
 +
#: this has more historical significance for archivematica than future.
 +
# standardOutputFile
 +
# standardErrorFile
 +
# execute
 +
#: linked to [http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/MCPClient/etc/archivematicaClientModules archivematicaClientModules]
 +
#: the client maps it to it's executable, and will run it as though on the command line with the given arguments below.
 +
#: The client can map these to anything callable at the command line.
 +
#: Whatever is called will need to return without human intervention, or the system will hang!
 +
# arguments
 +
#: arguments given to the executable
 +
#: some variables are replaced when the task is created. See replacement dics getReplacementDic():
 +
#:* [[http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/MCPServer/lib/unitSIP.py SIPs]]
 +
#:* [[http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/MCPServer/lib/unitTransfer.py Transfers ]]
 +
#:* [[http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/MCPServer/lib/unitFile.py Files]]
 +
#:* [[http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/MCPServer/lib/unitDIP.py Dips]]
 +
#: note: transfers use the word SIP instead of transfer for simplifying workflow migration from previous revisions of archivematica
 +
 +
==Workflow decision tools==
 +
This section defines the tools available to select the chain or next chain link to process.
 +
 +
===Watched Directories===
 +
Watched directories are watching for directories/files placed in them. When one is placed in them, it starts the corresponding Microservice Chain.
 +
====WatchedDirectories Fields====
 +
# watchedDirectoryPath
 +
#: The path to the directory. Starts with variable '%watchDirectoryPath%', which is replaced by the MCP with the location of the watched directories.
 +
# chain
 +
#: The pk of the MicroServiceChains to start processing down.
 +
# onlyActOnDirectories
 +
#: Always true for Archivematica 0.8
 +
#: Future or expanded use of MCP to allow for watching of individual files
 +
# expectedTypeTells the MCP what type of unit to expect
 +
#: two main unit types: SIPs and Transfers
 +
#: The MCP will try to match the directory to an existing unit, or create a unit to represent the directory.
 +
 +
====Restrictions====
 +
* A watched directory can not contain a watched directory.
 +
 +
===MicroServiceChainLinksExitCodes===
 +
This mechanism is used to provide an alternative to the default next chain link defined in the MicroServiceChainLink. The default is usually the default error condition, and then you define exit code 0 to go to the next chain in the link.
 +
 +
There are special circumstances, where you may desire the code to change the direction of the processing links. This can be done through the exit code. See Archivematica 0.8 release use of exitCode 179 and 0, in the checkForAccessDirectory microservice for an example.
 +
 +
===Choices===
 +
Pre 0.8 release of Archivematica, the MCP could only approve a microservice. In 0.8, that has changed to choices being their own steps/microservices.
 +
 +
Choices link to a microservice chain, and are defined in MicroServiceChainChoice. The concept being that the user is selecting a path to follow, and the paths are defined in the microservice chains.
 +
 +
===Magic Chain Links===
 +
Magic chain links involve getting the next chain link from the unit the job is operating on. I think they have the potential to be very useful when generating unit tests.
 +
In archivematica there are two key job types:    ( 3, 'assign magic link'),    ( 4, 'goto magic link').
 +
 +
A key advantage the magic links provide, is that they allow two or more workflows to share the same watched directory. The items within that watched directory have a flag set to say which link they should go to next.
 +
 +
=Creating your first workflow=
 +
Make sure you read the section "Read First" (above) first.
 +
 +
== The mock situation ==
 +
Collection of jpgs normalized to bmp for preservation poorly. Want to remove the bmps, from a transfer, then process it as a standard transfer. (will later be normalized to Uncompressed TIFF by archivematica).
 +
 +
== Creating a chain ==
 +
gedit /usr/share/archivematica/mysql
 +
view -> highlight mode -> source -> SQL
 +
 +
I find it easier to work back chronologically.
 +
The chronological order is:
 +
# Watched directory watched
 +
# Move to processing directory
 +
# Remove .bmp files.
 +
# Set permissions
 +
# Move to regular processing watched directory.
 +
 +
So reverse is:
 +
# Move to regular processing watched directory.
 +
# Remove .bmp files.
 +
# Move to processing directory
 +
# Set permissions
 +
# Watched directory watched
 +
 +
[http://youtu.be/haotj_NlbX0 video]
 +
* Paths may be different
 +
* patch from video video
 +
<pre>
 +
Index: src/MCPServer/share/mysql
 +
===================================================================
 +
--- src/MCPServer/share/mysql (revision 2328)
 +
+++ src/MCPServer/share/mysql (working copy)
 +
@@ -2774,12 +2774,62 @@
 +
 +
 +
 +
+-- Move to regular processing watched directory. --
 +
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
+    VALUES
 +
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/activeTransfers/standardTransfer/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
 +
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
+    VALUES
 +
+    (0,      LAST_INSERT_ID(), 'Move to standard transfer directory');
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, NULL);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
 +
+-- Remove .bmp files. --
 +
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
+    VALUES
 +
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
 +
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
 +
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
+    VALUES
 +
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
+-- Move to processing directory --
 +
+-- move to processing directory --
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
+-- Set permissions --
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, @setFilePermissionsTaskConfig, @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
+/*
 +
+Watched directory watched
 +
+/var/archivematica/sharedDirectory/watchedDirectories/example1
 +
+*/
 +
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp\'s before processing');
 +
+set @MicroServiceChain = LAST_INSERT_ID();
 +
 +
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
 +
+    VALUES ('%watchDirectoryPath%example1', @MicroServiceChain, @expectedTypeTransfer);
 +
 +
 +
 +
@@ -2803,6 +2853,11 @@
 +
 +
 +
 +
+
 +
+
 +
+
 +
+
 +
+
 +
-- DSPACE TRANSER --
 +
-- transfer processing complete --
 +
SET @microserviceGroup  = 'Complete transfer';
 +
</pre>
 +
 +
==using choices==
 +
===Continuing the example using choices ===
 +
<pre>
 +
Index: src/MCPServer/share/mysql
 +
===================================================================
 +
--- src/MCPServer/share/mysql (revision 2328)
 +
+++ src/MCPServer/share/mysql (working copy)
 +
@@ -2657,6 +2657,87 @@
 +
    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
+
 +
+-- move to processing directory --
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+
 +
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Continue processing normally');
 +
+set @ContinueProcessingNormallyMicroServiceChain = LAST_INSERT_ID();
 +
+
 +
+-- Remove .bmp files. --
 +
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
+    VALUES
 +
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
 +
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
 +
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
+    VALUES
 +
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
+
 +
+-- move to processing directory --
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
+
 +
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp files');
 +
+set @RemoveBMPfilesFirstMicroServiceChain = LAST_INSERT_ID();
 +
+
 +
+
 +
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
+    VALUES
 +
+    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
 +
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
+    VALUES
 +
+    (2,      LAST_INSERT_ID(), 'Workflow decision - remove .bmp files');
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
+    VALUES               
 +
+    (@MicroServiceChainLink, @ContinueProcessingNormallyMicroServiceChain);
 +
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
+    VALUES               
 +
+    (@MicroServiceChainLink, @RemoveBMPfilesFirstMicroServiceChain);
 +
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
+    VALUES               
 +
+    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
+
 +
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'create remove .bmp files?');
 +
+set @MicroServiceChain = LAST_INSERT_ID();
 +
+
 +
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
 +
+    VALUES ('%watchDirectoryPath%workFlowDecisions/removeBMPFiles/', @MicroServiceChain, @expectedTypeTransfer);
 +
+
 +
+
 +
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
+    VALUES
 +
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/workFlowDecisions/removeBMPFiles/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
 +
+Set @MovetoworkFlowDecisionsquarantineSIPdirectory = LAST_INSERT_ID();
 +
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
+    VALUES
 +
+    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-removeBMPFiles directory');
 +
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
+set @MicroServiceChainLink = LAST_INSERT_ID();
 +
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
+    VALUES (@MicroServiceChainLink, 0, NULL);
 +
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
+
 +
+
 +
+
 +
SET @microserviceGroup  = 'Include default Transfer processingMCP.xml';
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
@@ -2802,7 +2883,6 @@
 +
 +
 +
 +
-
 +
-- DSPACE TRANSER --
 +
-- transfer processing complete --
 +
SET @microserviceGroup  = 'Complete transfer';
 +
 +
</pre>
 +
 +
==Using Magic Links==
 +
Magic links are used so two workflows can share the same watched directory.
 +
[[http://archivematica.org/wiki/index.php?title=Creating_Custom_Workflows_0.8_alpha#Magic_Chain_Links see above]]
 +
They use units that can store a magic chain link (SIPs and Transfers).
 +
===Example===
 +
Let's look at where archivematica get's the user decision to send the transfer to quarantine or not.
 +
 +
* Both workflows start with tasktype 3, where they set the chain link to resume processing on. This is stored in the unit itself.
 +
* Both then move the their transfer to the quarantine workflow decision directory
 +
* Only one has the watched directory and task type 4
 +
*: This is because it only has to occur once. There is only one watched directory, and it is liked to a chain that will start by loading magic link from the unit.
 +
* Each has then finished loading their magic link, and is back on their perspective processing chain. Where they present the user with the appropriate choices for the chain.
 +
 +
<pre>
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (2,      LAST_INSERT_ID(), 'Workflow decision - create transfer backup');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @CreateTransferBackupMicroServiceChain);
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @DoNotCreateTransferBackupMicroServiceChain);
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
set @WorkflowdecisioncreateTransferBackupDefaultTransferMagicChainLink = @MicroServiceChainLink;
 +
 +
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, NULL, NULL);
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (4,      LAST_INSERT_ID(), 'Find type to process as');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
 +
INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'create transfer backup ?');
 +
set @MicroServiceChain = LAST_INSERT_ID();
 +
 +
INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
 +
    VALUES ('%watchDirectoryPath%workFlowDecisions/createTransferBackup/', @MicroServiceChain, @expectedTypeTransfer);
 +
 +
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/workFlowDecisions/createTransferBackup/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
 +
Set @MovetoworkFlowDecisionsquarantineSIPdirectory = LAST_INSERT_ID();
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-createTransferBackup directory');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
    VALUES (@MicroServiceChainLink, 0, NULL);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, @WorkflowdecisioncreateTransferBackupDefaultTransferMagicChainLink, NULL);
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (3,      LAST_INSERT_ID(), 'Designate to process as a standard transfer');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
</pre>
 +
 +
 +
<pre>
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (2,      LAST_INSERT_ID(), 'Workflow decision - send transfer to quarantine');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @QuarantineMicroServiceChain);
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @SkipQuarantineMicroServiceChain);
 +
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
 +
    VALUES               
 +
    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
set @WorkflowdecisionsendtransfertoquarantineDspace17TransferMagicChainLink = @NextMicroServiceChainLink;
 +
 +
 +
 +
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-quarantineSIP directory');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
    VALUES (@MicroServiceChainLink, 0, NULL);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
 +
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
 +
    VALUES
 +
    (NULL, NULL, NULL, FALSE, NULL, NULL, @WorkflowdecisionsendtransfertoquarantineDspace17TransferMagicChainLink, NULL);
 +
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
 +
    VALUES
 +
    (3,      LAST_INSERT_ID(), 'Designate to process as a DSpace 1.7 transfer');
 +
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)   
 +
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
 +
set @MicroServiceChainLink = LAST_INSERT_ID();
 +
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink)
 +
    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 +
set @NextMicroServiceChainLink = @MicroServiceChainLink;
 +
</pre>
 +
 +
=Creating a custom microservice=
 +
These will make heavy use of the replacement dics listed [http://archivematica.org/wiki/index.php?title=Creating_Custom_Workflows_0.8_alpha#StandardTasksConfigs_Fields above] in the standard tasks configs arguments. Remember SIP processing workflow, as opposed to Transfer processing, will need to work for all SIPs. If you require metadata to end up in the METS.xml file, or another metadata container in the AIP, we recomend creating a set of tables in the database(That represent a metadata standard), populate the tables during transfer processing, and modifying the create METS script to include your data. There is a handy tool [http://pim.fcla.edu/validate here] for mets validation.
 +
 +
== Python ==
 +
Most microservices are writen in python. You will probably be able to save some time by writing yours in python, and using functions already available to you. These are generally in the [http://code.google.com/p/archivematica/source/browse/tags/release-0.8-alpha/src/archivematicaCommon/lib/ archivematicaCommon directory].
 +
Some key reusable code I can think of off the top of my head, is the database interface, and the createEvent function in the databaseFunctions.py file.
 +
 +
===CreateEvent===
 +
Used to create premis events related to an object in the METS.xml file produced by archivematica.

Latest revision as of 15:42, 11 February 2020

Main Page > Development > Creating Custom Workflows

This page is no longer being maintained and may contain inaccurate information. Please see the Archivematica documentation for up-to-date information.

Read First[edit]

This page is used to described editing workflows in the archivematica system.

We strongly encourage code contribution. It requires use of the Development_environment

Overview[edit]

The MCP operates on a set of 'MicroService Chains' defined in the MCP database. Thes chains have a starting link, and default next link. The MCP will continue to process along these chains until it reaches a next chain link of Null. Note these chains can have branches, which will process a different set of commands.

The archivematica workflow is now broken into two parts. The first is the transfer area. This section of the workflow is to prepare objects for SIP processing in archivematica. Transfer processing handles the format the objects were received in, where SIP processing is a very narrow workflow, designed to create archivematica DIPs and AIPs.

key components[edit]

MicroServiceChainLinks Fields[edit]

  1. currentTask
    FK TasksConfigs pk
    The task that operates at this stage in the chain.
  2. defaultNextChainLink
    If the exit code of this Job is not defined in the MicroServiceChainLinksExitCodes, goto this next chain.
    The following are really only used for advanced/not implemented features.
  3. defaultPlaySound
  4. microserviceGroup
  5. reloadFileList
  6. defaultExitMessage

TasksConfigs Fields[edit]

  1. taskType
    The task type is highly important.
    select * from TaskTypes;
    +----+---------------------------------+
    | pk | description |
    +----+---------------------------------+
    | 0 | one instance |
    | 1 | for each file |
    | 2 | get user choice to proceed with |
    | 3 | assign magic link |
    | 4 | goto magic link |
    +----+---------------------------------+
  2. taskTypePKReference
    Used in combination with the taskType. The type allows the MCP code to map to a table, and the pkReference knows which entry in that table to look at.
  3. description
    a text description to appear in the dashboard

StandardTasksConfigs Fields[edit]

  1. File search filters
    filterFileEnd - useful for looking for extensions.
    filterFileStart
    filterSubDir - sub directory to operate on within the unit location
  2. requiresOutputLock
    boolean. used when logging to files. If a number of tasks are writing to the same file it's used.
    this has more historical significance for archivematica than future.
  3. standardOutputFile
  4. standardErrorFile
  5. execute
    linked to archivematicaClientModules
    the client maps it to it's executable, and will run it as though on the command line with the given arguments below.
    The client can map these to anything callable at the command line.
    Whatever is called will need to return without human intervention, or the system will hang!
  6. arguments
    arguments given to the executable
    some variables are replaced when the task is created. See replacement dics getReplacementDic():
    note: transfers use the word SIP instead of transfer for simplifying workflow migration from previous revisions of archivematica

Workflow decision tools[edit]

This section defines the tools available to select the chain or next chain link to process.

Watched Directories[edit]

Watched directories are watching for directories/files placed in them. When one is placed in them, it starts the corresponding Microservice Chain.

WatchedDirectories Fields[edit]

  1. watchedDirectoryPath
    The path to the directory. Starts with variable '%watchDirectoryPath%', which is replaced by the MCP with the location of the watched directories.
  2. chain
    The pk of the MicroServiceChains to start processing down.
  3. onlyActOnDirectories
    Always true for Archivematica 0.8
    Future or expanded use of MCP to allow for watching of individual files
  4. expectedTypeTells the MCP what type of unit to expect
    two main unit types: SIPs and Transfers
    The MCP will try to match the directory to an existing unit, or create a unit to represent the directory.

Restrictions[edit]

  • A watched directory can not contain a watched directory.

MicroServiceChainLinksExitCodes[edit]

This mechanism is used to provide an alternative to the default next chain link defined in the MicroServiceChainLink. The default is usually the default error condition, and then you define exit code 0 to go to the next chain in the link.

There are special circumstances, where you may desire the code to change the direction of the processing links. This can be done through the exit code. See Archivematica 0.8 release use of exitCode 179 and 0, in the checkForAccessDirectory microservice for an example.

Choices[edit]

Pre 0.8 release of Archivematica, the MCP could only approve a microservice. In 0.8, that has changed to choices being their own steps/microservices.

Choices link to a microservice chain, and are defined in MicroServiceChainChoice. The concept being that the user is selecting a path to follow, and the paths are defined in the microservice chains.

Magic Chain Links[edit]

Magic chain links involve getting the next chain link from the unit the job is operating on. I think they have the potential to be very useful when generating unit tests. In archivematica there are two key job types: ( 3, 'assign magic link'), ( 4, 'goto magic link').

A key advantage the magic links provide, is that they allow two or more workflows to share the same watched directory. The items within that watched directory have a flag set to say which link they should go to next.

Creating your first workflow[edit]

Make sure you read the section "Read First" (above) first.

The mock situation[edit]

Collection of jpgs normalized to bmp for preservation poorly. Want to remove the bmps, from a transfer, then process it as a standard transfer. (will later be normalized to Uncompressed TIFF by archivematica).

Creating a chain[edit]

gedit /usr/share/archivematica/mysql view -> highlight mode -> source -> SQL

I find it easier to work back chronologically. The chronological order is:

  1. Watched directory watched
  2. Move to processing directory
  3. Remove .bmp files.
  4. Set permissions
  5. Move to regular processing watched directory.

So reverse is:

  1. Move to regular processing watched directory.
  2. Remove .bmp files.
  3. Move to processing directory
  4. Set permissions
  5. Watched directory watched

video

  • Paths may be different
  • patch from video video
Index: src/MCPServer/share/mysql
===================================================================
--- src/MCPServer/share/mysql	(revision 2328)
+++ src/MCPServer/share/mysql	(working copy)
@@ -2774,12 +2774,62 @@
 
 
 
+-- Move to regular processing watched directory. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/activeTransfers/standardTransfer/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (0,      LAST_INSERT_ID(), 'Move to standard transfer directory');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, NULL);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
 
+-- Remove .bmp files. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+-- Move to processing directory --
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+-- Set permissions --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @setFilePermissionsTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+/*
+Watched directory watched
+/var/archivematica/sharedDirectory/watchedDirectories/example1
+*/
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp\'s before processing');
+set @MicroServiceChain = LAST_INSERT_ID();
 
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
+    VALUES ('%watchDirectoryPath%example1', @MicroServiceChain, @expectedTypeTransfer);
 
 
 
@@ -2803,6 +2853,11 @@
 
 
 
+
+
+
+
+
 -- DSPACE TRANSER --
 -- transfer processing complete --
 SET @microserviceGroup  = 'Complete transfer';

using choices[edit]

Continuing the example using choices[edit]

Index: src/MCPServer/share/mysql
===================================================================
--- src/MCPServer/share/mysql	(revision 2328)
+++ src/MCPServer/share/mysql	(working copy)
@@ -2657,6 +2657,87 @@
     VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Continue processing normally');
+set @ContinueProcessingNormallyMicroServiceChain = LAST_INSERT_ID();
+
+-- Remove .bmp files. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp files');
+set @RemoveBMPfilesFirstMicroServiceChain = LAST_INSERT_ID();
+
+
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (2,      LAST_INSERT_ID(), 'Workflow decision - remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @ContinueProcessingNormallyMicroServiceChain);
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @RemoveBMPfilesFirstMicroServiceChain);
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'create remove .bmp files?');
+set @MicroServiceChain = LAST_INSERT_ID();
+
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
+    VALUES ('%watchDirectoryPath%workFlowDecisions/removeBMPFiles/', @MicroServiceChain, @expectedTypeTransfer);
+
+
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/workFlowDecisions/removeBMPFiles/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
+Set @MovetoworkFlowDecisionsquarantineSIPdirectory = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-removeBMPFiles directory');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, NULL);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+
+
 SET @microserviceGroup  = 'Include default Transfer processingMCP.xml';
 INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
     VALUES
@@ -2802,7 +2883,6 @@
 
 
 
-
 -- DSPACE TRANSER --
 -- transfer processing complete --
 SET @microserviceGroup  = 'Complete transfer';

Using Magic Links[edit]

Magic links are used so two workflows can share the same watched directory. [see above] They use units that can store a magic chain link (SIPs and Transfers).

Example[edit]

Let's look at where archivematica get's the user decision to send the transfer to quarantine or not.

  • Both workflows start with tasktype 3, where they set the chain link to resume processing on. This is stored in the unit itself.
  • Both then move the their transfer to the quarantine workflow decision directory
  • Only one has the watched directory and task type 4
    This is because it only has to occur once. There is only one watched directory, and it is liked to a chain that will start by loading magic link from the unit.
  • Each has then finished loading their magic link, and is back on their perspective processing chain. Where they present the user with the appropriate choices for the chain.
INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (2,      LAST_INSERT_ID(), 'Workflow decision - create transfer backup');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @CreateTransferBackupMicroServiceChain);
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @DoNotCreateTransferBackupMicroServiceChain);
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
set @NextMicroServiceChainLink = @MicroServiceChainLink;
set @WorkflowdecisioncreateTransferBackupDefaultTransferMagicChainLink = @MicroServiceChainLink;


INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, NULL, NULL);
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (4,      LAST_INSERT_ID(), 'Find type to process as');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
set @MicroServiceChainLink = LAST_INSERT_ID();

INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'create transfer backup ?');
set @MicroServiceChain = LAST_INSERT_ID();

INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
    VALUES ('%watchDirectoryPath%workFlowDecisions/createTransferBackup/', @MicroServiceChain, @expectedTypeTransfer);


INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/workFlowDecisions/createTransferBackup/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
Set @MovetoworkFlowDecisionsquarantineSIPdirectory = LAST_INSERT_ID();
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-createTransferBackup directory');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
    VALUES (@MicroServiceChainLink, 0, NULL);
set @NextMicroServiceChainLink = @MicroServiceChainLink;

INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, @WorkflowdecisioncreateTransferBackupDefaultTransferMagicChainLink, NULL);
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (3,      LAST_INSERT_ID(), 'Designate to process as a standard transfer');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
set @NextMicroServiceChainLink = @MicroServiceChainLink;



INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (2,      LAST_INSERT_ID(), 'Workflow decision - send transfer to quarantine');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @QuarantineMicroServiceChain);
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @SkipQuarantineMicroServiceChain);
INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
    VALUES                
    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
set @NextMicroServiceChainLink = @MicroServiceChainLink;
set @WorkflowdecisionsendtransfertoquarantineDspace17TransferMagicChainLink = @NextMicroServiceChainLink;




INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-quarantineSIP directory');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
    VALUES (@MicroServiceChainLink, 0, NULL);
set @NextMicroServiceChainLink = @MicroServiceChainLink;

INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
    VALUES
    (NULL, NULL, NULL, FALSE, NULL, NULL, @WorkflowdecisionsendtransfertoquarantineDspace17TransferMagicChainLink, NULL);
INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
    VALUES
    (3,      LAST_INSERT_ID(), 'Designate to process as a DSpace 1.7 transfer');
INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
set @MicroServiceChainLink = LAST_INSERT_ID();
INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
set @NextMicroServiceChainLink = @MicroServiceChainLink;

Creating a custom microservice[edit]

These will make heavy use of the replacement dics listed above in the standard tasks configs arguments. Remember SIP processing workflow, as opposed to Transfer processing, will need to work for all SIPs. If you require metadata to end up in the METS.xml file, or another metadata container in the AIP, we recomend creating a set of tables in the database(That represent a metadata standard), populate the tables during transfer processing, and modifying the create METS script to include your data. There is a handy tool here for mets validation.

Python[edit]

Most microservices are writen in python. You will probably be able to save some time by writing yours in python, and using functions already available to you. These are generally in the archivematicaCommon directory. Some key reusable code I can think of off the top of my head, is the database interface, and the createEvent function in the databaseFunctions.py file.

CreateEvent[edit]

Used to create premis events related to an object in the METS.xml file produced by archivematica.

Retrieved from "https://wiki.archivematica.org/index.php?title=Creating_Custom_Workflows_0.8_alpha&oldid=13330"