Creating Custom Workflows 0.8 alpha

From Archivematica
Revision as of 03:15, 28 February 2012 by Joseph (talk | contribs)
Jump to navigation Jump to search

Main Page > Development > Creating Custom Workflows

Read First

This page is used to described editing workflows in the archivematica system.

Overview

The MCP operates on a set of 'MicroService Chains' defined in the MCP database. Thes chains have a starting link, and default next link. The MCP will continue to process along these chains until it reaches a next chain link of Null. Note these chains can have branches, which will process a different set of commands.

Workflow decision tools

This section defines the tools available to select the chain or next chain link to process.

Watched Directories

Watched directories are watching for directories/files placed in them. When one is placed in them, it starts the corresponding Microservice Chain.

MicroServiceChainLinksExitCodes

This mechanism is used to provide an alternative to the default next chain link defined in the MicroServiceChainLink. The default is usually the default error condition, and then you define exit code 0 to go to the next chain in the link.

There are special circumstances, where you may desire the code to change the direction of the processing links. This can be done through the exit code. See Archivematica 0.8 release use of exitCode 179 and 0, in the checkForAccessDirectory microservice for an example.

Choices

Pre 0.8 release of Archivematica, the MCP could only approve a microservice. In 0.8, that has changed to choices being their own steps/microservices.

Choices link to a microservice chain, and are defined in MicroServiceChainChoice. The concept being that the user is selecting a path to follow, and the paths are defined in the microservice chains.

Magic Chain Links

Magic chain links involve getting the next chain link from the unit the job is operating on. I think they have the potential to be very useful when generating unit tests. In archivematica there are two key job types: ( 3, 'assign magic link'), ( 4, 'goto magic link').

A key advantage the magic links provide, is that they allow two or more workflows to share the same watched directory. The items within that watched directory have a flag set to say which link they should go to next.

Creating your first workflow

Make sure you read the section "Read First" (above) first.

The mock situation

Collection of jpgs normalized to bmp for preservation poorly. Want to remove the bmps, from a transfer, then process it as a standard transfer. (will later be normalized to Uncompressed TIFF by archivematica).

Creating a chain

gedit /usr/share/archivematica/mysql view -> highlight mode -> source -> SQL

I find it easier to work back chronologically. The chronological order is: Watched directory watched Move to processing directory Remove .bmp files. Set permissions Move to regular processing watched directory.

So reverse is: Move to regular processing watched directory. Remove .bmp files. Move to processing directory Set permissions Watched directory watched

video

  • Paths may be different
  • patch from video video
Index: src/MCPServer/share/mysql
===================================================================
--- src/MCPServer/share/mysql	(revision 2328)
+++ src/MCPServer/share/mysql	(working copy)
@@ -2774,12 +2774,62 @@
 
 
 
+-- Move to regular processing watched directory. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/activeTransfers/standardTransfer/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (0,      LAST_INSERT_ID(), 'Move to standard transfer directory');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, NULL);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
 
+-- Remove .bmp files. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+-- Move to processing directory --
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+-- Set permissions --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @setFilePermissionsTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+/*
+Watched directory watched
+/var/archivematica/sharedDirectory/watchedDirectories/example1
+*/
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp\'s before processing');
+set @MicroServiceChain = LAST_INSERT_ID();
 
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
+    VALUES ('%watchDirectoryPath%example1', @MicroServiceChain, @expectedTypeTransfer);
 
 
 
@@ -2803,6 +2853,11 @@
 
 
 
+
+
+
+
+
 -- DSPACE TRANSER --
 -- transfer processing complete --
 SET @microserviceGroup  = 'Complete transfer';

using choices

Continuing the example using choices

Index: src/MCPServer/share/mysql
===================================================================
--- src/MCPServer/share/mysql	(revision 2328)
+++ src/MCPServer/share/mysql	(working copy)
@@ -2657,6 +2657,87 @@
     VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
 set @NextMicroServiceChainLink = @MicroServiceChainLink;
 
+
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Continue processing normally');
+set @ContinueProcessingNormallyMicroServiceChain = LAST_INSERT_ID();
+
+-- Remove .bmp files. --
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    ('.bmp', NULL, 'objects', TRUE, NULL, NULL, 'remove_v0.0',  '"%relativeLocation%"');
+SET @AssignfileUUIDstoobjects = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (1,      @AssignfileUUIDstoobjects, 'Remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+-- move to processing directory --
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)     
+    VALUES (@microserviceGroup, @moveToProcessingDirectoryTaskConfig, @defaultNextChainLink);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, @NextMicroServiceChainLink);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'Remove .bmp files');
+set @RemoveBMPfilesFirstMicroServiceChain = LAST_INSERT_ID();
+
+
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, '', '');
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (2,      LAST_INSERT_ID(), 'Workflow decision - remove .bmp files');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @ContinueProcessingNormallyMicroServiceChain);
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @RemoveBMPfilesFirstMicroServiceChain);
+INSERT INTO MicroServiceChainChoice (choiceAvailableAtLink, chainAvailable)
+    VALUES                
+    (@MicroServiceChainLink, @rejectSIPMicroServiceChain);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+INSERT INTO MicroServiceChains (startingLink, description) VALUES (@MicroServiceChainLink,  'create remove .bmp files?');
+set @MicroServiceChain = LAST_INSERT_ID();
+
+INSERT INTO WatchedDirectories (watchedDirectoryPath, chain, expectedType)
+    VALUES ('%watchDirectoryPath%workFlowDecisions/removeBMPFiles/', @MicroServiceChain, @expectedTypeTransfer);
+
+
+INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
+    VALUES
+    (NULL, NULL, NULL, FALSE, NULL, NULL, 'moveTransfer_v0.0', '"%SIPDirectory%" "%sharedPath%watchedDirectories/workFlowDecisions/removeBMPFiles/." "%SIPUUID%" "%sharedPath%" "%SIPUUID%" "%sharedPath%"');
+Set @MovetoworkFlowDecisionsquarantineSIPdirectory = LAST_INSERT_ID();
+INSERT INTO TasksConfigs (taskType, taskTypePKReference, description)
+    VALUES
+    (0,      @MovetoworkFlowDecisionsquarantineSIPdirectory, 'Move to workFlowDecisions-removeBMPFiles directory');
+INSERT INTO MicroServiceChainLinks (microserviceGroup, currentTask, defaultNextChainLink)    
+    VALUES (@microserviceGroup, LAST_INSERT_ID(), NULL);
+set @MicroServiceChainLink = LAST_INSERT_ID();
+INSERT INTO MicroServiceChainLinksExitCodes (microServiceChainLink, exitCode, nextMicroServiceChainLink) 
+    VALUES (@MicroServiceChainLink, 0, NULL);
+set @NextMicroServiceChainLink = @MicroServiceChainLink;
+
+
+
 SET @microserviceGroup  = 'Include default Transfer processingMCP.xml';
 INSERT INTO StandardTasksConfigs (filterFileEnd, filterFileStart, filterSubDir, requiresOutputLock, standardOutputFile, standardErrorFile, execute, arguments)
     VALUES
@@ -2802,7 +2883,6 @@
 
 
 
-
 -- DSPACE TRANSER --
 -- transfer processing complete --
 SET @microserviceGroup  = 'Complete transfer';

using magic links

Retrieved from "https://wiki.archivematica.org/index.php?title=Creating_Custom_Workflows_0.8_alpha&oldid=5518"