Difference between revisions of "MCPServer/TaskTypes"

From Archivematica
Jump to navigation Jump to search
(Added breadcrumbs, docs note)
(Added links to table description on MCP page)
Line 22: Line 22:
 
For each '''MicroServiceChainLink''' in the [[MCP | MCP Server ]], the TaskType determines what code will be executed for that task.  There are 13 task types.
 
For each '''MicroServiceChainLink''' in the [[MCP | MCP Server ]], the TaskType determines what code will be executed for that task.  There are 13 task types.
  
Reading about the [[MCP | MCP Server ]] is highly recommended to understand this page.
+
Reading about the [[MCP | MCP Server]] is highly recommended to understand this page.
  
 
== General ==
 
== General ==
Line 29: Line 29:
 
* '''Description in TaskTypes''': one instance
 
* '''Description in TaskTypes''': one instance
 
* '''LinkTaskMananger''': linkTaskManagerDirectories
 
* '''LinkTaskMananger''': linkTaskManagerDirectories
* '''TaskTypePKReference''': FK to '''StandardTasksConfigs'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#StandardTasksConfigs | StandardTasksConfigs]]
 
Run the command listed in StandardTasksConfigs exactly once.  TODO filterSubDir?  [[MCP/TaskTypes/one_instance | Old Notes]]
 
Run the command listed in StandardTasksConfigs exactly once.  TODO filterSubDir?  [[MCP/TaskTypes/one_instance | Old Notes]]
  
Line 36: Line 36:
 
* '''Description in TaskTypes''': for each file
 
* '''Description in TaskTypes''': for each file
 
* '''LinkTaskMananger''': linkTaskManagerFiles
 
* '''LinkTaskMananger''': linkTaskManagerFiles
* '''TaskTypePKReference''': FK to '''StandardTasksConfigs'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#StandardTasksConfigs | StandardTasksConfigs]]
 
Run the command once for each file associated with that Unit (SIP, DIP, Transfer).  Files can be filtered using filterFileEnd, filterFileStart (works on the basename of the file) or filterSubDir (selects only files in those directories)  [[MCP/TaskTypes/for_each_file | Old Notes]]
 
Run the command once for each file associated with that Unit (SIP, DIP, Transfer).  Files can be filtered using filterFileEnd, filterFileStart (works on the basename of the file) or filterSubDir (selects only files in those directories)  [[MCP/TaskTypes/for_each_file | Old Notes]]
  
Line 45: Line 45:
 
* '''Description in TaskTypes''': get user choice to proceed with
 
* '''Description in TaskTypes''': get user choice to proceed with
 
* '''LinkTaskMananger''': linkTaskManagerChoice
 
* '''LinkTaskMananger''': linkTaskManagerChoice
* '''TaskTypePKReference''': FK to '''MicroServiceChainChoice'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#MicroServiceChainChoice | MicroServiceChainChoice]]
 
* '''Use Case''': Run different MicroServiceChains based on user input
 
* '''Use Case''': Run different MicroServiceChains based on user input
 
Run a MicroServiceChain, depending on what the user chooses.  Each MicroServiceChainChoice knows where it should be displayed (choiceAvailableAtLink), and what chain to run if it is selected (chainAvailable).  The text to display comes from MicroServiceChains.description, as does the MicroServiceChainLink to run (MicroServiceChains.startingLink)
 
Run a MicroServiceChain, depending on what the user chooses.  Each MicroServiceChainChoice knows where it should be displayed (choiceAvailableAtLink), and what chain to run if it is selected (chainAvailable).  The text to display comes from MicroServiceChains.description, as does the MicroServiceChainLink to run (MicroServiceChains.startingLink)
Line 53: Line 53:
 
* '''Description in TaskTypes''': get replacement dic from user choice
 
* '''Description in TaskTypes''': get replacement dic from user choice
 
* '''LinkTaskMananger''': linkTaskManagerReplacementDicFromChoice
 
* '''LinkTaskMananger''': linkTaskManagerReplacementDicFromChoice
* '''TaskTypePKReference''': FK to '''MicroServiceChoiceReplacementDic'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#MicroServiceChoiceReplacementDic | MicroServiceChoiceReplacementDic]]
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you know all the options ahead of time.  Adds the user choice as a replacement variable.
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you know all the options ahead of time.  Adds the user choice as a replacement variable.
 
Adds a new replacement dict and value for the rest of the chain.  Future commands can use replacement %FOO% in their StandardTaskConfigs.arguments, and it will be replaced by the user's choice.  Each MicroServiceChoiceReplacementDic knows where it should be displayed (choiceAvailableAtLink), what text to display (description) and what replacement variables to add (replacementDic).
 
Adds a new replacement dict and value for the rest of the chain.  Future commands can use replacement %FOO% in their StandardTaskConfigs.arguments, and it will be replaced by the user's choice.  Each MicroServiceChoiceReplacementDic knows where it should be displayed (choiceAvailableAtLink), what text to display (description) and what replacement variables to add (replacementDic).
Line 63: Line 63:
 
* '''Description in TaskTypes''': Get microservice generated list in stdOut
 
* '''Description in TaskTypes''': Get microservice generated list in stdOut
 
* '''LinkTaskMananger''': linkTaskManagerGetMicroserviceGeneratedListInStdOut
 
* '''LinkTaskMananger''': linkTaskManagerGetMicroserviceGeneratedListInStdOut
* '''TaskTypePKReference''': FK to '''StandardTasksConfigs'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#StandardTasksConfigs | StandardTasksConfigs]]
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time.  Requires [[#Get User Choice - select from MicroService Generated List]]
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time.  Requires [[#Get User Choice - select from MicroService Generated List]]
 
Takes the output (stdout) of the command run, evaluates it into a single python object, and adds it to the chain's passVar as a choicesDic.  The output is almost always a dict of {'choice to display':'value for replacement dict'}.
 
Takes the output (stdout) of the command run, evaluates it into a single python object, and adds it to the chain's passVar as a choicesDic.  The output is almost always a dict of {'choice to display':'value for replacement dict'}.
Line 71: Line 71:
 
* '''Description in TaskTypes''': Get user choice from microservice generated list
 
* '''Description in TaskTypes''': Get user choice from microservice generated list
 
* '''LinkTaskMananger''': linkTaskManagerGetUserChoiceFromMicroserviceGeneratedList
 
* '''LinkTaskMananger''': linkTaskManagerGetUserChoiceFromMicroserviceGeneratedList
* '''TaskTypePKReference''': FK to '''StandardTasksConfigs'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#StandardTasksConfigs | StandardTasksConfigs]]
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time.  Adds the user choice as a replacement variable.  Requires [[#Get User Choice - select from MicroService Generated List]]
 
* '''Use Case''': Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time.  Adds the user choice as a replacement variable.  Requires [[#Get User Choice - select from MicroService Generated List]]
 
Looks for a choicesDic in the passVar, and displays the choices to the user.  Values to display are the choicesDic keys, and the choicesDic value is put in a replacement variable specified in StandardTasksConfig.execute.
 
Looks for a choicesDic in the passVar, and displays the choices to the user.  Values to display are the choicesDic keys, and the choicesDic value is put in a replacement variable specified in StandardTasksConfig.execute.
Line 83: Line 83:
 
* '''Description in TaskTypes''': linkTaskManagerSetUnitVariable
 
* '''Description in TaskTypes''': linkTaskManagerSetUnitVariable
 
* '''LinkTaskMananger''': linkTaskManagerSetUnitVariable
 
* '''LinkTaskMananger''': linkTaskManagerSetUnitVariable
* '''TaskTypePKReference''': FK to '''TasksConfigsSetUnitVariable'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#TasksConfigsSetUnitVariable | TasksConfigsSetUnitVariable]]
 +
In progress...
  
 
=== Get Unit Variable ===
 
=== Get Unit Variable ===
Line 89: Line 90:
 
* '''Description in TaskTypes''': linkTaskManagerUnitVariableLinkPull
 
* '''Description in TaskTypes''': linkTaskManagerUnitVariableLinkPull
 
* '''LinkTaskMananger''': linkTaskManagerUnitVariableLinkPull
 
* '''LinkTaskMananger''': linkTaskManagerUnitVariableLinkPull
* '''TaskTypePKReference''': FK to '''TasksConfigsUnitVariableLinkPull'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#TasksConfigsUnitVariableLinkPull | TasksConfigsUnitVariableLinkPull]]
 +
In progress...
  
 
== Transcoding ==
 
== Transcoding ==
Line 97: Line 99:
 
* '''Description in TaskTypes''': Split creating Jobs for each file
 
* '''Description in TaskTypes''': Split creating Jobs for each file
 
* '''LinkTaskMananger''': linkTaskManagerSplit
 
* '''LinkTaskMananger''': linkTaskManagerSplit
* '''TaskTypePKReference''': FK to '''TasksConfigsStartLinkForEachFile'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#TasksConfigsStartLinkForEachFile | TasksConfigsStartLinkForEachFile]]
 +
In progress...
  
 
=== Split based on File and Ruleset? ===
 
=== Split based on File and Ruleset? ===
Line 103: Line 106:
 
* '''Description in TaskTypes''': Split Job into many links based on file ID
 
* '''Description in TaskTypes''': Split Job into many links based on file ID
 
* '''LinkTaskMananger''': linkTaskManagerSplitOnFileIdAndruleset
 
* '''LinkTaskMananger''': linkTaskManagerSplitOnFileIdAndruleset
* '''TaskTypePKReference''': FK to '''StandardTasksConfigs'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#StandardTasksConfigs | StandardTasksConfigs]]
 +
In progress...
  
 
=== Transcode ===
 
=== Transcode ===
Line 109: Line 113:
 
* '''Description in TaskTypes''': Transcoder task type
 
* '''Description in TaskTypes''': Transcoder task type
 
* '''LinkTaskMananger''': linkTaskManagerTranscoderCommand
 
* '''LinkTaskMananger''': linkTaskManagerTranscoderCommand
* '''TaskTypePKReference''': FK to '''CommandRelationships'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#CommandRelationships | CommandRelationships]]
 +
In progress...
  
 
== Magic Links - Deprecated ==
 
== Magic Links - Deprecated ==
Line 117: Line 122:
 
* '''Description in TaskTypes''': assign magic link
 
* '''Description in TaskTypes''': assign magic link
 
* '''LinkTaskMananger''': linkTaskManagerAssignMagicLink
 
* '''LinkTaskMananger''': linkTaskManagerAssignMagicLink
* '''TaskTypePKReference''': FK to '''TasksConfigsAssignMagicLink'''
+
* '''TaskTypePKReference''': Foreign key to [[ MCP#TasksConfigsAssignMagicLink | TasksConfigsAssignMagicLink]]
 +
In progress...
  
 
=== Get Magic Link - Deprecated ===
 
=== Get Magic Link - Deprecated ===
Line 124: Line 130:
 
* '''LinkTaskMananger''': linkTaskManagerLoadMagicLink
 
* '''LinkTaskMananger''': linkTaskManagerLoadMagicLink
 
* '''TaskTypePKReference''': NULL
 
* '''TaskTypePKReference''': NULL
 
+
In progress...
  
  
 
[[Category:Development documentation]]
 
[[Category:Development documentation]]

Revision as of 15:25, 12 September 2013

Main Page > Development > Development documentation > MCP > TaskTypes

Design

This page proposes a new feature and reviews design options

Development

This page describes a feature that's in development

Documentation

This page documents an implemented feature

For each MicroServiceChainLink in the MCP Server , the TaskType determines what code will be executed for that task. There are 13 task types.

Reading about the MCP Server is highly recommended to understand this page.

General

Run once

  • UUID: 36b2e239-4a57-4aa5-8ebc-7a29139baca6
  • Description in TaskTypes: one instance
  • LinkTaskMananger: linkTaskManagerDirectories
  • TaskTypePKReference: Foreign key to StandardTasksConfigs

Run the command listed in StandardTasksConfigs exactly once. TODO filterSubDir? Old Notes

Run for each file

  • UUID: a6b1c323-7d36-428e-846a-e7e819423577
  • Description in TaskTypes: for each file
  • LinkTaskMananger: linkTaskManagerFiles
  • TaskTypePKReference: Foreign key to StandardTasksConfigs

Run the command once for each file associated with that Unit (SIP, DIP, Transfer). Files can be filtered using filterFileEnd, filterFileStart (works on the basename of the file) or filterSubDir (selects only files in those directories) Old Notes

User Choice

Old Notes

Get User Choice - select chain

  • UUID: 61fb3874-8ef6-49d3-8a2d-3cb66e86a30c
  • Description in TaskTypes: get user choice to proceed with
  • LinkTaskMananger: linkTaskManagerChoice
  • TaskTypePKReference: Foreign key to MicroServiceChainChoice
  • Use Case: Run different MicroServiceChains based on user input

Run a MicroServiceChain, depending on what the user chooses. Each MicroServiceChainChoice knows where it should be displayed (choiceAvailableAtLink), and what chain to run if it is selected (chainAvailable). The text to display comes from MicroServiceChains.description, as does the MicroServiceChainLink to run (MicroServiceChains.startingLink)

Get User Choice - select replacement dict

  • UUID: 9c84b047-9a6d-463f-9836-eafa49743b84
  • Description in TaskTypes: get replacement dic from user choice
  • LinkTaskMananger: linkTaskManagerReplacementDicFromChoice
  • TaskTypePKReference: Foreign key to MicroServiceChoiceReplacementDic
  • Use Case: Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you know all the options ahead of time. Adds the user choice as a replacement variable.

Adds a new replacement dict and value for the rest of the chain. Future commands can use replacement %FOO% in their StandardTaskConfigs.arguments, and it will be replaced by the user's choice. Each MicroServiceChoiceReplacementDic knows where it should be displayed (choiceAvailableAtLink), what text to display (description) and what replacement variables to add (replacementDic).

WARNING: The replacement variables will disappear when a new MicroServiceChain is started, because the Unit is recreated.

Generate User Choice in MicroService

  • UUID: a19bfd9f-9989-4648-9351-013a10b382ed
  • Description in TaskTypes: Get microservice generated list in stdOut
  • LinkTaskMananger: linkTaskManagerGetMicroserviceGeneratedListInStdOut
  • TaskTypePKReference: Foreign key to StandardTasksConfigs
  • Use Case: Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time. Requires #Get User Choice - select from MicroService Generated List

Takes the output (stdout) of the command run, evaluates it into a single python object, and adds it to the chain's passVar as a choicesDic. The output is almost always a dict of {'choice to display':'value for replacement dict'}.

Get User Choice - select from MicroService Generated List

  • UUID: 01b748fe-2e9d-44e4-ae5d-113f74c9a0ba
  • Description in TaskTypes: Get user choice from microservice generated list
  • LinkTaskMananger: linkTaskManagerGetUserChoiceFromMicroserviceGeneratedList
  • TaskTypePKReference: Foreign key to StandardTasksConfigs
  • Use Case: Run the same MicroServiceChainLinks in all cases, but with slightly different behavior based on user input, and you need to get the options at run time. Adds the user choice as a replacement variable. Requires #Get User Choice - select from MicroService Generated List

Looks for a choicesDic in the passVar, and displays the choices to the user. Values to display are the choicesDic keys, and the choicesDic value is put in a replacement variable specified in StandardTasksConfig.execute.

WARNING: The replacement variables will disappear when a new MicroServiceChain is started, because the Unit is recreated.

Unit Variable?

Old Notes

Set Unit Variable

  • UUID: 6f0b612c-867f-4dfd-8e43-5b35b7f882d7
  • Description in TaskTypes: linkTaskManagerSetUnitVariable
  • LinkTaskMananger: linkTaskManagerSetUnitVariable
  • TaskTypePKReference: Foreign key to TasksConfigsSetUnitVariable

In progress...

Get Unit Variable

  • UUID: c42184a3-1a7f-4c4d-b380-15d8d97fdd11
  • Description in TaskTypes: linkTaskManagerUnitVariableLinkPull
  • LinkTaskMananger: linkTaskManagerUnitVariableLinkPull
  • TaskTypePKReference: Foreign key to TasksConfigsUnitVariableLinkPull

In progress...

Transcoding

Old Notes

Split based on File?

  • UUID: 75cf8446-1cb0-474c-8245-75850d328e91
  • Description in TaskTypes: Split creating Jobs for each file
  • LinkTaskMananger: linkTaskManagerSplit
  • TaskTypePKReference: Foreign key to TasksConfigsStartLinkForEachFile

In progress...

Split based on File and Ruleset?

  • UUID: 405b061b-361e-4e75-be27-834a1bc25f5c
  • Description in TaskTypes: Split Job into many links based on file ID
  • LinkTaskMananger: linkTaskManagerSplitOnFileIdAndruleset
  • TaskTypePKReference: Foreign key to StandardTasksConfigs

In progress...

Transcode

  • UUID: 5e70152a-9c5b-4c17-b823-c9298c546eeb
  • Description in TaskTypes: Transcoder task type
  • LinkTaskMananger: linkTaskManagerTranscoderCommand
  • TaskTypePKReference: Foreign key to CommandRelationships

In progress...

Magic Links - Deprecated

Replaced by Unit Variables. Old Notes

Set Magic Link - Deprecated

  • UUID: 3590f73d-5eb0-44a0-91a6-5b2db6655889
  • Description in TaskTypes: assign magic link
  • LinkTaskMananger: linkTaskManagerAssignMagicLink
  • TaskTypePKReference: Foreign key to TasksConfigsAssignMagicLink

In progress...

Get Magic Link - Deprecated

  • UUID: 6fe259c2-459d-4d4b-81a4-1b9daf7ee2e9
  • Description in TaskTypes: goto magic link
  • LinkTaskMananger: linkTaskManagerLoadMagicLink
  • TaskTypePKReference: NULL

In progress...