MCPClient - Revision history

Sallain at 21:56, 11 February 2020

2020-02-11T21:56:22Z

Jdunham: /* Config File */

2018-05-11T17:16:37Z

Config File

Hbecker: /* Client scripts */ Expand client script info

2017-03-27T23:48:12Z

Client scripts: Expand client script info

Hbecker: /* Client scripts */ Expand client script info

2017-03-27T23:37:20Z

Client scripts: Expand client script info

Hbecker: /* Client scripts */ Expand client script info

2017-03-27T22:51:49Z

Client scripts: Expand client script info

Show changes

Hbecker: Add category

2017-03-10T19:56:01Z

Add category

Hbecker: /* Client script summaries */ Expand details on client scripts

2017-03-10T02:02:43Z

Client script summaries: Expand details on client scripts

Hbecker: Add config file section

2017-03-10T01:35:43Z

Add config file section

Hbecker: Add initial data

2017-03-08T18:58:49Z

Add initial data

New page

[[Main Page]] > [[Development]] > [[:Category:Development documentation|Development documentation]] > MCPClient

<div class="status">
<div>
Design
<div class="description">This page proposes a new feature and reviews design options</div>
</div>
<div>
Development
<div class="description">This page describes a feature that's in development</div>
</div>
<div class="active">
Documentation
<div class="description">This page documents an implemented feature</div>
</div>
</div>

Archivematica has one or more MCPClient instances to perform the actual work. They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.

== Client script summaries ==
Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.

New scripts are defined in <code>[https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/archivematicaClientModules src/MCPClient/lib/archivematicaClientModules]</code>, which is what is registered with Gearman on MCPClient startup.

{| class="wikitable" style="background-color:#ffeecc;" cellpadding="10";
| Improvement note: archivematicaClientModules lists both 'supportedCommandSpecial' and 'supportedCommands'. This distinction may have once been based on scripts that relied on external services, but serves no purpose now and should be removed.
|}

The name is what the [[MCPServer#StandardTasksConfigs | StandardTasksConfig]] table will refer to them as, and the value is the script that will be run. Some are defined as shell builtins (eg copy_v0.0 is cp). Most are paths to a script in the clientScripts directory, using the <code>%clientScriptsDirectory%</code> replacement variable. The name of the client script is usually the same as the name in archivematicaClientModules, but for very old scripts may have ‘archivematica’ at the beginning (eg createMETS_v2.0 = archivematicaCreateMETS2.py) or be named more pythonically (eg parseExternalMETS = parse_external_mets.py). Entries are added alphabetically.

The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen. Newer scripts may not have the version defined.

Notable client scripts include:
* createMETS_v0.0: Generates the transfer METS file
* createMETS_v2.0: Generates the AIP METS file
*

← Older revision		Revision as of 21:56, 11 February 2020
Line 1:		Line 1:
	[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > MCPClient		[[Main Page]] > [[Development]] > [[:Category:Development documentation\|Development documentation]] > MCPClient
		+
		+	<div style="padding: 10px 10px; border: 1px solid black; background-color: #F79086;">This page is no longer being maintained and may contain inaccurate information. Please see the [https://www.archivematica.org/docs/latest/ Archivematica documentation] for up-to-date information. </div> <p>

	<div class="status">		<div class="status">

← Older revision		Revision as of 17:16, 11 May 2018
Line 300:		Line 300:

	== Config File ==		== Config File ==
		+

	Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.		Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.

@@ Line 74: / Line 74: @@
 Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].
 === createEvent_v0.0 ===
@@ Line 142: / Line 159: @@
 Extracts files from a package (e.g. zip, tar, 7z etc) and optionally deletes the package.  Generates Events for the new files.
 === validateFile_v1.0 ===
@@ Line 154: / Line 186: @@
 Validates files are correct, where correctness is defined by the file format.
-=== createMETS_v0.0 ===
+=== examineContents_v0.0  ===
-* '''Purpose''': Generate the transfer METS file
+* '''Purpose''': Run bulk extractor for a detailed analysis of contents
-* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaCreateMETS.py archivematicaCreateMETS.py]
+* '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/examineContents.py examineContents.py]
 * '''Used in''': Transfer
-* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
+* '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
 * '''Event?''': No
 * '''FPR?''': No
-Creates the Transfer METS file. This will contain all the information generated on the transfer during processing, and is especially useful for backlogged transfers.
+Runs bulk extractor and stores the outputs in the logs directory.
 === elasticSearchIndex_v0.0 ===

@@ Line 51: / Line 51: @@
 * '''Used in''': Transfer, Ingest
 * '''Task type''': [[MCPServer/TaskTypes#Run for each file | per file]]
-* '''Event?''': ingestion/reingestion, possibly accession
+* '''Event?''': ingestion/reingestion, possibly registration
 * '''FPR?''': No
-This creates an entry in the Files table with the file's UUID, current & original paths and file group.  It also creates an 'ingestion' Event and an 'accession' Event if an accession ID was specified.  Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.
+This creates an entry in the Files table with the file's UUID, current & original paths and file group.  It also creates an 'ingestion' Event and an 'registration' Event if an accession ID was specified.  Updating the file group (eg original, preservation, submission documentation) can be disabled with <code>--disable-update-filegrpuse</code>.
 In ingest, is used on manually normalized files which may have been newly added, metadata and submission documentation.
@@ Line 74: / Line 74: @@
 Note this script will fail if there was a problem with [[#assignFileUUIDs_v0.0]].
 === archivematicaClamscan_v0.0 ===
@@ Line 87: / Line 98: @@
 This is run on incoming files, files after extraction, metadata files and submission documentation.  It is not run on normalized files.
 === identifyFileFormat_v0.0 ===
@@ Line 106: / Line 131: @@
 | Improvement Note: Only one identification tool can be run at a time currently. It would be better to allow a cascading of tools. E.g. if a file is identified as a video to subsequently run a tool specialized in identifying different types of video. Similarly, if the default tool failed, we could run a backup tool for a second opinion.
 |}
 === createMETS_v0.0 ===
@@ Line 144: / Line 191: @@
 * '''Script''': [https://github.com/artefactual/archivematica/blob/qa/1.x/src/MCPClient/lib/clientScripts/archivematicaMoveSIP.py archivematicaMoveSIP.py]
 * '''Used in''': Ingest
-* '''Task type''': [[MCPServer/TaskTypes#Run once  | once]]
+* '''Task type''': [[MCPServer/TaskTypes#Run once | once]]
 * '''Event?''': No
 * '''FPR?''': No
 Moves the whole SIP and updates the database with the new location relative to the shared directory.
 === transcribeFile_v0.0 ===

← Older revision		Revision as of 19:56, 10 March 2017
Line 119:		Line 119:
	\|-		\|-
	\|}		\|}
		+
		+	[[Category:Development documentation]]

@@ Line 18: / Line 18: @@
 Archivematica has one or more MCPClient instances to perform the actual work.  They are gearman worker implementations that inform the gearman server what tasks they can perform, and wait for the server to assign them a task. When a client starts, it connects to the specified gearman server and provides a list of modules they support. When the [[MCPServer]] informs the gearman server of a Task that the client supports and the gearman server assigns the job to the client, the client will process the Job, and return the results to the gearman server, which in turn will return them to the MCPServer.
-== Client script summaries ==
+== Client scripts ==
 Client scripts do the actual work in Archivematica. They are anything that can be run on the command line, from builtins like mv and cp, to custom-written scripts.
@@ Line 31: / Line 31: @@
 The version (eg copy_v0.0) was originally intended to be used to version the scripts as they changed, and be able to track those changes, but that did not happen.  Newer scripts may not have the version defined.
-Notable client scripts include:
+The list of client scripts is sorted roughly in order of appearance during processing
 == Config File ==

← Older revision		Revision as of 01:35, 10 March 2017
Line 35:		Line 35:
	* createMETS_v2.0: Generates the AIP METS file		* createMETS_v2.0: Generates the AIP METS file
	*		*
		+
		+
		+	== Config File ==
		+
		+	Several config settings are read from <code>/etc/archivematica/MCPClient/clientConfig.conf</code> on startup.
		+
		+	Variables in the MCPClient section:
		+
		+	{\|
		+	\|-
		+	! Variable !! Description !! Default value
		+	\|-
		+	\| MCPArchivematicaServer \|\| URL of the MCP gearman server. Must match the [[MCPServer#Config File \| server config file]]. \|\| localhost:4730
		+	\|-
		+	\| sharedDirectoryMounted \|\| Directory structure owned by Archivematica and shared between the MCPServer & MCPClient. Must match the [[MCPServer#Config File \| server config file]]. \|\| /var/archivematica/sharedDirectory/
		+	\|-
		+	\| archivematicaClientModules \|\| Path to the list of jobs to register with Gearman \|\| /usr/lib/archivematica/MCPClient/archivematicaClientModules
		+	\|-
		+	\| clientScriptsDirectory \|\| Path to the directory where client scripts are installed. Used when parsing archivematicaClientModules \|\| /usr/lib/archivematica/MCPClient/clientScripts/
		+	\|- style="background: #ffeecc;"
		+	\| LoadSupportedCommandsSpecial \|\| Whether or not to register the SupportedCommandsSpecial section of archivematicaClientModules. This should be removed. \|\| True
		+	\|-
		+	\| numberOfTasks \|\| Number of MCPClient workers to created. 0 detects the number of cores and uses that. \|\| 0
		+	\|-
		+	\| elasticsearchServer \|\| URL of the ElasticSearch server. \|\| localhost:9200
		+	\|- style="background: #ffeecc;"
		+	\| disableElasticsearchIndexing \|\| If true, do not index AIPs or Transfers in backlog. This should be removed, since ElasticSearch indexing is required \|\| False
		+	\|-
		+	\| temp_dir \|\| Path to the temporary usage directory. Should be in the shared directory \|\| /var/archivematica/sharedDirectory/tmp
		+	\|- style="background: #ffeecc;"
		+	\| kioskMode \|\| Dashboard setting that disables editing users. This should be removed, or at least moved to dashboard settings \|\| False
		+	\|-
		+	\| removableFiles \|\| List of filenames that are not archivally significant and can be removed. \|\| Thumbs.db, Icon, Icon\r, .DS_Store
		+	\|-
		+	\| django_settings_module \|\| Name of the Django settings module, so the client scripts can access the database via the Django ORM. \|\| settings.common
		+	\|-
		+	\|}