Performance Improvements - Revision history

Kelly at 21:30, 8 June 2018

2018-06-08T21:30:44Z

Kelly at 21:24, 8 June 2018

2018-06-08T21:24:06Z

Kelly at 20:52, 8 June 2018

2018-06-08T20:52:04Z

Kelly at 17:59, 8 June 2018

2018-06-08T17:59:35Z

Kelly at 17:58, 8 June 2018

2018-06-08T17:58:49Z

Kelly: Created page with "'''Make indexing configurable''' Make search an optional feature in Archivematica so that it can be run with ElasticSearch turned off. '''Make capture output configurable'''..."

2018-06-08T17:54:33Z

Created page with "'''Make indexing configurable''' Make search an optional feature in Archivematica so that it can be run with ElasticSearch turned off. '''Make capture output configurable'''..."

New page

'''Make indexing configurable'''
Make search an optional feature in Archivematica so that it can be run with ElasticSearch turned off.

'''Make capture output configurable'''

The next most place to start for performance improvements was selected: reducing processing time by changing how output streams are handled. In this phase, sending automatically writing standard out and standard error to the database was made configurable. When output capture is turned off, only a non-zero exist code (an error) is returned.

This option has been tested, [https://www.archivematica.org/en/docs/archivematica-1.7/admin-manual/installation-setup/customization/task-config/ documented ] for all deployment methods, and released in Archivematica 1.7.1

'''Make performance metrics accessible via the API'''

Whenever a preservation task is performed, Archivematica records its timing information (start and end times) in the MySQL database. Columbia University Library wants to be able to measure the processing time (performance) of Archivematica packages and their component microservices so that they can identify bottlenecks, estimate package processing times, and make informed decisions about their configuration.

The problem is that the relevant timing information is not exposed via Archivematica’s API endpoints and is only partially exposed via its GUI. In addition, since this information is internal (not exposed via a public API), it is subject to change and users are therefore wary of building features or implementing workflows that make use of it.

We are therefore implementing an API endpoint that returns processing performance details for a specified package (i.e., a transfer or an AIP) divided by microservice group. This endpoint will return the following data:

Phase of processing (transfer or ingest)
Microservice group
CPU time
Number of Tasks
Duration (wall time)

← Older revision		Revision as of 21:30, 8 June 2018
Line 29:		Line 29:
	*This feature will be released in the next minor release of Archivematica		*This feature will be released in the next minor release of Archivematica

−	~~'''Other performance improvements''' TODO~~

−	Filter tasks based on the availability of an FPR rule - this feature was analysed but development work was not started.	+	'''Filter tasks based on the availability of an FPR rule'''
		+
		+	This feature was analysed but development work was not started.
		+	Artefactual has published the basic analysis in the Archivematica wiki, making it possible for other Archivematica users to consider working on this feature
		+
		+	'''Run ‘remove unneeded files’ microservice once only'''
		+
		+	Similar to the ‘generate thumbnails’ feature. This feature was analysed but development work was not started.
		+	Artefactual has published the basic analysis in the Archivematica wiki, making it possible for other Archivematica users to consider working on this feature
		+
		+	'''Refactor mcpClient/clientScript structure to reduce database connections.'''
		+
		+	This feature was analysed and initial exploratory development work was done as part of phase 2. Based on these initial findings, additional development work was funded by Jisc. The feature has now been developed and is currently in the peer review phase. The feature is expected to be available in the next minor release (Archivematica 1.8.0)
		+
		+
		+	'''Additional Performance Improvement development'''
		+	Additional performance improvement work is being done by other Artefactual clients, primarily Jisc, including:
		+
		+	*Reduce size of METS and improve xml handling
		+	*Analysis of performance metrics reporting
		+	*The analysis methods developed in phase 2 will be used to quantify the results of these features
		+	*These features are expected to be released in the future, after then next minor release (i.e. in 1.9.0 or later)

← Older revision		Revision as of 21:24, 8 June 2018
Line 28:		Line 28:

	*This feature will be released in the next minor release of Archivematica		*This feature will be released in the next minor release of Archivematica
		+
		+	'''Other performance improvements''' TODO
		+
		+	Filter tasks based on the availability of an FPR rule - this feature was analysed but development work was not started.

← Older revision		Revision as of 20:52, 8 June 2018
Line 27:		Line 27:
	Duration (wall time)		Duration (wall time)

−	*This feature will be released in the next ~~major~~ release of Archivematica	+	*This feature will be released in the next minor release of Archivematica

@@ Line 3: / Line 3: @@
 Make search an optional feature in Archivematica so that it can be run with ElasticSearch turned off.
-The installation methods have been tested, [https://www.archivematica.org/en/docs/archivematica-1.7/user-manual/administer/dashboard-admin/#elasticsearch-indexing documented], and released in Archivematica 1.7.0
+*The installation methods have been tested, [https://www.archivematica.org/en/docs/archivematica-1.7/user-manual/administer/dashboard-admin/#elasticsearch-indexing documented], and released in Archivematica 1.7.0
@@ Line 10: / Line 10: @@
 The next most place to start for performance improvements was selected: reducing processing time by changing how output streams are handled.  In this phase, sending automatically writing standard out and standard error to the database was made configurable.  When output capture is turned off, only a non-zero exist code (an error) is returned.
-This option has been tested, [https://www.archivematica.org/en/docs/archivematica-1.7/admin-manual/installation-setup/customization/task-config/ documented ] for all deployment methods, and released in Archivematica 1.7.1
+*This option has been tested, [https://www.archivematica.org/en/docs/archivematica-1.7/admin-manual/installation-setup/customization/task-config/ documented ] for all deployment methods, and released in Archivematica 1.7.1
@@ Line 26: / Line 26: @@
 Number of Tasks
 Duration (wall time)

@@ Line 1: / Line 1: @@
 '''Make indexing configurable'''
 Make search an optional feature in Archivematica so that it can be run with ElasticSearch turned off.
 '''Make capture output configurable'''