Asynchronous Processing in Storage Service

From Archivematica
(Difference between revisions)
Jump to: navigation, search
(API backwards compatibility)
Line 20: Line 20:
 
** It may not be a good idea when the properties of the response are going to be significantly different, e.g. a deferred call may expect a document with a single UUID that can be used later to query the status of the call in the future, instead of the document originally requested. This needs to be analyzed individually for each of the endpoints affected.
 
** It may not be a good idea when the properties of the response are going to be significantly different, e.g. a deferred call may expect a document with a single UUID that can be used later to query the status of the call in the future, instead of the document originally requested. This needs to be analyzed individually for each of the endpoints affected.
 
* '''New''' <code>/api/v3</code>, similar API but everything is async.
 
* '''New''' <code>/api/v3</code>, similar API but everything is async.
 +
  
 
== List of endpoints affected ==
 
== List of endpoints affected ==
  
 
TODO.
 
TODO.
 +
 +
 +
== Backends ==
 +
 +
There are many different ways to architecture background jobs.
 +
 +
Frequently you need two components: a queue and a
 +
 +
==== Kafka ====
 +
 +
Properties:
 +
 +
* Acks are not necessary, consumers "commit" (using offsets).
 +
* Order is preserved (the log is append-only), but this may be only relevant in a stream-oriented system but not in a job queue.
 +
 +
Cons:
 +
 +
* One consumer per partition means that a long-running job keeps the consumer busy while other job may be dispatched to the same partition but can't be processed until the previous job is done.
 +
* Complexity on the client client (offset storage, at-least-once if the client can handle it).
 +
* Complexity on the operation layer (Zookeper, [https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool explicit partitioning].
 +
 +
==== disque ====
 +
 +
at-least-once.
 +
 +
disque is being written by @antirez (creator of Redis). It's an off-the-shelve job queue.
 +
[https://news.ycombinator.com/item?id=14770110 Redis 4.2 is going to bring disque as a module].
 +
 +
Pros:
 +
 +
* Complexity on the broker (ACKs, locks...), client has little room for errors.
 +
* Provides iterators to read dequeued messages efficiently (with filters).
 +
* at-least-once, managed by the broker efficiently.
 +
 +
Cons:
 +
 +
* Order is not guaranteed, but this makes the solution more scalable and redelivery can be done easily by the broker without extra effort on the client.

Revision as of 21:02, 15 August 2017


Contents

Synopsis

The Storage Service is implemented as a web application, written using the Django Framework. It exposes a REST API, which is consumed by other components in Archivematica (the dashboard, the automation tools) and can be used by third party applications as well. All of the AIP endpoints are currently synchronous - the http request made by a client blocks until the work is completed. Many of the tasks performed by the storage service involve significant disk i/o and can take a long time (minutes/hours) to complete.

Adding a way to perform work asynchronously is an important feature, probably a requirement before a 1.0.0 version of the Storage Service can be released.

A couple of experiments have been tried already:

API backwards compatibility

Possible approaches:

  • In /api/v2, add new endpoints that can do the job asynchronously. Maintain the existing endpoints.
  • In /api/v2, update existing parameters to accept a new query parameter ?async=true.
    • It may not be a good idea when the properties of the response are going to be significantly different, e.g. a deferred call may expect a document with a single UUID that can be used later to query the status of the call in the future, instead of the document originally requested. This needs to be analyzed individually for each of the endpoints affected.
  • New /api/v3, similar API but everything is async.

List of endpoints affected

TODO.

Backends

There are many different ways to architecture background jobs.

Frequently you need two components: a queue and a

Kafka

Properties:

  • Acks are not necessary, consumers "commit" (using offsets).
  • Order is preserved (the log is append-only), but this may be only relevant in a stream-oriented system but not in a job queue.

Cons:

  • One consumer per partition means that a long-running job keeps the consumer busy while other job may be dispatched to the same partition but can't be processed until the previous job is done.
  • Complexity on the client client (offset storage, at-least-once if the client can handle it).
  • Complexity on the operation layer (Zookeper, explicit partitioning.

disque

at-least-once.

disque is being written by @antirez (creator of Redis). It's an off-the-shelve job queue. Redis 4.2 is going to bring disque as a module.

Pros:

  • Complexity on the broker (ACKs, locks...), client has little room for errors.
  • Provides iterators to read dequeued messages efficiently (with filters).
  • at-least-once, managed by the broker efficiently.

Cons:

  • Order is not guaranteed, but this makes the solution more scalable and redelivery can be done easily by the broker without extra effort on the client.
Personal tools
Namespaces

Variants
Actions
Navigation
Toolbox