Asynchronous Processing in Storage Service

Synopsis

The Storage Service is implemented as a web application, written using the Django Framework. It exposes a REST API, which is consumed by other components in Archivematica (the dashboard, the automation tools) and can be used by third party applications as well. All of the AIP endpoints are currently synchronous - the http request made by a client blocks until the work is completed. Many of the tasks performed by the storage service involve significant disk i/o and can take a long time (minutes/hours) to complete.

Adding a way to perform work asynchronously is an important feature, probably a requirement before a 1.0.0 version of the Storage Service can be released.

A couple of experiments have been tried already:

• AIP Migration endpoint

API backwards compatibility

Possible approaches:

• In /api/v2, add new endpoints that can do the job asynchronously. Maintain the existing endpoints.
• In /api/v2, update existing parameters to accept a new query parameter ?async=true.
  • It may not be a good idea when the properties of the response are going to be significantly different, e.g. a deferred call may expect a document with a single UUID that can be used later to query the status of the call in the future, instead of the document originally requested. This needs to be analyzed individually for each of the endpoints affected.
• New /api/v3, similar API but everything is async.

List of endpoints affected

TODO.

Backends

There are many different ways to architecture background jobs. Solutions like Celery or similar introduce certain requirements.

Frequently you need two components: a queue (broker) and a result backend (job state).

Frameworks:

• Celery
• RQ
• Taskmaster
• ... https://www.fullstackpython.com/task-queues.html

RabbitMQ

...

Kafka

Pros:

• Acks are not necessary, consumers "commit" (using offsets).
• Order is preserved (the log is append-only), but this may be only relevant in a stream-oriented system but not in a job queue.

Cons:

• One consumer per partition means that a long-running job keeps the consumer busy while other job may be dispatched to the same partition but can't be processed until the previous job is done.
• Complexity on the client client (offset storage, at-least-once, exact-once can be possible!)
• Complexity on the operation layer (Zookeper, explicit partitioning).

disque

disque is being written by @antirez (creator of Redis). It's an off-the-shelve job queue. Redis 4.2 is going to bring disque as a module.

Pros:

• Complexity on the broker (ACKs, locks...), client has little room for errors.
• Provides iterators to read dequeued messages efficiently (with filters).
• at-least-once, managed by the broker efficiently.

Cons:

• Order is not guaranteed, but this makes the solution more scalable and redelivery can be done easily by the broker without extra effort on the client.
• disque 1.0 has not been released yet, only rc1. redis 4.2 + disque not available but in the roadmap.

PostgreSQL

Recent versions of this DB incorporate new solutions that allow developers to model job queues that can perform reasonably well - enough for our needs. Well-know open source apps like Concourse CI use PostgreSQL as a job queue.

Cons:

• Hard to get right (unless you're using libraries like que)