Asynchronous Processing in Storage Service - Revision history

Sevein: /* Backends */

2017-08-16T15:19:53Z

Backends

2017-08-16T04:21:21Z

2017-08-16T01:25:30Z

Backends

2017-08-16T01:23:41Z

2017-08-16T01:22:24Z

2017-08-16T01:11:35Z

2017-08-16T01:02:58Z

Kafka

2017-08-16T01:02:50Z

Backends

2017-08-16T01:02:12Z

2017-08-16T00:20:25Z

API backwards compatibility

@@ Line 27: / Line 27: @@
-== Backends ==
+== Frameworks ==
-There are many different ways to architecture background jobs. Solutions like Celery or similar introduce certain requirements.
+RQ, Taskmaster, [https://www.fullstackpython.com/task-queues.html etc...]
-Frequently you need two components: a queue (broker) and a result backend (job state).
+== Backends ==
-Frameworks:
+Based on our own goals, e.g. reliability over performance, at-least-once delivery, simplicity...
 ==== RabbitMQ ====

← Older revision		Revision as of 04:21, 16 August 2017
Line 79:		Line 79:

	* Hard to get right (unless you're using libraries like [https://github.com/chanks/que que])		* Hard to get right (unless you're using libraries like [https://github.com/chanks/que que])
		+
		+	==== NATS ====
		+
		+	It comes up all the time but NATS is not a great fit for the kind of job queue we need because it only implements at-most-once delivery. Great for other purposes.
		+
		+	==== NATS Streaming ====
		+
		+	* at-least-once delivery.
		+	* append-only like Kafka, with replay too, but offsets no required, the broker takes care of it (durable subscriptions).
		+	* is it a good fit for a job queue?

@@ Line 33: / Line 33: @@
 Frequently you need two components: a queue (broker) and a result backend (job state).
-https://www.fullstackpython.com/task-queues.html
+Frameworks:
 ==== RabbitMQ ====

@@ Line 29: / Line 29: @@
 == Backends ==
-There are many different ways to architecture background jobs.
+There are many different ways to architecture background jobs. Solutions like Celery or similar introduce certain requirements.
 Frequently you need two components: a queue (broker) and a result backend (job state).
 ==== RabbitMQ ====
 ==== Kafka ====
@@ Line 64: / Line 68: @@
 * '''disque 1.0 has not been released yet, only rc1. redis 4.2 + disque not available but in the roadmap.'''
-==== PostgreSQL ===
+==== PostgreSQL ====
 Recent versions of this DB incorporate new solutions that allow developers to model job queues that can perform [https://github.com/chanks/queue-shootout reasonably well] - enough for our needs. Well-know open source apps like Concourse CI [https://github.com/concourse/atc/blob/07252a072624ba6a528dcf8175c5fc54989ebbd1/db/worker_lifecycle.go use] PostgreSQL as a job queue.

@@ Line 32: / Line 32: @@
 Frequently you need two components: a queue (broker) and a result backend (job state).
 ==== Kafka ====
@@ Line 60: / Line 62: @@
 * Order is not guaranteed, but this makes the solution more scalable and redelivery can be done easily by the broker without extra effort on the client.

@@ Line 43: / Line 43: @@
 * One consumer per partition means that a long-running job keeps the consumer busy while other job may be dispatched to the same partition but can't be processed until the previous job is done.
-* Complexity on the client client (offset storage, at-least-once if the client can handle it).
+* Complexity on the client client (offset storage, at-least-once, [https://medium.com/@jaykreps/exactly-once-support-in-apache-kafka-55e1fdd0a35f '''exact-once''' can be possible!])
-* Complexity on the operation layer (Zookeper, [https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool explicit partitioning].
+* Complexity on the operation layer (Zookeper, [https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-5.AddPartitionTool explicit partitioning]).
 ==== disque ====

@@ Line 35: / Line 35: @@
 ==== Kafka ====
-Properties:
+Pros:
 * Acks are not necessary, consumers "commit" (using offsets).

@@ Line 20: / Line 20: @@
 ** It may not be a good idea when the properties of the response are going to be significantly different, e.g. a deferred call may expect a document with a single UUID that can be used later to query the status of the call in the future, instead of the document originally requested. This needs to be analyzed individually for each of the endpoints affected.
 * '''New''' <code>/api/v3</code>, similar API but everything is async.
 == List of endpoints affected ==
 TODO.

@@ Line 18: / Line 18: @@
 * In <code>/api/v2</code>, add new endpoints that can do the job asynchronously. Maintain the existing endpoints.
 * In <code>/api/v2</code>, update existing parameters to accept a new query parameter <code>?async=true</code>.
 * '''New''' <code>/api/v3</code>, similar API but everything is async.