'Start and monitor celery work-flows from Node.js API
Background
I am building a system that will perform lengthy file operations on demand. I'm thinking of having a few containers running Celery to perform the tasks. Because of the task complexity, I'm thinking of using some work-flow primitives, like group, chain and map. This will allow me to have simpler tasks, more granular retry and error handling (I believe). I will also use the Flower API.
The issue
The way I'm planning to kick off the tasks is via a Node.js API (using express), which means I can't use the client part of Celery, where the work-flow primitives are usually used. I was thinking of starting tasks using the Flower API.
I thought I could define a root task that itself starts a chain of other tasks, like this:
@app.task(track_started=True)
def A(arg):
perform_file_task(arg)
@app.task(track_started=True)
def B(arg):
perform_second_file_task(arg)
@app.task(track_started=True)
def root_task(arg):
chain = (A.si(arg) | B.se(arg))
chain()
This is how the Celery docs also suggest calling tasks from other tasks.
However, when I start root_task, even if A or B fails, the root task does not. To check whether the subtasks succeeded I have to use the task info endpoint and follow the children field of the response to reconstruct the task tree and check each one. However, that seems excessive and impractical.
I know I can use python client side, which can take care of all that (if I define the chain client side and call result.get() on the result, but I don't have python on the client side. Furthermore, the Celery docs discourage calling .get() in a task.
Is there a better way to run a task chain and check if everything succeeded in a succint way?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
