'Start and monitor celery work-flows from Node.js API

Background

I am building a system that will perform lengthy file operations on demand. I'm thinking of having a few containers running Celery to perform the tasks. Because of the task complexity, I'm thinking of using some work-flow primitives, like group, chain and map. This will allow me to have simpler tasks, more granular retry and error handling (I believe). I will also use the Flower API.

The issue

The way I'm planning to kick off the tasks is via a Node.js API (using express), which means I can't use the client part of Celery, where the work-flow primitives are usually used. I was thinking of starting tasks using the Flower API.

I thought I could define a root task that itself starts a chain of other tasks, like this:

@app.task(track_started=True)
def A(arg):
    perform_file_task(arg)


@app.task(track_started=True)
def B(arg):
    perform_second_file_task(arg)


@app.task(track_started=True)
def root_task(arg):
    chain = (A.si(arg) | B.se(arg))

    chain()

This is how the Celery docs also suggest calling tasks from other tasks.

However, when I start root_task, even if A or B fails, the root task does not. To check whether the subtasks succeeded I have to use the task info endpoint and follow the children field of the response to reconstruct the task tree and check each one. However, that seems excessive and impractical.

I know I can use python client side, which can take care of all that (if I define the chain client side and call result.get() on the result, but I don't have python on the client side. Furthermore, the Celery docs discourage calling .get() in a task.

Is there a better way to run a task chain and check if everything succeeded in a succint way?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source