'Celery task_routes and task_create_missing_queues note working

I have configured a CentOS server with (server A) :

  • Redis 6.2.6
  • Python 3.8.1
  • Celery 5.2.1
  • Flower

On another server (server B) I send tasks to redis that are executed by a worker on server A.

I want to try to implement multiple queues and change de name of the default queue : queues "high", "normal" and "low", with "normal" beeing the default queue (instead of celery queue).

If I hardcode the queue in the task, it works (they are send to the right queue).

But if I try to do it with "task_routes", it doesn't.

I tried to:

  • hardcode route in this settings
  • implement a routing function
  • implement a routing class with the routing function

But it never works. Moreover, even though I put the "task_create_missing_queue" parameter to False, all tasks are send to celery queue (the default queue is still celery and not normal...).

Any ideas ?

Here is the code for my celeryconfig.py file (with routing function:

from celery import Celery
from kombu import Exchange, Queue
from celery.exceptions import Reject
import re

task_create_missing_queues = False

task_queues = (
    Queue('high', Exchange('high'), routing_key='high'),
    Queue('normal', Exchange('normal'), routing_key='normal'),
    Queue('low', Exchange('low'), routing_key='low')
)

task_default_queue = 'normal'
task_default_exchange = 'normal'
task_default_routing_key = 'normal'


def route_tasks (name, args, kwargs, options, task=None, **kw):
    if ':' not in name:
        return {'queue': 'normal'}

    namespace, _ = name.split(':')
    return {'queue': namespace}

task_routes = (route_tasks,)

And my tasks are defined with a name like this with the decorator (when I hardcode the queue for each task, it is there that I add "queue="queue_name") (my celery app is called celery):

@celery.task(bind=True, name="high:long_task")

I can see in flower that the configuration is well taken into account.

My worker is started with multi and -Q high,normal,low.

Many thanks!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source