Celery: Temporary Auto-Delete Queues with TTL so I can round-robin large volumes of messages

Question

I'm building a service with Celery to process data from a large number of users. I want each user to be able to use this pipeline in a perfectly-fair round-robin fashion. It's infeasible for me to manually create a separate permanent queue for each user, so I rather do it dynamically.

In the perfect scenario, when a user wants to start work, Celery would either create a new queue or re-use the same queue previously created that user (idempotent). Then, I know Celery already has round-robin mechanisms built-in so processing is "fair". After the user leaves, the queue can be auto-deleted through a TTL so as not to not take up space in my messaging broker. This queue will be re-created when the user comes back and needs to do more work.

My current broker is RabbitMQ/AmazonMQ but I'm flexible and I can change this to Redis or SQS.

I'm looking to see if there's already an out-of-box solution for this. Otherwise, a manual solution could be to maintain a database table of active queues and use Beat to check activeness and TTL.

I stumbled upon these other posts but they haven't been super helpful as the proposed solutions are not significantly better than the manual queue table.

• RabbitMq Dynamically Add/Remove Queues/Consumers
• Auto Delete SQS queue