'Celery : Create only one queue per each concurrent worker

I have a flask application with two celery tasks. It is a user-based scraping job. So user give the CSV urls and based on that we do scraping for each user.

My Code is like follows:

@celery.task(bind=True,queue="imageScraping")
def saveUrlData(self,id, imgs,outputDir, isLast = False):
    #Save Image Data From URL to Directory and Insert Record in DB

@celery.task(bind=True, queue="pageScraping")
def startScraping(self,usrReqId,filename):
    #Do Page Scraping and parsing the data. Then pass it that to saveUrlData()

@app.route('/sendRequest', methods=['POST'])
def index():
    # User file receive. Store its information and calling startScraping()

I need to maintain a sequence in saveUrlData() for each user. So I only require one queue per user. I am starting celery from following commands.

celery -A app.celery worker --concurrency 1 -Q imageScraping

celery -A app.celery worker --concurrency 10 -Q pageScraping

I am using concurrency so I can pageScraping queue for multiple user. But for imageScraping if I use --concurrency 1 then both user's queues mixedup. If I use --concurrency 10 then a single user's queue is not maintained.

So basically I need an imgScraping queue for each pageScraping worker to maintain queue



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source