'Celery : Create only one queue per each concurrent worker
I have a flask application with two celery tasks. It is a user-based scraping job. So user give the CSV urls and based on that we do scraping for each user.
My Code is like follows:
@celery.task(bind=True,queue="imageScraping")
def saveUrlData(self,id, imgs,outputDir, isLast = False):
#Save Image Data From URL to Directory and Insert Record in DB
@celery.task(bind=True, queue="pageScraping")
def startScraping(self,usrReqId,filename):
#Do Page Scraping and parsing the data. Then pass it that to saveUrlData()
@app.route('/sendRequest', methods=['POST'])
def index():
# User file receive. Store its information and calling startScraping()
I need to maintain a sequence in saveUrlData() for each user. So I only require one queue per user. I am starting celery from following commands.
celery -A app.celery worker --concurrency 1 -Q imageScraping
celery -A app.celery worker --concurrency 10 -Q pageScraping
I am using concurrency so I can pageScraping queue for multiple user. But for imageScraping if I use --concurrency 1 then both user's queues mixedup. If I use --concurrency 10 then a single user's queue is not maintained.
So basically I need an imgScraping queue for each pageScraping worker to maintain queue
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
