Celery tasks stopped running in Django app deployed using Heroku

Question

We have a Django website deployed on Heroku (free plan) with a specific app for web crawlers that uses Celery and Redis. This app has celery tasks that add entries in our Postgres DB. Everything worked fine until a couple days ago when we updated some models and migrations from another unrelated app and suddenly all tasks stopped running (periodic tasks are registered, but don't run). The whole site is connected to Sentry and we didn't receive any error flags. We haven't changed anything in important files (such as settings, production config, requirements, tasks or procfile). Everything works fine locally.

We have tried several things like excluding all registered tasks from the Django admin and putting them again. Testing different times. Rolling back to a few weeks ago, before our migrations, to see if the tasks ran... But nothing seems to work and we are feeling a little desperate since without the workers we are being forced to manually scrape several websites.

We believe that the problem lies in the initalization of celery beat, and a log is provided here.

Django admin: django admin for periodic tasks

Procfile:

release: python manage.py migrate
web: newrelic-admin run-program gunicorn config.wsgi:application
worker: newrelic-admin run-program celery -A config --beat worker -l INFO

Celery.py (in our case its called Celery_app.py):

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
DJANGO_ENV = os.getenv("DJANGO_ENV", default="production")

if DJANGO_ENV == "production":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production")
elif DJANGO_ENV == "test":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.test")
elif DJANGO_ENV == "local":
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.local")
else:
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings.production")

app = Celery("our_app_name")

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object("django.conf:settings", namespace="CELERY")

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

worker dyno logs:

heroku logs --dyno worker
2022-03-22T20:09:08.429112+00:00 app[worker.1]: self.install_default_entries(self.schedule)
2022-03-22T20:09:08.429112+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 361, in schedule
2022-03-22T20:09:08.429113+00:00 app[worker.1]: self._schedule = self.all_as_schedule()
2022-03-22T20:09:08.429113+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/schedulers.py", line 245, in all_as_schedu
le
2022-03-22T20:09:08.429113+00:00 app[worker.1]: for model in self.Model.objects.enabled():
2022-03-22T20:09:08.429113+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/query.py", line 287, in __iter__
2022-03-22T20:09:08.429114+00:00 app[worker.1]: self._fetch_all()
2022-03-22T20:09:08.429114+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/query.py", line 1308, in _fetch_all
2022-03-22T20:09:08.429114+00:00 app[worker.1]: self._result_cache = list(self._iterable_class(self))
2022-03-22T20:09:08.429114+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/query.py", line 53, in __iter__
2022-03-22T20:09:08.429115+00:00 app[worker.1]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
2022-03-22T20:09:08.429115+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
2022-03-22T20:09:08.429115+00:00 app[worker.1]: cursor.execute(sql, params)
2022-03-22T20:09:08.429116+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py", line 500, in exe
cute
2022-03-22T20:09:08.429116+00:00 app[worker.1]: return real_execute(self, sql, params)
2022-03-22T20:09:08.429116+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
2022-03-22T20:09:08.429117+00:00 app[worker.1]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
2022-03-22T20:09:08.429117+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrapp
ers
2022-03-22T20:09:08.429118+00:00 app[worker.1]: return executor(sql, params, many, context)
2022-03-22T20:09:08.429118+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
2022-03-22T20:09:08.429118+00:00 app[worker.1]: return self.cursor.execute(sql, params)
2022-03-22T20:09:08.429123+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
2022-03-22T20:09:08.429123+00:00 app[worker.1]: raise dj_exc_value.with_traceback(traceback) from exc_value
2022-03-22T20:09:08.429123+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
2022-03-22T20:09:08.429124+00:00 app[worker.1]: return self.cursor.execute(sql, params)
2022-03-22T20:09:08.429124+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/database_psycopg2.py", line 64, in execute
2022-03-22T20:09:08.429124+00:00 app[worker.1]: return super(CursorWrapper, self).execute(sql, parameters, *args,
2022-03-22T20:09:08.429125+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/database_dbapi2.py", line 37, in execute
2022-03-22T20:09:08.429125+00:00 app[worker.1]: return self.__wrapped__.execute(sql, parameters,
2022-03-22T20:09:08.429125+00:00 app[worker.1]: django.db.utils.OperationalError: could not receive data from server: Bad file descriptor
2022-03-22T20:09:08.429126+00:00 app[worker.1]: SSL SYSCALL error: Bad file descriptor
2022-03-22T20:09:08.429126+00:00 app[worker.1]:
2022-03-22T20:09:08.441155+00:00 app[worker.1]: [2022-03-22 17:09:08,441: WARNING/Beat] Process Beat:
2022-03-22T20:09:08.442447+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] Traceback (most recent call last):
2022-03-22T20:09:08.442502+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/kombu/utils/objects.p
y", line 42, in __get__
2022-03-22T20:09:08.442502+00:00 app[worker.1]: return obj.__dict__[self.__name__]
2022-03-22T20:09:08.442554+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] KeyError: 'scheduler'
2022-03-22T20:09:08.442607+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] During handling of the above exception, another exception occurred:
2022-03-22T20:09:08.442646+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] Traceback (most recent call last):
2022-03-22T20:09:08.442710+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/ut
ils.py", line 84, in _execute
2022-03-22T20:09:08.442711+00:00 app[worker.1]: return self.cursor.execute(sql, params)
2022-03-22T20:09:08.442746+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/databa
se_psycopg2.py", line 64, in execute
2022-03-22T20:09:08.442746+00:00 app[worker.1]: return super(CursorWrapper, self).execute(sql, parameters, *args,
2022-03-22T20:09:08.442788+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/databa
se_dbapi2.py", line 37, in execute
2022-03-22T20:09:08.442788+00:00 app[worker.1]: return self.__wrapped__.execute(sql, parameters,
2022-03-22T20:09:08.442834+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] psycopg2.OperationalError: could not receive data from server: Bad file des
criptor
2022-03-22T20:09:08.442843+00:00 app[worker.1]: SSL SYSCALL error: Bad file descriptor
2022-03-22T20:09:08.442881+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] The above exception was the direct cause of the following exception:
2022-03-22T20:09:08.442923+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] Traceback (most recent call last):
2022-03-22T20:09:08.443008+00:00 app[worker.1]: [2022-03-22 17:09:08,442: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/billiard/process.py",
 line 327, in _bootstrap
2022-03-22T20:09:08.443008+00:00 app[worker.1]: self.run()
2022-03-22T20:09:08.443050+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/celery/beat.py", line
 707, in run
2022-03-22T20:09:08.443050+00:00 app[worker.1]: self.service.start(embedded_process=True)
2022-03-22T20:09:08.443092+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/celery/beat.py", line
 622, in start
2022-03-22T20:09:08.443092+00:00 app[worker.1]: humanize_seconds(self.scheduler.max_interval))
2022-03-22T20:09:08.443134+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/kombu/utils/objects.p
y", line 44, in __get__
2022-03-22T20:09:08.443134+00:00 app[worker.1]: value = obj.__dict__[self.__name__] = self.__get(obj)
2022-03-22T20:09:08.443175+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/celery/beat.py", line
 666, in scheduler
2022-03-22T20:09:08.443175+00:00 app[worker.1]: return self.get_scheduler()
2022-03-22T20:09:08.443216+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/celery/beat.py", line
 657, in get_scheduler
2022-03-22T20:09:08.443217+00:00 app[worker.1]: return symbol_by_name(self.scheduler_cls, aliases=aliases)(
2022-03-22T20:09:08.443258+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/sc
hedulers.py", line 231, in __init__
2022-03-22T20:09:08.443258+00:00 app[worker.1]: Scheduler.__init__(self, *args, **kwargs)
2022-03-22T20:09:08.443305+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/celery/beat.py", line
 257, in __init__
2022-03-22T20:09:08.443305+00:00 app[worker.1]: self.setup_schedule()
2022-03-22T20:09:08.443370+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/sc
hedulers.py", line 239, in setup_schedule
2022-03-22T20:09:08.443370+00:00 app[worker.1]: self.install_default_entries(self.schedule)
2022-03-22T20:09:08.443427+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/sc
hedulers.py", line 361, in schedule
2022-03-22T20:09:08.443427+00:00 app[worker.1]: self._schedule = self.all_as_schedule()
2022-03-22T20:09:08.443468+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django_celery_beat/sc
hedulers.py", line 245, in all_as_schedule
2022-03-22T20:09:08.443468+00:00 app[worker.1]: for model in self.Model.objects.enabled():
2022-03-22T20:09:08.443511+00:00 app[worker.1]: [2022-03-22 17:09:08,443: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/quer
y.py", line 287, in __iter__
2022-03-22T20:09:08.443511+00:00 app[worker.1]: self._fetch_all()
2022-03-22T20:09:08.444859+00:00 app[worker.1]: [2022-03-22 17:09:08,444: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/quer
y.py", line 1308, in _fetch_all
2022-03-22T20:09:08.444860+00:00 app[worker.1]: self._result_cache = list(self._iterable_class(self))
2022-03-22T20:09:08.444907+00:00 app[worker.1]: [2022-03-22 17:09:08,444: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/quer
y.py", line 53, in __iter__
2022-03-22T20:09:08.444907+00:00 app[worker.1]: results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
2022-03-22T20:09:08.444955+00:00 app[worker.1]: [2022-03-22 17:09:08,444: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/models/sql/
compiler.py", line 1156, in execute_sql
2022-03-22T20:09:08.444955+00:00 app[worker.1]: cursor.execute(sql, params)
2022-03-22T20:09:08.445001+00:00 app[worker.1]: [2022-03-22 17:09:08,444: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/sentry_sdk/integratio
ns/django/__init__.py", line 500, in execute
2022-03-22T20:09:08.445002+00:00 app[worker.1]: return real_execute(self, sql, params)
2022-03-22T20:09:08.445046+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/ut
ils.py", line 66, in execute
2022-03-22T20:09:08.445046+00:00 app[worker.1]: return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
2022-03-22T20:09:08.445098+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/ut
ils.py", line 75, in _execute_with_wrappers
2022-03-22T20:09:08.445099+00:00 app[worker.1]: return executor(sql, params, many, context)
2022-03-22T20:09:08.445145+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/ut
ils.py", line 84, in _execute
2022-03-22T20:09:08.445145+00:00 app[worker.1]: return self.cursor.execute(sql, params)
2022-03-22T20:09:08.445185+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/utils.py",
line 90, in __exit__
2022-03-22T20:09:08.445194+00:00 app[worker.1]: raise dj_exc_value.with_traceback(traceback) from exc_value
2022-03-22T20:09:08.445230+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/django/db/backends/ut
ils.py", line 84, in _execute
2022-03-22T20:09:08.445230+00:00 app[worker.1]: return self.cursor.execute(sql, params)
2022-03-22T20:09:08.445279+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/databa
se_psycopg2.py", line 64, in execute
2022-03-22T20:09:08.445280+00:00 app[worker.1]: return super(CursorWrapper, self).execute(sql, parameters, *args,
2022-03-22T20:09:08.445315+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/hooks/databa
se_dbapi2.py", line 37, in execute
2022-03-22T20:09:08.445315+00:00 app[worker.1]: return self.__wrapped__.execute(sql, parameters,
2022-03-22T20:09:08.445362+00:00 app[worker.1]: [2022-03-22 17:09:08,445: WARNING/Beat] django.db.utils.OperationalError: could not receive data from server: Bad f
ile descriptor
2022-03-22T20:09:08.445363+00:00 app[worker.1]: SSL SYSCALL error: Bad file descriptor
2022-03-22T20:09:10.408237+00:00 app[worker.1]: [2022-03-22 17:09:10,407: INFO/MainProcess] Connected to redis://:**-------our redis key ----------
//
2022-03-22T20:09:10.419752+00:00 app[worker.1]: [2022-03-22 17:09:10,419: INFO/MainProcess] mingle: searching for neighbors
2022-03-22T20:09:11.447857+00:00 app[worker.1]: [2022-03-22 17:09:11,447: INFO/MainProcess] mingle: all alone

We cannot figure it out why the worker is showing this "django.db.utils.OperationalError: could not receive data from server: Bad file descriptor" error or how to solve it, but we are able to add new entries on our DB just fine, the web dyno is working, and there are no error on deployment of new versions.

Can somebody help me out with this? I'll be happy to share more information in order to solve this issue :)