How to access Django models from a scheduled script?

Question

TL;DR:

I have an app that I want to run a routine every day at midnight (for that I'm using APScheduler). In the middle of this routine it's supposed to access data from a few Django models.

The logic is

Django is running > run apps.py > run scheduler.py > run routine.py > access models.py.

Exceptions raised at the bottom of the post.

- Here comes the details:

My directory is this:

myproject/
- manage.py

+ myproject/
-- settings.py
-- wsgi.py
-- ...

+ myapp/
-+ static/
-+ templates/
-- admin.py
-- apps.py
-- models.py
-- views.py
-- scheduler.py  #<<<<<<<<<<
-- routine.py    #<<<<<<<<<<
-- ...

myapp/models.py

class MyModel(models.Model):
    field1 = models.DateField(auto_now=True)
    field2 = models.DecimalField(max_digits=19, decimal_places=16)
    ...

myapp/apps.py

from django.apps import AppConfig
from .scheduler import ScheduledRoutine

class MyAppConfig(AppConfig):
    name = 'myapp'

    def ready(self):
        routine = ScheduledRoutine()
        routine.start()

myapp/scheduler.py

from .routine import MyRoutine
#from ..routine import MyRoutine   # See error nr3 <<<<<<<<<
from apscheduler.schedulers.background import BackgroundScheduler

class ScheduledRoutine(object):

    def start(self):
        self.scheduler = BackgroundScheduler()
        startdate = datetime.now() #For brevity assume datetime object
        self.scheduler.add_job(self.routine, 'interval', days=1, start_date=startdate)
        self.scheduler.start()

    def routine(self):
        data = MyRoutine()

myapp/routine.py

import os
os.environ["DJANGO_SETTINGS_MODULE"] = "myproject.settings"
import django
django.setup()

from .models import MyModel
#from myapp.models import MyModel  # See error nr3 <<<<<<<<<
    
class MyRoutine(object):
    def __init__(self, arg):
        self.arg = arg
        data = MyModel.objects.filter(reliable=True)
        self.do_something(data)

- Exceptions and what didn't work

I've tried several things already, those are the ones that I remember:

- Error Nr1:

With the current state of things (as seen above) this is the error:

Watching for file changes with StatReloader
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "C:\Python\Python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "C:\Python\Python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "C:\venv\lib\site-packages\django\utils\autoreload.py", line 54, in wrapper
    fn(*args, **kwargs)
  File "C:\venv\lib\site-packages\django\core\management\commands\runserver.py", line 109, in inner_run
    autoreload.raise_last_exception()
  File "C:\venv\lib\site-packages\django\utils\autoreload.py", line 77, in raise_last_exception
    raise _exception[1]
  File "C:\venv\lib\site-packages\django\core\management\__init__.py", line 337, in execute
    autoreload.check_errors(django.setup)()
  File "C:\venv\lib\site-packages\django\utils\autoreload.py", line 54, in wrapper
    fn(*args, **kwargs)
  File "C:\venv\lib\site-packages\django\__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "C:\venv\lib\site-packages\django\apps\registry.py", line 91, in populate
    app_config = AppConfig.create(entry)
  File "C:\venv\lib\site-packages\django\apps\config.py", line 90, in create
    module = import_module(entry)
  File "C:\Python\Python37\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\venv\myproject\myapp\apps.py", line 2, in <module>
    from .scheduler import ScheduledRoutine
  File "C:\venv\myproject\myapp\scheduler.py", line 6, in <module>
    from .routine import MyRoutine
  File "C:\venv\myproject\myapp\routine.py", line 10, in <module>
    django.setup()
  File "C:\venv\lib\site-packages\django\__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "C:\venv\lib\site-packages\django\apps\registry.py", line 83, in populate
    raise RuntimeError("populate() isn't reentrant")
RuntimeError: populate() isn't reentrant

- Error Nr2:

Since routine.py is inside myapp/ folder, and it tries to reference myproject/settings.py I tried adding . (also tried ..) to the reference like this:

os.environ["DJANGO_SETTINGS_MODULE"] = ".myproject.settings"

and this:

os.environ["DJANGO_SETTINGS_MODULE"] = "..myproject.settings"

Both returned the same exceptions:

TypeError: the 'package' argument is required to perform a relative import for '.myproject.settings'

- Error Nr3:

Due to the same problem mentioned above (the reference to settings.py that is myproject/settings.py) I tried moving the routine.py to the parent folder, therefore I needed to change the imports in routine.py and scheduler.py to their commented versions. That led to this exception:

ValueError: attempted relative import beyond top-level package

- Error Nr4:

Also, running django with the following lines commented (in routine.py)

#os.environ["DJANGO_SETTINGS_MODULE"] = "myproject.settings"
#import django
#django.setup()

raises django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet. which is pretty much how things started.

Edit: - Error Nr5: Using this in routine.py raises the same error as Nr1.

from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
application = get_wsgi_application()

To me it's seems clear that the issue is the way I'm trying to access the models in routine.py and not the scheduler. What else can I try?

Answer 1

You can try creating a Django custom command, so you can create a command like python manage.py myroutine and use a scheduler (like cron) to run it. If you do it, can avoid all this in your code:

os.environ["DJANGO_SETTINGS_MODULE"] = "myproject.settings"
import django
django.setup()

I know it isn't a exactly solution to your problem, but it's a workaround.