When I enable uwsgi thread support and start the scheduler the api stops working

Question

I'm new to python, flask, nginx and all that stuff.

I have a flask app that acts as API for a frontend. Also when the flask app is started I would like to start a scheduled task with APScheduler.

The problem is that when I enable uwsgi thread support and start the scheduler the api stops working (504 Gateway Time-out). But the scheduler works as seen in the logfile. When i remove the scheduler / thread support the api works but I obviously don't have the scheduler anymore. Somehow I suspect the scheduler prevents the flask app from being run properly?

As I am new to those technologies I'll post my setup below. If you need any more file infos please tell me.(The whole thing is run on a raspberry pi and the api is accessed from my pc over lan)

app.service

[Unit]
Description=uWSGI instance to serve app
After=network.target

[Service]
User=pi
Group=www-data
WorkingDirectory=/home/pi/flask
Environment="PATH=/home/pi/flask/appenv/bin"
ExecStart=/home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads

[Install]
WantedBy=multi-user.target

app.ini

[uwsgi]
module = wsgi:app

master = true
processes = 5

socket = /home/pi/flask/app.sock
chmod-socket = 660
vacuum = true

die-on-term = true

app.py

#!/usr/bin/env python3

from flask import Flask, request
from apscheduler.schedulers.background import BackgroundScheduler

import logging
logging.basicConfig(filename='logfile.log',level=logging.DEBUG)

from api.Controller import Controller
from Handler.Handler import Handler
from apscheduler.executors.pool import ThreadPoolExecutor, ProcessPoolExecutor

api_controller = Controller()
handler = Handler()

def startHandlerJob():
    handler.ExecuteAllSensors()

app = Flask( __name__ )


@app.route('/app')
def apiDefinition():
    return 'API Definition: GetHumidityValues, TODO'

@app.route( "/app/GetHumidityValues", methods=["GET"] )
def GetHumidityValues():
    logging.info("app.py: API-call GetHumidityValues")
    return api_controller.GetHumidityValues()


if (__name__ == "__main__"):
    app.run(host='0.0.0.0')

executors = {
    'default': ThreadPoolExecutor(20),
    'processpool': ProcessPoolExecutor(5)
}
job_defaults = {
    'coalesce': False,
    'max_instances': 1
}

scheduler = BackgroundScheduler(daemon=True, executors=executors, job_defaults=job_defaults)
scheduler.start()
scheduler.add_job(startHandlerJob,'cron', minute='*')

logfile.log

WARNING:apscheduler.scheduler:Execution of job "startHandlerJob (trigger: cron[minute=''], next run at: 2019-12-18 19:01:00 CET)" skipped: maximum number of running instances reached (1) DEBUG:apscheduler.scheduler:Next wakeup is due at 2019-12-18 19:02:00+01:00 (in 59.980780 seconds) DEBUG:apscheduler.scheduler:Looking for jobs to run WARNING:apscheduler.scheduler:Execution of job "startHandlerJob (trigger: cron[minute=''], next run at: 2019-12-18 19:02:00 CET)" skipped: maximum number of running instances reached (1) DEBUG:apscheduler.scheduler:Next wakeup is due at 2019-12-18 19:03:00+01:00 (in 59.979407 seconds)

systemctl status app

* app.service - uWSGI instance to serve app
   Loaded: loaded (/etc/systemd/system/app.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-12-18 18:40:57 CET; 23min ago
 Main PID: 21129 (uwsgi)
    Tasks: 8 (limit: 2200)
   Memory: 22.9M
   CGroup: /system.slice/app.service
           |-21129 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
           |-21148 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
           |-21149 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
           |-21150 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
           |-21151 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads
           `-21152 /home/pi/flask/appenv/bin/uwsgi --ini app.ini --enable-threads

Dec 18 18:40:57 raspberrypi uwsgi[21129]: mapped 386400 bytes (377 KB) for 5 cores
Dec 18 18:40:57 raspberrypi uwsgi[21129]: *** Operational MODE: preforking ***
Dec 18 18:40:59 raspberrypi uwsgi[21129]: WSGI app 0 (mountpoint='') ready in 2 seconds on interpreter 0xa6f900 pid: 21129 (default app)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: *** uWSGI is running in multiple interpreter mode ***
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI master process (pid: 21129)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 1 (pid: 21148, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 2 (pid: 21149, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 3 (pid: 21150, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 4 (pid: 21151, cores: 1)
Dec 18 18:40:59 raspberrypi uwsgi[21129]: spawned uWSGI worker 5 (pid: 21152, cores: 1)

The logfile shows that the scheduler is active. But when I try to use http:/raspberryipaddress/app the answer is a 504 Gateway Time-out response. When I remove the scheduler and disable thread support this call works as desired.

Any help is appreciated. Probably I am missing something obvious, because I am new to all those things. Thanks!

Answer 1

I hesitate a bit to put this as an answer but here goes...

In uwsgi app.ini after setting processes (5 in your case, I use 4) I also set threads = 2. I don't know if this has any direct relation to the --enable-threads option because that seems to be for your app to start it's own threads, but it might help for uwsgi to have its own threads per process. Also the UWSGI documentation points out (somewhere) that more processes are not necessarily better.

Also on that, your logfile shows warnings from the scheduler that the maximum number of running instances being reached. I take this means the job you've given the scheduler is not finishing before the next scheduled run (1 minute later)? If that's the case is it stuck somewhere, blocking everything else?

Finally, if all else fails, something else from the docs (UWSGI Security and availability)

A common problem with webapp deployment is “stuck requests”. All of your threads/workers are stuck (blocked on request) and your app cannot accept more requests. To avoid that problem you can set a harakiri timer.It is a monitor (managed by the master process) that will destroy processes stuck for more than the specified number of seconds (choose harakiri value carefully).

And finally, finally :) there's a top like monitoring tool for uwsgi which might be handy to see what's going on, just

pip install uwsgitop
uwsgitop 127.0.0.1:9191

plus telnetting into that adress:port apparently shows you a lot more info (haven't tried it myself).

Answer 2

I had exactly the same problem. The fix for me was to use ProcessPoolExecutor instead of ThreadPoolExecutor.

For some unknown reasons (at least unknown for me), when there were several threads in ThreadPoolExecutor, that damn problem showed itself.

After I changed most cpu-bound threads to processes (i.e. using ProcessPoolExecutor for those jobs), that damn problem disappeared ...