django + gunicorn: problem with async workers (gevent)

Question

I am having the exact same problem as this post Django+gunicorn+nginx upload large file 502 error. But the solution provided does not make the trick for me, maybe because it is fairly old.

I am using django, gunicorn, supervisor and nginx with python 3.6, I have installed gevent according to gunicorn's documentation, and my gunicorn config looks like that:

[program:gunicorn]
directory=/home/ubuntu/mysite
command=/home/ubuntu/env/bin/gunicorn --workers 3 --worker-class gevent --worker-connections=1000  --timeout 600 --bind unix:/home/ubuntu/mysite/app.sock app.wsgi:application

autostart=true
autorestart=true
stderr_logfile=/var/log/gunicorn/gunicorn.err.log
stdout_logfile=/var/log/gunicorn/gunicorn.out.log


[group:guni]
programs:gunicorn

and in gunicorn log, I get the error:

 File "/home/ubuntu/exo/lib/python3.6/site-packages/gunicorn/workers/ggevent.py", line 16, in <module>
    raise RuntimeError("gevent worker requires gevent 1.4 or higher")
RuntimeError: gevent worker requires gevent 1.4 or higher

[2020-05-16 13:22:40 +0000] [24451] [DEBUG] Current configuration:
  config: None
  bind: ['unix:/home/ubuntu/myapp/app.sock']
  backlog: 2048
  workers: 4
  worker_class: sync
  threads: 12
  worker_connections: 1000
  max_requests: 0
  max_requests_jitter: 0
  timeout: 600
  graceful_timeout: 90
  keepalive: 2
  limit_request_line: 4094
  limit_request_fields: 100
  limit_request_field_size: 8190
  reload: False
  reload_engine: auto
  reload_extra_files: []
  spew: False
  check_config: False
  preload_app: False
  sendfile: None
  reuse_port: False
  chdir: /home/ubuntu/myapp
  daemon: False
  raw_env: []
  pidfile: None
  worker_tmp_dir: None
  user: 0
  group: 0
  umask: 0
  initgroups: False
  tmp_upload_dir: None
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
  accesslog: None
  disable_redirect_access_to_syslog: False
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  errorlog: -
  loglevel: DEBUG
  capture_output: False
  logger_class: gunicorn.glogging.Logger
  logconfig: None
  logconfig_dict: {}
  syslog_addr: udp://localhost:514
  syslog: False
  syslog_prefix: None
  syslog_facility: user
  enable_stdio_inheritance: False
  statsd_host: None
  dogstatsd_tags:
  statsd_prefix:
  proc_name: None
default_proc_name: myapp.wsgi:application
  pythonpath: None
  paste: None
  on_starting: <function OnStarting.on_starting at 0x7f7836ae4bf8>
  on_reload: <function OnReload.on_reload at 0x7f7836ae4d08>
  when_ready: <function WhenReady.when_ready at 0x7f7836ae4e18>
  pre_fork: <function Prefork.pre_fork at 0x7f7836ae4f28>
  post_fork: <function Postfork.post_fork at 0x7f7836b000d0>
  post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f7836b001e0>
  worker_int: <function WorkerInt.worker_int at 0x7f7836b002f0>
  worker_abort: <function WorkerAbort.worker_abort at 0x7f7836b00400>
  pre_exec: <function PreExec.pre_exec at 0x7f7836b00510>
  pre_request: <function PreRequest.pre_request at 0x7f7836b00620>
  post_request: <function PostRequest.post_request at 0x7f7836b006a8>
  child_exit: <function ChildExit.child_exit at 0x7f7836b007b8>
  worker_exit: <function WorkerExit.worker_exit at 0x7f7836b008c8>
  nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f7836b009d8>
  on_exit: <function OnExit.on_exit at 0x7f7836b00ae8>
  proxy_protocol: False

Being a novice with this stuff it's really difficult to find out what's going on.

I have installed gevent see below:

Installing collected packages: greenlet, psutil, gevent
Successfully installed gevent-20.5.0 greenlet-0.4.15 psutil-5.7.0

UPDATE: I have modified gunicorn config.py file in /vev/lib/python3.6/site-packages/gunicorn/config.py

BASE_DIR = "/path/to/base/dir/"
sys.path.append(BASE_DIR)


bind = '127.0.0.1:8000'
backlog = 2048


import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 1000
timeout = 300
keepalive = 2

and then modified my supervisor/conf/gunicorn.conf file as such:

[program:gunicorn]

directory=/home/ubuntu/mysite
command=/home/ubuntu/exo/bin/gunicorn  --config /home/ubuntu/venv/lib/python3.6/site-packages/gunicorn/config.py  unix:/home/ubuntu/mysite/app.sock mysite.wsgi:application

autostart=true
autorestart=true
stderr_logfile=/var/log/gunicorn/gunicorn.err.log
stdout_logfile=/var/log/gunicorn/gunicorn.out.log


[group:guni]
programs:gunicorn

This still gives me the same error. I am not even sure that if gunicorn config file was the right file to add the modifications but at this point I am out of things to try, maybe a fresh eye could unblock situation, or I am also open for any substitute that someone may know of

Answer 1

You could also add sudo to the command to allow supervisor to be able to access the root user, which is the user that have access to the folder that contains the python libraries that's downloaded using pip3, which is /usr/local/lib/python3.8/dist-packages/gevent in my machine, so the final supervisor conf file will be:

[program:gunicorn]

directory=/home/ubuntu/mysite
command=sudo /home/ubuntu/exo/bin/gunicorn  --config /home/ubuntu/venv/lib/python3.6/site-packages/gunicorn/config.py  --bind unix:/home/ubuntu/mysite/app.sock mysite.wsgi:application

autostart=true
autorestart=true
stderr_logfile=/var/log/gunicorn/gunicorn.err.log
stdout_logfile=/var/log/gunicorn/gunicorn.out.log

[group:guni]
programs:gunicorn

In site-packages/gunicorn/:

bind = '127.0.0.1:8000'
backlog = 2048

import multiprocessing
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = 'gevent'
worker_connections = 1000
timeout = 300
keepalive = 2

Another solution:

You can Download gevent using python for the current user you are using in the supervisor config file, using this command:

pip install --user guni gevent 

Please, refer to this StackOverFlow Answer, for more info about --user,More info about --user Flag