Running any web server event loop on a secondary thread

Question

We have a rich backend application that handles messaging/queuing, database queries, and computer vision. An additional feature we need is tcp communication - preferably via http. The point is: this is not primarily a web application. We would expect to have a set of http channels set up for different purposes. Yes - we understand about messaging including topics and publish-subscribe: but direct tcp based request/response also has its place.

I have looked at and tried out a half dozen python http web servers. They either implicitly or explicitly describe a requirement to run the event loop on the main thread. This is for us a cart before the horse: the main thread is already occupied with other tasks including coordination of the other activities.

To illustrate the intended structure I will lift code from my aiohttp-specific question How to run an aiohttp web application in a secondary thread. In that question I tried running in another standalone script but on a subservient thread:

def runWebapp():
  from aiohttp import web

  async def handle(request):
      name = request.match_info.get('name', "Anonymous")
      text = "Hello, " + name
      return web.Response(text=text)

  app = web.Application()
  app.add_routes([web.get('/', handle),
                  web.get('/{name}', handle)])
  web.run_app(app)

if __name__ == '__main__':
  from threading import Thread
  t = Thread(target=runWebapp)
  t.start()
  print('thread started let''s nap..')
  import time
  time.sleep(50)

This gives error:

RuntimeError: There is no current event loop in thread 'Thread-1'.

This error turns out to mean "hey you're not running this on the main thread".

We can logically replace aiohttp with other web servers here. Are there any for which this approach of asking the web server's event handling loop to run on a secondary thread will work? So far I have also tried cherrypy, tornado, and flask.

Note that one prominent webserver that I have not tried is django. But that one seems to require an extensive restructuring of the application around the directory structures expected (/required?) for django. We would not want to do that given the application has a set of other purposes that supersede this sideshow of having http servers.

An approach that I have looked at is asyncio. I have not understood whether it can support running event loops on a side thread or not: if so then it would be an answer to this question.

In any case are there any web servers that explicitly support having their event loops off of the main thread?

Answer 1

Any Python program is run on a single thread which is the main. And when you create a Thread it does not mean that your program already uses two threads.

Unfortunately, it is not possible to use different event loops for every Thread but possible to do that using multiprocessing instead of threading. It allows creating its own event loop for every single Process.

from multiprocessing import Process
from aiohttp import web


def runWebapp(port):
    async def handle(request):
        name = request.match_info.get("name", "Anonymous")
        text = "Hello, " + name
        return web.Response(text=text)

    app = web.Application()
    app.add_routes([
        web.get("/", handle),
        web.get("/{name}", handle)
    ])
    web.run_app(app, port=port)


if __name__ == "__main__":
    p1 = Process(target=runWebapp, args=(8080,))
    p2 = Process(target=runWebapp, args=(8081,))
    p1.start()
    p2.start()