'Server closed connection unexpectedly with SQLAlchemy NullPool configuration

I have the following project setup:

  • We are using a python nameko framework to communicate microservices each other (using RabbitMQ as a transport)
  • In production containers are managed using a docker-swarm orchestrator.
  • PostgreSQL DB is used + SQLAlchemy ORM + nameko-sqlalchemy library
  • Database Connection Pool is configured as NullPool, so pooling is disabled and we checkout a connection on demand.

However, when we have huge RPC requests load (over 1k per second) we encounter this error occasionally:

server closed the connection unexpectedly
    This probably means the server terminated abnormally
    before or while processing the request.

We assumed that the problem can be with checking out and in connections to the database in a short time, so we have tried QueuePool(pool_size=N, max_overflow=M) (with pre_ping=True). We are sure that N + M is far below database max_connections. However, this setup causes over time a problem with:

sorry, too many clients already

Because there was a connection leakage... Probably (?) after nameko service restart, new idle connections were created without closing old ones, and over time our service has more connections created (usually in idle state) than database connection limit.

Does anybody use a similar application setup and has encountered this problem? Or have any tips?

pythonpostgresqlsqlalchemydocker-swarmnameko


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions