'Python, Share a MySQL connection in multiple threads

TL;DR

In short, I'm trying to open a single MySQL connection and share it with multiple threads, so that each thread creates and uses a cursor, allowing a second query to be placed before the end of the first query.

Is there any library or explanation on how to achieve threadsafety = 2 (PEP 249)?

Background

In detail, the behavior I want to achieve is: assuming we have two threads T1 and T2, if T1 executes first a query that takes 10s and T2 executes second a query that takes 1s, T2 should get the result before T1. This is critical as my application works with thousands of latency-sensitive threads. Each thread can't wait for other threads' queries and it's not possible to open a DB connection for each thread. Using a pool is also limiting, as eventually one thread will have to wait another thread's query (for pool size X, X+1 threads executing a query at the same time will have 1 thread waiting).

I learned that PEP 249 describes what I want to achieve as threadsafety = 2, and that other languages can achieve this behavior just fine. From all the Python <> MySQL libraries I looked into (mysql-connector-python, mysqldb, pymysql), all seem to allow only threadsafety = 1. My first question would be: is there any existing library or snippet that achieves threadsafety = 2?

Progress

I'm now moving towards overriding the mysql-connector-python library, so that hopefully after hours of painful debugging I can synchronize cursors' and their result sets.

Looks like that whenever a cursor needs to send a command to the socket, I'll have to check if there are any unread/incomplete packets. Meaning that the only moment a thread will have to wait is when there are packets in transit (cursor sending a query or the connection receiving the result of a query). Afterwards, I'll aim to make sure the correct cursor receives the correct result set.

Behaving as:

T1 executes a query -> lock the connection
T2 executes a query -> wait for T1 command to be sent

T1 finishes sending the command -> release the connection
T2 executes a query -> lock the connection and send command

T1 checks for result -> wait for T2 to release the connection lock
T2 finishes sending the command -> release the connection

T1 checks for result -> no result yet
T2 checks for result -> no result yet

Connection receives packets of T2 query after 1s -> read it and store internally

T1 checks for result -> no result yet
T2 checks for result -> gets result from internal storage

Connection receives packets of T2 query after 9s -> read it and store internally
T1 checks for result -> gets result from internal storage

Now, I'm aware that other languages achieve this behavior but I'm not familiar with the proper flow. I'm continuing my research and will keep this post updated.

My second question would be: what is the state-of-the-art approach for a same connection to send two queries from two cursors that race, and get the result in a safe way?

Thank you in advance.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source