'Python, Share a MySQL connection in multiple threads
TL;DR
In short, I'm trying to open a single MySQL connection and share it with multiple threads, so that each thread creates and uses a cursor, allowing a second query to be placed before the end of the first query.
Is there any library or explanation on how to achieve threadsafety = 2 (PEP 249)?
Background
In detail, the behavior I want to achieve is: assuming we have two threads T1 and T2, if T1 executes first a query that takes 10s and T2 executes second a query that takes 1s, T2 should get the result before T1. This is critical as my application works with thousands of latency-sensitive threads. Each thread can't wait for other threads' queries and it's not possible to open a DB connection for each thread. Using a pool is also limiting, as eventually one thread will have to wait another thread's query (for pool size X, X+1 threads executing a query at the same time will have 1 thread waiting).
I learned that PEP 249 describes what I want to achieve as threadsafety = 2, and that other languages can achieve this behavior just fine. From all the Python <> MySQL libraries I looked into (mysql-connector-python, mysqldb, pymysql), all seem to allow only threadsafety = 1. My first question would be: is there any existing library or snippet that achieves threadsafety = 2?
Progress
I'm now moving towards overriding the mysql-connector-python library, so that hopefully after hours of painful debugging I can synchronize cursors' and their result sets.
Looks like that whenever a cursor needs to send a command to the socket, I'll have to check if there are any unread/incomplete packets. Meaning that the only moment a thread will have to wait is when there are packets in transit (cursor sending a query or the connection receiving the result of a query). Afterwards, I'll aim to make sure the correct cursor receives the correct result set.
Behaving as:
T1 executes a query -> lock the connection
T2 executes a query -> wait for T1 command to be sent
T1 finishes sending the command -> release the connection
T2 executes a query -> lock the connection and send command
T1 checks for result -> wait for T2 to release the connection lock
T2 finishes sending the command -> release the connection
T1 checks for result -> no result yet
T2 checks for result -> no result yet
Connection receives packets of T2 query after 1s -> read it and store internally
T1 checks for result -> no result yet
T2 checks for result -> gets result from internal storage
Connection receives packets of T2 query after 9s -> read it and store internally
T1 checks for result -> gets result from internal storage
Now, I'm aware that other languages achieve this behavior but I'm not familiar with the proper flow. I'm continuing my research and will keep this post updated.
My second question would be: what is the state-of-the-art approach for a same connection to send two queries from two cursors that race, and get the result in a safe way?
Thank you in advance.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
