'How to efficiently make REST API call using aiohttp
I trying to write a script which will listen to some TCP message and do REST API to a server using the message. Each REST API call will take approximately 1 second. I want to make it efficient that it process atleast 1K messages per second. I am using asyncio and aiohttp to achieve this concurrently but not getting the expected performance. . I think I am making some fundamental mistake but not able to figure it out.
async def get_rest(session, name, msg) -> None:
url = "base_url" + msg
print(f"Consumer {name} before making REST API")
async with session.get(url, verify_ssl=False) as resp:
res = await resp.text()
print(f"Consumer {name} done making REST API")
async def consume(name: int, task_queue: asyncio.Queue) -> None:
session = aiohttp.ClientSession()
print(f"Consumer {name} started")
while True:
msg = await task_queue.get()
await get_rest(session, name, msg)
class TCPHandlerClass():
def __init__(self, task_queue):
self.task_queue = task_queue
async def handle_data(self, reader, writer):
while True:
data = await reader.read(8192)
if not data:
break
data = data.strip().decode("utf-8")
lines = data.split("\n")
for line in lines:
await self.task_queue.put(line)
async def asyncio_tcp_syslog_server(self) -> None:
server = await asyncio.start_server(self.handle_data, '0.0.0.0', 5140)
addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets)
async with server:
await server.serve_forever()
async def main():
task_queue = asyncio.Queue()
tcpHandler = TCPHandlerClass(task_queue)
producers = [asyncio.create_task(tcpHandler.asyncio_tcp_syslog_server())]
consumers = [asyncio.create_task(consume(n,task_queue)) for n in range(100)]
await asyncio.gather(*producers)
await task_queue.join() # Implicitly awaits consumers, too
for c in consumers:
c.cancel()
if __name__ == "__main__":
asyncio.run(main())
Output:
2022-03-25 05:33:30,660 INFO [11569] logger: Consumer 43 before making REST API
2022-03-25 05:33:30,660 INFO [11569] logger: Consumer 44 before making REST API
2022-03-25 05:33:30,661 INFO [11569] logger: Consumer 45 before making REST API
2022-03-25 05:33:30,661 INFO [11569] logger: Consumer 46 before making REST API
2022-03-25 05:33:30,661 INFO [11569] logger: Consumer 47 before making REST API
2022-03-25 05:33:30,661 INFO [11569] logger: Consumer 48 before making REST API
[>>>>>>>>>>>>>>>>>>>>>100 REST API query done <<<<<<<<<<<<<<<<<<<<<]
2022-03-25 05:33:30,674 INFO [11569] logger: Consumer 99 before making REST API
2022-03-25 05:33:32,361 INFO [11569] logger: Consumer 25 done making REST API <<< Started getting response
2022-03-25 05:33:32,361 INFO [11569] logger: Consumer 25 before making REST API
2022-03-25 05:33:32,368 INFO [11569] logger: Consumer 20 done making REST API
2022-03-25 05:33:32,368 INFO [11569] logger: Consumer 20 before making REST API
2022-03-25 05:33:32,369 INFO [11569] logger: Consumer 15 done making REST API
2022-03-25 05:33:32,369 INFO [11569] logger: Consumer 15 before making REST API
2022-03-25 05:33:32,423 INFO [11569] logger: Consumer 10 done making REST API
2022-03-25 05:33:32,423 INFO [11569] logger: Consumer 10 before making REST API
2022-03-25 05:33:32,424 INFO [11569] logger: Consumer 22 done making REST API
2022-03-25 05:33:32,435 INFO [11569] logger: Consumer 4 done making REST API
2022-03-25 05:33:32,703 INFO [11569] logger: Consumer 56 done making REST API
2022-03-25 05:33:32,723 INFO [11569] logger: Consumer 11 done making REST API
2022-03-25 05:33:32,750 INFO [11569] logger: Consumer 7 done making REST API
2022-03-25 05:33:32,794 INFO [11569] logger: Consumer 16 done making REST API
2022-03-25 05:33:32,801 INFO [11569] logger: Consumer 0 done making REST API\
2022-03-25 05:33:32,823 INFO [11569] logger: Consumer 5 done making REST API
2022-03-25 05:33:32,836 INFO [11569] logger: Consumer 32 done making REST API
[ >>>>>>>>>>>>>>> 100 Messages done <<<<<<<<<<<<<<<<<<<<< ]
2022-03-25 05:33:35,000 INFO [11569] logger: Consumer 64 done making REST API
2022-03-25 05:33:35,071 INFO [11569] logger: Consumer 46 done making REST API
2022-03-25 05:33:35,395 INFO [11569] logger: Consumer 60 done making REST API
2022-03-25 05:33:35,424 INFO [11569] logger: Consumer 43 done making REST API
I see the all API query is being sent around the same second(05:33:30) and expect the results to be received within ~3 seconds since each REST call will take approx 1 second. But here it take around ~5-7 seconds to complete. How can I make it more efficient ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
