'Getting Bad Gateway and Connection reset by peer using asyncio + aiohttp
Is there some async expert with sharp eyes around? I am using asyncio (Python 3.9) with aiohttp (v3.8.1) to fetch multiple urls asynchronously through a proxy, but this implementation seems to fail pretty often throwing reset by peer and Bad Gateway errors. The same proxy works well using Scrapy spiders so there must be something wrong with my async implementation.
My code:
from asgiref.sync import async_to_sync
import asyncio
import aiohttp
async def async_fetch_url(session, url: str, method: str = "get"):
async with getattr(session, method)(url, proxy="https://someproxy.com/") as response:
return await response.json(content_type=None)
async def my_async_fetch(urls: list, method: str = "get"):
async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(force_close=True)) as session:
return await asyncio.gather(*[async_fetch_url(session, url, method=method) for url in urls])
def my_sync_func():
urls = [
'https://somesite.com/1',
'https://somesite.com/2'
]
result = async_to_sync(my_async_fetch)(urls)
Error logs:
[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
async with getattr(session, method)(url, **kwargs) as response:
File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1275, in _create_proxy_connection
resp = await proxy_resp.start(conn)
File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
message, payload = await protocol.read() # type: ignore[union-attr]
File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
[2022-04-28 21:47:28,577: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:43,163: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
async with getattr(session, method)(url, **kwargs) as response:
File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
_, proto = await self._create_proxy_connection(req, traces, timeout)
File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1288, in _create_proxy_connection
raise ClientHttpProxyError(
aiohttp.client_exceptions.ClientHttpProxyError: 502, message='Bad Gateway', url=URL('https://someproxy.com')
Which could be the issue?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
