'Getting Bad Gateway and Connection reset by peer using asyncio + aiohttp

Is there some async expert with sharp eyes around? I am using asyncio (Python 3.9) with aiohttp (v3.8.1) to fetch multiple urls asynchronously through a proxy, but this implementation seems to fail pretty often throwing reset by peer and Bad Gateway errors. The same proxy works well using Scrapy spiders so there must be something wrong with my async implementation.

My code:

from asgiref.sync import async_to_sync
import asyncio
import aiohttp


async def async_fetch_url(session, url: str, method: str = "get"):
    async with getattr(session, method)(url, proxy="https://someproxy.com/") as response:
        return await response.json(content_type=None)


async def my_async_fetch(urls: list, method: str = "get"):
    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(force_close=True)) as session:
        return await asyncio.gather(*[async_fetch_url(session, url, method=method) for url in urls])


def my_sync_func():
    urls = [
        'https://somesite.com/1',
        'https://somesite.com/2'
    ]
    result = async_to_sync(my_async_fetch)(urls)

Error logs:

[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
  File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
    async with getattr(session, method)(url, **kwargs) as response:
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
    _, proto = await self._create_proxy_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1275, in _create_proxy_connection
    resp = await proxy_resp.start(conn)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer


[2022-04-28 21:47:28,577: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:43,163: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
  File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
    async with getattr(session, method)(url, **kwargs) as response:
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
    _, proto = await self._create_proxy_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1288, in _create_proxy_connection
    raise ClientHttpProxyError(
aiohttp.client_exceptions.ClientHttpProxyError: 502, message='Bad Gateway', url=URL('https://someproxy.com')

Which could be the issue?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source