Получение плохого шлюза и сброса соединения сверстником с использованием asyncio + aiohttp

Есть ли здесь какой-нибудь эксперт по async с острыми глазами? Я использую asyncio (Python 3.9) с aiohttp (v3.8.1) для асинхронной выборки нескольких урлов через прокси, но эта реализация, похоже, довольно часто дает сбой, выбрасывая reset by peer и Bad Gateway errors. Тот же самый прокси отлично работает с пауками Scrapy, так что должно быть что-то не так с моей асинхронной реализацией.

Мой код:

from asgiref.sync import async_to_sync
import asyncio
import aiohttp


async def async_fetch_url(session, url: str, method: str = "get"):
    async with getattr(session, method)(url, proxy="https://someproxy.com/") as response:
        return await response.json(content_type=None)


async def my_async_fetch(urls: list, method: str = "get"):
    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(force_close=True)) as session:
        return await asyncio.gather(*[async_fetch_url(session, url, method=method) for url in urls])


def my_sync_func():
    urls = [
        'https://somesite.com/1',
        'https://somesite.com/2'
    ]
    result = async_to_sync(my_async_fetch)(urls)

Журналы ошибок:

[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:28,576: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
  File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
    async with getattr(session, method)(url, **kwargs) as response:
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
    _, proto = await self._create_proxy_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1275, in _create_proxy_connection
    resp = await proxy_resp.start(conn)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client_reqrep.py", line 898, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
  File "/usr/local/lib/python3.9/site-packages/aiohttp/streams.py", line 616, in read
    await self._waiter
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer


[2022-04-28 21:47:28,577: ERROR/MainProcess] ???[???]: [Errno 104] Connection reset by peer
[2022-04-28 21:47:43,163: ERROR/MainProcess] ???[???]: Traceback (most recent call last):
  File "/app/cubist_backend/helpers.py", line 176, in async_fetch_url
    async with getattr(session, method)(url, **kwargs) as response:
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 905, in _create_connection
    _, proto = await self._create_proxy_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1288, in _create_proxy_connection
    raise ClientHttpProxyError(
aiohttp.client_exceptions.ClientHttpProxyError: 502, message='Bad Gateway', url=URL('https://someproxy.com')

В чем может быть проблема?

Вернуться на верх