'Using requests_html with arender() function gives RuntimeError: 'Event loop is closed'
I'm trying to scrape Amazon asynchronously with requests-html package. See the following code:
import datetime
from requests_html import AsyncHTMLSession
import asyncio
asins = ['B09GB6GSMM', 'B09GB7BCQH', 'B075D23KXX']
async def get_price_and_title_for_asin(s, asin):
r = await s.get(f'https://www.amazon.de/dp/{asin}')
await r.html.arender()
try:
price = r.html.find("#corePrice_feature_div > div > span > span")[0].text
except:
price = r.html.find(".priceToPay > span")[0].text
title = r.html.find('#productTitle')[0].text
date = datetime.datetime.now().replace(microsecond=0)
print(asin, date, price)
return [title, price]
async def main(asins):
s = AsyncHTMLSession()
coroutines = [get_price_and_title_for_asin(s, asin) for asin in asins]
return await asyncio.gather(*coroutines)
asyncio.run(main(asins))
and I get the following Error:
B09GB6GSMM 2022-04-05 17:26:28 109,90€
B075D23KXX 2022-04-05 17:26:31 339,00€
B09GB7BCQH 2022-04-05 17:26:31 239,00€
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Users\Christian\anaconda3\lib\site-packages\pyppeteer\launcher.py", line 153, in _close_process
self._loop.run_until_complete(self.killChrome())
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 591, in run_until_complete
self._check_closed()
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'Launcher.killChrome' was never awaited
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Users\Christian\anaconda3\lib\site-packages\pyppeteer\launcher.py", line 153, in _close_process
self._loop.run_until_complete(self.killChrome())
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 591, in run_until_complete
self._check_closed()
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "C:\Users\Christian\anaconda3\lib\site-packages\pyppeteer\launcher.py", line 153, in _close_process
self._loop.run_until_complete(self.killChrome())
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 591, in run_until_complete
self._check_closed()
File "C:\Users\Christian\anaconda3\lib\asyncio\base_events.py", line 508, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
So the program is printing out the 3 lines as required, but then throws an Error. Does anyone know how to solve this? I checked several other posts on similar topics but nothing worked for me.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
