'python requests-html Chromium process leaking
My program cannot run thought the entire loop because a leak crashes it before it gets to the end.
I have the following script:
from requests_html import HTMLSession
from bs4 import BeautifulSoup
import requests
for x in range(9376,23534):
session = HTMLSession()
r = session.get('https://someexampleurl.com/yadayada/database1/{}'.format(x))
r.html.render() # this call executes the js in the page
soup = BeautifulSoup(r.html.html, features="lxml")
r.close()
print(x)
name = "\n".join([img['alt'] for img in soup.find_all('img', alt=True)])
name = name[1:]
name = name[:-1]
url = "\n".join([img['src'] for img in soup.find_all('img', alt=True)])
def solve_fast(s):
ind1 = s.find('\n')
ind2 = s.rfind('\n')
return s[ind1+1:ind2]
url = solve_fast(url)
url = url[0:41] + "1" + url[41+1: ]
url = url[0:42] + "2" + url[42+1: ]
url = url[0:43] + "8" + url[43+1: ]
img_data = requests.get(url)
with open('local_database1/{}{}.avif'.format(x,name), 'wb') as handler:
handler.write(img_data.content)
img_data.close()
When ran in a loop the chromium process stacks up infinitely until the program crashes, I can't see where I am not closing the connection to the request.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
