'Dynamic Web scrapping - Chromedriver security

I am trying to web-scrapp a dynamic page, simple urllib request gets me results from fist page only, instead of returning the whole set.

from urllib import request
from bs4 import BeautifulSoup

URL = "https://www.olx.pl/d/nieruchomosci/mieszkania/warszawa/"
get_url = request.urlopen(URL)
get_page = get_url.read()
get_url.close()
print(get_page)

I was about to implement selenium and chromedriver for the dynamic webscrapping, but then I read about security of this solution. Chromedriver should never be run on account with privileges, best option would be virtual machine.

As I read forum posts, almost all solutions to dynamic web scrapping involve chromedriver / selenium/ scrapy. It makes me wonder if all users set up firewall or VM to download data.

Is there any other, safer solution you would recommend for dynamic web scrapping?

selenium security

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Dynamic Web scrapping - Chromedriver security

Sources

Related Questions