Selenium-js: Firefox: Error: TimedPromise timed out after 300000 ms

Question

I am trying to run my selenium javascript on the site bet365.com. I am using Firefox (geckodriver), I tried both headless and normal but for understanding/debugging the problem the non-headless-mode is helpful.

This is the code:

const driver = await new Builder().forBrowser("firefox").build();
await driver.get("https://bet365.com");

The problem is that the site is not loading:

After 5 mins I then end up with the error

TimeoutError: TimedPromise timed out after 300000 ms
    at Object.throwDecodedError (C:\1_code\RFB\git\webscraper-srv\node_modules\selenium-webdriver\lib\error.js:517:15)
    at parseHttpResponse (C:\1_code\RFB\git\webscraper-srv\node_modules\selenium-webdriver\lib\http.js:671:13)
    at Executor.execute (C:\1_code\RFB\git\webscraper-srv\node_modules\selenium-webdriver\lib\http.js:597:28)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Driver.execute (C:\1_code\RFB\git\webscraper-srv\node_modules\selenium-webdriver\lib\webdriver.js:729:17)
    at async C:\1_code\RFB\git\webscraper-srv\scripts\seleniumUtils.js:190:3
    at async XWrap.<anonymous> (C:\1_code\RFB\git\webscraper-srv\scripts\utils.js:127:60)
    at async XWrap.<anonymous> (C:\1_code\RFB\git\webscraper-srv\scripts\utils.js:127:60)
    at async XWrap.<anonymous> (C:\1_code\RFB\git\webscraper-srv\scripts\utils.js:127:60)
    at async XWrap.<anonymous> (C:\1_code\RFB\git\webscraper-srv\scripts\utils.js:127:60) {
  remoteStacktrace: 'WebDriverError@chrome://marionette/content/error.js:181:5\n' +
    'TimeoutError@chrome://marionette/content/error.js:450:5\n' +
    'bail@chrome://marionette/content/sync.js:229:19\n'

I tried visiting a different site with selenium and they work perfectly so I don't think its a problem with my setup. If I try visiting the site with my normal Firefox Browser it works too. I also tried manually searching for the page in the browser which is opened by the program and it leads to another endless loop. But manually opening other pages in this browser opened by selenium works fine.

Is it possible for the server of a webpage to detect browsers that were started using selenium? I always thought the only way to detect webscrapers was by looking at the frequency it is visiting and the clicks the scraper is doing on a page but it is the first time I visited the page with Selenium...

If the server doesn't allow these kind of requests, is there any way to still scrape data from this webpage? I also already tried opening it in headless mode...

This is the (entire) Firefox Network tab when its stuck loading (sometimes it looks a little different):

This is the (begin of the) Firefox Network tab when its loaded normally:

I circled the requests that might be causing the problem. In the bottom left of the browser it's telling me the entire time that it's Transfering data from ff.kis.v2.scr.kaspersky-labs.com. I tried deactivating kaspersky on my machine and also let the progamme run on a machine without kaspersky so I am not quite sure why this request is made. It might have to do something with https validation but I am not sure.

Another interesting thing is that the response of the first request to www.bet365.com/ looks like this (even with selenium): Meaning that it does actually reach the server but it just sends a loading screen. Also the following requests get the same response as with the normal browser. Only the requests with status 101 don't give back any response, unlike with the normal browser.

Last interesting thing is this request www.bet365.com/increment?desktop-site-loaded_11=1. It is only made when starting it with the selenium browser, not when opening the site with a normal browser. This might mean that its not a loading problem but that its actively blocking the request and telling the backend to increase a counter of requests that were blocked.

Any ideas how I can get the code working or why this problem comes up?

Answer 1

try

driver.execute_script(f"location.href='{url}';")

instand of

driver.get