scrape live scores from oddsportal live odds page using requests

Question

I want to scrape inplay odds and scores. I succeed to get live odds data using the below code, but without finding live scores:

import requests, re, time
from bs4 import BeautifulSoup

url = f"https://fb.oddsportal.com/feed/livegames/live/1/0.dat?_{int(time.time() * 1000)}"
headers = {'User-Agent': 'curl/7.64.0','Referer': 'https://www.oddsportal.com/inplay-odds/live-now/soccer/'}
r = requests.get(url, headers=headers)

live_html = re.findall(r'<table class=.*table>', r.text)[0].replace("\\","")
soup = BeautifulSoup(live_html, 'html.parser')

I tried to search from from Developper Tools > Sources > Page, but can't find any source that provide live scores

Answer 1

The Live-odds or score on any website come through web sockets and thus cannot be scraped using any normal approach but there are some tricks to do that given that website does have a very good authentication protocol. you can refer to the link and may try it for your use case. https://towardsdatascience.com/websocket-retrieve-live-data-f539b1d9db1e

Answer 2

Where there is a will there is a way.

Have you tried using Selenium? (I've heard it might be slower however)

If not try OCR you can decipher the text for every frame change.

Answer 3

Use python's websocket-client-package to retrieve the LIVE data.

First, you need to copy your web browser’s header to here and use json.dumps to convert it into the string format. Additionally, you have to do a handshake i.e., sending messages to the website and receiving messages when connecting to the websocket.

After that, create the connection to the server by using create_connection. Then, perform the handshake by sending the message, and you will be able to see the data on your side.