How can I let a webpage update before scraping it?

Question

I'm trying to scrape the current rank of a player from lolprofile.net, but it's never up to date, so I somehow have to hit the "update" button before getting my data. I've tried scraping the 'updated' page the button leads to, without getting different results. Suspected the site might take some time to refresh the data and therefore added a timer, nothing happened differently. My code:

from bs4 import BeautifulSoup import requests import time

url = ['https://lolprofile.net/de/summoner/euw/Sidestep%20Sabrina#update', 'https://lolprofile.net/de/summoner/euw/Diadormus#update', 'https://lolprofile.net/de/summoner/euw/DeluxeFuture#update', 'https://lolprofile.net/de/summoner/euw/Natron#update', 'https://lolprofile.net/de/summoner/euw/51l3nc3#update']

for x in url:

html_text = requests.get(x).text
time.sleep(5)
soup = BeautifulSoup(html_text, 'lxml')

player_name = soup.find('h1').text
rank = soup.find('span', class_ = 'tier').text
LP = soup.find('span', class_ = 'lp').text

print(f'{player_name} ist {rank} mit {LP}')

input()

How should I approach this problem? There's been a similar question asked here but it never got answered.

Answer 1

If you use the browser development tools (Network tab) you can see that the "update" button sends an AJAX request to the server to a different address to the one that shows if you hover over it.

Therefore you need to be using that address to get the updated details. In this case the correct address is: https://lolprofile.net/index.php?page=summoner&ajaxr=1&region=euw&name=Sidestep%20Sabrina