'Webscaping table data with drop down menu help in either Pandas, Beautiful Soup or Selenium
I am trying to scrape data from this website:
https://www.shanghairanking.com/rankings/grsssd/2021
Initially pandas gets me out the gates and I can scrape the table but I am struggling with the drop down menus. I want to select the options next to the total score box which are PUB, CIT, etc. When I inspect the element it looks like maybe Javascript and the usual methods of interating over these options don't work. I have tried Beutifalsoup and most recently Selenium to select the drop downs by hand. This works for the default table data '''
import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import Select
driver = webdriver.Chrome('/Users/martinbell/Downloads/chromedriver')
driver.get('https://www.shanghairanking.com/rankings/grsssd/2021')
submit = driver.find_element_by_xpath("//input[@value='CIT']").click()
''' Doesn't get me anywhere.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
