'Web Scrape - Scrape Fields with Same Tags
I get to a page where there are many rows of data per page.
My code gets to each row, and I am able to scrape the title of each row.
However, all the data after that, all looks to have the same tag names (for example the author and program number etc)
Based on this, how do I scrape all the data within each row.
from selenium import webdriver
from bs4 import BeautifulSoup
import time
driver = webdriver.Chrome()
baseURL = 'https://index.mirasmart.com/aan2022/'
for x in range (1,3):
driver.get(f'https://index.mirasmart.com/aan2022/SearchResults.php?pg={x}')
time.sleep(3)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')
eachRow=soup.find_all('div',class_='full search-result')
for item in eachRow:
title=item.find('h2').text
print(title)
Solution 1:[1]
You could use :-soup-contains to target by text
for result in soup.select('.detail'):
print('title: ', result.find_previous('h2').text)
for item in ['Author:', 'Session Name:', 'Author Disclosures:', 'Topic:', 'Program Number:', 'Author Institution:']:
try:
print(item, result.select_one(f'.cell:-soup-contains("{item}")').find_next('span').text)
except:
print(f'{item} not found')
print()
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | QHarr |
