'BeautifulSoup web scrape href issue
I am writing a web scraper and am struggling to get the href link from a web page. The URL is https://fortune.com/100-fastest-growing-companies/2021/search/. I am trying to get the href pull all of the article links. Here is an example of an href I am trying to pull"
<a class="searchResults__cellWrapper--39MAj" trackerdata="[object Object] href="https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/"><div class="searchResults__cellContent--3WEWj"><span><div>Atlanticus Holdings</div></span></div></a>
Here is my code:
import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re
URL = "https://fortune.com/100-fastest-growing-companies/2021/search/"
headers={'User-Agent':'mozilla/5.0'}
page = requests.get(URL,headers=headers)
soup = BeautifulSoup(page.text, "html.parser")
links = []
for link in soup.findAll(class_="searchResults__cellWrapper--39MAj"):
links.append((link.get('href')))
print(links)
When I run my code, it gives me nothing. Can someone point me in the right direction?
Solution 1:[1]
This page uses JavaScript to add elements
But it reads data as JSON from
and it can be used to get links
import requests
url = 'https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM='
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
data = response.json()
for item in data[1]['items']:
print(item['permalink'])
Result:
https://fortune.com/company/progressive/100-fastest-growing-companies/
https://fortune.com/company/adobe-systems/100-fastest-growing-companies/
https://fortune.com/company/salesforce-com/100-fastest-growing-companies/
https://fortune.com/company/newmont-mining/100-fastest-growing-companies/
https://fortune.com/company/lennar/100-fastest-growing-companies/
https://fortune.com/company/kla-tencor/100-fastest-growing-companies/
https://fortune.com/company/jd-com/100-fastest-growing-companies/
https://fortune.com/company/jacobs-engineering-group/100-fastest-growing-companies/
https://fortune.com/company/cigna/100-fastest-growing-companies/
https://fortune.com/company/charles-river-laboratories-intl/100-fastest-growing-companies/
https://fortune.com/company/advanced-micro-devices/100-fastest-growing-companies/
https://fortune.com/company/paypal-holdings/100-fastest-growing-companies/
https://fortune.com/company/netflix/100-fastest-growing-companies/
https://fortune.com/company/alphabet/100-fastest-growing-companies/
https://fortune.com/company/amazon-com/100-fastest-growing-companies/
https://fortune.com/company/veeva-systems/100-fastest-growing-companies/
https://fortune.com/company/align-technology/100-fastest-growing-companies/
https://fortune.com/company/etsy/100-fastest-growing-companies/
https://fortune.com/company/lululemon-athletica/100-fastest-growing-companies/
https://fortune.com/company/trade-desk/100-fastest-growing-companies/
https://fortune.com/company/emergent-biosolutions/100-fastest-growing-companies/
https://fortune.com/company/hibbett/100-fastest-growing-companies/
https://fortune.com/company/business-first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/marinemax/100-fastest-growing-companies/
https://fortune.com/company/spirit-of-texas-bancshares/100-fastest-growing-companies/
https://fortune.com/company/ies-holdings/100-fastest-growing-companies/
https://fortune.com/company/brooks-automation/100-fastest-growing-companies/
https://fortune.com/company/mid-penn-bancorp/100-fastest-growing-companies/
https://fortune.com/company/strategic-education/100-fastest-growing-companies/
https://fortune.com/company/waterstone-financial/100-fastest-growing-companies/
https://fortune.com/company/g-willi-food-international/100-fastest-growing-companies/
https://fortune.com/company/marketaxess-holdings/100-fastest-growing-companies/
https://fortune.com/company/ameris-bancorp/100-fastest-growing-companies/
https://fortune.com/company/shyft-group/100-fastest-growing-companies/
https://fortune.com/company/smartfinancial/100-fastest-growing-companies/
https://fortune.com/company/superior-group-of-companies/100-fastest-growing-companies/
https://fortune.com/company/meta-financial-group/100-fastest-growing-companies/
https://fortune.com/company/addus-homecare/100-fastest-growing-companies/
https://fortune.com/company/metropolitan-bank-holding/100-fastest-growing-companies/
https://fortune.com/company/1-800-flowers-com/100-fastest-growing-companies/
https://fortune.com/company/fs-bancorp/100-fastest-growing-companies/
https://fortune.com/company/arch-capital-group/100-fastest-growing-companies/
https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/
https://fortune.com/company/b-riley-financial/100-fastest-growing-companies/
https://fortune.com/company/joint-corp/100-fastest-growing-companies/
https://fortune.com/company/quidel/100-fastest-growing-companies/
https://fortune.com/company/acm-research/100-fastest-growing-companies/
https://fortune.com/company/victory-capital-holdings/100-fastest-growing-companies/
https://fortune.com/company/freedom-holding/100-fastest-growing-companies/
https://fortune.com/company/staar-surgical/100-fastest-growing-companies/
https://fortune.com/company/neurocrine-biosciences/100-fastest-growing-companies/
https://fortune.com/company/kornit-digital/100-fastest-growing-companies/
https://fortune.com/company/enstar-group/100-fastest-growing-companies/
https://fortune.com/company/logitech-international/100-fastest-growing-companies/
https://fortune.com/company/pjt-partners/100-fastest-growing-companies/
https://fortune.com/company/mvb-financial/100-fastest-growing-companies/
https://fortune.com/company/repligen/100-fastest-growing-companies/
https://fortune.com/company/harborone-bancorp/100-fastest-growing-companies/
https://fortune.com/company/capstar-financial-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssr-mining/100-fastest-growing-companies/
https://fortune.com/company/merchants-bancorp/100-fastest-growing-companies/
https://fortune.com/company/boston-beer/100-fastest-growing-companies/
https://fortune.com/company/first-horizon-national-corp/100-fastest-growing-companies/
https://fortune.com/company/lhc-group/100-fastest-growing-companies/
https://fortune.com/company/mercury-systems/100-fastest-growing-companies/
https://fortune.com/company/national-bank-holdings/100-fastest-growing-companies/
https://fortune.com/company/facebook/100-fastest-growing-companies/
https://fortune.com/company/monolithic-power-systems/100-fastest-growing-companies/
https://fortune.com/company/perkinelmer/100-fastest-growing-companies/
https://fortune.com/company/malibu-boats/100-fastest-growing-companies/
https://fortune.com/company/svb-financial-group/100-fastest-growing-companies/
https://fortune.com/company/first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/veritex-holdings/100-fastest-growing-companies/
https://fortune.com/company/myr-group/100-fastest-growing-companies/
https://fortune.com/company/mi-homes/100-fastest-growing-companies/
https://fortune.com/company/slm/100-fastest-growing-companies/
https://fortune.com/company/dow-chemical/100-fastest-growing-companies/
https://fortune.com/company/fox-factory-holding/100-fastest-growing-companies/
https://fortune.com/company/lgi-homes/100-fastest-growing-companies/
https://fortune.com/company/take-two-interactive-software/100-fastest-growing-companies/
https://fortune.com/company/paylocity-holding/100-fastest-growing-companies/
https://fortune.com/company/alarm-com-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssc-technologies-holdings/100-fastest-growing-companies/
https://fortune.com/company/stonex-group/100-fastest-growing-companies/
https://fortune.com/company/century-communities/100-fastest-growing-companies/
https://fortune.com/company/teradyne/100-fastest-growing-companies/
https://fortune.com/company/nmi-holdings/100-fastest-growing-companies/
https://fortune.com/company/virtu-financial/100-fastest-growing-companies/
https://fortune.com/company/ubiquiti/100-fastest-growing-companies/
https://fortune.com/company/scotts-miracle-gro/100-fastest-growing-companies/
https://fortune.com/company/vertex-pharmaceuticals/100-fastest-growing-companies/
https://fortune.com/company/medifast/100-fastest-growing-companies/
https://fortune.com/company/epam-systems/100-fastest-growing-companies/
https://fortune.com/company/medpace-holdings/100-fastest-growing-companies/
https://fortune.com/company/green-brick-partners/100-fastest-growing-companies/
https://fortune.com/company/leucadia-national/100-fastest-growing-companies/
https://fortune.com/company/fortinet/100-fastest-growing-companies/
https://fortune.com/company/appfolio/100-fastest-growing-companies/
https://fortune.com/company/kinsale-capital-group/100-fastest-growing-companies/
https://fortune.com/company/pennymac-financial-services/100-fastest-growing-companies/
To get other information you may have to digg in JSON.
for item in data[1]['items']:
print(item['permalink'])
for f in item['fields']:
if f['key'] == 'fgc_industry':
print(f['value'])
if f['key'] == 'rank':
print(f['value'])
if f['key'] == 'title':
print(f['value'])
Url has field token but I'm not sure if it may change value. You would have to observer it in browser for few requests. If it changes then you may have to search it in original HTML.
After decoding base64
import base64
base64.b64decode('Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=')
it gives 'fortune:BpsrfkMd#yJwcZI!4qj2wD93' and this values you may have to search in HTML.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |