'Scrapy - Web scraping returning empty list from crypto exchange site
I usually have success when I web scrape but having trouble with this one. I'm assuming I'm getting blocked or they have security measures.
I'm trying to get the APY rates from https://www.okcoin.com/earn . You don't need an account to login and should be fairly straight forward so here's my code (the xpath I've provided is the table under Other offers):
from requests_html import HTMLSession
from scrapy import Selector
def url_headers(url):
"""
Creates headers for each url to be scraped.
param url: webpage for scraping
:return:
"""
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Firefox/68.0',
'Accept': '*/*',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate',
'Referer': url
}
# Get HTML version of document
sess = HTMLSession()
res = sess.get(url, headers=headers)
# Convert to text and extract whole document
selector = Selector(text=res.content)
return selector
sel = url_headers("https://www.okcoin.com/earn")
apy = sel.xpath(
'/html/body/div[2]/div/div/div/div[2]/div/div/div[3]/div/div[3]'
).extract()
print(apy)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
