'How to load infinite scroll pages faster?
https://immutascan.io/address/0xac98d8d1bb27a94e79fbf49198210240688bb1ed
This URL has 100k+ rows that I'm trying to scrape. They go back about a month (1/10/2022 I believe) but load in badges of 7-8.
Right now I have a macro slowly scrolling down the page, which is working, but takes about 8-10 hours per day's worth of rows.
As of now, when new rows load there are 2-3 items that load immediately and then a few that load over time. I don't need the parts that load slowly and would like them to load faster or not at all.
Is there a way that I can prevent elements from loading to speed up the loading of additional rows?
I'm using an autohotkey script that scrolls down with the mouse-wheel and that's been working best.
I've also tried a Chrome extension but that was slower.
I found a python script at one point but it wasn't any faster than autohotkey.
Answer: Immutable X has an API so I'm using that instead of this site that does the same thing. Here's the working code:
import requests
import time
import pandas as pd
import time
URL = "https://api.x.immutable.com/v1/orders"
bg_output = []
params = {'status': 'filled',
'sell_token_address': '0xac98d8d1bb27a94e79fbf49198210240688bb1ed'}
with requests.Session() as session:
while True:
(r := session.get(URL, params=params)).raise_for_status()
data = r.json()
for value in data['result']:
orderID = value['order_id']
info = value["sell"]["data"]["properties"]["name"]
wei = value["buy"]["data"]["quantity"]
decimals = value["buy"]["data"]["decimals"]
spacer = "."
eth = float(wei[decimals:] + spacer + wei[:decimals])
print(f'Count={len(bg_output)},Order ID={orderID}, Info={info}, Eth={eth}')
bg_output.append(f'Count={len(bg_output)},Order ID={orderID}, Info={info}, Eth={eth}')
timestr = time.strftime("%Y%m%d")
pd.DataFrame(bg_output).to_csv('bg_output' + timestr + '.csv')
#print(len(bg_output))
time.sleep(1)
if (cursor := data.get('cursor')):
params['cursor'] = cursor
else:
print(bg_output)
break
print(bg_output)
print("END")
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
