'How to load infinite scroll pages faster?

https://immutascan.io/address/0xac98d8d1bb27a94e79fbf49198210240688bb1ed

This URL has 100k+ rows that I'm trying to scrape. They go back about a month (1/10/2022 I believe) but load in badges of 7-8.

Right now I have a macro slowly scrolling down the page, which is working, but takes about 8-10 hours per day's worth of rows.

As of now, when new rows load there are 2-3 items that load immediately and then a few that load over time. I don't need the parts that load slowly and would like them to load faster or not at all.

Is there a way that I can prevent elements from loading to speed up the loading of additional rows?

I'm using an autohotkey script that scrolls down with the mouse-wheel and that's been working best.

I've also tried a Chrome extension but that was slower.

I found a python script at one point but it wasn't any faster than autohotkey.

Answer: Immutable X has an API so I'm using that instead of this site that does the same thing. Here's the working code:

import requests
import time
import pandas as pd
import time

URL = "https://api.x.immutable.com/v1/orders"
bg_output = []
params = {'status': 'filled',
          'sell_token_address': '0xac98d8d1bb27a94e79fbf49198210240688bb1ed'}

with requests.Session() as session:
    while True:
        (r := session.get(URL, params=params)).raise_for_status()
        data = r.json()
        for value in data['result']:
            orderID = value['order_id']
            info = value["sell"]["data"]["properties"]["name"]
            wei = value["buy"]["data"]["quantity"]
            decimals = value["buy"]["data"]["decimals"]
            spacer = "."
            eth = float(wei[decimals:] + spacer + wei[:decimals])
            print(f'Count={len(bg_output)},Order ID={orderID}, Info={info}, Eth={eth}')
            bg_output.append(f'Count={len(bg_output)},Order ID={orderID}, Info={info}, Eth={eth}')
            timestr = time.strftime("%Y%m%d")
            pd.DataFrame(bg_output).to_csv('bg_output' + timestr + '.csv')
            #print(len(bg_output))
        time.sleep(1)    
        if (cursor := data.get('cursor')):
            params['cursor'] = cursor
        else:
            print(bg_output)
            break
            
print(bg_output)
print("END")


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source