How to scrape this website items with beautiful soup?

Question

Hi I am trying to scrape all the products on this site. https://segari.id/

However the url is static, and also soup when I tried scraping doesn't work on this site. Even if it does, how do I infinitely scroll to the bottom to get the items? What is the recommended way to scrape all the items?

Here's my current code:

#user agent

from fake_useragent import UserAgent
import requests
from bs4 import BeautifulSoup
import pandas as pd
from datetime import datetime

def userAgent(URL):
    dateNow = datetime.date(datetime.now())
    ua = UserAgent()
    USER_AGENT = ua.random
    headers = {"User-Agent" : str(USER_AGENT),"Accept-Encoding": "*","Connection": "keep-alive"}
    print(USER_AGENT)
    resp = requests.get(URL, headers=headers)
    soup = BeautifulSoup(resp.content, "html.parser")
    if resp.status_code != 200:
        print(f'error 200:{URL}')
        urlError = pd.DataFrame({'url':[URL],
                                'date':[dateNow] 
                                })
        responseCode=500
        urlError.to_csv('errorUrl/errorUrl.csv', mode='a', index=False, header=False)
    elif resp.status_code == 200:
        responseCode=200
    return soup,responseCode

#scraping component
URL = https://segari.id/
soup = userAgent(URL)
title = soup.find_all('div', {"class":"ProductCard_productName__1fPfF"})
listTitle=[]
for t in title:
    listTitle.append(t)
list = pd.DataFrame(listTitle)

Answer 1

You can grab data using only requests module as data is dynamically loaded by Javascript from API calls json response along with GET method.

import requests
data = requests.get('https://api-v2.segari.id/v1.1/products/price?agentId=311&size=40&page=0&paginationType=slice&deliveryDate=2022-05-12&deliveryServiceType=NEXT_DAY_DELIVERY&availableDeliveryDates=2022-05-12,2022-05-13,2022-05-14').json()
for item in data['data']['data']:
    title=item['productDTO']['name']
    print(title)

Output:

Ayam Broiler Probiotic Utuh
Ayam Broiler Utuh Premium
Ayam Kampung Utuh
Dada Ayam Boneless Frozen
Fillet Ayam
Paha Ayam Boneless Frozen
Kentang Dieng AB
Kentang Rendang
Kurma Ajwa Curah
Kurma Tunisia
Kurma Tunisia 500 gram
Lemon Import Imperfect
Pear Xiang Lie Imperfect
Ayam Giling
Bone Marrow Sum Sum Sapi Lokal     
Daging Kerang Hijau
Daging Sapi Import Giling
Daging Sapi Import Gulai Value Pack
Ikan Gabus Laut Fillet
Tulang Kaldu Sapi Lokal
Bumbu Ayam Garam Qian Ji
Beras Putih 1 kg Cap Bunga Setra Ramos
Ikan Dori Fillet
ABC Squash Delight Syrup Orange 460 ml Bundle 3
Ampela Ayam
Anggur Crimson
Anggur HIjau Autumn Crisp
Anggur Hijau Calmeria
Anggur Merah Red Globe Premium
Apel Envy
Apel Fuji
Apel Fuji Rosy Blush
Apel Fuji Wang Shan
Ati Ayam
Ayam Broiler Potong 4
Ayyomi Telur Ayam Kampoeng
Ayyomi Telur Ayam Negeri
Ayyomi Telur Omega 3
Baby Buncis Kenya
Bawang Bombay