'My code cannot append next pages tables to the end of the list
I trying to scrap all tables of 8 pages but my code just scrap 1st table. It can move to other pages also it works individually on each page but it cannot scrap all pages.
data_ingram = []
n = 1
for i in range(1,pagenum+1):
driver.get(f"https://www.ingrammicro.com/IMD_WASWeb/jsp/search/Results.jsp?cache=900&key=%3Bmoc.orcimmargni.etaroproc.D7801LQSWHCSU%3Ans%3B48334463074612032GVDM%3Asaila&type=1&perf=0&user=VG2302&kwds=&siskwds=&PerP=25&cate=&sCat=&mVnd=&tab=vendTab&pBgn=&pEnd=&iStk=&prom=&intl=2&acad=2&pc=&mac=&unix=2&auth=&spec=&cnsr=&dcon=&nDys=&dDys=&fCls=&fVal=&level=&page={n}&sCls=OrderNbr|ProductDes&sVal=%2B%2B")
html = driver.page_source
tables = pd.read_html(html)
data = tables[11]
data_ingram.append(data)
n += 1
df_ingram = pd.DataFrame(data_ingram[0])
df_ingram.drop_duplicates()
df_ingram
Solution 1:[1]
Instead of this line:
df_ingram = pd.DataFrame(data_ingram[0])
Use this:
df_ingram = pd.concat([pd.DataFrame(x) for x in data_ingram])
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | richardec |
