'how to download and save image into a folder from scraped url from ebay product page
I am trying to scrape links of the product images from ebay and then save the first 4 image into a folder. However, usually there is url of each image shown in the html and then we can easily scrape this. But with eBay I noticed that the links of the image only show in the html when we scroll to the image.
Below is my code for this. I am able to get all the images links and also able to download the first one. but the rest of them are not being downloaded into the folder. But I printed the links and they all are being scraped. However, I used the same script and used Pyautogui to click to the next images and it downloads the images fine. But that is not good way as the next button appears in different positions so only way I see is to click the buttons through selenium but it is not downloading the images when I do this?
maybe is there any other way to scrape the images that I cannot see? or something wrong in my code?
import textwrap
import os
import requests
from bs4 import BeautifulSoup
from selenium import webdriver
import csv
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
import pyautogui
import time
import codecs
from PIL import Image
from selenium.common.exceptions import NoSuchElementException
opts = Options()
opts.add_experimental_option('debuggerAddress', 'localhost:2323')
driver = webdriver.Chrome(executable_path="C:\chromedriver_win32\chromedriver.exe", chrome_options=opts)
act = ActionChains(driver)
product_data_table_upload = '/Users/singh/PycharmProjects/ebayscraper/product_url_tables_shpock.csv'
product_data_cabinets_upload = '/Users/singh/PycharmProjects/ebayscraper/product_url_cabinets_shpock.csv'
images_folder = '/Users/singh/PycharmProjects/ebayscraper/images_folder'
Upload_image_script_autoit = 'C:\\Users\\singh\\PycharmProjects\\etsywebautomation\\EtsyProductUploadScript.exe'
types_of_encoding = ["utf8", "cp1252"]
for encoding_type in types_of_encoding:
with codecs.open(product_data_cabinets_upload, 'r', encoding=encoding_type, errors='replace') as csv_file:
csv_reader = csv.reader(csv_file)
next(csv_reader)
for line in csv_reader:
driver.implicitly_wait(5)
driver.get('https://www.shpock.com/en-gb/sell')
def image_link_extraction():
os.chdir(images_folder)
driver.execute_script('window.open("https://www.google.com/");')
driver.switch_to.window(driver.window_handles[1])
driver.get(line[0])
for i in range(1, 6):
image_url = driver.find_element_by_xpath('//img[contains(@id, "icImg")]')
name = image_url.get_attribute('alt')
url = image_url.get_attribute('src')
next_button = driver.find_element_by_xpath(
"//button[contains(@aria-disabled, 'false') and contains(@role, 'button')]")
next_button.click()
with open(name.replace(' ', '-').replace('/', '') + '.jpg', 'wb') as f:
im = requests.get(url)
f.write(im.content)
print('writing: ', name)
print(name, url)
time.sleep(60)
driver.close()
driver.switch_to.window(driver.window_handles[0])
image_link_extraction()
Here is the image of where there is only 1 link and no other links for the other images and the links only appear when we scroll to them and is replaced where the current url is. Product page image
Here is the html as well:
</div><div id="test"><!-- test --></div><script type="text/javascript">var sPT = new Date().getTime(),picTimer,picTimer1,picTimer2;</script>
<button aria-disabled="false" id="linkMainImg" style="display: block; cursor: default; text-decoration: none;" aria-label="Item image. Opens image gallery.">
<div id="mainImgHldr" class="img500 vi-img-gallery-fixed-height-500 " style="width:500px" title="">
<!-- <span id="mainImgHldr" style="display: inline-block;"> -->
<h4 id="mainImageHeader" class="vi-img-gallery-header" aria-live="polite"></h4>
<img id="icThrImg" class="img img500 vi-hide-mImgThr vi-img-gallery-vertical-align " style="" src="https://p.ebaystatic.com/aw/pics/s.gif" imgsel="0" alt="Image is loading" />
<img id="icImg" class="img img500 vi-img-gallery-vertical-align " itemprop="image" src="https://i.ebayimg.com/images/g/TPEAAOSw7odiAQfD/s-l500.jpg" style="display:none;" onload="picTimer=new Date().getTime();" onerror=if(trackingUtil)trackingUtil('VI_MAIN_IMG_ON_ERROR') clk="" alt="Ergonomic Office Chair, Wheel, Mesh Back, Adjustable Height Home" />
<!-- </span> -->
<div id="vi-img-gallery-overlay--dim" class="vi-img-overlay--dim vi-gallery-hidden"></div>
<svg id="vi-img-gallery-video-icon" class="vi-gallery-hidden icon--video-play" aria-hidden="true" focusable="false">
<use xlink:href="#icon--video-play"/>
</svg>
<div id="vi-img-overlay--trans" class="vi-img-overlay--trans"></div>
</div>
</button>
<div class="hero-image-arrows visible">
<button class="prev-arr navigation-image-arr" aria-disabled="false" role="button" aria-label="Previous image - Item images" title="Previous image - Item images">
<svg class="gallery-svg" viewBox="0 0 12 20" width="8" height="14" focusable="false" aria-hidden="true">
<path d="M10.0002222 0L0 9.9997778 10.0002222 20l1.6142581-1.6142581-8.4370764-8.3859641 8.4370764-8.4375209" fill-rule="evenodd" fill="#545658"></path>
</svg>
</button>
<button class="next-arr navigation-image-arr" aria-disabled="false" role="button" aria-label="Next image - Item images" title="Next image - Item images">
<svg class="gallery-svg" viewBox="0 0 12 20" width="8" height="14" focusable="false" aria-hidden="true">
<path d="M1.5622222 0L0 1.5622222 8.3853333 10 0 18.3857778 1.5622222 20l10-10" fill-rule="evenodd" fill="#545658"></path>
</svg>
</button>
</div>
<span id="imgNATxt" class="imgNa">Image not available</span>
<span id="varImgNATxt" class="imgNa" style="display:none">Photos not available for this variation</span>
<noscript><img id="icImg" class="img img500" itemprop="image" src="https://i.ebayimg.com/images/g/TPEAAOSw7odiAQfD/s-l500.jpg" style="" clk="" alt="Ergonomic Office Chair, Wheel, Mesh Back, Adjustable Height Home" /></noscript>
<script type="text/javascript">
function picOnLoad(isSetClkId){
var elem = document.getElementById('icThrImg');
var pic = document.getElementById('icImg');
elem.style.display = 'none';
pic.style.display = '';
if(isSetClkId) {
pic.setAttribute('clk', elem.getAttribute('imgsel'));
Lastly, here is the link to the product page I am trying to get. But you can choose any product on ebay.
I appreciate any help on this. I have been struggling to solve this for a week now. This is my last resort. I am quite new to coding so I hope I was able to clearly explain my issue. Please let me know if you have any questions. Thank you
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
