'Can't find button by CSS Selenium

Goal

I'm trying to create a list of source image URLs for the first n images in google search results, in this case for the search term "cat"

Approach

I'm doing this with Selenium and Google Chrome. My method is

  1. Search the term
  2. Click on the first image. This pops up a bigger preview
  3. Get the source URL of this image and write it to a file
  4. Click the "next" button in the popup image
  5. Repeat steps 3 & 4 for n-1 more times

Problem

Everything works fine until the second time I try to click the "next" button, and I'm not exactly sure what the problem is. I tried debugging, finding the element by XPATH multiple ways, finding it by CSS selector multiple ways, but I can't figure it out for the life of me.

Has anybody dealt with any issues like this? Or perhaps have a smarter way of doing what I'm trying to accomplish? Thanks in advance!

Code

        # Function parameters
        timeout_delay = 20
        full_size_delay = 0.1
        n=10



        # Search "cat" in google image search
        url = f"https://www.google.com/search?q=cat&tbm=isch"
        self.wd.get(url)

        # Find the first image and click on it
        # self.wd.find_element
        first_image = WebDriverWait(self.wd,
                                  timeout_delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".rg_i.Q4LuWd")))
        first_image.click()

        # Open up a file so we can write to it
        with open(f'cat.txt', 'a') as f:
            # For as many images as we elect to save
            for i in range(n):
                time.sleep(0.5)
                # Find the full-size popup 
                full_size = self.wd.find_elements(By.CLASS_NAME, "n3VNCb")

                # Find the associated URL of the image and write it to the file
                for j in full_size:
                    time.sleep(full_size_delay)
                    pic_url = j.get_attribute("src")
                    if "http" in pic_url:
                        #print(pic_url)
                        f.write(pic_url + "\n")
                        break

                # Find the "next image" right arrow button in the popup to go to the next image
                #next_button = self.wd.find_element(By.CSS_SELECTOR, ".knIqbf")
                next_button = WebDriverWait(self.wd, 5).until(
                    EC.element_to_be_clickable((By.CSS_SELECTOR, "a.knIqbf.SIwKhe")))
                #next_button = self.wd.find_element(By.XPATH, "a[jsname='gxjVle']")

                next_button.click()


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source