Python Selenium take screenshots and Save as PDF for windows opened with document.write()

Question

I'm using Selenium with Python (in Jupyter notebook). I have a number of tabs open, say 5 tabs (with all elements already finished loading) and I would like to cycle through them and do 2 things:

1. Take a screenshot,
2. (As a bonus) Print each to PDF using Chrome's built-in Save as PDF function using A4 landscape, normal scaling, and a specified default directory, with no user interaction required.

(In the code below I focus on the screenshot requirement, but would also very much like to know how to Save it as a PDF)

This code enables looping through the tabs:

numTabs = len(driver.window_handles)
for x in range(numTabs):
    driver.switch_to.window(driver.window_handles[x])    
    time.sleep(0.5)     

However, if I try to add a driver.save_screenshot() call as shown below, the code seems to halt after taking the first screenshot. Specifically, '0.png' is created for the first tab (with index 0) and it switches to the next tab (with index 1) but stops processing further. It doesn't even cycle to the next tab.

numTabs = len(driver.window_handles)
for x in range(numTabs):
    driver.switch_to.window(driver.window_handles[x])    
    driver.save_screenshot(str(x) + '.png') #screenshot call
    time.sleep(0.5)    

Edit1: I modified the code as shown below to start taking the screenshots from window_handles[1] instead of [0] as I don't really need the screenshot from [0], but now not even a single screenshot is generated. So it seems that the save_screenshot() call doesn't work after even the initial switch_to.window() call.

tabs = driver.window_handles
for t in range(1, len(tabs)):
    print("Processing tab " + tabs[t]) 
    driver.switch_to.window(tabs[t])  
    driver.save_screenshot(str(t) + '.png') #screenshot call, but the code hangs. No screenshot taking, no further cycling through tabs.

Edit2: I've found out why my code is "hanging", no matter which method of printing to PDF or taking screenshots I'm using. I mentioned earlier that the new tabs were opened via clicking on buttons from the main page, but upon closer inspection, I see now that the content of the new tabs is generated using document. write(). There's some ajax code that retrieves waybillHTML content that is then written into a new window using document.write(waybillHTML)

For more context, this is an orders system, with the main page with a listing of orders, and a button next to each order that opens up a new tab with a waybill. And the important part is that the waybills are actually generated using document. write() triggered by the button clicks. I notice that the "View page source" option is greyed out when right-clicking in the new tabs. When I use switch_to.window() to switch to one of these tabs, the Page.printToPDF times out after 300 (seconds I suppose).

---------------------------------------------------------------------------
TimeoutException                          Traceback (most recent call last)
<ipython-input-5-d2f601d387b4> in <module>
     14     driver.switch_to.window(handles[x])
     15     time.sleep(2)
---> 16     data = driver.execute_cdp_cmd("Page.printToPDF", printParams)
     17     with open(str(x) + '.pdf', 'wb') as file:
     18         file.write(base64.b64decode(data['data']))

...

TimeoutException: Message: timeout: Timed out receiving a message from renderer: 300.000
  (Session info: headless chrome=96.0.4664.110)

So my refined question should be how to use Page.printToPDF to print a page in a new window (that is generated dynamically with document. write()) without timing out?

One approach I tried was to do this:

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(options=chrome_options, desired_capabilities=caps)

referring to: this question

but the problem is this is too 'aggressive' and prevents the code from logging into the ordering system and doing the navigating & filtering necessary to get the browser to the point where I get the orders listing page with the waybill generating buttons (i.e. the original setup in this question).

Edit3: At this point, I've tried something as simple as just getting the page source

try:
    pageSrc = driver.find_element(By.XPATH, "//*").get_attribute("outerHTML")
    print(pageSrc)

of the dynamically generated tabs (long after they had completed rendering and I can see the content on the screen (not using headless for this stage of the debugging)) and even this itself is throwing a TimeoutException, so I don't think it's an issue with waiting for content to load. Somehow the driver is unable to see the content. It might be something peculiar with how these pages are generated - I don't know. All the methods suggested in the answers for taking screenshots and saving PDFs are good I'm sure for otherwise normal windows. With Chrome the View page source remains greyed out, but I can see regular HTML content using Inspect.

Edit4: Using Chrome's Inspection function, the page source of the dynamically generated page has this HTML structure:

Curiously, "View page source" remains greyed out even after I'm able to Inspect the contents:

Edit5: Finally figured it out. When I clicked on the button for generating the new tab, the site would make an ajax call to fetch the HTML content of the new page, and document.write it to a new tab. I suppose this is outside the scope of selenium to handle. Instead, I imported selenium-wire, used driver.wait_for_request to intercept the ajax call, parsed the response containing the HTML code of the new tab, and dumped the cleaned HTML into a new file. From then on generating the PDF can be easily handled using a number of ways as others have already suggested.

Answer 1

I'm a little confused on these part of your questions:

Take a screen shot,

(As a bonus) Print each to PDF using Chrome's built in Save as PDF function using A4 landscape, normal scaling, and a specified default directory, with no user interaction required.

The function save_screenshot saves an image file to your file system. To convert this image file to a PDF you would have to open it and write it out to PDF file.

That task is easy, using various Python PDF modules. I have code for this so let me know if you need it and I will add it the code below.

Concerning printing the webpages in the tabs as a PDFs you would use execute_cdp_cmd with Page.printToPDF. The code below can be modified to support your unknown urls scenario via the button clicks. If you need help with this let me know.

import base64
import traceback
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import TimeoutException

chrome_options = Options()
chrome_options.add_argument("--start-maximized")
chrome_options.add_argument("--disable-infobars")
chrome_options.add_argument("--disable-extensions")
chrome_options.add_argument("--disable-popup-blocking")

# headless mode is required for this method of printing
chrome_options.add_argument("--headless")

# disable the banner "Chrome is being controlled by automated test software"
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation'])

driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)

# replace this code with your button code
###
driver.get('https://abcnews.go.com')

urls = ['https://www.bbc.com/news', 'https://www.cbsnews.com/', 'https://www.cnn.com', 'https://www.newsweek.com']
for url in urls:
    driver.execute_script(f"window.open('{url}','_blank')")
    # I'm using a sleep statement, which can be replaced with 
    # driver.implicitly_wait(x_seconds) or even a 
    # driver.set_page_load_timeout(x_seconds) statement
    sleep(5)
###


# A4 print parameters
params = {'landscape': False,
          'paperWidth': 8.27,
          'paperHeight': 11.69}

# get the open window handles, which in this care is 5
handles = driver.window_handles
size = len(handles)

# loop through the handles 
for x in range(size):
    try: 
       driver.switch_to.window(handles[x])
       # adjust the sleep statement as needed
       # you can also replace the sleep with 
       # driver.implicitly_wait(x_seconds)
       sleep(2)
       data = driver.execute_cdp_cmd("Page.printToPDF", params)
       with open(f'file_name_{x}.pdf', 'wb') as file:
          file.write(base64.b64decode(data['data']))
       # adjust the sleep statement as needed
       sleep(3)
     except TimeoutException as error:
        print('something went wrong')
        print(''.join(traceback.format_tb(error.__traceback__)))

driver.close()
driver.quit()

Here is one of my previous answers that might be useful:

• Selenium print PDF in A4 format

Answer 2

As your usecase is to Print each to PDF using Chrome's built in Save as PDF function or take a screen shot, instead of opening all the additional links at the same time you may like to open the links in the adjascent tab one by one and take the screenshot using the following Locator Strategies:

• Code Block:

  num_tabs_to_open = len(elements_href)
  windows_before  = driver.current_window_handle
  # open the links in the adjascent tab one by one to take screenshot
  for href in elements_href:
      i = 0
      driver.execute_script("window.open('" + href +"');")
      windows_after = driver.window_handles
      new_window = [x for x in windows_after if x != windows_before][0]
      driver.switch_to.window(new_window)
      driver.save_screenshot(f"image_{str(i)}.png")
      driver.close()
      driver.switch_to.window(windows_before)
      i = i+1
  

---

References

You can find a relevant detailed discussion in:

• InvalidSessionIdException: Message: invalid session id taking screenshots in a loop using Selenium and Python

Answer 3

Update 2

Based on your recent Edit 3, I now get the source for the new window by retrieving it with an AJAX call. The main page that the driver gets is:

test.html

<!doctype html>
<html>
<head>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta charset="utf-8">
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/1.12.4/jquery.min.js"></script>
<script>
function makeRequest() {
    var req = jQuery.ajax({
        'method': 'GET',
        'url': 'http://localhost/Booboo/test/testa.html'
        });
    req.done(function(html) {
        let w = window.open('', '_blank');
        w.document.write(html);
        w.document.close();
    });
}
</script>
<body>
</body>
<script>
$(function() {
   makeRequest();
});
</script>
</html>

And the document it is retrieving, testa.html, as the source for the new window is:

testa.html

<!doctype html>
<html>
<head>
<meta name=viewport content="width=device-width,initial-scale=1">
<meta charset="utf-8">
</head>
<body>
<h1>It works!</h1>
</body>
</html>

And finally the Selenium program gets test.html and enters a loop until it detects that there are now two windows. It then retrieves the source of the second window and takes a snapshot as before using Pillow and image2Pdf.

from selenium import webdriver
import time

def save_snapshot_as_PDF(filepath):
    """
    Take a snapshot of the current window and save it as filepath.
    """
    from PIL import Image
    import image2pdf
    from tempfile import mkstemp
    import os

    if not filepath.lower().endswith('.pdf'):
        raise ValueError(f'Invalid or missing filetype for the filepath argument: {filepath}')

    # Get a temporary file for the png
    (fd, file_name) = mkstemp(suffix='.png')
    os.close(fd)
    driver.save_screenshot(file_name)
    img = Image.open(file_name)
    # Remove alpha channel, which image2pdf cannot handle:
    background = Image.new('RGB', img.size, (255, 255, 255))
    background.paste(img, mask=img.split()[3])
    background.save(file_name, img.format)
    # Now convert it to a PDF:
    with open(filepath, 'wb') as f:
        f.write(image2pdf.convert([file_name]))
    os.unlink(file_name) # delete temporary file


options = webdriver.ChromeOptions()
options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)

try:
    driver.get('http://localhost/Booboo/test/test.html')
    trials = 10
    while trials > 10 and len(driver.window_handles) < 2:
        time.sleep(.1)
        trials -= 1
    if len(driver.window_handles) < 2:
        raise Exception("Couldn't open new window.")
    driver.switch_to.window(driver.window_handles[1])
    print(driver.page_source)
    save_snapshot_as_PDF('test.pdf')
finally:
    driver.quit()

Prints:

<html><head>
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta charset="utf-8">
</head>
<body>
<h1>It works!</h1>


</body></html>