Python Selenium - How to fix WebDriverException: Message: target frame detached

Question

I am coding a program that crawls through a search query.

1. scraps all titles from the first query page
2. clicks each link of each title and scraps main body of the link and exits
3. moves onto the next page

It works fine for most, and it did for 480 titles last night. But with the same code, it keeps returning the following.

And at different titles, so once, an error might pop up on the 8th title of the 2nd query page, and another time, an error pops up on 6th title of first page and so.

I am utterly baffled as to why it was fine before, and why it is acting the way it is now. I would appreciate some pointers.

I am working with Anaconda - Jupyter Notebook, Selenium (4.1.0.) and Python.

WebDriverException                        Traceback (most recent call last)
<ipython-input-100-21e892bd2f24> in <module>
      1 #to_next_page(browser)
----> 2 crawl_page(browser)
      3 #x = XPath_finder(10, browser)
      4 #x.click()

<ipython-input-95-32c6665608a1> in crawl_page(browser)
     10     # 본문 구하는 부분
     11     for i, _ in enumerate(page_titles, 1):
---> 12         XPATH = XPath_finder(i, browser)
     13         XPATH.click()
     14         bs2 = BeautifulSoup(browser.page_source, 'html.parser')

<ipython-input-96-1904e01f1b97> in XPath_finder(i, browser)
      1 def XPath_finder(i, browser):
----> 2     elements = browser.find_elements(By.CLASS_NAME, 'txt_wrap')
      3     target = elements[i-1].find_element(By.TAG_NAME, 'a')
      4     return target

C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_elements(self, by, value)
   1277         # Return empty list if driver returns null
   1278         # See https://github.com/SeleniumHQ/selenium/issues/4555
-> 1279         return self.execute(Command.FIND_ELEMENTS, {
   1280             'using': by,
   1281             'value': value})['value'] or []

C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in execute(self, driver_command, params)
    422         response = self.command_executor.execute(driver_command, params)
    423         if response:
--> 424             self.error_handler.check_response(response)
    425             response['value'] = self._unwrap_value(
    426                 response.get('value', None))

C:\ProgramData\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py in check_response(self, response)
    245                 alert_text = value['alert'].get('text')
    246             raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 247         raise exception_class(message, screen, stacktrace)
    248 
    249     def _value_or_default(self, obj: Mapping[_KT, _VT], key: _KT, default: _VT) -> _VT:

WebDriverException: Message: target frame detached
  (Session info: chrome=100.0.4896.88)```

Answer 1

As you are seeing the error message...

WebDriverException: Message: target frame detached

...indiscriminately one possible reason can be incompatibility between the version of the binaries you are using.

---

Solution

Ensure that:

• Selenium is upgraded to current release Version 4.1.3.
• If you are using ChromeDriver and google-chrome combo, ensure that:
  • ChromeDriver is updated to current ChromeDriver v100.0 level.
  • Chrome Browser is updated to current chrome=100.0 (as per chromedriver=100.0.4896.60 release notes).
• If you are using GeckoDriver and firefox combo, ensure that:
  • GeckoDriver is updated to current GeckoDriver v0.31.0 level.
  • Firefox Browser is updated to current firefox=99.0.1.

---

tl; dr

Supported platforms