I have created a web scraper that reads a dynamic table on a webpage that is live updating. I'm not sure the best way to do this... so I decided to just read the table every 30 seconds and take the results from there. Anyway, the website I am using to webscrape is pretty buggy and an error message pops up pretty often on the webpage:
error message
This error message causes my code to stop due to it reporting an error. Ideally, I would like to just move on and reload the webpage 30 seconds later and just miss out on this scrape. Currently, my code returns an error that it can't find the table I am looking for and then the code stops.
Traceback (most recent call last):
File "C:\Users\New Jaden\Documents\GitHub\dfk\test.py", line 108, in <module>
make_data()
File "C:\Users\New Jaden\Documents\GitHub\dfk\test.py", line 91, in make_data
print(type(wd.page_source))
File "C:\Users\New Jaden\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 926, in page_source
return self.execute(Command.GET_PAGE_SOURCE)['value']
File "C:\Users\New Jaden\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 424, in execute
self.error_handler.check_response(response)
File "C:\Users\New Jaden\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 246, in check_response
raise exception_class(message, screen, stacktrace, alert_text) # type: ignore[call-arg] # mypy is not smart enough here
selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: DataTables warning: table id=saleAuctionTable - Requested unknown parameter '13' for row 0, column 13. For more information about this error, please see
http://datatables.net/tn/4
Message: unexpected alert open: {Alert text : DataTables warning: table id=saleAuctionTable - Requested unknown parameter '13' for row 0, column 13. For more information about this error, please see http://datatables.net/tn/4}
My code snippet would be as follows:
def make_data():
options = webdriver.ChromeOptions()
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
url = 'https://dfktavern.com/saleAuction-alert'
wd = webdriver.Chrome('chromedriver', options=options)
wd.get(url)
time.sleep(5)
html = wd.page_source
df = pd.read_html(html)
The error happens on the line:
html = wd.page_source
I basically am just looking for, instead of my code breaking, for my code to just ignore the rest and wait for the next call to be made to do this again.
Related
This is my current code, but whenever I run it I get an error on the last line
stale element reference: element is not attached to the page document
from selenium import webdriver
url = "https://otctransparency.finra.org/otctransparency/OtcDownload"
driver.get(url)
driver.maximize_window()
driver.implicitly_wait(5)
agree = driver.find_elements_by_xpath("//button[#class='btn btn-warning']")[0]
agree.click()
nonats = driver.find_element_by_link_text('OTC (Non-ATS) Download')
nonats.click()
driver.find_element_by_xpath("//img[#src='./assets/icon_download.png']").click()
driver.switch_to.window(driver.window_handles[0])
driver.find_element_by_xpath("(//div[#class='checkbox-inline'])[2]").click()
driver.find_element_by_xpath("(//div[#class='checkbox-inline'])[1]").click()
driver.implicitly_wait(5)
button = driver.find_element_by_xpath("//img[#src='./assets/icon_download.png']")
print(button.is_displayed())
button.click()
When I run my code in debugging mode line by line, everything works fine without any errors. Any help would be great.
Edit: This is my stack trace
Traceback (most recent call last):
File "C:\Users\derpe\Desktop\python projects personal\testing finra\untitled1.py", line 31, in <module>
button.click()
File "C:\Users\derpe\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\derpe\anaconda3\lib\site-packages\selenium\webdriver\remote\webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "C:\Users\derpe\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\derpe\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
StaleElementReferenceException: stale element reference: element is not attached to the page document
(Session info: chrome=86.0.4240.75)
The checkbox clicks are triggering a page refresh while you're searching for the download link.
Call sleep to allow the refresh to finish.
driver.implicitly_wait(5)
import time # add this
time.sleep(1) # add this
button = driver.find_element_by_xpath("//img[#src='./assets/icon_download.png']")
print(button.is_displayed())
button.click()
I tried other selenium waits, but they did not work for me. Probably because the element search succeeds before the refresh starts but still clicks to late.
We have a Selenium process that fetches data from our data provider for further processing. The process tries to get all historical data, and since we do not have functionality to reliably filter out data that has been edited recently, we must get full data dumps from time to time to ensure that no records are dropped.
The Selenium process is written as a Python script. It goes to a specific website, logs in, presses a button that triggers report generation, and then downloads the report. The issue is that for full dump reports, the report generation has started to take longer than the default Selenium timeouts.
We use the Google Chrome driver, but would be willing to change the browser/driver if only other drivers have the ability to fully disable timeouts.
Below is a simplified, stripped-down version of the code with just the bare bones needed to appropriately describe this issue. The places where the report times out are marked out. It's waiting for a response from the website after clicking a button. How to disable that, and why what we currently do (can be seen below) is not enough?
from selenium import webdriver
def selectPopUp(driver, main_window_handle, maxWait=100):
"""
Selects a pop-up window. If no pop-up window appears, raises an
Exception.
:param maxWait: how long to wait (in seconds) before raising the error?
"""
popup_window_handle = None
i = 0
while (not popup_window_handle) and (i < maxWait):
for handle in driver.window_handles:
if handle != main_window_handle:
popup_window_handle = handle
break
# Wait a litle and re-iterate.
time.sleep(1)
i += 1
if popup_window_handle != None:
driver.switch_to.window(popup_window_handle)
return popup_window_handle
else:
raise Exception("No pop-up window appeared although the program was expecting one.")
def selectOption(dropdown, option, possibilities):
"""
Validates and selects an option from a drop-down.
:param dropdown: the Selenium reference to the dropdown selection.
:param option: the name of the option we'll select
:param possibilities: the possibilities we allow to choose from --
will raise an Exception
if `option` is not in `possibilities`.
:returns: Selenium selection.
"""
# Select the default Excel option
if option in possibilities:
return dropdown.find_element_by_xpath("//option[#value='" + option + "']")\
.click()
else:
raise Exception("Invalid choice! Use one of the following: \n\n " + \
"\n ".join(possibilities))
chromeOptions = webdriver.ChromeOptions()
prefs = {"download.default_directory" : DOWNLOAD_DIRECTORY}
chromeOptions.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(executable_path=settings.chromeDriver,
chrome_options=chromeOptions)
# Set an infite timeout time so that the slow Crystal reports
# would stand a chance of not timing out too.
driver.set_page_load_timeout(9999999999)
driver.set_script_timeout(9999999999)
# We go to the relevant website - that part is edited out.
# Once in the website, we select the current window handle
# to be able to come back to it after navigating through pop-ups.
main_window_handle = driver.current_window_handle
reportDropDown = driver.find_element_by_id(REPORT_ELEMENT_ID)
selectedReport = reportDropDown.find_element_by_xpath("//option[#value='" + SELCTED_REPORT_TITLE + "']")
selectedReport.click()
########################################################################
# >>>>>>>> The first place where the process keeps timing out. <<<<<<< #
########################################################################
# Click on the export button to open export pop-up.
exportButton = driver.find_element_by_name(r'crytlViewer$ctl02$ctl00')
exportButton.click()
# Now a pop-up with a download button appears. Select it.
popUpHandle = selectPopUp(driver, main_window_handle)
# Now, select a particular download format.
formatDropDown = driver.find_element_by_id("exportFormatList")
selectedFormat = selectOption(formatDropDown, reportFormat, REPORT_FORMAT_LIST)
# Download the report.
driver.find_element_by_id("submitexport").click()
#########################################################################
# >>>>>>>> The second place where the process keeps timing out. <<<<<<< #
#########################################################################
An representative example of an error message we get would be the following.
Traceback (most recent call last):
File "/home/ubuntu/main/oodle/core/utils.py", line 296, in repeatUntilSuccess
return func()
File "/home/ubuntu/main/oodle/salesforce/data_feed.py", line 277, in <lambda>
cleanUp=True),
File "/home/ubuntu/main/oodle/salesforce/data_feed.py", line 322, in pushReports
'liveEnvironment': self.liveEnvironment}]\
File "/home/ubuntu/main/oodle/core/reporting.py", line 1160, in __getitem__
self.handlers[name].PreparePandas(**paramDict)
File "/home/ubuntu/main/oodle/reports/vienna_salesforce_data_feed/Salesforce_LoanObject_Parsed.py", line 38, in PreparePandas
loan = self.manager.handlers[crystalReport].PreparePandas()
File "/home/ubuntu/main/oodle/core/reporting.py", line 1231, in PreparePandas
return self.TransformRaw(self.GetRaw(fileFrom))
File "/home/ubuntu/main/oodle/core/reporting.py", line 1387, in GetRaw
self.PrepareExcel(fileFrom)
File "/home/ubuntu/main/oodle/core/reporting.py", line 1367, in PrepareExcel
fileTo=fileTo)
File "/home/ubuntu/main/oodle/vienna/crystal.py", line 293, in downloadReport
self.downloadReport_no_error_handling(reportTitle, reportFormat, fileTo)
File "/home/ubuntu/main/oodle/vienna/crystal.py", line 247, in downloadReport_no_error_handling
self.driver.find_element_by_id("submitexport").click()
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
TimeoutException: Message: timeout
(Session info: chrome=60.0.3112.78)
(Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.4.0-1052-aws x86_64)
After edits suggested by Alexey Dolgopolov, we get the following error message, which is slightly different from the previous one:
Traceback (most recent call last):
File "/home/ubuntu/main/oodle/core/utils.py", line 296, in repeatUntilSuccess
return func()
File "/home/ubuntu/main/oodle/salesforce/data_feed.py", line 277, in <lambda>
cleanUp=True),
File "/home/ubuntu/main/oodle/salesforce/data_feed.py", line 322, in pushReports
'liveEnvironment': self.liveEnvironment}]\
File "/home/ubuntu/main/oodle/core/reporting.py", line 1160, in __getitem__
self.handlers[name].PreparePandas(**paramDict)
File "/home/ubuntu/main/oodle/reports/vienna_salesforce_data_feed/Salesforce_LoanObject_Parsed.py", line 38, in PreparePandas
loan = self.manager.handlers[crystalReport].PreparePandas()
File "/home/ubuntu/main/oodle/core/reporting.py", line 1231, in PreparePandas
return self.TransformRaw(self.GetRaw(fileFrom))
File "/home/ubuntu/main/oodle/core/reporting.py", line 1387, in GetRaw
self.PrepareExcel(fileFrom)
File "/home/ubuntu/main/oodle/core/reporting.py", line 1367, in PrepareExcel
fileTo=fileTo)
File "/home/ubuntu/main/oodle/vienna/crystal.py", line 318, in downloadReport
try:
File "/home/ubuntu/main/oodle/vienna/crystal.py", line 254, in downloadReport_no_error_handling
exportButton.click()
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 77, in click
self._execute(Command.CLICK_ELEMENT)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 494, in _execute
return self._parent.execute(command, params)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/home/ubuntu/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
TimeoutException: Message: timeout
(Session info: chrome=60.0.3112.78)
(Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 4.10.0-26-generic x86_64)
When you find_element_by_<whatever> an implicitly_wait timeout plays. The default value is 0.
I guess, you don't need to disable timeouts. Try using Explicit Wait. Read about it http://selenium-python.readthedocs.io/waits.html and https://seleniumhq.github.io/selenium/docs/api/py/webdriver_support/selenium.webdriver.support.wait.html#module-selenium.webdriver.support.wait.
And use something like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
...
selectedReport.click()
exportButton = WebDriverWait(driver, 9999, 5).until(
EC.visibility_of_element_located(
(By.NAME, r'crytlViewer$ctl02$ctl00')
)
I'm having a lot of troubles to find a clean and reliable way to wait in my Python Selenium tests. I've been using Implicit wait for a long time and for some reason it started being unstable so I switched to explicit wait. For some reason, I can't figure out a way for my tests to work 100% of the time. Right now, even though the code being tested and the test itself does not change, problems are happening again and again. (I'd say it works 50% of the time). When problems are happening, it's usually and the same lines. Here are some of them :
for th in td_list:
if(th.text == "18"):
th.click()
break
time.sleep(3)
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.ID, "closeModalDisplayBill"))
)
Here the "closeModalDisplayBill" is a button that has to be pushed. The problem it raises is the following :
Traceback (most recent call last):
File "./scripts/test-sl/SeleniumTest.py", line 54, in test_selenium_launcher
frontAdminErr = SlFrontAdminErr(driver, self.add)
File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontAdminErr.py", line 45, in __init__
driver.find_element_by_id("closeModalAddComp").click()
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 78, in click
self._execute(Command.CLICK_ELEMENT)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
return self._parent.execute(command, params)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotInteractableException: Message:
Another place where problems are recurrent is when Selenium try to open a modal to edit some stuff (clicking or clearing inputs is pretty unstable). Here is the code :
time.sleep(5)
driver.implicitly_wait(15)
driver.find_element_by_id("btnLogin").click()
time.sleep(1)
driver.find_element_by_id("login-email").clear()
And here is the log when the bug happen (trying to clear the login-email input field) :
Traceback (most recent call last):
File "./scripts/test-sl/SeleniumTest.py", line 39, in test_selenium_launcher
frontUserTest = SlFrontUser(self, driver, self.add)
File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontUser.py", line 21, in __init__
driver.find_element_by_id("login-email").clear()
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 93, in clear
self._execute(Command.CLEAR_ELEMENT)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
return self._parent.execute(command, params)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated
The same test can be successful four or five times in a row and then bug the same amount of time. (Everything here is tested from my own computer). When those tests are done through Jenkins it gets worse. Some of the time the full suite of test can be done in 10 or 8 minutes, but some of the time thoses tests will be done/failing after 30 minutes. I figure that the slow factor on Jenkins can be one of the reason my tests are unstable, but that doesn't explain why those bug appear regularly on my own computer.
All those tests are launched from another Python Script that initiate the firefox driver instance and then launch all the tests like such :
class SeleniumTestLauncher(unittest.TestCase):
environmnent = "envName"
add = ""
quickbooks_url = "https://developer.intuit.com/"
port = ""
driver = webdriver.Firefox()
base_url = ""
def setUpAdd(self):
self.driver.implicitly_wait(30)
self.verificationErrors = []
self.accept_next_alert = True
def test_selenium_launcher(self):
driver = self.driver
### -- Here just call every selenium test -- ###
## -- Test Front User -- ##
frontUserTest = SlFrontUser(self, driver, self.add)
SECOND EDIT :
As suggested I removed all the implicitly_wait() for explicit_wait. Seems a bit more stable but I still encounters bugs. For instance :
element = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.ID, "closeModalDisplayBill"))
)
driver.find_element_by_id("closeModalDisplayBill").click()
Even though the driver should wait for 'closeModalDisplayBill' to be clickable to actually attempt to click, I get this error :
Traceback (most recent call last):
File "./scripts/test-sl/SeleniumTest.py", line 53, in test_selenium_launcher
frontUserTest = SlFrontUser(self, driver, self.add)
File "/Users/JNEB/WebstormProjects/backend/backend/scripts/test-sl/SlFrontUser.py", line 37, in __init__
driver.find_element_by_id("closeModalDisplayBill").click()
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 78, in click
self._execute(Command.CLICK_ELEMENT)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webelement.py", line 499, in _execute
return self._parent.execute(command, params)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 297, in execute
self.error_handler.check_response(response)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotInteractableException: Message:
EDIT 3 :
Using only explicit wait is as unstable as before. Most of the errors I run into are those : "InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated". I don't understand why Selenium tries to click on the "input field" even though I use an explicit_wait. For instance :
time.sleep(3)
element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "login-email"))
)
driver.find_element_by_id("login-email").clear()
Here I ask selenium to wait for 'login-email' to be present, and if it is, to click on it. The wait returns before the 20 seconds timeout but the .clear() function throw a "InvalidElementStateException: Message: Element is not currently interactable and may not be manipulated".
closeModalDialog suggests that your application is based on some Ajax framework.
Automating Ajax applications using Selenium can sometimes be a big challenge. An element seems to be visible and clickable at time X when Webriver is trying to locate the element, but there are still some pending javascript code which modifies the DOM and will change this element at time X+10 miliseconds when the test is trying to click on the element, and Webdriver in this case usually throws one of: ElementNotVisibleException, InvalidElementStateException, StaleElementReferenceException, ElementNotInteractableException.
The test is working fine on a local (fast) machine, but fails on much slower virtual machines, or when the application is under heavy load and it's response times increase.
First of all discuss this issue with your developers.
Our appliaction is based on JSF + Primefaces framework and we had many issues like above while trying to automate our product. We have discussed the issues with our developers, and they put simple Status component in the header on each page, which indicates whether there are ongoing (active) ajax request or not. Secondly, since Primefaces is using jQuery (many ajax framoworks are using jQuery), we are also checking if jQuery status is ready (for details see this answer
We have created a simple waitForPrimefacesAndJQuery function that waits unitil Primefaces status and jQuery status are ready, and have been using this function everywhere alongsize with standard WebDriverWait.
This has solved 99,9% of issues.
There are still some components which don't want to cooperate though.
For them we usually use a brute-force method - simply cath the exception and try again a few times, for example in this way:
for(int i=0; i <= 5; i++ ){
try{
findElement( element ).click();
break;
} catch( ElementNotVisibleException | InvalidElementStateException
| StaleElementReferenceException | ElementNotInteractableException )
{
// swallow the exception, wait some time and try again
sleep( 500 );
}
}
I am trying to track down an error in Python Selenium. The error message is..
Traceback (most recent call last):
File "C:\myscipt\main.py", line 110, in <module>
source_mysource(func1, func2, func3, func4, func5, func6, func7, func8, func9)
File "C:\myscipt\sources\functions.py", line 132, in source_mysource
current_url = driver.current_url
File "C:\Users\tom\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 454, in current_url
return self.execute(Command.GET_CURRENT_URL)['value']
File "C:\Users\tom\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 201, in execute
self.error_handler.check_response(response)
File "C:\Users\tom\AppData\Local\Programs\Python\Python35\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 193, in check_response
raise exception_class(message, screen, stacktrace, value['alert'].get('text'))
selenium.common.exceptions.UnexpectedAlertPresentException: Alert Text: To display this page, Firefox must send information that will repeat any action (such as a search or order confirmation) that was performed earlier.
<super: <class 'WebDriverException'>, <UnexpectedAlertPresentException object>>
Is the error saying that the line current_url = driver.current_url is the one that is triggering the error or is it the next line?
When you use .current_url property of a WebDriver instance, a GET_CURRENT_URL webdriver command is sent, which is triggering an error in your case. What happens is, the currently opened web-page is not the current top-level browsing context because of the alert being present, and, according to the specification, it should and fails with an error.
In other words, current URL cannot be retrieved when there is an opened active alert.
I am trying to hit next url using python and selenium IDE.
It is showing only first link then error for next URL.I do not know what I am missing here. Can anyone help me out?
Here is my code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
#base Url
baseurl="http://www.happytrips.com/"
driver = webdriver.Firefox()
driver.get(baseurl)
driver.implicitly_wait(2)
link_name=driver.find_elements_by_xpath(".//*[#id='container']/div[3]/div/nav/div/ul/li/a")
for tab1 in link_name:
driver.implicitly_wait(3)
tab1_destination=tab1.get_attribute('href')
print tab1_destination
driver.get(tab1_destination)
Output
http://timesofindia.indiatimes.com/
Traceback (most recent call last):
File "happytest1.py", line 13, in <module>
tab1_destination=tab1.get_attribute('href')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 97, in get_attribute
resp = self._execute(Command.GET_ELEMENT_ATTRIBUTE, {'name': name})
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 402, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: Element not found in the cache - perhaps the page has changed since it was looked up
Stacktrace:
at fxdriver.cache.getElementAt (resource://fxdriver/modules/web-element-cache.js:8953)
at Utils.getElementAt (file:///tmp/tmpH_Pmgs/extensions/fxdriver#googlecode.com/components/command-processor.js:8546)
at WebElement.getElementAttribute (file:///tmp/tmpH_Pmgs/extensions/fxdriver#googlecode.com/components/command-processor.js:11746)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpH_Pmgs/extensions/fxdriver#googlecode.com/components/command-processor.js:12274)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmpH_Pmgs/extensions/fxdriver#googlecode.com/components/command-processor.js:603)
When you do a get of another page, all existing elements become invalid. You need to get all of the URLs before switching to a new page.
So, add another loop. The first loop iterates over the elements to save each URL, and the second loop iterates over the list of URLs rather than over the list of elements.