Based on every answer and piece of documentation I've seen, the following should wait for the element at xpath path:
delay = some amount of time way longer than I know is needed
driver = webdriver.Firefox()
driver.get(url)
wait = WebDriverWait(driver, delay, ignored_exceptions=NoSuchElementException)
wait.until(EC.presence_of_element_located(driver.find_element_by_xpath(path)))
but no matter what I do it always throws a NoSuchElementException immediately after getting the url. And before anyone marks this as a duplicate or calls me out on it, I am aware of this answer and although the element I'm looking for is in some kind of wrapper, I got the same problem when trying the above looking for that wrapper instead (also it works if I just provide a normal sleep call instead of the Expected Condition stuff which makes me think I don't need to manually enter into the wrapper). What's the point of having a function for waiting for an element to be loaded that doesn't wait until the element is loaded? Any help in solving this problem would be greatly appreciated.
I use this java code for similar situations:
private Wait<WebDriver> staleWait = new FluentWait<>(getDriver())
.withTimeout(WAIT_INTERVAL, TimeUnit.SECONDS)
.pollingEvery(POLLING_INTERVAL, TimeUnit.MILLISECONDS)
.ignoring(NoSuchElementException.class)
.ignoring(StaleElementReferenceException.class);
protected WebElement visibilityOf(WebElement webElement) {
staleWait.until((ExpectedCondition<Boolean>) webDriver -> {
try {
return element(webElement).isDisplayed();
} catch (StaleElementReferenceException e) {
return false;
} catch (NoSuchElementException ne) {
return false;
}
});
return element(webElement);
}
Related
I am trying to check if web page is loaded completed or not (i.e. checking that all the control is loaded) in selenium.
I tried below code:
new WebDriverWait(firefoxDriver, pageLoadTimeout).until(
webDriver -> ((JavascriptExecutor) webDriver).executeScript("return document.readyState").equals("complete"));
but even if page is loading above code does not wait.
I know that I can check for particular element to check if its visible/clickable etc but I am looking for some generic solution
As you mentioned if there is any generic function to check if the page has completely loaded through Selenium the answer is No.
First let us have a look at your code trial which is as follows :
new WebDriverWait(firefoxDriver, pageLoadTimeout).until(webDriver -> ((JavascriptExecutor) webDriver).executeScript("return document.readyState").equals("complete"));
The parameter pageLoadTimeout in the above line of code doesn't really reseambles to actual pageLoadTimeout().
Here you can find a detailed discussion of pageLoadTimeout in Selenium not working
Now as your usecase relates to page being completely loaded you can use the pageLoadStrategy() set to normal [ the supported values being none, eager or normal ] using either through an instance of DesiredCapabilities Class or ChromeOptions Class as follows :
Using DesiredCapabilities Class :
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.remote.DesiredCapabilities;
public class myDemo
{
public static void main(String[] args) throws Exception
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
DesiredCapabilities dcap = new DesiredCapabilities();
dcap.setCapability("pageLoadStrategy", "normal");
FirefoxOptions opt = new FirefoxOptions();
opt.merge(dcap);
WebDriver driver = new FirefoxDriver(opt);
driver.get("https://www.google.com/");
System.out.println(driver.getTitle());
driver.quit();
}
}
Using ChromeOptions Class :
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;
import org.openqa.selenium.PageLoadStrategy;
public class myDemo
{
public static void main(String[] args) throws Exception
{
System.setProperty("webdriver.gecko.driver", "C:\\Utility\\BrowserDrivers\\geckodriver.exe");
FirefoxOptions opt = new FirefoxOptions();
opt.setPageLoadStrategy(PageLoadStrategy.NORMAL);
WebDriver driver = new FirefoxDriver(opt);
driver.get("https://www.google.com/");
System.out.println(driver.getTitle());
driver.quit();
}
}
You can find a detailed discussion in Page load strategy for Chrome driver (Updated till Selenium v3.12.0)
Now setting PageLoadStrategy to NORMAL and your code trial both ensures that the Browser Client have (i.e. the Web Browser) have attained 'document.readyState' equal to "complete". Once this condition is fulfilled Selenium performs the next line of code.
You can find a detailed discussion in Selenium IE WebDriver only works while debugging
But the Browser Client attaining 'document.readyState' equal to "complete" still doesn't guarantees that all the JavaScript and Ajax Calls have completed.
To wait for the all the JavaScript and Ajax Calls to complete you can write a function as follows :
public void WaitForAjax2Complete() throws InterruptedException
{
while (true)
{
if ((Boolean) ((JavascriptExecutor)driver).executeScript("return jQuery.active == 0")){
break;
}
Thread.sleep(100);
}
}
You can find a detailed discussion in Wait for ajax request to complete - selenium webdriver
Now, the above two approaches through PageLoadStrategy and "return jQuery.active == 0" looks to be waiting for indefinite events. So for a definite wait you can induce WebDriverWait inconjunction with ExpectedConditions set to titleContains() method which will ensure that the Page Title (i.e. the Web Page) is visible and assume the all the elements are also visible as follows :
driver.get("https://www.google.com/");
new WebDriverWait(driver, 10).until(ExpectedConditions.titleContains("partial_title_of_application_under_test"));
System.out.println(driver.getTitle());
driver.quit();
Now, at times it is possible though the Page Title will match your Application Title still the desired element you want to interact haven't completed loading. So a more granular approach would be to induce WebDriverWait inconjunction with ExpectedConditions set to visibilityOfElementLocated() method which will make your program wait for the desired element to be visible as follows :
driver.get("https://www.google.com/");
WebElement ele = new WebDriverWait(driver, 10).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("xpath_of_the_desired_element")));
System.out.println(ele.getText());
driver.quit();
References
You can find a couple of relevant detailed discussions in:
Selenium IE WebDriver only works while debugging
Selenium how to manage wait for page load?
I use selenium too and I had the same problem, to fix that I just wait also for the jQuery to load.
So if you have the same issue try this also
((Long) ((JavascriptExecutor) browser).executeScript("return jQuery.active") == 0);
You can wrap both function in a method and check until both page and jQuery is loaded
Implement this, Its working for many of us including me. It includes Web Page wait on JavaScript, Angular, JQuery if its there.
If your Application is containing Javascript & JQuery you can write code for only those,
By define it in single method and you can Call it anywhere:
// Wait for jQuery to load
{
ExpectedCondition<Boolean> jQueryLoad = driver -> ((Long) ((JavascriptExecutor) driver).executeScript("return jQuery.active") == 0);
boolean jqueryReady = (Boolean) js.executeScript("return jQuery.active==0");
if (!jqueryReady) {
// System.out.println("JQuery is NOT Ready!");
wait.until(jQueryLoad);
}
wait.until(jQueryLoad);
}
// Wait for ANGULAR to load
{
String angularReadyScript = "return angular.element(document).injector().get('$http').pendingRequests.length === 0";
ExpectedCondition<Boolean> angularLoad = driver -> Boolean.valueOf(((JavascriptExecutor) driver).executeScript(angularReadyScript).toString());
boolean angularReady = Boolean.valueOf(js.executeScript(angularReadyScript).toString());
if (!angularReady) {
// System.out.println("ANGULAR is NOT Ready!");
wait.until(angularLoad);
}
}
// Wait for Javascript to load
{
ExpectedCondition<Boolean> jsLoad = driver -> ((JavascriptExecutor) driver).executeScript("return document.readyState").toString()
.equals("complete");
boolean jsReady = (Boolean) js.executeScript("return document.readyState").toString().equals("complete");
// Wait Javascript until it is Ready!
if (!jsReady) {
// System.out.println("JS in NOT Ready!");
wait.until(jsLoad);
}
}
Click here for Reference Link
Let me know if you stuck anywhere by implementing.
It overcomes the use of Thread or Explicit Wait.
public static void waitForPageToLoad(long timeOutInSeconds) {
ExpectedCondition<Boolean> expectation = new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver driver) {
return ((JavascriptExecutor) driver).executeScript("return document.readyState").equals("complete");
}
};
try {
System.out.println("Waiting for page to load...");
WebDriverWait wait = new WebDriverWait(Driver.getDriver(), timeOutInSeconds);
wait.until(expectation);
} catch (Throwable error) {
System.out.println(
"Timeout waiting for Page Load Request to complete after " + timeOutInSeconds + " seconds");
}
}
Try this method
This works for me well with dynamically rendered websites:
Wait for complete page to load
WebDriverWait wait = new WebDriverWait(driver, 50);
wait.until((ExpectedCondition<Boolean>) wd -> ((JavascriptExecutor) wd).executeScript("return document.readyState").equals("complete"));
Make another implicit wait with a dummy condition which would always fail
try {
wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[contains(text(),'" + "This text will always fail :)" + "')]"))); // condition you are certain won't be true
}
catch (TimeoutException te) {
}
Finally, instead of getting the html source - which would in most of one page applications would give you a different result , pull the outerhtml of the first html tag
String script = "return document.getElementsByTagName(\"html\")[0].outerHTML;";
content = ((JavascriptExecutor) driver).executeScript(script).toString();
There is a easy way to do it. When you first request the state via javascript, it tells you that the page is complete, but after that it enters the state loading. The first complete state was the initial page!
So my proposal is to check for a complete state after a loading state. Check this code in PHP, easily translatable to another language.
$prevStatus = '';
$checkStatus = function ($driver) use (&$prevStatus){
$status = $driver->executeScript("return document.readyState");
if ($prevStatus=='' && $status=='loading'){
//save the previous status and continue waiting
$prevStatus = $status;
return false;
}
if ($prevStatus=='loading' && $status=='complete'){
//loading -> complete, stop waiting, it is finish!
return true;
}
//continue waiting
return false;
};
$this->driver->wait(20, 150)->until($checkStatus);
Checking for a element to be present also works well, but you need to make sure that this element is only present in the destination page.
Something like this should work (please excuse the python in a java answer):
idle = driver.execute_async_script("""
window.requestIdleCallback(() => {
arguments[0](true)
})
""")
This should block until the event loop is idle which means all assets should be loaded.
Using this link I want all reviews from that page.
I have used xpaths(given in sample code) to click load more until it disappear from that page,but my solution fails and giving following errors.
Error- Message: Element is no longer attached to the DOM Stacktrace
or
in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
Sample Code with xpaths
Either
driver.execute_script('$("div.load-more").click();')
or
xpath_content='//div[#class = "load-more"]'
driver.find_element_by_xpath(xpath_content).click()
Is there any solution which may not fail in any case? How can I click on load more until it disappear form that page or Is there any other way to get all reviews from this page?
One more thing I am using firepath to generate review's xpath which is .//*[#id='reviews-container']/div1/div[3]/div1/div/div/div[3]/div/div1/div
Is there a way to get our own xpath instead using firepath?
This is a java solution for your problem. You can use the same logic for python as well
public static void loadAll(WebDriver driver) {
while (true) {
//Using findElements to get list of elements so that it wont throw exception if element is not present
List<WebElement> elements = driver.findElements(By.xpath("//div[#class='load-more']"));
//If the size is zero that means load more element is not present so breaking the loop
if (elements.isEmpty()) {
break;
}
//Assigning first element to a variable
WebElement loadEl = elements.get(0);
//Getting text of element
String text = loadEl.getText().toLowerCase();
//check if text contains load more, as, if it is loading it will have ... ,so we cant click at that time
if (text.contains("load more")) {
loadEl.click();
}
//if text contains 1 to 4 means [for ex "Load More 4"] this is the last click so breaking the loop
if (text.matches("load more [1-4]")) {
break;
}
}
System.out.println("Done");
}
I have pieced together a web crawler with Selenium that uses XPath to find elements. On the web page I'm scraping, there are two possible layouts that are loaded, depending on the content.
If I run my code on the wrong layout, I get the error: Message: Unable to locate element: {"method":"xpath","selector":
How can I create a try/except (or similar) that tries an alternative xpath, if the first xpath is not present? Or if none is present, continue on to the next section of code?
I haven't got experience with Python so I'm not able to write you an example code, but you should create two try/catch (or in this case try/except) block where you try to find your element with find_element_by_xpath. After that catch the NoSuchElementException and you can work with the WebElement(s).
In JAVA it looks something like this:
Boolean isFirstElementExist, isSecondElementExist = true;
WebElement firstElement, secondElement;
try {
firstElement = driver.findElement(By.xpath("first xpath"));
} catch (NoSuchElementException e) {
isFirstElementExist = false;
}
try {
secondElement = driver.findElement(By.xpath("second xpath"));
} catch (NoSuchElementException e) {
isSecondElementExist = false;
}
//... work with the WebElement
Here is the simple solution:
try:
element_in_layout_1 = driver.find_element_by_xpath("my_xpath_1")
except:
try:
element_in_layout_2 = driver.find_element_by_xpath("my_xpath_2")
except:
print "No Element was found"
pass
else:
print "Element in Layout 2 was found"
else:
print "Element in Layout 1 was found"
I'm writing a test script using selenium in python. I have a web-page containing a tree-view object like this:
I want to traverse over the menu to go to the desired directory. Respective HTML code for plus/minus indications is this:
<a onclick="changeTree('tree', 'close.gif', 'open.gif');">
<img id="someid" src="open.gif" />
</a>
The src attribute of the image can be either open.gif or close.gif.
I can detect weather there is a plus or minus by simply checking the src attribute of the img tag. I can also easily access to the parent tag, a, by using .find_element_by_xpath("..").
The problem is that I can't perform the click action not on the img nor the a tag.
I'v tried webdriver.Actions(driver).move_to_element(el).click().perform(); but it did not work.
I think I should mention that there is no problem in accessing the elements, since I can print all their attributes; I just can't perform actions on them. Any help?
EDIT 1:
Here's the js code for collapsing and expanding the tree:
function changeTree(tree, image1, image2) {
if (!isTreeviewLocked(tree)) {
var image = document.getElementById("treeViewImage" + tree);
if (image.src.indexOf(image1)!=-1) {
image.src = image2;
} else {
image.src = image1;
}
if (document.getElementById("treeView" + tree).innerHTML == "") {
return true;
} else {
changeMenu("treeView" + tree);
return false;
}
} else {
return false;
}
}
EDIT 2:
I Googled for some hours and I found out that there is a problem about triggering the Javascript events and the click action from web-driver. Additionally I have a span tag in my web-page that has an onclick event and I also have this problem on it.
After some tries like .execute_script("changeTree();"), .submit(), etc, I have solved the issue by using the ActionChains class. Now, I can click in all elements that they have java-script events as onclick. The code that I have used is this:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('someURL')
el = driver.find_element_by_id("someid")
webdriver.ActionChains(driver).move_to_element(el).click(el).perform()
I don't know if it occurred just to me or what, but I found out that I should find the element right before the key command; otherwise the script does not perform the action. I think it would be related to staling elements or something like that; anyway, thanks all for their attention.
for i in driver.find_elements_by_class_name("endorse-count"):
try:
i.click()
except:
continue
elem = WebDriverWait(driver, 100).until(EC.presence_of_element_located((By.CLASS_NAME, "dialog-window")))
src = elem.get_attribute("innerHTML")
add_skill(name, src)
WebDriverWait(driver, timeout=10)
I'm getting the following error while running the above code -
selenium.common.exceptions.StaleElementReferenceException: Message: u'Element is no longer attached to the DOM' ; Stacktrace:
at fxdriver.cache.getElementAt (resource://fxdriver/modules/web_element_cache.js:7646)
for line -
src = elem.get_attribute("innerHTML")
I'm running this code on LinkedIn user profile page, after logging in.
I tried putting the following line of code after "i.click()" -
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
But then I see that function "add_skill(name, src)" is not called and none of the code after driver.manage() is called, though for loop and further i.click() work fine.
Selenium is trying to complete actions (such as clicking a button or link) before verifying that the target element has rendered on the page. Selenium can be more patient, but you have to explicitly ask him to be.
For example, if you are testing something that makes an AJAX request, you can try something like this (in Ruby):
# timeout is in seconds
def wait_for_ajax(timeout=x)
time_limit, interval = (Time.now + timeout), 0.5
loop do
break if #driver.execute_script "return jQuery.active == 0"
sleep interval
raise "Wait for AJAX timed out after waiting for #{timeout} seconds" if Time.now > time_limit
end
end
To ensure your tests are fully comprehensive, always make Selenium waits for elements to load before running a task.
I had faced a similar issue and tried refreshing the page before finding that element, and it worked...
driver.navigate().refresh();
Though I couldnt reason out how this worked.
If this works for you as well, please let me know. I just want to learn more about this exception.
you can refer this page to learn about a similar issue
I had a similar problem when trying to execute some javascript (IJavaScripExecutor). I created an IWebElement and passed that to the JSE and that failed for me. When I moved the driver.FindElement(BySelector) into my JSE call, then it worked. (C# code ahead.)
Instead of:
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
IWebElement tableEl = driver.FindElement(selector);
js.ExecuteScript(script, tableEl);
I had to do:
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
js.ExecuteScript(script, driver.FindElement(selector));
You may have to do something similar: move your selector or element creation onto the same line as what you are trying to do. Or, maybe, in your case:
src = driver.find_element_by_class_name("dialog-window").get_attribute("innerHTML")
Upon closer inspection, that's what looks to be your problem, there's a stale web element object when you try to use the get_attribute method.