We Keep Coding

Selenium can't find dynamially loaded input field element even after waiting for that specific element

I'm trying to access an input field of username (of login page) with Selenium. The page is JavaScript based.
driver.get() wait by default to load the complete page. In my case, it is unable to load that.
I can inspect the element on browser (firefox) and I get this.
<input type="text" autocomplete="username" name="username">
I tried to wait for that specific element with EC.presence_of_element_located.
Code trials:
driver = webdriver.Firefox()
driver.get(url)
delay = 10 # seconds
try:
myElem = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.NAME, 'username')))
print("Page is ready!")
except TimeoutException:
print("Loading took too much time!")
print(driver.page_source)
I get Loading took too much time!. Even though the element is there as I can inspect it in the browser. I also tried EC.presence_of_element_located((By.TAG_NAME, 'input'))) but It also can't find the tag.
Update from the comments: url='https://drive.inditex.com/drfrcomr/login'

To interact a clickable element instead of presence_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:
Using NAME:
myElem = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "username")))
Using CSS_SELECTOR:
myElem = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='username']")))
Using XPATH:
myElem = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[#name='username']")))
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
However on accessing the website I find the username field implemented through Windows Authentication
Solution
Incase of Basic Authentication you need to embed the username and password within the url as follows:
driver.get("http://username:password#drive.inditex.com/drfrcomr/login")
Update
Additionally the username field is within nested #shadow-root (open)
To send a character sequence to the username field you have to use shadowRoot.querySelector() and you can use the following Locator Strategy:
Code Block:
driver.get("http://username:password#drive.inditex.com/drfrcomr/login")
time.sleep(10)
item = driver.execute_script('''return document.querySelector('glyph-splash.hydrated').shadowRoot.querySelector('glyph-login-form.hydrated').shadowRoot.querySelector('input[name="username"]')''')
item.send_keys("ShaidaMuhammad")

You can do as stated here, so your code would looks like
url='https://user:password#drive.inditex.com/drfrcomr/login'
driver.get(url)
I tried with Chrome and it worked for me!

If you can get around that popup you may try:
myElem = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, "//*[.='Username']")))

Selenium Not Finding Element Present in HTML Even After Waiting for DOM to update

I am trying to scrape information on a website where the information is not immediately present. When you click a certain button, the page begins to load new content on the bottom of the page, and after it's done loading, red text shows up as "Assists (At Least)". I am able to find the first button "Go to Prop builder", which doesn't immediately show up on the page, but after the script clicks the button, it times out when trying to find the "Assists (At Least)" text, in spite of the script sleeping and being present on the screen.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get('https://www.bovada.lv/sports/basketball/nba')
# this part succeeds
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located(
(By.XPATH, "//span[text()='Go to Prop builder']")
)
)
element.click()
time.sleep(5)
# this part fails
element2 = WebDriverWait(driver, 6).until(
EC.visibility_of_element_located(
(By.XPATH, "//*[text()='Assists (At Least)']")
)
)
time.sleep(2)
innerHTML = driver.execute_script('return document.body.innerHTML')
driver.quit()
soup = BeautifulSoup(innerHTML, 'html.parser')

The problem is the Assist element is under a frame. You need to switch to the frame like this:
frame = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME,"player-props-frame")))
driver.switch_to.frame(frame)

Increase the timeout to confirm the timeout provided is correct, You can also confirm using debug mode. If still issue persist, please check "Assists (At Least)" element do not fall under any frame.
You can also share the DOM and proper error message if issue not resolved.

I have a couple of suggestions you could try,
Make sure that the content loaded at the bottom of the is not in a frame. If it is, you need to switch to the particular frame
Check the XPath is correct, try the XPath is matching from the Developer Console
Inspect the element from the browser, once the Developer console is open, press CTRL +F and then try your XPath. if it's not highlighting check frames

Check if there is are any iframes in the page, search for iframe in the view page source, and if you find any for that field which you are looking for, then switch to that frame first.
driver.switch_to.frame("name of the iframe")
Try adding a re-try logic with timeout, and a refresh button if any on the page
st = time.time()
while st+180>time.time():
try:
element2 = WebDriverWait(driver, 6).until(
EC.visibility_of_element_located(
(By.XPATH, "//*[text()='Assists (At Least)']")
)
)
except:
pass

The content you want is in an iFrame. You can access it by switching to it first, like this:
iframe=driver.find_element_by_css_selector('iframe[class="player-props-frame"]')
driver.switch_to.frame(iframe)

Round brackets are the issue here (at least in some cases...). If possible, use .contains selector:
//*[contains(text(),'Assists ') and contains(text(),'At Least')]

How to Fix Python Selenium Can't Click Target Element?

My Python Selenium can't click target element, instead seem click element behind target element?
I try to click, or enter text in 'drop-down meun', but I find that I am result in clicking element behind this 'drop-down. I know this is element behind it, because there is advistering area behind, and the result top-up showing the same advistering material. Here is my code:
# info for login
my_email = 'my_email'
my_passcode = 'my_passcode'
email_url = r'https://www.gmx.com/#.1559516-header-navlogin2-1'
# start driver and open url
driver = webdriver.Chrome(chrome_path)
driver.get(email_url)
# input email account
xpath = r'//*[#id="login-email"]'
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, xpath)))
target = driver.find_element_by_xpath(xpath)
actions = ActionChains(driver)
actions.move_to_element(target).perform()
actions.click().send_keys(my_email).perform()
# input passcode and hit 'enter' to login
xpath = r'//input[#id="login-password"]'
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, xpath)))
target = driver.find_element_by_xpath(xpath)
actions = ActionChains(driver)
actions.move_to_element(target).perform()
actions.click().send_keys(my_passcode).send_keys(Keys.ENTER).perform()
It happens to me for some other site when the site appear to have 'two layers' (not sure if I am using the right word). I can process anything on top layer, and only result in activate anything behind it. Many thank when provide solution!!

You don't need to use this ActionChains class, all you need to do can be done using WebElement.send_keys() and WebElement.click()
You don't need to re-find the element after using WebDriverWait as it returns the WebElement in case of success
I fail to see clickign on login button anywhere in your script, you can locate the relevant button using XPath contains() function like:
xpath = r'//button[contains(#class,"login-submit")]'
and then just call click() function on the resulting variable
Example suggested code:
You don't need to use this ActionChains class, all you need to do can be done using WebElement.send_keys() and WebElement.click()
You don't need to re-find the element after using WebDriverWait as it returns the WebElement in case of success
I fail to see clickign on login button anywhere in your script, you can locate the relevant button using XPath contains() function like:
xpath = r'//button[contains(#class,"login-submit")]'
and then just call click() function on the resulting variable
Example suggested code:
# input email account
xpath = r'//*[#id="login-email"]'
target = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, xpath)))
target.send_keys(my_email)
# input passcode and hit 'enter' to login
xpath = r'//input[#id="login-password"]'
target = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, xpath)))
target.send_keys(my_passcode)
xpath = r'//button[contains(#class,"login-submit")]'
target = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, xpath)))
target.click()

Selenium explicit wait doesnt work

I have a code that tell Selenium to wait until an element is clickable but for some reason, Selenium doesnt wait but instead, click that element and raise a Not clickable at point (x, y) immediately. Any idea how to fix this ?
x = '//*[#id="arrow-r"]/i'
driver = webdriver.Chrome(path)
driver.get('https://www.inc.com/inc5000/list/2017')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, x)))
driver.find_element_by_xpath(x).click()

EC.element_to_be_clickable check if element is visible and enabled. In terms of visibility it doesn't cover scenario when element is behind other. Maybe your page use something like blockUI widget and click() occurs before the cover disappears. You can check if element is truly clickable by enriching EC.element_to_be_clickable((By.XPATH, x)) check with assertion that ensure element is not behind by other. In my projects I use implementation as below:
static bool IsElementClickable(this RemoteWebDriver driver, IWebElement element)
{
return (bool)driver.ExecuteScript(#"
(function(element){
var rec = element.getBoundingClientRect();
var elementAtPosition = document.elementFromPoint(rec.left+rec.width/2, rec.top+rec.height/2);
return element == elementAtPosition || element.contains(elementAtPosition);
})(arguments[0]);
", element);
}
This code is in C# but I'm sure you can easily translate into your programming language of choice.
UPDATE:
I wrote a blog post about problems related to clicking with selenium framework https://cezarypiatek.github.io/post/why-click-with-selenium-so-hard/

Here is a link to the 'waiting' section for the Python Selenium docs: Click here
Wait will be like:
element = WebDriverWait(driver, 10).until(
EC.visibility_of((By.XPATH, "Your xpath"))
)
element.click();

Select iframe using Python + Selenium

So, I was absolutely baffled as to how to do this in Selenium, and couldn't find the answer anywhere, so I'm sharing my experience.
I was trying to select an iframe and having no luck (or not repeatably anyway). The HTML is:
<iframe id="upload_file_frame" width="100%" height="465px" frameborder="0" framemargin="0" name="upload_file_frame" src="/blah/import/">
<html>
<body>
<div class="import_devices">
<div class="import_type">
<a class="secondary_button" href="/blah/blah/?source=blah">
<div class="import_choice_image">
<img alt="blah" src="/public/images/blah/import/blah.png">
</div>
<div class="import_choice_text">Blah Blah</div>
</a>
</div>
</div>
</body>
</html>
The Python code (using the selenium library) was trying to find this iframe using this:
#timed(650)
def test_pedometer(self):
sel = self.selenium
...
time.sleep(10)
for i in range(5):
try:
if sel.select_frame("css=#upload_file_frame"): break
except: pass
time.sleep(10)
else: self.fail("Cannot find upload_file_frame, the iframe for the device upload image buttons")
Repeated fails with every combination of Selenium commands I could find.
The occasional success would not be reproducible, so perhaps it was some sort of race condition or something? Never did find the right way to get it in selenium proper.

This worked for me with Python (v. 2.7), webdriver & Selenium when testing with iframes and trying to insert data within an iframe:
self.driver = webdriver.Firefox()
## Give time for iframe to load ##
time.sleep(3)
## You have to switch to the iframe like so: ##
driver.switch_to.frame(driver.find_element_by_tag_name("iframe"))
## Insert text via xpath ##
elem = driver.find_element_by_xpath("/html/body/p")
elem.send_keys("Lorem Ipsum")
## Switch back to the "default content" (that is, out of the iframes) ##
driver.switch_to.default_content()

If iframe is dynamic node, it's also possible to wait for iframe appearence explicitly and then switch to it using ExpectedConditions:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
driver = webdriver.Chrome()
driver.get(URL)
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("iframe_name_or_id"))
If iframe doesn't have #id or #name it can be found as common WebElement using driver.find_element_by_xpath(), driver.find_element_by_tag_name(), etc..:
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_xpath("//iframe[#class='iframe_class']")))
To switch back from iframe:
driver.switch_to.default_content()

What finally worked for me was:
sel.run_script("$('#upload_file_frame').contents().find('img[alt=\"Humana\"]').click();")
Basically, don't use selenium to find the link in the iframe and click on it; use jQuery. Selenium has the capability to run an arbitrary piece of javascript apparently (this is python-selenium, I am guessing the original selenium command is runScript or something), and once I can use jQuery I can do something like this: Selecting a form which is in an iframe using jQuery

You don't need to use JavascriptExecutor. All you needed to do was switch into the frame and then switch back out, like so:
// do stuff on main window
driver.switch_to.frame(frame_reference)
// then do stuff in the frame
driver.switch_to.default_content()
// then do stuff on main window again
As long as you are careful with this, you will never have a problem. The only time I always use a JavascriptExecutor is to get window focus since I think using Javascript is more reliable in that case.

To shift Selenium's focus within the <iframe> you can use either of the following Locator Strategies:
Using ID:
driver.switch_to.frame("upload_file_frame")
Using CSS_SELECTOR:
driver.switch_to.frame(driver.find_element(By.CSS_SELECTOR, "iframe#upload_file_frame"))
Using XPATH:
driver.switch_to.frame(driver.find_element(By.XPATH, "//iframe[#id='upload_file_frame']"))
Ideally, you have to induce WebDriverWait for the desired frame to be available and switch to it and you can use either of the following Locator Strategies:
Using ID:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID,"upload_file_frame")))
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe#upload_file_frame")))
Using XPATH:
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='upload_file_frame']")))
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Reference
You can find a couple of relevant discussions in:
Ways to deal with #document under iframe
Switch to an iframe through Selenium and python

Selenium's selectFrame command accepts all the standard locators like css=, but it also has a an extra set of locators that work specifically with FRAME and IFRAME elements.
As the doc says:
selectFrame ( locator ) Selects a frame within the current window.
(You may invoke this command multiple times to select nested frames.)
To select the parent frame, use "relative=parent" as a locator; to
select the top frame, use "relative=top". You can also select a frame
by its 0-based index number; select the first frame with "index=0", or
the third frame with "index=2".
You may also use a DOM expression to identify the frame you want
directly, like this: dom=frames["main"].frames["subframe"]
Arguments: locator - an element locator identifying a frame or iframe
In general, you'll have better luck using the specialized locators, especially if you establish the right context first (e.g., select_frame("relative=top"); select_frame("id=upload_file_frame");).

you can use this simple code to look for the iframe using xpath
sample use
set_iframe("/html/body/div[2]/table/")
def set_iframe(xpath):
print('set_iframe')
driver.switch_to.default_content()
seq = driver.find_elements_by_tag_name('iframe')
for x in range(0, len(seq)):
driver.switch_to.default_content()
print ("iframe-"+str(x))
try:
driver.switch_to.frame(int(x))
driver.find_element_by_xpath(xpath)
return str(x)
except:
continue