Have searched through some of the previous questions and have not been able to rectify - I'm a complete newbie to this, so please forgive the ignorance... Trying to select the third 'Radio' button on the page using the following:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
import smtplib
import time
date = time.strftime("%d %b %Y" + ", " + "%H:%M%p")
print (time.strftime("%d %b %Y" + ", " + "%H:%M%p"))
driver=webdriver.Chrome()
driver.maximize_window()
driver.get("http://www.water.nsw.gov.au/water-licensing/registers")
driver.implicitly_wait(10)
driver.find_element_by_xpath('// [#id="wizard_tabs"]/div/div[1]/div/ul/li[3]/input').click()
The result is:
"Message: no such element: Unable to locate element:
{"method":"xpath","selector":"//[#id="wizard_tabs"]/div/div[1]/div/ul/li[3]/input"}"
Have tried waiting for longer (500), but it doesn't make a difference. When I 'inspect' the page that has opened, the xpath is still the same. I know there's a simple answer and I'm hoping you internet legends can help!
A few points of order:
There is no point in implicitly waiting when you can just perform an explicit wait (WebdriverWait.until) on a unique element on that page.
If the webdriver says the element isn't there, then the locator you are looking for is not there, at least not where it is actually looking. iframes can make you think you are looking in one place when really the webdriver is looking in another.
Always confirm your locators in the browser the moment you suspect something is off. Or just every time, it will save all that time you spend running your script, launching your browser, and loading the page every time as you tinker. Go to your browser's dev tools, click on the console tab, and type $$("css selector here") or $x("xpath here") and hit enter. If an element is returned, click the arrow to expand it and hover over what displays. Is it highlighting the desired element on the page? Then and ONLY then are you using a good locator.
If the dev tools console says your locator is good, but python still won't find it - even after a WebdriverWait - find the iframe that is stealing your focus and use webdriver.switch_to(iframe_name) to steal your focus back to the iframe with your element.
Practice less-brittle XPATHs. They can be quite surgical and readable with minimal effort. I would be willing to bet your input has something we could use to find it directly. If not, an element right above or below it in the hierarchy likely does. Post the HTML of your desired element and we'll know for sure.
To handle an iframe issue:
iframe_element = driver.find_element_by_css_selector("iframe[name='name_of_the_iframe']")
driver.switch_to.frame(iframe_element)
# Now find your element
Related
I am trying to get data from a Power Bi table. There are some elements that appear when hovering over a table. When I right click on ... I don't see Inspect Element. However, when I left click on this element, I can see a menu, and if I right click on any items, I can see Inspect element.
My first question, is why I don't see Inspect Element in the right click menu for all elements in the browser. Am I somehow able to open this ... menu programmatically in Selenium?
the Export Data element only appears in HTML after the first left click. I'm assuming this is created using Javascript and in order to export data with Selenium I would have to programmatically instantiate this by clicking on the ... menu. Is selenium capable of triggering javascript functions that generate more html code in a dynamic webpage? Or do I need to somehow click on the ... element.
If I can execute a javascript function, how can I find out in Edge the javascript function that gets executed and how can I replicate this function in Selenium
Essentially, if I try to find the Export data element in Selenium, it is not able to find it, unless I set a breakpoint before search, then in EdgeDriver I open this menu, and then I can find it and click it through Python
If all else fails, can I programmatically open the left click menu by automating a mouse click at certain coordinates in Selenium?
1.1 why I don't see Inspect Element in the right click menu for all elements:
PowerBi has its own context menu so they suppress the browsers context menu. If the element is tricky to find the dev tools, you can press Ctrl + Shift + C (while dev tools is open) and then click the desired element. Your mouse needs to be already over the element before pressing the key combination.
1.2 Am I somehow able to open this ... menu programmatically in Selenium?
Seems a little tricky, but could work if you first find the title of that area and move the mouse there, like described here: https://stackoverflow.com/a/8261754/12914172
Then your element should be in the html and you can find it hopefully by its class name vcMenuBtn that seems to be unique on that page. But you need to verify that.
2. Is selenium capable of triggering javascript functions that generate more html code in a dynamic webpage? Or do I need to somehow click on the ... element.
Selenium is able to execute javascript like desribed here: https://stackoverflow.com/a/70544802/12914172
However in your sample, and I was quickly checking the PowerBI online page, this looks like a whole lot of reverse engineering to understand and can sometimes be dangerous as well. I would go for hoover over the area find the ... and click it.
3. How can I find out in Edge the javascript function that gets executed
In dev tools you can set breakpoints to debug the steps the pages does after an action. But again, I would not invest to much time in that.
4. Can I programmatically open the left click menu by automating a mouse click at certain coordinates in Selenium?
Yes but this never works as good as the way described above. If you still want to give it a try, maybe that answer helps: https://stackoverflow.com/a/26385456/12914172
Many thanks to r000bin, this solution works for me, downloading data from PowerBI using Selenium for Python:
import selenium, mouse, time
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
url = 'https://dataport.gasunie.nl/Vulling-Gasopslagen-Nederland'
driver = selenium.webdriver.Chrome(service=Service())
driver.get(url)
time.sleep(4)
#driver.fullscreen_window()
#driver.switch_to.window(driver.current_window_handle)
time.sleep(4)
iframe = driver.find_elements(By.TAG_NAME, 'iframe')
assert len(iframe)==1
driver.switch_to.frame(iframe[0])
time.sleep(4)
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
actions.move_to_element_with_offset(driver.find_element(By.TAG_NAME, 'html'), 0,0)
actions.move_by_offset('5', '5').click().perform()
time.sleep(4)
button = driver.find_element(By.CLASS_NAME, 'vcMenuBtn')
button.click()
button = driver.find_element(By.ID, '0')
button.click()
# 4 tabs and 1 enter
time.sleep(4)
for n in range(4):
element = driver.switch_to.active_element
time.sleep(2)
element.send_keys(Keys.TAB)
time.sleep(2)
element = driver.switch_to.active_element
time.sleep(2)
element.send_keys(Keys.ENTER)
driver.close()
Disclaimer: I'm coming back to scripting after more than a decade so apologies in advance for such a basic question but help is much needed and appreciated.
I recently tried to find scripts to automate my job hunt, to that end I found a script that would help me login to a job portal and apply to jobs matching my criteria.
But I believe the script is not updated because I'm coming across an error when I'm running it:
NoSuchElementException: no such element: Unable to locate element:
{"method":"xpath","selector":"//*[#id="root"]/div[2]/div[2]/div/form/div[2]/input"}
(Session info: chrome=98.0.4758.109)
I believe this is in regard to the following lines of code:
driver.get("https://www.naukri.com")
driver.find_element_by_xpath('//*[#id="login_Layer"]/div').click()
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[2]/div/form/div[2]/input').send_keys("YOUR NAUKRI LOGIN ID")
driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[2]/div/form/div[3]/input').send_keys("YOUR NAUKRI PASSWORD")
time.sleep(5)
driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[2]/div/form/div[6]/button').click()
The script is able to go to Naukri.com (the job portal) and find the path for login_Layer ID, which opens the login sidebar. But then it is not able to find the username and password field. Maybe because the webpage has changed.
I'm trying to inspect the page elements and find the right xpath but having no luck. Again, haven't touched XML/HTML or web development of any kind for over a decade (and was a novice to begin with) so finding it hard.
Any guidance or help will be really appreciated.
Looking forward to your answers.
Thanks in advance!
Yeah your XPaths are outdated.
If you are on a chromium browser, right click on the element you want the XPath for and click inspect. If your DevTools are not yet open, do this twice. Then it will highlight your element in the page source. After that do a right click on the highlighted element, copy, and "Copy XPath" to update your script.
driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[2]/div/form/div[2]/input').send_keys("YOUR NAUKRI LOGIN ID")
This syntax should still work, but modern guidelines want you to use this:
from selenium.webdriver.common.by import By
driver.find_element(By.XPATH, '//*[#id="root"]/div[3]/div[2]/div/form/div[2]/input').send_keys("YOUR NAUKRI LOGIN ID")
wait=WebDriverWait(driver,20)
driver.get("https://www.naukri.com")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'#login_Layer > div'))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'form > div:nth-child(2) > input'))).send_keys("user")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'form > div:nth-child(3) > input'))).send_keys("pw")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.btn-primary.loginButton'))).click()
You'd want to use webdriver waits and switch up your xpaths.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Your XPATHs are actually incorrect.
What I would suggest you do is to write the most resilient XPath possible since the longer the XPath, the more brittle it becomes. Having the XPath based on a big HTML tag hierarchy is usually not a good idea since if there are new tags introduced in between (a new div tag) it will break. Which I believe what happened to this application also.
To avoid this you can use below xpaths.
driver.find_element_by_xpath("//label[text()='Email ID / Username']/../input").send_keys("YOUR NAUKRI LOGIN ID")
driver.find_element_by_xpath("//label[text()='Password']/../input").send_keys("YOUR NAUKRI PASSWORD")
I need to click the first item in the menu in a webpage with Python3 using Selenium.
I manage to log-in and navigate to the required page using Selenium, but there I get stuck: it looks like Selenium can't find any element in the page beyond the very first div in body.
I tried to find the element by ID, class, xpath, selector... The problem is probably not about that. I thought it could be about an iframe, but the content I need does not seem to be in one.
I guess that the problem is that the element I need to find is visible in the devtools, but not in the page source, so Selenium just can't see it - does this make sense? If so, can this be fixed?
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
self.driver.get("my site")
# log-in website and navigate to needed page
# [...]
# find element in page
# this works
first_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper")
# this does not work
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout.flex.flex-col.overflow-x-hidden.h-display-flex.h-flex-direction-column.h-screen")
Edit
The problem is most likely due to a dynamic webpage with parts of the DOM tree attached later on by a script. I downloaded a local version of page.html, removed scripts, and successfully found the sought-after element in the local page with
from selenium import webdriver
from selenium.webdriver.common.by import By
from pathlib import Path
driver = webdriver.Chrome()
html_file = Path.cwd() / "page.html"
driver.get(html_file.as_uri())
my_element = driver.find_element(By.CSS_SELECTOR, "[title='my-title']")
The exact same driver.find_element query won't work on the online page. I'm trying to implement a waiting condition as suggested in Misc08's answer.
I guess that the problem is that the element I need to find is visible
in the devtools, but not in the page source, so Selenium just can't
see it - does this make sense? If so, can this be fixed?
No, this does not make sense, since Selenium is executing a full browser in the background like you are using when you investigate the page source with the devtools.
But you have some options to narrow your problem. The first thing you can do, is to print the source the webdriver is "seeing" in this moment:
print(driver.page_source)
If you see the elements you are looking for in the page source, than you should try to improve your selector. It is helpful to go down the DOM step by step. Look for an upper element in the page tree first. If this works, try to find the next child element, then the next child, and so on. You can check if selenium found the element like this:
try:
myelement = driver.find_element(By.CSS_SELECTOR, 'p.content')
print("Found :)")
except NoSuchElementException:
print("No found :(")
By the way, i think your CSS selector is by far to complex, just use on CSS class, not all of them:
second_div = driver.find_element(By.CSS_SELECTOR, "#app-wrapper > div.layout")
But there might be the case, where the elements you are looking for, are not present in the page source from the beginning on. Dynamic webpages getting more and more popular. In this case parts of the DOM tree are attached later on by a script. So you have to wait for the execution of the scripts, before you can find this "dynamic" elements. One dirty and unreliable option is to just add a sleep() here. Much better is to to use an explicit waiting condition, see https://selenium-python.readthedocs.io/waits.html
I am trying to access PenFed in order to get my current outstanding amount. I have done quite a bit of research and unfortunately, I am still stumped. I am using Python Selenium and I am trying to click on the initial login button on the side in order to see the username field. This is the element's HTML code:
Login
When I try to run the following code:
driver.find_element_by_id("mobile-login").click()
I get the following error:
selenium.common.exceptions.ElementNotVisibleException: Message: element not interactable
Even when I try to use WebDriver Wait functions such as these:
try:
WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.ID, "mobile-login"))).click()
except ElementNotVisibleException:
WebDriverWait(driver, 5).until(EC.element_to_be_clickable((By.ID, "mobile-login"))).click()
No matter how long I make them wait, I get a timeout message:
raise TimeoutException(message, screen, stacktrace) selenium.common.exceptions.TimeoutException: Message:
All of my research says that invoking a wait function should fix it but it doesn't work for me. I also read that there might be an image overlay on top of the element that I would have to invoke before clicking on the button, but I didn't see anything in the website code as well. If I am testing it out, the only way that I am able to click on the button through code is if I physically click on it first, so I am unaware of anything else I can use. Thank you in advance for the help!
UPDATE: I have discovered that the following code works for me:
element = driver.find_element_by_id("mobile-login")
driver.execute_script("$(arguments[0]).click();", element)
But I do not know what the execute_script actually does. Can someone explain that piece of code works or if any other alternatives work for them?
The code you specified is JQuery. execute_script(p1, p2) runs a js script, where p1 is the script (in your case a JQuery line that clicks the element) and p2 is the desired element. It seems like you shouldn't need p2 if arguments[0] is equal to "element," but I'm not totally sure.
One potential fix is to use a counter for the number of times you clicked the element. If the counter reaches a certain number and the page doesn't change (you can check by finding a unique element/value on your current page), then you know it's not clickable.
Good luck!
The desired element is a dynamic element so to locate the element you have to induce WebDriverWait for the element to be clickable and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument('start-maximized')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www.penfed.org/')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.pfui-button.login-slide-button.pfui-button-login.dtm-global-nav[data-id='Open Log In Drawer']"))).click()
Browser Snapshot:
The link you are attempting to click is for the mobile site and is not visible if you are viewing the site at desktop resolutions. If you shrink your browser down until it changes layout, you will see the LOGIN button appear that corresponds to that link. That's why you are getting the ElementNotVisibleException.
To your second question, the reason that using .execute_script() works is that it executes JS directly and can click on anything... hidden or not. Selenium was designed to interact with the page as a user would so it won't let you click invisible elements, etc.
If you are intending for your script to act like a user, you will want to avoid using .execute_script() because it allows you to do things on the page that a user cannot do.
If you want to log in like a desktop user would, you need to click the "LOG IN" button using the CSS selector below
button[data-id='Open Log In Drawer']
That will open a side panel where you can enter your username, etc. and log in. FYI... you will likely need a wait to give the panel a chance to open before continuing the log in process.
I'm trying to click on the webpage "https://2018.navalny.com/hq/arkhangelsk/" from the website's main page. However, I get this error
selenium.common.exceptions.ElementNotInteractableException: Message:
There's nothing after "Message:"
My code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
browser = webdriver.Firefox()
browser.get('https://2018.navalny.com/')
time.sleep(5)
linkElem = browser.find_element_by_xpath("//a[contains(#href,'arkhangelsk')]")
type(linkElem)
linkElem.click()
I think xpath is necessary for me because, ultimately, my goal is to click not on a single link but on 80 links on this webpage. I've already managed to print all the relevant links using this :
driver.find_elements_by_xpath("//a[contains(#href,'hq')]")
However, for starters, I'm trying to make it click at least a single link.
Thanks for your help,
The best way to figure out issues like this, is to look at the page source using developer tools of your preferred browser. For instance, when I go to this page and look at HTML tab of the Firebug, and look for //a[contains(#href,'arkhangelsk')] I see this:
So the link is located within div, which is currently not visible (in fact entire sub-section starting from div with id="hqList" is hidden). Selenium will not allow you to click on invisible elements, although it will allow you to inspect them. Hence getting element works, clicking on it - does not.
What you do with it depends on what your expectations are. In this particular case it looks like you need to click on <label class="branches-map__toggle-label" for="branchesToggle">Список</label> to get that link visible. So add this:
browser.find_element_by_link_text("Список").click();
after that you can click on any links in the list.