Python Selenium - Cannot find element on page - python

I've perused SO for quite a while and cannot find the exact or similar solution to my current problem. This is my first post on SO, so I apologize if my formatting is off.
The Problem -
I'm trying to find a button on a webpage to punch me into a timeclock automatically. I am able to sign in and navigate to the correct page (it seems the page is dynamically loaded, as switching from different tabs like "Time Management" or "Pay Period" do not change the URL).
Attempts to solve -
I've tried using direct and indirect XPaths, CSS Selectors, IDs, Classes, Names, and all have failed. Included below are the different code attempts to find the button, and also a snippet of code including the button.
Button - HTML
Full Page HTML Source Code
<td>
<a onclick="return OnEmpPunchClick2(this);" id="btnEMPPUNCH_PUNCH" class="timesheet button icon " href="javascript:__doPostBack('btnEMPPUNCH_PUNCH','')">
<span> Punch</span></a>
<input type="hidden" name="hdfEMPPUNCH_PUNCH" id="hdfEMPPUNCH_PUNCH" value="0">
</td>
Attempts - PYTHON - ALL FAIL TO FIND
#All these return: "Unable to locate element"
self.browser.find_element_by_id("btnEMPPUNCH_PUNCH")
self.browser.find_element_by_xpath("//a[#id='btnEMPPUNCH_PUNCH']")
self.browser.find_element_by_css_selector('#btnEMPPUNCH_PUNCH')
#I attempted a manual wait:
wait=WebDriverWait(self.browser,30)
button = wait.until(expected_conditions.element_to_be_clickable((By.CSS_SELECTOR,'#btnEMPPUNCH_PUNCH')))
#And even manually triggering the script:
self.browser.execute_script("javascript:__doPostBack('btnEMPPUNCH_PUNCH','')")
self.browser.execute_script("__doPostBack('btnEMPPUNCH_PUNCH','')")
#Returns Message: ReferenceError: __doPostBack is not defined
None of these work, and I cannot seem to figure out why that is. Any help will be greatly appreciated!

Related

Selenium: Selecting Items from a List with No IDs

I'm experimenting with python + selenium, but I'm having trouble navigating list items when there is no ID provided for the elements.
In short, the page I'm interested in contains a list with two elements (see below): "Pitch View" and "List View". By default, "Pitch View" is selected, but I need the list view.
<div class="sc-bdnxRM ichxnR">
<div>
<ul class="Tabs__TabList-sc-1e6ubpf-0 eSBWKp"> [flex]
<li class="Tab__Item-sc-19t48gi-0 bsocgQ">
::marker
<a class="Tab__Link-sc-19t48gi-1 dDKNAk" href="#pitch">Pitch View></a> [event]
</li>
<li class="Tab__Item-sc-19t48gi-0 bsocgQ">
::marker
<a class="Tab__Link-sc-19t48gi-1 xSQCR" href="#list">List View></a> [event]
</li>
</ul>
...
Sorry, a screen shot would have been cleaner, but I don't seem to have permission.
I can load up the page, and I'm able to interact with all the elements. Switching to the "List View" manually is not an issue once that's done, but I can't seem to get Selenium to change to the view automatically. I'm getting TimeoutException errors, but the real issue is that I'm not providing the right element tags, so Selenium can't navigate correctly. Most of my attempts have been variations on the code shown below.
element=WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"//a[#class='Tab__Link-sc-19t48gi-1 xSQCR']")))
element.click()
If I can get to the "List View", I'll be fine from there as retrieving data from the tables isn't a problem once they are rendered.
My background isn't in web programming, so apologies if this is a simple question. If anyone is able to assist, that would be very helpful. Thanks!
Update:
It took a bit of tweaking, but I did eventually get it to work after exploring the link provided by Ahmed. Examples below using both the absolute as well as relative xpaths in case anyone else is stuck on this:
Absolute path
driver.find_element_by_xpath("/html/body/main/div/div[2]/div[2]/div[1]/div[4]/div/div/ul/li[2]/a").click()
Relative Path
driver.find_element_by_xpath('//a[contains(#href,"#list")]').click()
Thanks to both Ahmed and Dan.
You can target the href and get the results you want.
[href=‘#list’]

Selenium in Python: ElementNotInteractableException: Message: element not interactable

I know there are lots of posts about this exception, and I've read lots of them and tried their suggestions but they don't seem to work. Maybe you guys can see what I'm doing wrong.
I'm trying to scrape this page: https://www.kichink.com/stores/barshop
Specifically the information inside the little popup that apppears when clicking the "i" button next to the green "SEGUIR" button.
The HTML for that button is this:
<button id="about" class="btn btn-default btn-info-store" data-toggle="popover" data-original-title="" title=""></button>
I've tried many things to click it, but I just keep getting the element not interactable exception.
My last attempt was with:
element_boton = wd.find_element_by_class_name("btn.btn-default.btn-info-store")
element_boton.click()
The button seems to be correctly found, but I just can't click it.
Any suggestions?
Well, I have no idea why (so maybe someone can shed some light) but after using the full xpath (for some reason not even the relative xpath worked, it had to be the full one) to search for the button, it just started working, and i was able to click it.
element_boton = wd.find_element_by_xpath('/html/body/div[1]/div[3]/div/header/div/nav[2]/div/div['
'2]/div/ul[2]/li[1]/button')

How to Bypass confirm age model through scrapy

If you go to the site, you'd notice that there is an age confirmation window which I want to bypass through scrapy but I messed up with that and I had to move on to selenium webdriver and now I'm using
driver.find_element_by_xpath('xpath').click()
to bypass that age confirmation window. Honestly I don't want to go with selenium webdriver because of its time consumption. Is there any way to bypass that window?
I searched a lot in stackoverflow and google
but didn't get any answer which may resolves my problem. If you've any link or idea of resolving it by Scrapy, that'd be appreciated. A single helpful comment will be up-voted!
To expand on Chillie's answer.
The age verification is irrelavant here. The data you are looking for is loaded via AJAX request:
See related question: Can scrapy be used to scrape dynamic content from websites that are using AJAX? to understand how they work.
You need to figure out how https://ns5bwtai8m-dsn.algolia.net/1/indexes/*/queries?x-algolia-agent=Algolia%20for%20vanilla%20JavaScript%203.19.1&x-algolia-application-id=NS5BWTAI8M&x-algolia-api-key=e676b05f3844d3adf54a29732af6e43c url works and how can you retrieve in it scrapy.
But the age verification "window" is just a div that gets hidden when you press the button, not a real separate window:
<div class="age-check-modal" id="age-check-modal">
You can use the browser's Network tab in developer tools to see that no new info is uploaded or sent when you press the button. So everything is already loaded when you request a page. The "popup" is not even a popup, just an element whose display is changed to none when you click the button.
So Scrapy doesn't really care what's meant to be displayed as long as all html is loaded. If the elements are loaded, they are accessible. Or have you seen some information being unavailable without pressing the button?
You should inspect the html code more to see what each website does, this might make your scraping tasks easier.
Edit: After inspecting the original html you can see the following:
<div class="products-list">
<div class="products-container-block">
<div class="products-container">
<div id="hits" class='row'>
</div>
</div>
</div>
</div>
You can also see a lot of JS script tags.
The browser element inspector shows us the following:
The ::before part gives away that this was manipulated by JS, as you cannot do this with simple CSS. See Granitosaurus' answer for details on this.
What this means is that you need to somehow execute the arbitrary JS code on those pages. So you either need a solution with Scrapy, or just use Selenium, as many do, and as you already have.

Focus on current Screen Selenium

I am trying to test the functionality of a developing webpage by using selenium in python.
This webpage have several instances where ids/names are repeated.
For example:
<input class="" name="title1" type="text">
This line of code is repeated but linked to different input fields throughout the code.
Therefore, when I try to test the webpage by using:
driver.find_element_by_name("elname").send_keys("BOb")
it seems to be looking for the first instance of the name that is in the code instead of focusing in on the current screen and inputting my desired input. This screen is not a window. So I can not switch to a window.
Is there a way to cause the driver to only focus on the current screen?
Is the "current screen" really a browser alert, or is it simply html that is designed to look like a dialog box in the browser window? It's it's really HTML, then these individual input elements probably have parents with a unique name.
<div id="first_dialog"><input class="" name="title1" type="text"></div>
so you would limit your search by that:
driver.find_element_by_id("first_dialog").find_element_by_name("elname").send_keys("BOb")
If your goal is to interact with the current element in focus use driver.switch_to.active_element. This returns the current active element with which you can interact.

Cannot switch to a frame

I am having super difficulty understanding a problem I have while working on automating a page using Chromedriver. I am in the login page and here is how the HTML for the page looks:
<frame name="mainFrame" src>
<body>
<table ..>
<tr>
<td ..>
<input type="password" name="ui_pws">
</td>
..
..
..
</frame>
This is gist, the page of course has multiple tables, divs, etc ...
I am trying to enter the password in the input element using xpath //input[#name="ui_pws"].
But the element was not found.
So I thought it might be because of wrong frame and I tried:
driver.switch_to_frame('mainFrame')
and it failed with NoSuchFrameException.
So I switched to:
main_frame = driver.find_element_by_xpath('//frame[#name="mainFrame"]')
driver.switch_to_frame(main_frame)
Then to cross verify I got the current frame element using:
current_frame = driver.execute_script("return window.frameElement")
And to my surprise I got two different elements when printed it out.
Now I am really confused as to what I should be doing to switch frames or access the password field in the webpage. I have had 4 cups of coffee since morning and still have a brain freeze.
Can anyone please guide me with this?
You can try, this is in Java should be almost similar in python too
driver.switchTo().defaultContent();
WebElement frameElement = driver.findElement(By.xpath("//frame[#name='mainFrame']"));
drive.switchTo().frame(frameElement);
SwitchTo defaultContent helps bring in focus properly, and later we can switch to the desired frame in the window.
driver.switchTo().frame(driver.findElement(By.xpath("//frame[# name='mainFrame']")));
//perform operation which you want to perform on web elements present inside the frame(mainFrame), once you finish your operation come back to default
driver.switchTo().defaultContent();

Categories