How to scroll down an element on page - python

I would like to scrape names of companies from https://justjoin.it/, however my scraper has a problem with page scrolling. I usually use the following method to scroll down page:
driver.execute_script("window.scrollTo(0, x)")
Unfortunately, on this page execute_script method doesn't work.
I tried also to manipulate the element with table od names by send_keys method, but my script raises error:
elem = driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[1]/div/div[2]/div[1]/div')
elem.send_keys(Keys.END)
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Does anyone have an idea how to handle with this?

Try this
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")

As you wrote, you first have to click on the element with the list of jobs offers and after that scroll to the desired element.
The scrolling will work fine in this case

Related

Error when switching to iframe with Python Selenium

Currently I'm attemping to switch from my default content, to the only iframe in the website. I don't know if it's how the site is coded, but I can't access via DOM any element.
This is the HTML structure of the site:
XPATH of site is //*[#id="iframeBody"] (When I paste this in the element inspector, I get the correct iframe). So, if I try to switch using frame_to_be_available_and_switch_to_it, this is the output:
try:
WebDriverWait(self.driver, 10).until(ec.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[#id='iframeBody' and #name='body']")))
except Exception as e:
print(e)
>>> Message: javascript error: Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element'.
>>>(Session info: chrome=79.0.3945.88)
I also tried creating an iframe element variable, finding it via ID and XPATH and then I've used switch_to(element). Getting the same result. When I print this variable, the element is actually found:
# Also tried finding with id
element = self.driver.find_element_by_xpath('//iframe[#id='iframeBody')
print(element)
<selenium.webdriver.remote.webelement.WebElement (session="9184691b1fdcccc15dd36bbcb914ac8b", element="1ef77729-8a6e-4d3c-98bd-c95878437585")>
But when I try to switch to this iframe, I get the same result as above.
For some reason, this site is not letting me use the DOM data, actually, when I try to click a button I need to use action chains, because I get the same error.
Anybody can help me?
Did you put enough time before switching ?
also, you could directly switch as well:
driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element_by_id("iframeBody"))

Selenium click on XPath button

I have the following code when I inspect on Chrome.
<span id="button-1111-btnInnerEl" class="x-btn-inner x-btn-inner-center" unselectable="on" style="">New Email</span>
I need to click on the label "New Email", but how should invoke it in Selenium (I'm using Python).
def CreateMail():
EmailButton="//*[contains(text(),'New Email')]"
driver.find_elements_by_xpath(EmailButton) // there is no method to enable click.
You can use execute_script
driver.execute_script("document.getElementById('button-1111-btnInnerEl').click()")
driver.find_element_by_id("button-1111-btnInnerEl").click()
Thanks all for your help. Finally i found the answer to my question.I had to add a wait statement, before finding the key. key wasn't present when the page loads, so had to wait a little bit to find the correct key.
def CreateMail():
try:
element = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.ID, "button-1143-btnInnerEl")))
driver.find_element_by_id("button-1143-btnInnerEl").click()
except TimeoutException:
print ("Loading took too much time!")
Hope This XPath will work for you.If you want to validate xpath using your chrome browser just paste this text on your chrome console $x("//*[text()='New Email']") and check how many elements found using this XPath
driver.find_elements_by_xpath("//span[text()='New Email']")
Your statement : "driver.find_elements_by_xpath(EmailButton)" . Click does not work on group of elements. It is actionable only on single element. So you use a singular finder.
driver.find_**element**_by_id(EmailButton).click()
As per the HTML you have shared, the id attribute looks dynamic to me. So we must construct a dynamic xpath or css. Additionally instead of find_elements we have to use find_element so a single WebElement is returned and we can invoke the click() method. Finally, if you look at the node properly, the unselectable attribute is on so we will take help of JavascriptExecutor as follows :
myElement = driver.find_element_by_xpath("//span[starts-with(#id, 'button-')][#class='x-btn-inner x-btn-inner-center']")
driver.execute_script("arguments[0].click();", myElement);
Actually there is different way for locating elements
but in your case there is a ID,
so you can prefer id if its exists
find solution below :
driver.find_element_by_id("button-1111-btnInnerEl").click()

Python selenium test - Facebook code generator XPATH

I'm trying to get the XPATH for Code Generator field form (Facebook) in order to fill it (of course before I need to put a code with "numbers").
In Chrome console when I get the XPATH I get:
//*[#id="approvals_code"]
And then in my test I put:
elem = driver.find_element_by_xpath("//*[#id='approvals_code']")
if elem: elem.send_keys("numbers")
elem.send_keys(Keys.ENTER)
With those I get:
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
What means wrong field name. Does anyone know how to properly get a XPATH?
This error usually comes if element is not present in the DOM.
Or may be element is in iframe.

Using python selenium to click an element not visible

I am using python selenium on browser to fill in some form. I was trying to select an element in the drop-down list,
0
but if I try to find it by text using this script:
browser.find_element_by_link_text("0").click()
it result an error:
"unknown error: Element is not clickable at point (498, 612). Other element would receive the click: ..."
and if try to find it by class name:
browser.find_element_by_class_name("dropdown-toggle").click()
it result in another error: "element not visible"
is there any way I can click onto that drop-down list? Thanks very much.
I had a similar problem. You can execute a script to change the visibility of that element once you find it and then click it.
driver.execute_script("arguments[0].style.visibility = 'visible';",myElement)
myElement.click()
Try to find it by Xpath searching for partial class and text at the same time:
browser.find_element_by_xpath(//a[contains(#class, 'dropdown-toggle select') and contains(text(), '0')]).click();
you should get the element by xpath and then press it.
browser.find_element_by_xpath(xpath).
read here how to get the xpath: http://www.wikihow.com/Find-XPath-Using-Firebug
Thanks for the feedback, I've found the problem which was due to the new script being loaded by javascript as triggered by some clicks. I've created the following code to capture the exception and retrying until the element is ready:
while True:
try:
time.sleep(1)
browser.find_element_by_xpath("//div[#class='tckt']/a").click()
print("found")
break
except ElementNotVisibleException:
print("notvisible")
except WebDriverException:
print("clickduplicate")
I read it on articles says this happened a lot on radio button for Chrome webdriver, hope this helps.
Firstly click the parent element by finding it, using it's xpath->find the element you want to click within the parent element like the way you did. i.e
browser.find_element_by_link_text("0").click()
Hope this 2 steps will work for you. Otherwise please post the url with the code you've tried-It'll be easy to find out the issue.

Selenium (python): can't switch to iframe (name is dynamically generated)

I'm having problem selecting the iframe and accessing the different elements inside it. The iframe name is dynamically generated (e.g. frame11424758092173 or frame0005809321 or frame32138092173). The problem is that Selenium can't find the iframe no matter what i do....
switching to most recent frame doesn't work:
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to_frame(iframe)
Waiting for frame gets a timeout exception:
try:
iframe = WebDriverWait(driver, 5).until(EC.frame_to_be_available_and_switch_to_it(By.TAG_NAME('iframe')))
except:
logger.error(traceback.format_exc())
The following lines of code also times out:
try:
iframe = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.TAG_NAME, u"iframe")))
driver.switch_to_frame(iframe)
except:
logger.error(traceback.format_exc())
I have also tried iterating through the frames but it can't find it. The returned list is empty
iframes = driver.find_elements_by_tag_name('iframe')
#iframes is empty
really need some help...
Have you tried locating the iframe by its XPath and using the contains method?:
iframe = driver.find_element_by_xpath('//iframe[contains(#name, "frame")]')
driver.switch_to_frame(iframe)
Now you can access elements within the iframe.
To exit the iframe use:
driver.switch_to_default_content()
The contains method lets you get an element by a partial attribute value. Pretty useful for dynamically generated IDs, names, etc. You can search by other attributes as well using XPath. For example, say your iframe element has the attribute value = "3". You could use:
iframe = driver.find_element_by_xpath('//iframe[contains(#name, "frame")][#value = "3"]')
driver.switch_to_frame(iframe)
This approach can be used with any number of attributes as well.
You could also try getting the element by its selector. Keep in mind that this limits what you can do with it:
driver.execute_script('document.querySelector("INSERT SELECTOR HERE").doSomething();')
To get the Selector and/or XPath you're going to want to inpect the element using your browser (Chrome in my case). Right click on the element. Click Inspect. Then right click on the HTML element and click Copy > Copy Xpath or Copy > Copy Selector.
If that doesn't work for me, my last resort is to go the url of the iframe.To get that, you need to right-click on the area of the webpage where the iframe exists and click View Frame Source. It'll then lead you to a new page. The url of that page will be shown in the top of the browser after view-source:. You can then simply navigate to that url:
driver.get('insert url of iframe here')
And now you have access to the elements within the iframe. I do not recommend this approach if you are manipulating elements within the iframe and then exiting the iframe. Your changes will get lost. This will only work if you are scraping info off of that iframe, NOT if you are manipulating the elements within. Finding the iframe element and switching into it is usually better and safer.

Categories