We Keep Coding

How to scroll down an element on page

I would like to scrape names of companies from https://justjoin.it/, however my scraper has a problem with page scrolling. I usually use the following method to scroll down page:
driver.execute_script("window.scrollTo(0, x)")
Unfortunately, on this page execute_script method doesn't work.
I tried also to manipulate the element with table od names by send_keys method, but my script raises error:
elem = driver.find_element_by_xpath('//*[#id="root"]/div[2]/div[1]/div/div[2]/div[1]/div')
elem.send_keys(Keys.END)
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Does anyone have an idea how to handle with this?

Try this
driver.execute_script("window.scrollTo(0,document.body.scrollHeight)")

As you wrote, you first have to click on the element with the list of jobs offers and after that scroll to the desired element.
The scrolling will work fine in this case

Selenium selects multiple elements with shadow dom

I am working with the chrome://extensions page and I met some problems when trying to access a specific element using selenium. I am trying to get the extension id of an extension that I installed. The work flow is as follows:
1. I go to the chrome://extensions page by using driver.get()
2. I get the "developer mode" toggle by using selenium element selector, and click the toggle so that extension id gets shown on the page.
3. I want to use multiple elements selector to select all "extensions-item" element on the page, find the specific extension that I installed, and retrieve the id. But I am stuck at this part because I am able to select one element but not multiple elements for some reason.
I have the following script:
def expand_shadow_element(driver, element):
shadow_root = driver.execute_script(
'return arguments[0].shadowRoot', element)
return shadow_root
def get_extension_id(driver):
root1 = driver.find_element_by_tag_name("extensions-manager")
shadow1 = expand_shadow_element(driver, root1)
shadow2 = shadow1.find_element_by_css_selector("cr-view-manager")
root3 = shadow2.find_element_by_id("items-list")
shadow3 = expand_shadow_element(driver, root3)
i = shadow3.find_element_by_tag_name("extensions-item") # This works fine, but it can only retrieve the first "extensions-item". What if I want the second one?
print("finding single element successful")
items = shadow3.find_elements_by_tag_name("extensions-item") # This line throws an error
for item in items:
shadow4 = expand_shadow_element(driver, item)
if shadow4.find_element_by_id("name").text != "Chrome Automation Extension":
return "Success"
raise "Error"
The error message is as follows:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"extensions-item"}
(Session info: chrome=78.0.3904.97)
Besides, I used find_element_by_tag_name instead of others because others do not work. To my understanding, even if I believe it is essentially the same as find_element_by_css_selector, but using this instead will throw element not found error. I do not understand why.
Anyone has any idea? I just don't know why I am able to locate one element but not a list of them. I am guessing it is because the way I manipulated shadow-dom, but through searching on the Internet I was not able to find another way of manipulating it. Any help is appreciated, thank you!

How to obtain required content from "find_element_by_xpath"

I'm trying to use selenium & python to auto get part of the html content.
I use find xpath like below ways to sort out the price from specified flight number, but always get failed result " unable to locate "
Anyone could shed some lights on it ?
element_price = driver.find_element_by_xpath("//div[#id='flight_MU5401']")
element_price.find_element_by_xpath(".//span[#class='base_price02']")
It's the html

If I were to guess, I would say that you are probably getting that error because the element isn't loaded when your code runs. It's probably a slower loading page. You should try adding a wait and see if that helps.
You can also simplify your locator by using a CSS selector and only scraping the page once.
price = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#flight_MU5401 span.base_price02")).text

Python selenium test - Facebook code generator XPATH

I'm trying to get the XPATH for Code Generator field form (Facebook) in order to fill it (of course before I need to put a code with "numbers").
In Chrome console when I get the XPATH I get:
//*[#id="approvals_code"]
And then in my test I put:
elem = driver.find_element_by_xpath("//*[#id='approvals_code']")
if elem: elem.send_keys("numbers")
elem.send_keys(Keys.ENTER)
With those I get:
StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
What means wrong field name. Does anyone know how to properly get a XPATH?

This error usually comes if element is not present in the DOM.
Or may be element is in iframe.

Using python selenium to click an element not visible

I am using python selenium on browser to fill in some form. I was trying to select an element in the drop-down list,
0
but if I try to find it by text using this script:
browser.find_element_by_link_text("0").click()
it result an error:
"unknown error: Element is not clickable at point (498, 612). Other element would receive the click: ..."
and if try to find it by class name:
browser.find_element_by_class_name("dropdown-toggle").click()
it result in another error: "element not visible"
is there any way I can click onto that drop-down list? Thanks very much.

I had a similar problem. You can execute a script to change the visibility of that element once you find it and then click it.
driver.execute_script("arguments[0].style.visibility = 'visible';",myElement)
myElement.click()

Try to find it by Xpath searching for partial class and text at the same time:
browser.find_element_by_xpath(//a[contains(#class, 'dropdown-toggle select') and contains(text(), '0')]).click();

you should get the element by xpath and then press it.
browser.find_element_by_xpath(xpath).
read here how to get the xpath: http://www.wikihow.com/Find-XPath-Using-Firebug

Thanks for the feedback, I've found the problem which was due to the new script being loaded by javascript as triggered by some clicks. I've created the following code to capture the exception and retrying until the element is ready:
while True:
try:
time.sleep(1)
browser.find_element_by_xpath("//div[#class='tckt']/a").click()
print("found")
break
except ElementNotVisibleException:
print("notvisible")
except WebDriverException:
print("clickduplicate")
I read it on articles says this happened a lot on radio button for Chrome webdriver, hope this helps.

Firstly click the parent element by finding it, using it's xpath->find the element you want to click within the parent element like the way you did. i.e
browser.find_element_by_link_text("0").click()
Hope this 2 steps will work for you. Otherwise please post the url with the code you've tried-It'll be easy to find out the issue.