We Keep Coding

Parsing nested elements using selenium not working - python

The picture attached show the structure of the HTML page that I am trying to scrape:
First I retrieve the element league-item and then I am looking for the i item with class name : 'ds-icon-material league-toggle-icon'
Selenium is telling me that it cannot find any item with such name.
Here is my code:
path = r"""chromedriver.exe"""
driver = webdriver.Chrome(executable_path=path)
driver.get(_1bet)
time.sleep(5)
#a = driver.find_element_by_class_name('box-content.box-bordered.box-stick.box-bordered-last')
league1 = driver.find_elements_by_class_name('league-list')[0]
league1.find_element_by_class_name("ds-icon-material league-toggle-icon")
Can you please help me? I dont understand why it isn't working.
Thanks
NB: The website I'm scraping is: https://1bet.com/ca/sports/tennis?time_range=all

I can't access that web page so I can only guess what is going there.
I can figure 2 problems here:
To select element inside element it's better to use XPath starting with a dot .
The element you trying to access having 2 class names. You should use css selector or XPath to locate element according to multiple class names.
So I suggest you trying this:
league1 = driver.find_elements_by_class_name('league-list')[0]
league1.find_element_by_xpath(".//i[#class='ds-icon-material league-toggle-icon']")

Selenium expects single class name - and it adds dot at the beginning to create CSS selector.
But "ds-icon-material league-toggle-icon" is two classes and it will add dot befor first class but not before second class and this makes proble.
You may use directly css selector with all dots
.find_element_by_css_selctor(".ds-icon-material.league-toggle-icon")
or you have to trick Selenium and add missing dots between classes
.find_element_by_class_name("ds-icon-material.league-toggle-icon")
I can't connect with this page to confirm that this is all.

InvalidSelectorException Error while trying to get text from div class in Selenium Python

I'm trying to get text using Selenium WebDriver and here is my code. Please note that I don't want to use XPath, because in my case the ID gets changed on every relaunch of the web page.
My code:
driver.find_element_by_class_name("05uR6d").text
HTML:
<div class="O5uR6d">to fasten stuff</div>
Error:
selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified (Session info: chrome=88.0.4324.150)
Error is specific to the line of code I mentioned above.
How can I fix this?

Use this xpath:
driver.find_element_by_xpath("//div[contains(text(),'to fasten stuff')]")
Or this CSS:
driver.find_element_by_css_selector(".O5uR6d")
If both won't work, improve your question by adding more data of HTML you are looking at.

It can be done using multiple ways let me try to explain most of them.
Get element by class name.
this is the most easiest solution to get any element by class name you can simply do is:
driver.find_element_by_class_selector('foo');
Get Element by xpath
This is a bit tricky one, you can apply xpath either the class name, title, id or whatever remains same. it also works even if there's a text inside your div. For example:
driver.find_element_by_xpath("//tagname[#attribute='value']")
or in your case:
driver.find_element_by_xpath("//div['class='O5uR6d']")
or you can do something like #vitaliis said
driver.find_element_by_xpath("//div[contains(text(),'to fasten stuff')]")
You can read more about xpath and how to find it on this link
Get Elements by ID:
You can also get the element from id if there's any that's static:
driver.find_element_by_id('baz')
Get Elements by Name:
Get Elements by name using the following syntax:
driver.find_element_by_name('bazz')
Using CSS Selectors:
You can also use the css selectors to find the elements. Consider a following tag that has some attributes:
<p class="content">Site content goes here.</p>
You can get this element by:
driver.find_element_by_css_selector('p.content')
You can read more about it over here

what to do for dynamically changing xpaths in python using selenium?

I have a xpath as:
//*[#id="jobs-search-box-keyword-id-ember968"]
The number 968 constantly keeps on changing after every reload.
Rest of the string remains constant.
How to I find the constantly changing xpath?

You can use partial id with contains()
//*[contains(#id, "jobs-search-box-keyword-id-ember")]

You can try using starts-with below,
//*[starts-with(#id,'jobs-search-box-keyword-id-ember')]

The details provided is insufficient to to provide the accurate result. Still you can follow the below code references
In //*[#id="jobs-search-box-keyword-id-ember968"] the last number 968 keeps changing. but if you make this like //*[starts-with(#id,'jobs-search-box-keyword-id-ember')] then there might be possibility that you can have more then one element with the same partial is i.e. jobs-search-box-keyword-id-ember in this case it will locate on 1st matching element. that may not be your expected one
Use the tag name lets say element is an input tag whose id is jobs-search-box-keyword-id-ember968
Xpath - //input[starts-with(#id,'jobs-search-box-keyword-id-ember')]
CSS - input[id^='jobs-search-box-keyword-id-ember']
Use the relevant parent element to make this more specific. e.g the element is in parent tag <div class="container">
Xpath- //div[#class='container']//input[starts-with(#id,'jobs-search-box-keyword-id-ember')]
CSS - div.container input[id^='jobs-search-box-keyword-id-ember']

This worked for me:
Locator:
JOBS_SEARCH_BOX_XPATH = "//*[contains(#id,'jobs-search-box-keyword-id-ember')]"
Code:
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, JOBS_SEARCH_BOX_XPATH)))
.send_keys("SDET")

Find all elements that start with 'button-'

I am using Selenium to try and get all ID elements that that start with "button-". What I have tried so far was to use regex to match the "button-" but I get an error stating that TypeError: Object of type 'SRE_Pattern' is not JSON serializable. My code so far is:
all_btns = self.driver.find_elements_by_id(re.compile('^button-?'))
But as mentioned that raises an error. What is the appropriate way of getting all elements when you don't know the full ID, class, css selector etc.?

You could use find_element_by_xpath and starts-with:
find_elements_by_xpath('//*[starts-with(#id, "button-")]')
//* will match any elements
[starts-with(#id, "button-")] will filter the elements with a property id that starts with button-

Clément's answers works just fine, but there is also a way to do this with css selectors:
*[id^='button'].
* matches all tags, just like in xpath and ^= means 'starts with'

Can't get the value of an attribute with Selenium RC using xpath

I'm trying to get the first href attribute in a page using Selenium RC (in Python):
sel.get_text("xpath=//#href")
this returns an empty string.
However, an identical xpath on the same page inside Firefox (using the "View XPath" extension) yields the correct value.
I've tried fiddling with it, but the same happens for other attributes (eg #class) -- is there something awfully wrong with selenium or am I overlooking something trivial here?

Solved by using selenium's get_attribute e.g. sel.get_attribute("xpath=//a#href") for a nodes.

In Selenium RC, you can use the get_attribute function as below.
AttrValue = sel.get_attribute("//li[#id='result_0']/div/div[3]/div/a#href")
where //li[#id='result_0']/div/div[3]/div/a is the xpath.
xpath= is not required inside the function.

I think this applies to all types of elements:
For example: for an input type of element
selenium.getAttribute("//input#value");