I am using Selenium to try and get all ID elements that that start with "button-". What I have tried so far was to use regex to match the "button-" but I get an error stating that TypeError: Object of type 'SRE_Pattern' is not JSON serializable. My code so far is:
all_btns = self.driver.find_elements_by_id(re.compile('^button-?'))
But as mentioned that raises an error. What is the appropriate way of getting all elements when you don't know the full ID, class, css selector etc.?
You could use find_element_by_xpath and starts-with:
find_elements_by_xpath('//*[starts-with(#id, "button-")]')
//* will match any elements
[starts-with(#id, "button-")] will filter the elements with a property id that starts with button-
Clément's answers works just fine, but there is also a way to do this with css selectors:
*[id^='button'].
* matches all tags, just like in xpath and ^= means 'starts with'
Related
Using XPATH with Python, do I really need to use get() or getall() or does the xpath string suffice
For example, is this ok?
product_links = response.xpath('//a[contains(#class,"box_product")]/#href')
or do I really need to use
product_links = response.xpath('//a[contains(#class,"box_product")]/#href').getall()
Or is it so that when using an attribute #attribute works, but to retrieve the data (text) within the html tags itself then we use get() and getall()
question: when do I need to use variant 1 /#href or variant 2 /#href').getall()?
The goal is to obtain a workable array of links
Calling response.xpath('//a[contains(#class,"box_product")]/#href') gives you only an instance of Selector (i.e. a recipe for getting the results you want) instead of the actual results.
To get the actual results, you need to call either get(), which will give you only the first match, or getall(), which will return all matches.
So for your use case, go with getall().
=====================
Example and read more # https://www.pythongasm.com/introduction-to-scrapy/
I'm using Python with webdriver
I'm trying to locate element with find element with xpath and get :"unexpected EOF while parsing (, line 1)"
When I try to locate it by finding element with class name, it works well.
The problem is that I can't use it since I have several classes with the same name.
Here is xpath:
//*[#id="j_id0:j_id5:j_id6:j_id36"]/div/div/div[2]/div[2]/div[2]/ul/li[1]/div/svg/g/g[1]/path[1]
here is the class name : st0
here is my code line:
ss = self.SELENIUM_DRIVER.find_element_by_xpath('//*[#id="j_id0:j_id5:j_id6:j_id36"]/div/div/div[2]/div[2]/div[2]/ul/li[1]/div/svg/g/g[1]/path[1]')
-Which does not work
ddd = self.SELENIUM_DRIVER.find_element_by_class_name('st0')
-Which works
This is the html:
Thanks in advance
The following query will look for a path with the parent tree of div/svg/g/g[1], with a class equalling st0 and at index [1]:
self.SELENIUM_DRIVER.find_element_by_xpath('//div/svg/g/g[1]/path[#class=st0][1]')
If the parent tree is unique from earlier on, it may be a good idea to make the query shorter, e.g.:
self.SELENIUM_DRIVER.find_element_by_xpath('//g/g[1]/path[#class=st0][1]')
I think this is the problem with your provided Id j_id0:j_id5:j_id6:j_id36 and as you are saying class name is same for some other elements as well, So you should try using xpath by passing index as below :-
self.SELENIUM_DRIVER.find_element_by_xpath('(//path[#class=st0])[1]')
Now you can locate your desire element by just changing index value.
Hope it works...:)
In the case that I want the first use of class so I don't have to guess the find_elements_by_xpath(), what are my options for this? The goal is to write less code, assuring any changes to the source I am scraping can be fixed easily. Is it possible to essentially
find_elements_by_css_selector('source[1]')
This code does not work as is though.
I am using selenium with Python and will likely be using phantomJS as the webdriver (Firefox for testing).
In CSS Selectors, square brackets select attributes, so your sample code is trying to select the 'source' type element with an attribute named 1, eg
<source 1="your_element" />
Whereas I gather you're trying to find the first in a list that looks like this:
<source>Blah</source>
<source>Rah</source>
If you just want the first matching element, you can use the singular form:
element = find_element_by_css_selector("source")
The form you were using returns a list, so you're also able to get the n-1th element to find the nth instance on the page (Lists index from 0):
element = find_elements_by_css_selector("source")[0]
Finally, if you want your CSS selectors to be completely explicit in which element they're finding, you can use the nth-of-type selector:
element = find_element_by_css_selector("source:nth-of-type(1)")
You might find some other helpful information at this blog post from Sauce Labs to help you write flexible selectors to replace your XPath.
I'm trying to make a test for my site. Having troubles on some user form. The trick is, that the number of text fields in the form varies depending in user options (disabled ones are present in the code, but have a style <displayed: none;> tag), so I'm trying to find more flexible approach than locating every element one-by-one and filling the forms with try/except blocks.
I'm using an xpath locator
text_fields = driver.find_elements_by_xpath("//div[#class='form-line']/div[#class='form-inputs']/input[#type='text' and not(ancestor::div[#style='display: none;'])]")
The trouble is that firebug locates only needed elements, but when I use it my selenium script, printing the list of text_fields gives me all the elements, even without a <displayed: none;> tag
How can I get only visible elements?
PS Sorry for my bad English ^_^
You can get all the form elements the usual way, then iterate on the list and remove those elements that do not return true on is_displayed().
Try the contains() method:
text_fields = driver.find_elements_by_xpath(
"//div[#class='form-line']/div[#class='form-inputs']/input[#type='text' and
not(ancestor::div[contains(#style, 'display: none;')])]")
The important part is:
div[contains(#style, 'display: none;')]
Note, that if the style contains the string display:none; or display:none, the selector won't match.
I use the following and it works great.
self.assertTrue(driver.find_element_by_xpath("//div[#id='game_icons']/div/div[2]/div/a/img"))
This is for Selenium and Python of course.
I'm trying to reach a nested class, originally I used xPath but it returned an empty list, so I went through the classes individually, and one of them has an issue where selenium can't find it.
Up until Price4 it works fine, but it can't seem to find Price5
So, if you want to get the text from the last element containing the price you can define
String lastPriceXpath = "(//*[#class='css-1m1f8hn'])[last()]"
String lastPrice = driver.findElement(By.xpath(lastPriceXpath)).getText()
The syntax above is in Java but I hope you will be able to convert it to python, it's quite similar