I'm searching for a tag in class.I tried many methods but I couldn't get the value.
see source code
The data I need is inside the "data-description".
How can i get the "data-description" ?
I Tried some method but didn't work
driver.find_element_by_name("data-description")
driver.find_element_by_css_selector("data-description")
I Solved this method:
icerisi = browser.find_elements_by_class_name('integratedService ')
for mycode in icerisi:
hizmetler.append(mycode.get_attribute("data-description"))
Thanks for your help.
I think css selector would work best here. "data-description" isn't an element, it's an attribute of an element. The css selector for an element with a given attribute would be:
[attribute]
Or, to be more specific, you could use:
[attribute="attribute value"]
Here's a good tip:
Most web browsers have a way of copying an elements Selector or XPATH. For example, in Safari if you view the source code then right-click on an element it will give you the option to copy it. Then select XPATH or Selector and in your code use driver.find_element_by_xpath() or driver.find_element_by_css_selector(). I am certain Google Chrome and Firefox have similar options.
This method is not always failsafe, as the XPATH can be very specific, meaning that slight changes to the website will cause your script to crash, but it is a quick and easy solution, and is especially useful if you don't plan on reusing your code months or years later.
Related
I'm trying to learn how to click around using selenium. I have tried some different websites like reddit, google etc without success.
driver.get('https://www.dropbox.com/login')
driver.find_element_by_xpath('//a[#href="' + 'https://www.dropbox.com/forgot?email_from_login=' + '"]').click()
and
continue_link = driver.find_element_by_partial_link_text('Sign in')
They both exist but neither works. What am I doing wrong?
You're being a little specific in your xpath, so there're many more ways you might bugger something up. Using the xpath you can simply do:
driver.find_element_by_xpath("//*[#class='forgot-password-link']").click()
I make no assumptions, but just in case you aren't already, there's a very handy tool in Chrome's inspect element which lets you click an element and jumps to its node in the inspector.
Can you try the following:
from selenium.webdriver.common.by import By
driver.findElement(By.cssSelector(".login-button.signin-button.button-primary")).click()
You can use a shorter single class selector to Sign In button
driver.find_element_by_css_selector(".login-button").click()
Sign in with Google
driver.find_element_by_css_selector(".sign-in-text").click()
For forgot password
driver.find_element_by_css_selector(".forgot-password-link").click()
Single class css selectors will be the fastest method (faster than xpath and compound class)
for chrome, I install ChroPath to find elements on the page.
I want to find XPath for like elements on Instagram Page, but seems that not work :
//span[contains(#class,'glyphsSpriteHeart__outline__24__grey_9 u-__7')]
also, I try it :
/html[1]/body[1]/div[3]/div[1]/div[2]/div[1]/article[1]/div[2]/section[1]/span[1]/button[1]/span[1]
when selenium click :
elenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"div._2dDPU.vCf6V div.zZYga div.PdwC2._6oveC article.M9sTE.L_LMM.JyscU div.eo2As section.ltpMr.Slqrh span.fr66n button.coreSpriteHeartOpen.oF4XW.dCJp8 > span.glyphsSpriteHeart__outline__24__grey_9.u-__7"}
how can I find XPath? any good extension or something?
how can I find XPath? any good extension or something?
You cannot "find" the Xpath of an element. There are many, many XPath's that will find any element. Some will be stable, others will be unstable. The decision on which Xpath to use is based upon your understanding and experience of Selenium, and you understanding of how the Application Under Test is written and behaves.
If you are looking for a tool to experiment with different XPaths, then Chrome's built-in Developer Tools Console allows you to test both Xpath & CSS Selectors;
In your specific scenario about finding elements by class name, then CSS Selector is a much better choice than XPath as CSS selectors will treat multiple classes as an array where as XPath sees "class" as a literal string, hence why you needed to use "contains".
This might help:
https://selectorgadget.com/
This as well, to understand what you are manipulating:
https://www.w3schools.com/xml/xpath_syntax.asp
As for your example where you go down the tree using index numbers (ie: /html[1]/body[1]), A slight change in the site will make your script to fail. Find a way to build something more robust! Also have a look at CSS selectors if you object's appearance is known in advance.
To get all like buttons on instagram use css selector below:
span[aria-label="Like"]
You can get some helpful details here: https://www.w3schools.com/cssref/css_selectors.asp
My code is:
driver.get("http://www.thegoodguys.com.au/buyonline/SearchDisplay?pageSize=16&beginIndex=0&searchSource=Q&sType=SimpleSearch&resultCatEntryType=2&showResultsPage=true&pageView=image&searchTerm=laptops")
link=();
linkPrice=();
price=();
productName=[];
Site='Harvey Norman'
link=driver.find_elements_by_class_name("photo")
linkPrice=driver.find_elements_by_class_name("product-title")
price=driver.find_elements_by_xpath("//div[#class='purchase']/span/span")
I am not sure whether the supplied xpath and class_name are correct. Could some one verify them and please let me know how to find them
In firefox you can simply use the developer tools or firebug to check the html for classes and element ids. Following the link in your question I can find a class called photo but for linkPrice and price you should use other classes.
Try:
price=driver.find_elements_by_class_name("price")
linkPrice=driver.find_elements_by_class_name("addtocart")
Which gives me:
price[0].text
u'$496'
linkPrice[0].text
u'ADD TO CART'
You can verify Xpath using developer tools console in chrome e.g $x("//foo") or $(".foo")
Firebug for Firefox will also let you verify
Also browsers will suggest Xpath for you but these are often verbose and unstable so would recommend hand crafting
I am scraping individual listing pages from justproperty.com (individual listing from the original question no longer active).
I want to get the value of the Ref
this is my xpath:
>>> sel.xpath('normalize-space(.//div[#class="info_div"]/table/tbody/tr/td[norma
lize-space(text())="Ref:"]/following-sibling::td[1]/text())').extract()[0]
This has no results in scrapy, despite working in my browser.
The following works perfectly in lxml.html (with modern Scrapy uses):
sel.xpath('.//div[#class="info_div"]//td[text()="Ref:"]/following-sibling::td[1]/text()')
Note that I'm using // to get between the div and the td, not laying out the explicit path. I'd have to take a closer look at the document to grok why, but the path given in that area was incorrect.
Don't create XPath expression by looking at Firebug or Chrome Dev Tools, they're changing the markup. Remove the /tbody axis step and you'll receive exactly what you're look for.
normalize-space(.//div[#class="info_div"]/table/tr/td[
normalize-space(text())="Ref:"
]/following-sibling::td[1]/text())
Read Why does my XPath query (scraping HTML tables) only work in Firebug, but not the application I'm developing? for more details.
Another XPath that gets the same thing: (.//td[#class='titles']/../td[2])[1]
I tried your XPath using XPath Checker and it works fine.
I run the following and it works fine:
hxs.select('//h1').extract()
However this
hxs.select('//div[#class="ClassName"]/text()').extract()
returns [].
Is my syntax wrong? I'm unsure why the div class isn't working (it's definitely there!).
Without the html is a difficult to answers this.
I recommend to use firefox + xpath addon , I use firepath to evaluate xpaths on webpages. Check for other addons that allow you to create xpath from firebug view.
Your syntax seems to be correct.