We Keep Coding

Not able to extract data using scrapy with class names containing spaces and hyphens

I am new to scrapy and I have to extract text from a tag with multiple class names, where the class names contain spaces and hyphens.
Example:
<div class="info">
<span class="price sale">text1</span>
<span class="title ng-binding">some text</span>
</div>
When i use the code:
response.xpath("//span[contains(#class,'price sale')]/text()").extract()
I am able to get text1 but when I use:
response.xpath("//span[contains(#class,'title ng-binding')]/text()").extract()
I get an empty list. Why is this happening and how to handle this?

The expression you're looking for is:
//span[contains(#class, 'title') and contains(#class, 'ng-binding')]
I highly suggest XPath visualizer, which can help you debug xpath expressions easily. It can be found here:
http://xpathvisualizer.codeplex.com/
Or with CSS try
response.css("span.title.ng-binding")
Or there is a chance that element with ng-binding is loaded via Javascript/Ajax hence not included in initial server response.

You can replace the spaces with "." in your code when using response.css().
In your case you can try:
response.css("span.title.ng-binding::text").extract()
This code should return the text you are looking for.

Python Selenium: How do I click a link in a drop down menu with css selector?

The HTML looks like this:
<span class="MenuIcons searchButton"></span>
... (some stuff)
<a data-bind="" url="/ParagonLS/Search/Property.mvc/Index/1" tabdescription="RESIDENTIAL" subtabdescription="Criteria" subtabmaxallowed="3" targetex="" rel="" class=" SearchByClass1 " subtabgroup="true" subtabgroupadd="true" subtabstartindex="0" fullwindow="False" hideaddressbar="False">TEXT</a>
I can get to the span using:
driver.find_element_by_css_selector(".MenuIcons.searchButton")
But since the span is a drop down menu I need to get to the inner element, but don't know how since it has spaces around its class name. What do I do?

import time
driver.find_element_by_css_selector(".MenuIcons.searchButton").click()
time.sleep(1)
driver.find_element_by_partial_link_text("TEXT").click()
You can do this and click the link.

I suggest you to use xpath instead since the class contains space.
//a[contains(#class,'SearchByClass1')]
Text based search is also another possibility.
//a[.='TEXT']
Edit
Executing javascript since the element is hidden as per OP's comment
test = driver.execute_script("return document.querySelector(\"a[class*='SearchByClass1']\").innerHTML;");
print(test)
print
TEXT

Python Selenium find element by link text contains a string with wildcard or regex

I have a HTML snippet like this:
<span class="line S_line1">
评论
<em>1</em>
</span>
The thing is that number in <em>1</em> is not predictable or sometime just omit, I want to find this element by
driver.find_element_by_link_text(u'评论*')
But it didn't work, is there a way to do that with a wildcard or regex?

driver.find_element_by_partial_link_text(u'评论')
You can using partial_link_text.This way you can find a link with changing content using some part which is always constant.

scrapy xpath selector without same DIV ID

i want to select all divs that a first part of their ID is "edit" using scrapy/XPATH.
For example:
<div id="edit3423432">...</div>
<div id="edit0036594">...</div>
For divs which have same id i use this code:
hxs.select('.//div[contains(#id,"testid")]')
But now how can i select all divs that have the first four characters equal to "edit"?

Xpath has a special function called starts-with, that would be pretty ideal here. Here's an example of how to use it:
hxs.select('.//div[starts-with(#id, 'edit')]')
Hope that helps, let me know if you have any questions.

Select second element that is after first one

I have 2 divs in code under 1 li. I need to select second one:
<li>
<div id='1'>Stable Text</div>
<div>Unstable Text</div>
</li>
I can find only first one using text's name, as its stable. But I need second one.
using xpath with //li/div[2] will not work because place of this data is not stable.

You can use following-sibling.
//div[text() = 'Stable Text']/following-sibling::div
You can have a look here for more information.

you can try using a cssSelector as well,
li > div#1 + div
Here '+' is used to locate following sibling..you can refer this for more info on selectors.