We Keep Coding

Confirm preceding and following sibling in XPATH

I've got the below statement to check that 2 conditions exist in a
an element:
if len(driver.find_elements(By.XPATH, "//span[text()='$400.00']/../following-sibling::div/a[text()='Buy']")) > 0:
elem = driver.find_element(By.XPATH, "//span[text()='$400.00']/../following-sibling::div/a[text()='Buy']")
I've tried a few variations, including "preceding sibling::span[text()='x'", but can't seem to get the syntax correct or if I'm going about it the right way.
HTML is below. the current find_elements(By.XPATH...) correctly finds the "Total" and "Buy" class, I would like to add $20.00 in the "price" class as a condition also.
<ul>
<li class="List">
<div class="List-Content row">
<div class="Price">"$20.00"</div>
<div class="Quantity">10</div>
<div class="Change">0%</div>
<div class="Total">
<span>$400.00</span>
</div>
<div class="Buy">
<a class="Button">Buy</a>
</div>
</div>
</li>
</ul>

Using built in ElementTree
import xml.etree.ElementTree as ET
html = '''<li class="List">
<div class="List-Content row">
<div class="Price">"$20.00"</div>
<div class="Quantity">10</div>
<div class="Change">0%</div>
<div class="Total"><span>$400.00</span></div>
<div class="Buy"><a class="Button">Buy</a></div>
</div>
<div class="List-Content row">
<div class="Price">"$27.00"</div>
<div class="Quantity">10</div>
<div class="Change">0%</div>
<div class="Total"><span>$400.00</span></div>
<div class="Buy"><a class="Button">Buy</a></div>
</div>
</li>'''
items = {'Total':'$400.00','Buy':'Buy','Price':'"$20.00"'}
root = ET.fromstring(html)
first_level_divs = root.findall('div')
for first_level_div in first_level_divs:
results = {}
for k,v in items.items():
div = first_level_div.find(f'.div[#class="{k}"]')
one_level_down = len(list(div)) > 0
results[k] = list(div)[0].text if one_level_down else div.text
if results == items:
print('found')
else:
print('not found')
results = {}
output
found
not found

Given this HTML snippet
<ul>
<li class="List">
<div class="List-Content row">
<div class="Price">"$20.00"</div>
<div class="Quantity">10</div>
<div class="Change">0%</div>
<div class="Total"><span>$400.00</span></div>
<div class="Buy"><a class="Button">Buy</a></div>
</div>
</li>
</ul>
I would use this XPath:
buy_buttons = driver.find_elements(By.XPATH, """//div[
contains(#class, 'List-Content')
and div[#class = 'Price'] = '$20.00'
and div[#class = 'Total'] = '$400.00'
]//a[. = 'Buy']""")
for buy_button in buy_buttons:
print(buy_button)
The for loop replaces your if len(buy_buttons) > 0 check. It won't run when there are no results, so the if is superfluous.

Beautifulsoup get element with the same class

I'm having trouble parsing HTML elements with "class" attribute using Beautifulsoup.
The html code is like this :
<div class="info-item">
<div class="item-name">Model</div>
<div class="item-content">XPANDER 1.5L GLX</div>
</div>
<div class="info-item">
<div class="item-name">Transmission</div>
<div class="item-content"> MT </div>
</div>
<div class="info-item">
<div class="item-name">Engine Capacity (cc)</div>
<div class="item-content">1499 cc</div>
</div>
<div class="info-item">
<div class="item-name">Fuel</div>
<div class="item-content">Bensin </div>
</div>
I nead to get data (XPANDER 1.5L GLX, MT, 1499, Gasoline)
I try with script detail.find(class_='item-content') just only get XPANDER 1.5L GLX
please help

Use .find_all() or .select():
from bs4 import BeautifulSoup
html_doc = """
<div class="info-item">
<div class="item-name">Model</div>
<div class="item-content">XPANDER 1.5L GLX</div>
</div>
<div class="info-item">
<div class="item-name">Transmission</div>
<div class="item-content"> MT </div>
</div>
<div class="info-item">
<div class="item-name">Engine Capacity (cc)</div>
<div class="item-content">1499 cc</div>
</div>
<div class="info-item">
<div class="item-name">Fuel</div>
<div class="item-content">Bensin </div>
</div>
"""
soup = BeautifulSoup(html_doc, "html.parser")
items = [
item.get_text(strip=True) for item in soup.find_all(class_="item-content")
]
print(*items)
Prints:
XPANDER 1.5L GLX MT 1499 cc Bensin
Or:
items = [item.get_text(strip=True) for item in soup.select(".item-content")]

You can try this
soup = BeautifulSoup(html, "html.parser")
items = [item.text for item in soup.find_all("div", {"class": "item-content"})]
find_all retreives all occurences

Look for element in string and return element + following two characters

The following html appears as a string in my code. That is okay, what I need though is how to get:
"class="company-image company-34""
for each company-## there is also a price found in this tag further below in the HTML:
class="small-12 medium-4 cell text-right" data-after="kr./år">1.813
I tried following code:
for x in html:
if "company-image company" in x:
print("Oh yes")
else:
print("Nahh")
but it doesn't really work. My thinking is I look for everytime "company-image company" is mentioned and get the whole string and the following numbers after, it is always two numbers ##. And whenever it is found I look for "data-after="kr./år"" and get the numbers following. Eventually this would end in a for loop, as there are multiple companies and prices.
<app-offer-match _ngcontent-vdv-c20="" _nghost-vdv-c22="" class="ng-star-inserted">
<div _ngcontent-vdv-c22="" class="box">
<!---->
<div _ngcontent-vdv-c22="" class="line1">
<div _ngcontent-vdv-c22="" class="company-image company-34"><img _ngcontent-vdv-c22="" src="/assets/images/companies/34.svg"></div>
<div _ngcontent-vdv-c22="" class="button compare">Sammenlign </div>
</div>
<div _ngcontent-vdv-c22="" class="line2">
<div _ngcontent-vdv-c22="" class="container-button">
<div _ngcontent-vdv-c22="" class="button mini-accordion"></div>
</div>
<div _ngcontent-vdv-c22="" class="container-insurance-list">
<!---->
<div _ngcontent-vdv-c22="" class="indbo ng-star-inserted">
<div _ngcontent-vdv-c22="" class="grid-x container-product-overview">
<div _ngcontent-vdv-c22="" class="small-5 cell detail"><span _ngcontent-vdv-c22="">Indbo</span>
<!----><span _ngcontent-vdv-c22="" class="ng-star-inserted">Kongshaven 3</span>
</div>
<div _ngcontent-vdv-c22="" class="small-6 cell">
<div _ngcontent-vdv-c22="" class="grid-x price">
<div _ngcontent-vdv-c22="" class="small-12 medium-8 cell text-right" data-after="kr.">Selvrisiko 2.199</div>
<div _ngcontent-vdv-c22="" class="small-12 medium-4 cell text-right" data-after="kr./år">1.813 </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</app-offer-match>
EDIT: Added desired output.
Desired output would be a pandas dataframe of:
Company Price
company-image company-34 1.813
EDIT 2:
It looks like an xml, that's because I formatted it like that for you guys. WHen I output it, it is of type STR, thank you.

Try this:
company = """[your string above]"""
import lxml.html as lh
import pandas as pd
doc = lh.fromstring(company)
columns = ["Company", "Price"]
rows = []
targets = doc.xpath('//div[contains(#class,"company-image company")]')
for target in targets:
row = []
row.append(target.attrib['class'])
price = target.xpath('../following-sibling::div//div[#data-after="kr./år"]')[0]
row.append(price.text)
rows.append(row)
rows
pd.DataFrame(rows,columns=columns)
Output:
Company Price
0 company-image company-34 1.813

Extracting the right elements by text and span / Beautiful Soup / Python

Im trying to scrape following data:
Cuisine: 4.5
Service: 4.0
Quality: 4.5
But im having issues to scrape the right data. I tried following two Codes:
for bewertungen in soup.find_all('div', {'class' : 'histogramCommon bubbleHistogram wrap'}):
if bewertungen.find(text='Cuisine'):
cuisine = bewertungen.find(text='Cuisine')
cuisine = cuisine.next_element
print("test " + str(cuisine))
if bewertungen.find_all(text='Service'):
for s_bewertung in bewertungen.find_all('span', {'class':'ui_bubble_rating'}):
s_speicher = s_bewertung['alt']
In the first if i get no result. In the second If i get the right elements but i get all 3 results but i can not define which ones belongs to which text (Cuisine, Service, Quality)
Can someone give me an advice how to get the right data?
I put at the bottom the html code.
<div class="histogramCommon bubbleHistogram wrap">
<div class="colTitle">\nGesamtwertung\n</div>
<ul class="barChart">
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Cuisine</span>
</div>
<div class="wrap row part ">
<span alt="4.5 of five" class="ui_bubble_rating bubble_45"></span>
</div>
</div>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Service</span>
</div>
<div class="wrap row part ">
<span alt="4.0 of five" class="ui_bubble_rating bubble_40"></span>
</div>
</div>
</li>
<li>
<div class="ratingRow wrap">
<div class="label part ">
<span class="text">Quality</span>
</div>
<div class="wrap row part "><span alt="4.5 of five" class="ui_bubble_rating bubble_45"></span></div>
</div>
</li>
</ul>
</div>

Try this. According to the snippet you have pasted above, the following code should work:
from bs4 import BeautifulSoup
soup = BeautifulSoup(content,"lxml")
for item in soup.select(".ratingRow"):
category = item.select_one(".text").text
rating = item.select_one(".row span")['alt'].split(" ")[0]
print("{} : {}".format(category,rating))
Another way would be:
for item in soup.select(".ratingRow"):
category = item.select_one(".text").text
rating = item.select_one(".text").find_parent().find_next_sibling().select_one("span")['alt'].split(" ")[0]
print("{} : {}".format(category,rating))
Output:
Cuisine : 4.5
Service : 4.0
Quality : 4.5

lxml xpath() not returning what I expect

I am following a guide (http://docs.python-guide.org/en/latest/scenarios/scrape/) to scrape a website (https://www.brookfieldproperties.com/portfolio/toronto/bay-adelaide-east/) and have gone through the lxml package website and can't figure out what's going wrong.
I have this code:
from lxml import html
import requests
page = requests.get('https://www.brookfieldproperties.com/portfolio/toronto/bay-adelaide-east/')
tree = html.fromstring(page.content)
floor = tree.xpath('//div[#class="column floor"]/text()')
sf = tree.xpath('//div[#class="column rsf"]/text()')
but floor and sf return a list of '\n\t\t\t\t' values, not an integer which you'd expect looking at the html from the actual website ("20" and "5117" in the below case):
<div class="availabilityWrap">
<h3>Availabilities</h3>
<div class="availabilityRow headerRow">
<div class="column floor">
<a href="/media/img/asset/pdf/BAC-ET-_20th_Floor_-_5100sf.pdf"
target='blank'><img src="/static/images/pdf.png" class="floorPDF" />20</a>
</div>
<div class="column rsf">
<p><b>5117</b></p>
</div>
<div class="column divisible">
<p><b>yes</b></p>
</div>
<div class="column date">
<p><b>05/01/2017</b></p>
</div>
<div class="column space">
<p><b>Office</b></p>
</div>
<div class="column description">
<p><b>model suite</b></p>
</div>
<div class="column rent">
<p><b>$26.55</b></p>
</div>
</div>
Shouldn't it just be returning all text in the "column floor" div class? Any help would be great.

floor = tree.xpath('normalize-space(//div[#class="column floor"])')
The div contains the \n\t to get new lines and space, Those are text too, you can concatenate all the text and remove the whitespace by using normalize-space() function
In [14]: '''<div class="column floor">
...:
...: <a href="/media/img/asset/pdf/BAC-ET-_20th_Floor_-_5100sf.pdf"
...: target='blank'><img src="/static/images/pdf.png" class="floorPDF" />20</a>
...:
...: </div>'''
Out[14]: '<div class="column floor">\n\n <a href="/media/img/asset/pdf/BAC-ET-_20th_Floor_-_5100sf.pdf"\n target=\'blank\'><img src="/static/images/pdf.png" class="floorPDF" />20</a>\n\n </div>'
EDIT:
for div in tree.xpath('//div[#class="column floor"]'):
print(div.xpath('normalize-space(.)')) # `.` means current node