I have pretty printed a content in this way using this code. This code prints everything out, how do I print a specific location using IF ? Such as Upper Bukit Timah, West Coast...
Area: Upper Bukit Timah
Summary: Cloudy
Latitude: 1.356084
Longitude: 103.768873
Area: West Coast
Summary: Cloudy
Latitude: 1.30039493
Longitude: 103.7504196
Area: Woodlands
Summary: Cloudy
Latitude: 1.44043052
Longitude: 103.7878418
Area: Yishun
Summary: Cloudy
Latitude: 1.42738834
Longitude: 103.8290405
import urllib2
from BeautifulSoup import BeautifulStoneSoup #Using bs3
url="https://api.projectnimbus.org/neaodataservice.svc/NowcastSet"
request = urllib2.Request(url)
request.add_header("accept", "*/*")
request.add_header('AccountKey', "OSJeROQjTg4v7Ec3kiecjw==")
request.add_header('UniqueUserID', "00000000000000000000000000000001")
result = urllib2.urlopen(request)
xml_str = result.read()
soup = BeautifulStoneSoup(xml_str)
prop_list = []
for content in soup.findAll("m:properties"):
props = {}
for prop in content.findChildren():
props[prop.name[2:]] = prop.text
prop_list.append(props)
for prop in sorted(prop_list):
print "Area: %(area)s\nSummary: %(summary)s\nLatitude: %(latitude)s\nLongitude: %(longitude)s\n" % prop
Well, you'd have to add an if statement to your final for loop, checking whether the current entry in in some positive list. Something like this:
areas_to_print = ["Upper Bukit Timah", "West Coast", "Woodlands", "Yishun"]
for prop in sorted(prop_list):
if prop["area"] in areas_to_print:
print "Area: %(area)s\nSummary: %(summary)s\nLatitude: %(latitude)s\nLongitude: %(longitude)s\n" % prop
Alternatively, you could just as well add that same if statement to your first for loop, so only those entries are added to the prop_list in the first place.
for prop in sorted(prop_list):
if (prop["area"] == "whatever u wnt or someother condition"):
print "Area: %(area)s\nSummary: %(summary)s\nLatitude: %(latitude)s\nLongitude: %(longitude)s\n" % prop
Related
When I execute this code...
from bs4 import BeautifulSoup
with open("games.html", "r") as page:
doc = BeautifulSoup(page, "html.parser")
titles = doc.select("a.title")
prices = doc.select("span.price-inner")
for game_soup in doc.find_all("div", {"class": "game-options-wrapper"}):
game_ids = (game_soup.button.get("data-game-id"))
for title, price_official, price_lowest in zip(titles, prices[::2], prices[1::2]):
print(title.text + ',' + str(price_official.text.replace('$', '').replace('~', '')) + ',' + str(
price_lowest.text.replace('$', '').replace('~', '')))
The output is...
153356
80011
130187
119003
73502
156474
96592
154207
155123
152790
165013
110837
Call of Duty: Modern Warfare II (2022),69.99,77.05
Red Dead Redemption 2,14.85,13.79
God of War,28.12,22.03
ELDEN RING,50.36,48.10
Cyberpunk 2077,29.99,28.63
EA SPORTS FIFA 23,41.99,39.04
Warhammer 40,000: Darktide,39.99,45.86
Marvels Spider-Man Remastered,30.71,27.07
Persona 5 Royal,37.79,43.32
The Callisto Protocol,59.99,69.41
Need for Speed Unbound,69.99,42.29
Days Gone,15.00,9.01
But I'm trying to get the value next to the other ones on the same line
Expected output:
Call of Duty: Modern Warfare II (2022),69.99,77.05,153356
Red Dead Redemption 2,14.85,13.79,80011
...
Even when adding game_ids to print(), it spams the same game id for each line.
How can I go about resolving this issue?
HTML file: https://jsfiddle.net/m3hqy54x/
I feel like all 3 details (title, price_official, price_lowest) are probably all in a shared container. It would be better to loop through these containers and select the details as sets from each container to make sure the wight prices and titles are being paired up, but I can't tell you how to do that without seeing at least a snippet from (or all of) "games.html"....
Anyway, assuming that '110837\nCall of Duty: Modern Warfare II (2022)' is from the first title here, you can rewrite your last loop as something like:
for z in zip(titles, prices[::2], prices[1::2]):
z, lw = list(z), ''
for i in len(z):
if i == 0: # title
z[0] = ' '.join(w for w in z[0].text.split('\n', 1)[-1] if w)
if '\n' in z[0].text: lw = z[0].text.split('\n', 1)[0]
continue
z[i] = z[i].text.replace('$', '').replace('~', '')
print(','.join(z+[lw]))
Added EDIT: After seeing the html, this is my suggested solution:
for g in doc.select('div[data-container-game-id]'):
gameId = g.get('data-container-game-id')
title = g.select_one('a.title')
if title: title = ' '.join(w for w in title.text.split() if w)
price_official = g.select_one('.price-wrap > div:first-child span.price')
price_lowest = g.select_one('.price-wrap > div:first-child+div span.price')
if price_official:
price_official = price_official.text.replace('$', '').replace('~', '')
if price_lowest:
price_lowest = price_lowest.text.replace('$', '').replace('~', '')
print(', '.join([title, price_official, price_lowest, gameId]))
prints
Call of Duty: Modern Warfare II (2022), 69.99, 77.05, 153356
Red Dead Redemption 2, 14.85, 13.79, 80011
God of War, 28.12, 22.03, 130187
ELDEN RING, 50.36, 48.10, 119003
Cyberpunk 2077, 29.99, 28.63, 73502
EA SPORTS FIFA 23, 41.99, 39.04, 156474
Warhammer 40,000: Darktide, 39.99, 45.86, 96592
Marvel's Spider-Man Remastered, 30.71, 27.07, 154207
Persona 5 Royal, 37.79, 43.32, 155123
The Callisto Protocol, 59.99, 69.41, 152790
Need for Speed Unbound, 69.99, 42.29, 165013
Days Gone, 15.00, 9.01, 110837
Btw, this might look ok for just four values, but if you have a large amount of details that you want to extract, you might want to consider using a function like this.
for imo in imos:
...
...
keys_div= soup.find_all("div", {"class","col-4 keytext"})
values_div = soup.find_all("div",{"class","col-8 valuetext"})
for key, value in zip(keys_div, values_div):
print(key.text + ": " + value.text)
'......
Output:
Ship Name: MAERSK ADRIATIC
Shiptype: Chemical/Products Tanker
IMO/LR No.: 9636632
Gross: 23,297
Call Sign: 9V3388
Deadweight: 37,538
MMSI No.: 566429000
Year of Build: 2012
Flag: Singapore
Status: In Service/Commission
Operator: Handytankers K/S
Shipbuilder: Hyundai Mipo Dockyard Co Ltd
ShipType: Chemical/Products Tanker
Built: 2012
GT: 23,297
Deadweight: 37,538
Length Overall: 184.000
Length (BP): 176.000
Length (Reg): 177.460
Bulbous Bow: Yes
Breadth Extreme: 27.430
Breadth Moulded: 27.400
Draught: 11.500
Depth: 17.200
Keel To Mast Height: 46.900
Displacement: 46565
T/CM: 45.0
This is the output for one imo, i want to store this output in dataframe and write to csv, the csv will have the keytext as header and value text as rows for all the IMO's please help me on how to do it
All you have to do is add the results to a list and then output that list to a dataframe.
import pandas as pd
filepath = r"C\users\test\test_file.csv"
output_data = []
for imo in imos:
keys_div = [i.text for i in soup.find_all("div", {"class","col-4 keytext"})]
values_div = [i.text for i in soup.find_all("div",{"class","col-8 valuetext"})]
dict1 = dict(zip(keys_div, values_div))
output_data.append(dict1)
df = pd.DataFrame(output_data)
df.to_csv(filepath, index=False)
I've been working on a function which will update two dictionaries (similar authors, and awards they've won) from an open text file. The text file looks something like this:
Brabudy, Ray
Hugo Award
Nebula Award
Saturn Award
Ellison, Harlan
Heinlein, Robert
Asimov, Isaac
Clarke, Arthur
Ellison, Harlan
Nebula Award
Hugo Award
Locus Award
Stephenson, Neil
Vonnegut, Kurt
Morgan, Richard
Adams, Douglas
And so on. The first name is an authors name (last name first, first name last), followed by awards they may have won, and then authors who are similar to them. This is what I've got so far:
def load_author_dicts(text_file, similar_authors, awards_authors):
name_of_author = True
awards = False
similar = False
for line in text_file:
if name_of_author:
author = line.split(', ')
nameA = author[1].strip() + ' ' + author[0].strip()
name_of_author = False
awards = True
continue
if awards:
if ',' in line:
awards = False
similar = True
else:
if nameA in awards_authors:
listawards = awards_authors[nameA]
listawards.append(line.strip())
else:
listawards = []
listawards.append(line.strip()
awards_authors[nameA] = listawards
if similar:
if line == '\n':
similar = False
name_of_author = True
else:
sim_author = line.split(', ')
nameS = sim_author[1].strip() + ' ' + sim_author[0].strip()
if nameA in similar_authors:
similar_list = similar_authors[nameA]
similar_list.append(nameS)
else:
similar_list = []
similar_list.append(nameS)
similar_authors[nameA] = similar_list
continue
This works great! However, if the text file contains an entry with just a name (i.e. no awards, and no similar authors), it screws the whole thing up, generating an IndexError: list index out of range at this part Zname = sim_author[1].strip()+" "+sim_author[0].strip() )
How can I fix this? Maybe with a 'try, except function' in that area?
Also, I wouldn't mind getting rid of those continue functions, I wasn't sure how else to keep it going. I'm still pretty new to this, so any help would be much appreciated! I keep trying stuff and it changes another section I didn't want changed, so I figured I'd ask the experts.
How about doing it this way, just to get the data in, then manipulate the dictionary any ways you want.
test.txt contains your data
Brabudy, Ray
Hugo Award
Nebula Award
Saturn Award
Ellison, Harlan
Heinlein, Robert
Asimov, Isaac
Clarke, Arthur
Ellison, Harlan
Nebula Award
Hugo Award
Locus Award
Stephenson, Neil
Vonnegut, Kurt
Morgan, Richard
Adams, Douglas
And my code to parse it.
award_parse.py
data = {}
name = ""
awards = []
f = open("test.txt")
for l in f:
# make sure the line is not blank don't process blank lines
if not l.strip() == "":
# if this is a name and we're not already working on an author then set the author
# otherwise treat this as a new author and set the existing author to a key in the dictionary
if "," in l and len(name) == 0:
name = l.strip()
elif "," in l and len(name) > 0:
# check to see if recipient is already in list, add to end of existing list if he/she already
# exists.
if not name.strip() in data:
data[name] = awards
else:
data[name].extend(awards)
name = l.strip()
awards = []
# process any lines that are not blank, and do not have a ,
else:
awards.append(l.strip())
f.close()
for k, v in data.items():
print("%s got the following awards: %s" % (k,v))
As mentioned in the previous questions, I am using Beautiful soup with python to retrieve weather data from a website.
Here's how the website looks like:
<channel>
<title>2 Hour Forecast</title>
<source>Meteorological Services Singapore</source>
<description>2 Hour Forecast</description>
<item>
<title>Nowcast Table</title>
<category>Singapore Weather Conditions</category>
<forecastIssue date="18-07-2016" time="03:30 PM"/>
<validTime>3.30 pm to 5.30 pm</validTime>
<weatherForecast>
<area forecast="TL" lat="1.37500000" lon="103.83900000" name="Ang Mo Kio"/>
<area forecast="SH" lat="1.32100000" lon="103.92400000" name="Bedok"/>
<area forecast="TL" lat="1.35077200" lon="103.83900000" name="Bishan"/>
<area forecast="CL" lat="1.30400000" lon="103.70100000" name="Boon Lay"/>
<area forecast="CL" lat="1.35300000" lon="103.75400000" name="Bukit Batok"/>
<area forecast="CL" lat="1.27700000" lon="103.81900000" name="Bukit Merah"/>`
<channel>
I managed to retrieve the information I need using these codes :
import requests
from bs4 import BeautifulSoup
import urllib3
#getting the ValidTime
r = requests.get('http://www.nea.gov.sg/api/WebAPI/?
dataset=2hr_nowcast&keyref=781CF461BB6606AD907750DFD1D07667C6E7C5141804F45D')
soup = BeautifulSoup(r.content, "xml")
time = soup.find('validTime').string
print "validTime: " + time
#getting the date
for currentdate in soup.find_all('item'):
element = currentdate.find('forecastIssue')
print "date: " + element['date']
#getting the time
for currentdate in soup.find_all('item'):
element = currentdate.find('forecastIssue')
print "time: " + element['time']
for area in soup.find('weatherForecast').find_all('area'):
area_attrs_li = [area.attrs for area in soup.find('weatherForecast').find_all('area')]
print area_attrs_li
Here are my results :
{'lat': u'1.34039000', 'lon': u'103.70500000', 'name': u'Jurong West',
'forecast': u'LR'}, {'lat': u'1.31200000', 'lon': u'103.86200000', 'name':
u'Kallang', 'forecast': u'LR'},
How do I remove u' from the result? I tried using the method I found while googling but it doesn't seem to work
I'm not strong in Python and have been stuck at this for quite a while.
EDIT : I tried doing this :
f = open("C:\\scripts\\nea.csv" , 'wt')
try:
for area in area_attrs_li:
writer = csv.writer(f)
writer.writerow( (time, element['date'], element['time'], area_attrs_li))
finally:
f.close()
print open("C:/scripts/nea.csv", 'rt').read()
It worked however, I would like to split the area apart as the records are duplicates in the CSV :
Thank you.
EDIT 1 -Topic:
You're missing escape characters:
C:\scripts>python neaweather.py
File "neaweather.py", line 30
writer.writerow( ('time', 'element['date']', 'element['time']', 'area_attrs_li') )
writer.writerow( ('time', 'element[\'date\']', 'element[\'time\']', 'area_attrs_li')
^
SyntaxError: invalid syntax
EDIT 2:
if you want to insert values:
writer.writerow( (time, element['date'], element['time'], area_attrs_li) )
EDIT 3:
to split the result to different lines:
for area in area_attrs_li:
writer.writerow( (time, element['date'], element['time'], area)
EDIT 4:
The splitting is not correct at all, but it shall give a better understanding of how to parse and split data to change it for your needs.
to split the area element again as you show in your image, you can parse it
for area in area_attrs_li:
# cut off the characters you don't need
area = area.replace('[','')
area = area.replace(']','')
area = area.replace('{','')
area = area.replace('}','')
# remove other characters
area = area.replace("u'","\"").replace("'","\"")
# split the string into a list
areaList = area.split(",")
# create your own csv-seperator
ownRowElement = ';'.join(areaList)
writer.writerow( (time, element['date'], element['time'], ownRowElement)
Offtopic:
This works for me:
import csv
import json
x="""[
{'lat': u'1.34039000', 'lon': u'103.70500000', 'name': u'Jurong West','forecast': u'LR'}
]"""
jsontxt = json.loads(x.replace("u'","\"").replace("'","\""))
f = csv.writer(open("test.csv", "w+"))
# Write CSV Header, If you dont need that, remove this line
f.writerow(['lat', 'lon', 'name', 'forecast'])
for jsontext in jsontxt:
f.writerow([jsontext["lat"],
jsontext["lon"],
jsontext["name"],
jsontext["forecast"],
])
I've isolated a line of HTML procured from BeautifulSoup that i want to run regex on, but I keep getting AttributeError: 'NoneType' object has no attribute 'groups'
I read another stackoverflow question (using regex on beautiful soup tags) but I can't see what I need to do to fix my version of this issue.
this is my relevant part of the code (url is provided):
with rob's correct regex update still throwing dat attribute error:
soup = BeautifulSoup(urlopen(url).read()).find("div",{"id":"page"})
addy = soup.find("p","addy").em.encode_contents()
extracted_entities = re.match(r'\$([\d.]+)\. ([^,]+), ([\d-]+)', addy)
extracted_entities.groups()
price = extracted_entities[0]
location = extracted_entities[1]
phone = extracted_entities[2]
addy seems to be what I want, returning:
$10. 2109 W. Chicago Ave., 773-772-0406, theoldoaktap.com
$9. 800 W. Randolph St., 312-929-4580, aucheval.tumblr.com
$9.50. 445 N. Clark St., 312-334-3688, rickbayless.com
and so on, when i print it.
what's going on here? thanks in advance, all.
The issue seems to be a stray " in your RegEx pattern that I don't see in your example output.
match = re.match(r'\$([\d.]+)\. ([^,]+), ([\d-]+)', addy)
if match:
extracted_entities = match.groups()
else:
raise Exception("RegEx didn't match '%s'" % addy)
Should work:
>>> f = """$10. 2109 W. Chicago Ave., 773-772-0406, theoldoaktap.com
... $9. 800 W. Randolph St., 312-929-4580, aucheval.tumblr.com
... $9.50. 445 N. Clark St., 312-334-3688, rickbayless.com"""
>>> l = f.splitlines()
>>> for i in l:
... r = re.match(r'\$([\d.]+)\. ([^,]+), ([\d-]+)', i)
... if r:
... print "GOT IT", r.groups()
... else:
... print "NO GOT IT", i
...
GOT IT ('10', '2109 W. Chicago Ave.', '773-772-0406')
GOT IT ('9', '800 W. Randolph St.', '312-929-4580')
GOT IT ('9.50', '445 N. Clark St.', '312-334-3688')