I'm trying to parse a website with the requests module:
import requests
some_data = {'a':'',
'b':''}
with requests.Session() as s:
result = s.post('http://website.com',data=some_data)
print(result.text)
The page is responding as below:
{
"arrangetype":"U",
"list": [
{
"product_no":43,
"display_order":4,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
},
{
"product_no":44,
"display_order":6,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
}
],
"length":2
}
I found that instead of parsing full HTML, it would be better to deal with the response as all the data I want is in that response.
What I want to get is a list of the values of product_no, so the expected result is:
[43,44]
How do I do this?
Convert your JSON response to a dictionary with json.loads(), and collect your results in a list comprehension.
Demo:
from json import loads
data = """{
"arrangetype":"U",
"list": [
{
"product_no":43,
"display_order":4,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
},
{
"product_no":44,
"display_order":6,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
}
],
"length":2
}"""
json_dict = loads(data)
print([x['product_no'] for x in json_dict['list']])
# [43, 44]
Full Code:
import requests
from json import loads
some_data = {'a':'',
'b':''}
with requests.Session() as s:
result = s.post('http://website.com',data=some_data)
json_dict = loads(result.text)
print([x["product_no"] for x in json_dict["list"]])
Related
I have a json (url = http://open.data.amsterdam.nl/ivv/parkeren/locaties.json) and I want to print all 'title', 'adres', 'postcode'. How can I do that?
I want to print it like this:
title.
adres.
postcode.
title.
adres.
postcode.
so among themselves
I hope you can help me with this
import urllib, json
url = "http://open.data.amsterdam.nl/ivv/parkeren/locaties.json"
import requests
search = requests.get(url).json()
print(search['title'])
print(search['adres'])
print(search['postcode'])
Using print(json.dumps(r, indent=4)) you can see that the structure is
{
"parkeerlocaties": [
{
"parkeerlocatie": {
"title": "Fietsenstalling Tolhuisplein",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.9032801,52.3824545]}",
...
}
},
{
"parkeerlocatie": {
"title": "Fietsenstalling Paradiso",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.8833735,52.3621851]}",
...
}
},
So to access the inner properties, you need to follow the JSON path
import requests
url = ' http://open.data.amsterdam.nl/ivv/parkeren/locaties.json'
search = requests.get(url).json()
for parkeerlocatie in search["parkeerlocaties"]:
content = parkeerlocatie['parkeerlocatie']
print(content['title'])
print(content['adres'])
print(content['postcode'])
print()
I'm trying to parse the following JSON data and get URL values using python Function.From the below JSON example I would like to get the URL from under the Jobs tag and store it in 2 arrays. 1 array will store URL that has color tag and other will store URL that do not have color tag. Once the 2 arrays are ready I would like to return these two arrays. I'm very new to python and need some help with this.
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"actions":[ ],
"description":"This is a TSG level folder.",
"displayName":"CONSOLIDATED",
"displayNameOrNull":null,
"fullDisplayName":"CONSOLIDATED",
"fullName":"CONSOLIDATED",
"name":"CONSOLIDATED",
"url":"https://cyggm.com/job/CONSOLIDATED/",
"healthReport":[
{
"description":"Projects enabled for building: 187 of 549",
"iconClassName":"icon-health-20to39",
"iconUrl":"health-20to39.png",
"score":34
}
],
"jobs":[
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"name":"yyfyiff",
"url":"https://tdyt.com/job/
CONSOLIDATED/job/yfiyf/"
},
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"name":"Ops-Prod-Jobs",
"url":"https://ygduey.com/job/
CONSOLIDATED/job/Ops-Prod-Jobs/"
},
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"name":"TEST-DATA-MGMT",
"url":"https://futfu.com/job/
CONSOLIDATED/job/TEST-DATA-MGMT/"
},
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"name":"TESTING-OPS",
"url":"https://gfutfu.com/job/
CONSOLIDATED/job/TESTING-OPS/"
},
{
"_class":"com.cloudbees.hudson.plugins.folder.Folder",
"name":"Performance_Engineering Team",
"url":"https://ytdyt.com/job/
CONSOLIDATED/job/Performance_Engineering%20Team/"
},
{
"_class":"hudson.model.FreeStyleProject",
"name":"test",
"url":"https://tduta.com/job/
CONSOLIDATED/job/test/",
"color":"notbuilt"
}
],
"primaryView":{
"_class":"hudson.model.AllView",
"name":"all",
"url":"https://fuyfi.com/job/
CONSOLIDATED/"
},
"views":[
{
"_class":"hudson.model.AllView",
"name":"all",
"url":"https://utfufu.com/job/
CONSOLIDATED/"
}
]
}
The following is the python code I used to get the jobs data but then I'm not able to iterate through the jobs data to get all URL. I'm only getting 1 at a time if I change the code
req = requests.get(url, verify=False, auth=(username, password))
j = json.loads(req.text)
jobs = j['jobs']
print(jobs[1]['url'])
I'm getting 2nd URL here but no way to check if this entry has color tag
First of all, your JSON is improperly formatted. You will have to use a JSON formatter to check its validity and fix any issues.
That said, you'll have to read in the file as a string with
In [87]: with open('data.json', 'r') as f:
...: data = f.read()
...:
Then using the json library, load the data into a dict
In [88]: d = json.loads(data)
You can then use 2 list comprehensions to get the data you want
In [90]: no_color = [record['url'] for record in d['jobs'] if 'color' not in record]
In [91]: color = [record['url'] for record in d['jobs'] if 'color' in record]
In [93]: no_color
Out[93]:
['https://tdyt.com/job/CONSOLIDATED/job/yfiyf/',
'https://ygduey.com/job/CONSOLIDATED/job/Ops-Prod-Jobs/',
'https://futfu.com/job/CONSOLIDATED/job/TEST-DATA-MGMT/',
'https://gfutfu.com/job/CONSOLIDATED/job/TESTING-OPS/',
'https://ytdyt.com/job/CONSOLIDATED/job/Performance_Engineering%20Team/']
In [94]: color
Out[94]: ['https://tduta.com/job/CONSOLIDATED/job/test/']
I need an help on reading a JSON from a URL, which has the below JSON in it:
{
"totalItems":2,
"#href":"/classes/dsxplan:Program",
"#id":"dsxplan:Program",
"#mask":"dsplan:MVMask.WorkPackage.Complex",
"#type":"Collection",
"#code":200,
"#context":{
"dsxplan":"xplan",
"dsplan":"plan",
"dspol":"pol",
"image":{
"#id":"dspol:image",
"#type":"#id"
},
"dskern":"kern"
},
"member":[
{
"dsplan:actualType":{
"#href":"/resources/dsxplan:Program",
"#id":"dsxplan:Program",
"#mask":"dskern:Mask.Default",
"image":"iconProgram.png"
},
"dskern:owner":{
"#href":"/resources/dskern:Person.Creator",
"#id":"dskern:Person.Creator",
"#mask":"dskern:MVMask.Person.Complex",
"dsplan:actualType":{
"#href":"/resources/foaf:Person",
"#id":"foaf:Person",
"#mask":"dskern:Mask.Default"
}
},
"dspol:modificationDate":"2017-09-08T17:54:36.786Z",
"#href":"/resources/dsxplan:DSLCProgram.R-399",
"#id":"dsxplan:DSLCProgram.R-399",
"#mask":"dsplan:MVMask.WorkPackage.Complex",
"#etag":"7412df19-1dde-4245-b40b-5dd86dbbe3f1"
},
{
"dsplan:actualType":{
"#href":"/resources/dsxplan:Program",
"#id":"dsxplan:Program",
"#mask":"dskern:Mask.Default",
"image":"iconProgram.png"
},
"dskern:owner":{
"#href":"/resources/dskern:Person.Creator",
"#id":"dskern:Person.Creator",
"#mask":"dskern:MVMask.Person.Complex",
"dsplan:actualType":{
"#href":"/resources/foaf:Person",
"#id":"foaf:Person",
"#mask":"dskern:Mask.Default"
}
},
"dspol:modificationDate":"2017-09-08T17:54:36.786Z",
"#href":"/resources/dsxplan:xComModel2017program.R-394",
"#id":"dsxplan:xComModel2017program.R-394",
"#mask":"dsplan:MVMask.WorkPackage.Complex",
"#etag":"7412df19-1dde-4245-b40b-5dd86dbbe3f1"
}
]
}
I just need to read this json from a link provided. I tried the below code:
import urllib.request
request= urllib.request.Request("https://dummy_link")
response = urllib.request.urlopen(request)
input = (response.read().decode('utf-8'))
json.loads(input)
This code throws me this error:
"JSONDecodeError: Expecting value: line 9 column 1 (char 12)"
Could you please help me get this right? I really appreciate the help.!!
You could use Requests library which is more simple than urllib:
For instance:
import requests
r = requests.get('https://dummy_link')
obj = r.json()
EDIT
If you want to use urllib, you can do as below:
import urllib.request
import json
with urllib.request.urlopen("https://dummy_link") as f:
content = f.read()
obj = json.loads(content)
There is no need to convert the binary content to unicode string.
There is an urllib howto in the official documentation.
I'm new in this Python world, I'm trying to use an API to make basic currency calculations. I can get the output like:
{'USD': 1.13}
this but I want it just to be
1.13
The code:
import requests
inputCurrency = 'EUR'
outputCurrency = 'USD'
p = {"inpc":inputCurrency, "outc":outputCurrency}
url = 'https://somewebsite/api/data'
r = requests.get(url, params=p)
print(r.json())
The server returned a JSON object. The .json() method of your r response decodes it, and returns the decoded object, which is a Python dict.
You want the value corresponding to the 'USD' key.
Just do:
import requests
inputCurrency = 'EUR'
outputCurrency = 'USD'
p = {"inpc":inputCurrency, "outc":outputCurrency}
url = 'https://somewebsite/api/data'
response = requests.get(url, params=p)
json_data =response.json()
print(json_data['USD'])
If the structure of the data is more complicated, as in your comment:
json_data = { "status": 1, "data": [ { "time": "2015-08-30T07:56:28.000Z", "usd": 1.17 }, { "time": "2015-08-30T08:56:28.000Z", "usd": 1.27 }, { "time": "2015-08-30T09:56:28.000Z", "usd": 1.28 }]}
you could extract the relevant part:
data = json_data['data']
which is a list of dictionaries. You can then print the first one:
print(data[0]['usd'])
# 1.27
or print them all:
for day_value in data:
print(day_value['usd'])
I am returning a JSON object from a requests call. I would like to get all the values from it and store them in a flat array.
My JSON object:
[
{
"link": "https://f.com/1"
},
{
"link": "https://f.com/2"
},
{
"link": "https://f.com/3"
}
]
I would like to store this as:
[https://f.com/things/1, https://f.com/things/2, https://f.com/things/3]
My code is as follows.. it is just printing each link out:
import requests
import json
def start_urls_data():
url = 'http://106309.n.com:3000/api/v1/product_urls?q%5Bcompany_in%5D%5B%5D=F'
headers = {'X-Api-Key': '1', 'Content-Type': 'application/json'}
r = requests.get(url, headers=headers)
start_urls_data = json.loads(r.content)
for i in start_urls_data:
print i['link']
You can use a simple list comprehension:
data = [
{
"link": "https://f.com/1"
},
{
"link": "https://f.com/2"
},
{
"link": "https://f.com/3"
}
]
print([x["link"] for x in data])
This code just loops through the list data and put the value of the key link from the dict element to a new list.