How can I output the value of 'Number#en' from my response? I am struggling to understand the nested structure. Thanks
Response from my api
{
'count': 1, 'total': 1,
'data': [
{'id': '6a3d7026-43f3-67zt-9211-99dfc6fee82e',
'name': 'test',
'properties': {'Description#en': 'test', 'Number#en': '20934120'}}],
what I have trying to print the value
response = requests.get(url, headers=headers, data=payload)
data_text = json.loads(response.text)
print(data_text[data]['properties.Number#en'])
data_text['data'] is a list of dictionary so to access the Number#en, you should use
data_text['data'][0]['properties']['Number#en']
Related
So, I had this code and it worked perfectly fine:
def upload_to_imgur(url):
j1 = requests.post(
"https://api.imgur.com/3/upload.json",
headers=headers,
data={
'key': api_key,
'image': b64encode(requests.get(url).content),
'type': 'base64',
'name': '1.jpg',
'title': 'Picture no. 1'
}
)
data = json.loads(j1.text)['data']
return data['link']
Now, I wanted to make all this async, but that doesn't really seem to work. Here's my current code:
async def async_upload_to_imgur(url):
image = await get_as_base64(url)
j1 = await aiohttp.ClientSession().post(
"https://api.imgur.com/3/upload.json",
headers=headers,
data={
'key': api_key,
'image': image,
'type': 'base64',
'name': '1.jpg',
'title': 'Picture no. 1'
}
)
data = await j1.json()
return data['link']
The get_as_base64-function is not the issue.
Header and Api Key are the same
This is the output when I print "data" in the second example.
{'data': {'error': {'code': 1001, 'message': 'File was not uploaded correctly', 'type': 'Exception_Logged', 'exception': {}}, 'request': '/3/upload.json', 'method': 'POST'}, 'success': False, 'status': 500}
It has to be the get_as_base64, for 2 reasons -:
1.) It says the file was not uploaded correctly meaning it doesn't support the URL or something like that
2.) I think you need to link a file not a link containing the file in it.
So I'm trying to scrape a table from this API:
https://api.pbpstats.com/get-wowy-combination-stats/nbaTeamId=1610612743&Season=201819&SeasonType=Playoffs&PlayerIds=203999,1627750,200794
But I'm having trouble getting the headers as a nice list like ['Players On', 'Players Off', 'Minutes', 'NetRtg', 'OffRtg', 'DefRtg'] for my eventual dataframe because the headers are their own class and not part of the other class results.
My current code looks like:
import requests
url = 'https://api.pbpstats.com/get-wowy-combination-stats/nba?TeamId=1610612743&Season=2018-19&SeasonType=Playoffs&PlayerIds=203999,1627750,200794'
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
# grab table
table = response.json()['results'][0]
#grab headers
headers = response.json()['headers']
And when I print(headers) I get [{'field': 'On', 'label': 'Players On'}, {'field': 'Off', 'label': 'Players Off'}, {'field': 'Minutes', 'label': 'Minutes', 'type': 'number'}, {'field': 'NetRtg', 'label': 'NetRtg', 'type': 'decimal'}, {'field': 'OffRtg', 'label': 'OffRtg', 'type': 'decimal'}, {'field': 'DefRtg', 'label': 'DefRtg', 'type': 'decimal'}].
Is a good way to get these into a list like ['Players On', 'Players Off', 'Minutes', 'NetRtg', 'OffRtg', 'DefRtg'] so I can then create a dataframe?
Thank you!
Just extract out all the values with a specific key out of the headers list
and make your dictionary
import requests
url = 'https://api.pbpstats.com/get-wowy-combination-stats/nba?TeamId=1610612743&Season=2018-19&SeasonType=Playoffs&PlayerIds=203999,1627750,200794'
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
#grab table
table = response.json()['results'][0]
#grab headers
headers = response.json()['headers']
#Extracting all values with every key into a dictionary
results = {}
for header in headers:
for k,v in header.items():
results.setdefault(k,[])
results[k].append(v)
#Remove duplicate elements from the list of values
results = {k:list(set(v)) for k,v in results.items()}
print(results)
The output will look like
{
'field': ['Minutes', 'Off', 'On', 'DefRtg', 'NetRtg', 'OffRtg'],
'label': ['Minutes', 'DefRtg', 'Players On', 'NetRtg', 'OffRtg', 'Players Off'],
'type': ['decimal', 'number']
}
list comprehension to iterate through should do the trick:
import requests
url = 'https://api.pbpstats.com/get-wowy-combination-stats/nba?TeamId=1610612743&Season=2018-19&SeasonType=Playoffs&PlayerIds=203999,1627750,200794'
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
# grab table
table = response.json()['results'][0]
#grab headers
headers = response.json()['headers']
headers = [each['label'] for each in headers ]
I have an API response for listing out information of all Volumes. I want to loop through the response and get the value of the name and assign each one of them dynamically to each url.
This is my main API endpoint which returns the following:
[{'source': None, 'serial': '23432', 'created': '2018-11-
12T04:27:14Z', 'name': 'v001', 'size':
456456}, {'source': None, 'serial': '4364576',
'created': '2018-11-12T04:27:16Z', 'name': 'v002',
'size': 345435}, {'source': None, 'serial':
'6445645', 'created': '2018-11-12T04:27:17Z', 'name': 'v003', 'size':
23432}, {'source': None,
'serial': 'we43235', 'created': '2018-11-12T04:27:20Z',
'name': 'v004', 'size': 35435}]
I'm doing this to get the value of 'name'
test_url = 'https://0.0.0.0/api/1.1/volume'
test_data = json.loads(r.get(test_url, headers=headers,
verify=False).content.decode('UTF-8'))
new_data = [{
'name': value['name']
} for value in test_data]
final_data = [val['name'] for val in new_data]
for k in final_data:
print(k)
k prints out all the values in name, but i'm stuck at where i want to be able to use it in assigning different API endpoints. Now, k returns
v001
v002
v003
v004
I want to assign each one of them to different endpoints like below:
url_v001 = test_url + v001
url_v002 = test_url + v002
url_v003 = test_url + v003
url_v004 = test_url + v004
I want this to be dynamically done, because there may be more than 4 volume names returned by my main API.
It wouldn't be good to do that, but the best way is to use a dictionary:
d={}
for k in final_test:
d['url_'+k] = test_url + k
Or much better in a dictionary comprehension:
d={'url_'+k:test_url + k for k in final_test}
And now:
print(d)
Both reproduce:
{'url_v001': 'https://0.0.0.0/api/1.1/volumev001', 'url_v002': 'https://0.0.0.0/api/1.1/volumev002', 'url_v003': 'https://0.0.0.0/api/1.1/volumev003', 'url_v004': 'https://0.0.0.0/api/1.1/volumev004'}
To use d:
for k,v in d.items():
print(k+',',v)
Outputs:
url_v001, https://0.0.0.0/api/1.1/volumev001
url_v002, https://0.0.0.0/api/1.1/volumev002
url_v003, https://0.0.0.0/api/1.1/volumev003
url_v004, https://0.0.0.0/api/1.1/volumev004
I'm doing some scraping and looking at pages like this one (https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897), but I have not been able to fully retrieve the JSON content.I have tried using both of the following sets of code, but each returns an incomplete JSON object:
url = 'https://www.endomondo.com/rest/v1/users/%s/workouts/%s'%(string_use_user, string_use_workout)
print(url)
response = urlopen(url)
try:
reader = codecs.getreader("utf-8")
print(reader(response))
jsonresponse = json.load(reader(response))
print(jsonresponse)
and similarly using the response library instead of urllib also fails to retrieve the full JSON
url = 'https://www.endomondo.com/rest/v1/users/%s/workouts/%s'%(string_use_user, string_use_workout)
print("using this url %s"%url)
r = requests.get(url)
try:
print(r.json())
jsonresponse = r.json()# json.loads(response.read())
In both cases I get about 1/4 of the JSON. For example, in this case:
https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897
I received:
{'feed_id': 281475471235835, 'id': 526622897, 'duration': 4082.0, 'local_start_time': '2015-05-21T09:30:45.000+02:00', 'calories': 1073.0, 'tagged_users': [], 'altitude_max': 69.9523, 'sport': 0, 'distance': 11.115419387817383, 'altitud\
e_min': 14.9908, 'include_in_stats': True, 'hydration': 0.545339, 'start_time': '2015-05-21T07:30:45.000Z', 'ascent': 137.162, 'is_live': False, 'pb_count': 2, 'playlist': [], 'is_peptalk_allowed': False, 'weather': {'wind_speed': 11, '\
temperature': 12, 'wind_direction': 13, 'type': 3, 'humidity': 81}, 'speed_max': 24.8596, 'author': {'name': 'gfdgfd', 'id': 20261627, 'last_name': 'gdsgsk', 'gender': 0, 'expand': 'abs', 'picture': {'url': 'https://www.endom\
ondo.com/resources/gfx/picture/18511427/thumbnail.jpg'}, 'first_name': 'gdsgds', 'viewer_friendship': 1, 'is_premium': False}, 'sharing': [{'share_time': '2015-05-21T08:45:19.000Z', 'type': 0, 'share_id': 1635690786663532}], 'show_map':\
0, 'pictures': [], 'hashtags': [], 'descent': 150.621, 'speed_avg': 9.80291763746756, 'expand': 'full', 'show_workout': 0, 'points': {'expand': 'ref', 'id': 2199549878449}}
I am not receiving the long arrays within the data. I am also not even recovering all of the non-array data.
I ran the original page through a JSON validator, and it's fine. Similarly, I ran the JSON I do receive through a validator, and it's also fine - it doesn't show any signs of missing things unless I compare with the original.
I would appreciate any advice about how to troubleshoot this. Thanks.
Looks like this API is doing some User-Agent sniffing and only sending the complete content for what it considers to be actual web browsers.
Once you set a User-Agent header with the UA string of a common browser, you get the full response:
>>> UA = 'Mozilla/5.0 (X11; Linux x86_64; rv:36.0) Gecko/20100101 Firefox/36.0'
>>> url = 'https://www.endomondo.com/rest/v1/users/20261627/workouts/526622897'
>>> r = requests.get(url, headers={'User-Agent': UA})
>>>
>>> print len(r.content)
96412
See the requests docs for more details on setting custom headers.
I am working with the request module within python to grab certain fields within the JSON response.
import json
fn = 'download.json'
data = json
response = requests.get('http://api.appannie.com/v1/accounts/1000/apps/mysuperapp/sales?break_down=application+iap&start_date=2013-10-01&end_date=2013-10-02', \
auth=('username', 'password'))
data = response.json()
print(data)
This works in python, as the response is the following:
{'prev_page': None, 'currency': 'USD', 'next_page': None, 'sales_list': [{'revenue': {'ad': '0.00', 'iap': {'refunds': '0.00', 'sales': '0.00', 'promotions': '0.00'}, 'app': {'refunds': '0.00', 'updates': '0.00', 'downloads': '0.00', 'promotions': '0.00'}},
'units': {'iap': {'refunds': 0, 'sales': 0, 'promotions': 0}, 'app': {'refunds': 0, 'updates': 0, 'downloads': 2000, 'promotions': 0}}, 'country': 'all', 'date': 'all'}], 'iap_sales': [], 'page_num': 1, 'code': 200, 'page_index': 0}
The question is how do I parse this to get my downloads number within the 'app' block - namely the "2000" value?
After the response.json() data is already a dictionary otherwise response.json() would raise an exception. Therefore you can access it just like any other dictionary.
You can use the loads() method of json -
import json
response = requests.get('http://api.appannie.com/v1/accounts/1000/apps/mysuperapp/sales?break_down=application+iap&start_date=2013-10-01&end_date=2013-10-02',
auth=('username', 'password'))
data = json.loads(response.json()) # data is a dictionary now
sales_list = data.get('sales_list')
for sales in sales_list:
print sales['revenue']['app']
You can use json.loads:
import json
import requests
response = requests.get(...)
json_data = json.loads(response.text)
This converts a given string into a dictionary which allows you to access your JSON data easily within your code.