I am a new programmer and I'm learning the request module. I'm stuck on the fact that I don't know how to get a specific part of a json response, I think it's called a header? or its the thing inside of a header? I'm not sure. But the API returns simple json code. This is the api
https://mcapi.us/server/status?ip=mc.hypixel.net
for more of a example, lets say it returns this json code from the api
{"status":"success","online":true"}
And I wanted to get the "online" response, how would I do that?
And this is the code im currently working with.
import requests
def main():
ask = input("IP : ")
response = requests.get('https://mcapi.us/server/status?ip=' + ask)
print(response.content)
main()
And to be honest, I don't even know if this is json. I think it is but the api page says its cors? if it isn't I'm sorry.
In your example you have a dictionary with key "online"
You need to parse it first with .json() and then you can get it in form dict[key]
In your case
response = requests.get('https://mcapi.us/server/status?ip=' + ask).json()
print(response["online"])
or in case of actual content
response = requests.get('https://mcapi.us/server/status?ip=' + ask).json()
print(response["content"])
Related
learning to work with the request library and pandas but have been struggling to get past the starting point even with a good amount of examples online.
I am trying to extract NBA shot data from the URL below using a GET request, and then turn it into a DataFrame:
def extractData():
Harden_data_url = "https://stats.nba.com/events/?flag=3&CFID=33&CFPARAMS=2017-18&PlayerID=201935&ContextMeasure=FGA&Season=2017-18§ion=player&sct=hex"
response = requests.get(Harden_data_url)
data = response.json()
shots = data['resultSets'][0]['rowSet']
headers = data['resultSets'][0]['headers']
df = pandas.DataFrame.from_records(shots, columns = headers)
However I get this error starting on line 2 "response = requests.get(url)"
ValueError: No JSON object could be decoded
I imagine I am missing something basic, any debugging help is appreciated!
The problem is that you are using the wrong URL for fetching the data.
The URL you used was for the HTML, which is in charge of the layout of the site. The data comes from a different URL, which fetches it in JSON format.
The correct URL for the data you are looking for is this:
https://stats.nba.com/stats/shotchartdetail?CFID=33&CFPARAMS=2017-18&ContextMeasure=FGA&DateFrom=&DateTo=&EndPeriod=10&EndRange=28800&GameID=&GameSegment=&GroupQuantity=5&LastNGames=0&LeagueID=00&Location=&Month=0&OnOff=&OpponentTeamID=0&Outcome=&PORound=0&Period=0&PlayerID=201935&PlayerPosition=&RangeType=0&RookieYear=&Season=2017-18&SeasonSegment=&SeasonType=Regular+Season&StartPeriod=1&StartRange=0&TeamID=0&VsConference=&VsDivision=
If you run it on the browser, you can see only the raw JSON data, which is exactly what you will get in your code, and make it work properly.
This blog post explains the method to find the data URL, and although the API has changed a little since the post was written, the method still works:
http://www.gregreda.com/2015/02/15/web-scraping-finding-the-api/
I've been asked to swap over from urllib2 to the requests library because the library is simpler to use and doesn't cause exceptions.
I can get the HTTP error code with response.status_code but I don't see a way to get the error message for that code. Normally, I wouldn't care, but I'm testing an API and the string is just as important.
Does anybody know of a simple way to get that string? I'm expecting 2 pieces something like:
'400':'Bad Request'
This is NOT a DUPLICATE
Some of the codes being returned have unique strings being sent by the application that I am testing. These strings cannot be looked up using this method: requests.status_codes._codes[error][0] Since the string is dynamically coming from the back end server. I was able to get this information using urllib using this method:
import urllib2
...
opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx))
except urllib2.HTTPError as err:
try: error_message = err.read()
...
The question now is... is there a method of getting the dynamic http error string? Thanks so much for being patient. The previous issue was closed so quickly I never got a chance to look at the answer, test it and re-ask by cleaning up the description.
response = requests.get(url)
error_message = response.reason
In HTTPResponse there's a reason attribute that returns the reason phrase from the response's status line. In the requests library the requests.Response class has an equivalent reason attribute that returns the same thing. Both should return the information from the response, not a fixed string based on the code.
This is my first couple of weeks coding; apologies for a basic question.
I've managed to parse the 'WorldNews' subreddit json, identify the individual children (24 of them as I write) and grab the titles of each news item. I'm now trying to create an array from these news titles. The code below does print the fifth title ([4]) to command line every 2-3 attempts (otherwise provides the error below). It will also not print more than one title at a time (for example if I try[2,3,4] I will continuously get the same error).
The error I get when doesn't compile:
in <module> Children = theJSON["data"]["children"] KeyError: 'data'
My script:
import requests
import json
r = requests.get('https://www.reddit.com/r/worldnews/.json')
theJSON = json.loads(r.text)
Children = theJSON["data"]["children"]
News_list = []
for post in Children:
News_list.append (post["data"]["title"])
print News_list [4]
I've managed to find a solution with the help of Eric. The issue here was in fact not related to the key, parsing or presentation of the dict or array. When requesting a Url from reddit and attempting to print the json string output we encounter an HTTP Error 429. Fixing this is simple. The answer was found on this redditdev thread.
Solution: by adding an identifier for the device requesting the Url ('User-agent' in header) it runs smoothly and works every time.
import requests
import json
r = requests.get('https://www.reddit.com/r/worldnews.json', headers = {'User-agent': 'Chrome'})
theJSON = json.loads(r.text)
print theJSON
This means that the payload you got didn't have a data key in it, for whatever reason. I don't know about Reddit's JSON API; I tested the request and saw that you were using the correct keys. The fact that you say your code works every few times tells me that you're getting a different response between requests. I can't reproduce it, I tried making the request over and over and checking for the correct response. If I had to guess why you'd get something different I'd say it'd have to be either rate limiting or a temporary 503 (Reddit having issues.)
You can guard against this by either catching the KeyError or using the .get method of dictionaries.
Catching KeyError:
try:
Children = theJSON["data"]["children"]
except KeyError:
print 'bad payload'
return
Using .get:
Children = theJSON.get("data", {}).get("children")
if not Children:
print 'bad payload'
return
I'm trying to understand what exactly I'm getting back when I make a POST request using the Requests module — is it always JSON? Seems like every response I get appears to be JSON, but I'm not sure.
Where r is my response object, when I do:
print r.apparent_encoding
It always seems to return ascii
And when I try type():
>>>print type(r)
<class 'requests.models.Response'
I pasted the output from print r.text into a JSON validator, and it reported no errors. So should I assume Requests is providing my with JSON objects here?
A response can be anything. If you've posted to a REST endpoint, it will usually respond with JSON. If so, requests will detect that and allow you to decode it via the .json() method.
But it's perfectly possible for you to post to a normal web URL, in effect pretending to be a browser, and unless the server is doing something really clever it will just respond with the standard HTML it would serve to the browser. In that case, doing response.json() will raise a ValueError.
No, the response text for a POST request is totally up to the web service. A good REST API will always respond with JSON, but you will not always get that.
Example
A common pattern in PHP is
<?php
$successful_whatever = false;
if (isset($_POST['whatever'])) {
# put $_POST['whatever'] in a database
$successful_whatever = true;
}
echo $twig->render('gallery.twig',
array('successful_whatever' => $successful_whatever));
?>
As you can see the response text will be a rendered template (HTML). I'm not saying it is good, just that it is common.
I would like to create a python script that makes a series of web requests to endpoints that will spit out JSON. After receiving the response, I want to validate that it is well-formed JSON, and not some error page. Also, I will need to insert an API key into the request header to get a proper response. what is the best way to go about doing this in python?
Thanks!
To validate the JSON you can use http://python-jsonschema.readthedocs.org/en/latest/
To insert the API key in the header you can use Requests http://docs.python-requests.org/en/latest/user/quickstart/
Validate the json use json.dump(), if your json object incorect it's return the error, handle the error using try catch and answer to your second question , i think you talking about X-AUTH-TOKEN , if your request return json object use this header.
headers = {
"Content-Type": "application/json",
"X-AUTH-TOKEN": your API Token
}
otherwise use
"Content-type":"application/x-www-form-urlencoded"