I am currently writing a script that sends a request to a specific webpage and returns a JSON response. The issue is that multiple of the same requests come back, and some are HTML and one is JSON. I've been researching how to keep checking until a valid JSON response is returned, but no luck. Here is what I have currently:
response = requests.get('http://www.samplewebpage.com')
inputJSON = json.loads(response.text)
exampleList = list(inputJSON['metaData'].values())
outputArray = []
Is there an easy way to loop through the json.loads to wait until the response is a actual JSON?
Thanks in advance.
found = False
while not found:
response = requests.get('URL')
try:
inputJSON = json.loads(response.text)
found = True
print('valid JSON')
except:
print('not valid JSON')
pass
Related
this is my first ever question on here.
Currently I am looking into the Python requests module. I am trying to automate a task for which i need to pass on a csrf token.
The csrf token can be found in the payload of a previous request.
How can I extract the value out of the Payload?
Example Payload:
value1: ABCD
value2: EFGH
csrf_token: the token I am looking for
value3: false
Sorry if this is a dumb or easy question but I am not able to solve it right now.
Thanks for your help!
Here are some examples of where the data you might be looking for is located.
import requests
response = requests.get('http://some_url')
# raise an exception if the status code != 200
response.raise_for_status()
# the contents of the response. (bytes)
contents = response.contents
# the contents of the response if the contents are
# formatted as JSON. (dictionary)
contents = response.json()
# the headers of the response. (dictionary)
headers = response.headers
# You also have to consider if you are using the correct HTTP protocol
payload = {}
response = requests.put('http://some_url', json=payload)
response = requests.post('http://some_url', json=payload)
# the responses are going to function just like when using the "get" protocol
I am trying to parse json data from a site that shows values of crypto currency. I am trying to parse it using python. I am a little lost on how to show the output.
API: https://min-api.cryptocompare.com/data/price?fsym=XMR&tsyms=USD
# code starts below
import requests
# Set the request parameters
url = 'https://min-api.cryptocompare.com/data/price?fsym=XMR&tsyms=USD'
# Fetch url
print("Fetching url..")
# Do the HTTP get request
response = requests.get(url, verify=True) #Verify is check SSL certificate
# Error handling
# Check for HTTP codes other than 200
if response.status_code != 200:
print('Status:', response.status_code, 'Problem with the request. Exiting.')
exit()
# Decode the JSON response into a dictionary and use the data
USD = response.json()
output = USD[0]['USD']
print('Output USD:'), USD
# code ends
I am getting a response code 200 because IDLE tries to exit. The code is based off another project and I don't believe I am parsing json correctly?
The problem is here:
if response.status_code != 200:
print('Status:', response.status_code, 'Problem with the request. Exiting.')
exit()
The way you've indented, Python will always call exit(). You want it to call exit() only if there was actually an error, like so:
if response.status_code != 200:
print('Status:', response.status_code, 'Problem with the request. Exiting.')
exit()
However, you have another problem. You're trying to assign "output" the value of an object in USD; USD doesn't exist:
data = response.json()
output = USD[0]['USD'] #"USD" doesn't exist. It's the value you're trying to find, not the json that contains the object itself.
print('Output USD:'), USD #"USD" doesn't exist. I think you meant to print "output" here, which is the value you're setting in the line above.
Instead, try this:
data = response.json()
output = data["USD"]
print('Output USD:'), output
Your exit() line is not indented correctly. Try this:
if response.status_code != 200:
print('Status:', response.status_code, 'Problem with the request. Exiting.')
exit()
Additionally, even though you are parsing the JSON correctly, you incorrectly use the resulting data. Try this:
output = USD['USD'] # Note: "[0]" not required
print('Output USD:', USD) # Note: ", USD" inside the parentheses
the function looks like this:
import requests
import json
def parse(s):
r = requests.post('http://166.111.139.15:9000/?properties%3d%7b%22annotators%22%3a%22tokenize%2cssplit%2cpos%2clemma%2cparse%22%2c%22outputFormat%22%3a%22json%22%7d%0a', data=s)
return r.json()
print parse("I am a student")
And when I use it in Django, the web page shows:
"No JSON object could be decoded".Why?
No JSON object could be decoded is an exception message raised at r.json(). If your response is not a valid json object you can still retrieve it with r.text. Even if you are sure your response is always a valid json object, you should still check whether server returned a success code. If there's an internal server error (code 500), you won't get a valid json response!
import requests
def parse(s)
r = requests.post('http://someserver.com', data=s)
if r.status_code !== 200:
return "There was a problem: {} !".format(r.text)
return r.json()
My webservice should receive calls in these two formats: application/x-www-form-urlencoded and content-type application/json.
The code below works correctly for the forms. However, it doesn't work for the json ones. Apparently I need to use request.args.get for it.
Is there a way to modify the code so that the same method can receive calls in these two formats?
#app.route("/api/<projectTitle>/<path:urlSuffix>", methods=['POST'])
def projectTitlePage(projectTitle, urlSuffix):
apiKey = request.form.get('apikey')
userId = databaseFunctions.getApiKeyUserId(apiKey)
userInfo = databaseFunctions.getUserInfo(userId)
projectId = databaseFunctions.getTitleProjectId(projectTitle)
projectInfo = databaseFunctions.getProjectInfo(projectId)
databaseFunctions.addUserHit(userId, projectId)
databaseFunctions.addProjectHit(userId)
print request.form.to_dict(flat=False)
try:
r = requests.post(projectInfo['secretUrl'], data=request.form.to_dict(flat=False))
except Exception, e:
return '/error=Error'
return r.text
Try to get the JSON using Request.get_json(); an exception is raised if that fails, after which you can fall back to using request.form:
from flask import request
from werkzeug.exceptions import BadRequest
try:
data = request.get_json()
apiKey = data['apikey']
except (TypeError, BadRequest, KeyError):
apiKey = request.form['apikey']
If the mimetype is not application/json, request.get_json() returns None; trying to use data['apikey'] then results in a TypeError. The mimetype being correct but the JSON data being invalid gives you a BadRequest, and all other invalid return values either result in a KeyError (no such key) or a TypeError (object doesn't support indexing by name).
The other option would be to test the request.mimetype attribute:
if request.mimetype == 'application/json':
data = request.get_json()
apiKey = data['apiKey']
else:
apiKey = request.form['apikey']
Either way, if there is no valid JSON data or form data was posted but there is no apikey entry or an unrelated mimetype was posted, a BadRequest exception will be raised and a 400 response is returned to the client.
I'm not incredibly familiar with Flask in particular, but based on their documentation you should be able to do something like
content = request.headers['CONTENT-TYPE']
if content[:16] == 'application/json':
# Process json
else:
# Process as form-encoded
My approach to fetch the json and form-data regardless of the header was like
data = request.get_json() or request.form
key = data.get('key')
I am trying to parse a response.text that I get when I make a request using the Python Requests library. For example:
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
self.response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
print self.response.text
When I execute this method, the output is:
{"request":"POST /login","result":"success"}
I would like to check whether "result" equals "success", ignoring whatever comes before.
The manual suggests: if self.response.status_code == requests.codes.ok:
If that doesn't work:
if json.loads(self.response.text)['result'] == 'success':
whatever()
Since the output, response, appears to be a dictionary, you should be able to do
result = self.response.json().get('result')
print(result)
and have it print
'success'
If the response is in json you could do something like (python3):
import json
import requests as reqs
# Make the HTTP request.
response = reqs.get('http://demo.ckan.org/api/3/action/group_list')
# Use the json module to load CKAN's response into a dictionary.
response_dict = json.loads(response.text)
for i in response_dict:
print("key: ", i, "val: ", response_dict[i])
To see everything in the response you can use .__dict__:
print(response.__dict__)
import json
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
#this line converts the response to a python dict which can then be parsed easily
response_native = json.loads(response.text)
return self.response_native.get('result') == 'success'
I found another solution. It is not necessary to use json module. You can create a dict using dict = eval(whatever) and return, in example, dict["result"]. I think it is more elegant. However, the other solutions also work and are correct
Put in the return of your method like this:
return self.response.json()
If you wanna looking for more details, click this following link:
https://www.w3schools.com/python/ref_requests_response.asp
and search for json() method.
Here is an code example:
import requests
url = 'https://www.w3schools.com/python/demopage.js'
x = requests.get(url)
print(x.json())
In some cases, maybe the response would be as expected. So It'd be great if we can built a mechanism to catch and log the exception.
import requests
import sys
url = "https://stackoverflow.com/questions/26106702/how-do-i-parse-a-json-response-from-python-requests"
response = requests.get(url)
try:
json_data = response.json()
except ValueError as exc:
print(f"Exception: {exc}")
# to find out why you have got this exception, you can see the response content and header
print(str(response.content))
print(str(response.headers))
print(sys.exc_info())
else:
if json_data.get('result') == "success":
# do whatever you want
pass