I am using http.client to hit a POST api and get the json response. My code is working correctly when I print response.read(). However, for some reason, this response is limited to only 20 results and the total count of results is over 20,000. I want to get the complete response in a variable using response.read().decode(), I am hoping that the variable will contain the complete json string. The issue is that I am getting an empty string when I used decode(). How do I get this done? How do I get the complete results?
import http.client
host = 'jooble.org'
key = 'API_KEY'
connection = http.client.HTTPConnection(host)
#request headers
headers = {
"Content-type": "application/json"}
#json query
body = '{ "keywords": "sales", "location": "MA"}'
connection.request('POST','/api/' + key, body, headers)
response = connection.getresponse()
print(response.status, response.reason)
print(response.read())
print(response.read().decode())
Don't call response.read() twice. response is a stream, so each call to read() continues from where the previous one ended. Since the first call is reading the entire response, the second one doesn't read anything.
If you want to print the encoded and decoded response, assign response.read() to a variable, then decode that.
data = response.read()
print(data)
print(data.decode())
But this can be done more simply using the requests module.
import requests
host = 'jooble.org'
key = 'API_KEY'
body = { "keywords": "sales", "location": "MA"}
response = requests.post(f'https://{host}/api/{key}', json=body)
print(response.content)
Note that in this version body is a dictionary, not a string. The json parameter automatically converts it to JSON.
Related
I was trying to automate the task of pushing some files to various folder in a repo. I tried using Rest API provided by azure. When using Pushes Create API for the same, from the docs this is the content in the request body
snapshot of request body
This is the snapshot of python code that I wrote:
The code that I am using
In the above code repositeries_url contains the valid API URL.
When I run the code I am getting response code 400 and printing the JSON gives me
{'$id': '1', 'innerException': None, 'message': 'The body of the request contains invalid Json.\r\nParameter name: contentStream', 'typeName': 'Microsoft.TeamFoundation.SourceControl.WebServer.InvalidArgumentValueException, Microsoft.TeamFoundation.SourceControl.WebServer', 'typeKey': 'InvalidArgumentValueException', 'errorCode': 0, 'eventId': 0}
Why is this error coming and how can I rectify this error?
I found below mistake in your python code.
In your code, you defined payload["commits"] as commits array.
But you mistakenly assign temp as its value. You need to append temp to payload["commits"] array. ie. payload["commits"].append(temp)
Also if you want to use json in the request.post method to post the data. You can directly pass the payload object. Like below:
response = requests.post(url = repo_url, json = payload, headers = header)
Or you can use data in request.post method to post the json string. See below:
response = requests.post(url = repo_url, data= jsonPayload, headers = header)
See below full code:
....
temp["changes"].append(aksh)
payload["commits"].append(temp) ##append temp to commits array
jsonPayload=json.dumps(payload)
#pass payload to json parameter directly
response = requests.post(url = repo_url, json= payload, headers = header)
#you can also use data
#response = requests.post(url = repo_url, data= jsonPayload, headers = header)
I'm sending a request for a set of images to one of my API's. The API returns these images in a JSON format. This format contains data about the resource together with a single property that represents the image in Base64.
An example of the JSON being returned.
{
"id": 548613,
"filename": "00548613.png",
"pictureTaken": "2020-03-30T11:38:21.003",
"isVisible": true,
"lotcode": 23,
"company": "05",
"concern": "46",
"base64": "..."
}
The correct content of the Base64
The incorrectly parsed Base64
This is done with the Python3 requests library. When i receive a successful response from the API i attempt to decode the body to JSON using:
url = self.__url__(f"/rest/all/V1/products/{sku}/images")
headers = self.__headers__()
r = requests.get(url=url, headers=headers)
if r.status_code == 200:
return r.json()
elif r.status_code == 404:
return None
else:
raise IOError(
f"Error retrieving product '{sku}', got {r.status_code}: '{r.text}'")
Calling .json() results in the Base64 content being messed up, some parts are not there, and some are replaced with other characters. I tried manually decoding the content using r.content.decode() with the utf-8 and ascii options to see if this was the problem after seeing this post. Sadly this didn't work.
I know the response from the server is correct, it works with Postman, and calling print(r.content) results in a JSON document containing the valid Base64.
How would i go about de-serializing the response from the API to get the valid Base64?
import base64
import re
...
b64text = re.search(b"\"base64\": \"(?P<base>.*)\"", r.content, flags=re.MULTILINE).group("base")
decode = base64.b64decode(b64text).decode(utf-8)
Since you're saying "calling print(r.content) results in the valid Base64", it's just a matter of decoding the base64.
Trying to output my response from Microsoft's Computer Vision API to a .json file, it works with all of the other APIs I've been using so far. With the code below, directly from Microsoft's documentation, I get an error:
Error: the JSON object must be str, not 'bytes'
Removing the parsed = json.loads(data) and using print(json.dumps(data, sort_keys=True, indent=2)) prints out the information for the image that I want, but also says Error and is prefixed with
b
denoting it's in bytes and ending with
is not JSON serializable
I'm just trying to find out how I can get the response into a .json file like i'm able to do with other APIs and am at a loss for how I can possible convert this in a way that will work.
import http.client, urllib.request, urllib.parse, urllib.error, base64, json
API_KEY = '{API_KEY}'
uri_base = 'westus.api.cognitive.microsoft.com'
headers = {
'Content-Type': 'application/json',
'Ocp-Apim-Subscription-Key': API_KEY,
}
params = urllib.parse.urlencode(
{
'visualFeatures': 'Categories, Description, Color',
'language': 'en',
}
)
body = "{'url': 'http://i.imgur.com/WgPtc53.jpg'}"
try:
conn = http.client.HTTPSConnection(uri_base)
conn.request('POST', '/vision/v1.0/analyze?%s' % params, body, headers)
response = conn.getresponse()
data = response.read()
# 'data' contains the JSON data. The following formats the JSON data for display.
parsed = json.loads(data)
print ("Response:")
print (json.dumps(parsed, sort_keys=True, indent=2))
conn.close()
except Exception as e:
print('Error:')
print(e)
Shortly after posting the question, I realized I had missed looking for something: just converting bytes to a string.
found this Convert bytes to a Python string
and was able to modify my code to:
parsed = json.loads(data.decode('utf-8'))
And it seems to have resolved my issue. Now error-free and able to export to .json file like I needed.
Here is a snippet of code:
api_url = {url}
auth_head = {key: value} << my api authentication header
data = {'title':'Python'} << the valid json object format that the api accepts
Here is what I tried:
data = json.dumps(data)
json_obj = json.loads(data)
response = requests.post(api_url, headers = auth_head, data = json_obj)
print(response.text)
Here is the output:
"400 Bad request"
To make sure the url and my api token key work I tried GET:
response = requests.get(api_url, headers = auth_head)
print(response.status_code)
Output:
"200"
So the url and api token work fine. I feel I am not sending the correct json object construct that the api wants
You don't need to send data as json. Just send data as dictionary. It will accept. requests automatically will encode your dictionary to json
I am attempting to get an auth token from The Trade Desk's (sandbox) api but I get back a 400 response stating:
"Error reading Content-Type 'application/json' as JSON: Unexpected
character encountered while parsing value: L. Path '', line 0,
position 0."
Whole response.json():
{u'ErrorDetails': [{u'Reasons': [u"Error reading Content-Type 'application/json' as JSON: Unexpected character encountered while parsing value: L. Path '', line 0, position 0."], u'Property': u'TokenRequest'}], u'Message': u'The request failed validation. Please check your request and try again.'}
My script (runnable):
import requests
def get_token():
print "Getting token"
url = "https://apisb.thetradedesk.com/v3/authentication"
headers = {'content-type': 'application/json'}
data = {
"Login":"logintest",
"Password":"password",
"TokenExpirationInMinutes":60
}
response = requests.post(url, headers=headers, data=data)
print response.status_code
print response.json()
return
get_token()
Sandbox docs here
I believe this means my headers var is not being serialized correctly by requests, which seems impossible, or not being deserialized correctly by The Trade Desk. I've gotten into the requests lib but I can't seem to crack it and am looking for other input.
You need to do
import json
and convert your dict into json:
response = requests.post(url, headers=headers, data=json.dumps(data))
Another way would be to explicitely use json as parameter:
response = requests.post(url, headers=headers, json=data)
Background: In the prepare_body method of requests a dictionary is explicitely converted to json and a content-header is also automatically set:
if not data and json is not None:
content_type = 'application/json'
body = complexjson.dumps(json)
If you pass data=data then your data will be only form-encoded (see http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests). You will need to explicitely convert it to json, if you want json to be the content-type of your http body.
Your follow-up question was about why headers don't have to be converted to json. Headers can be simply passed as dictionary into the request. There's no need to convert it to json. The reason is implementation specific.