This may be a silly question but I am really confused(I am a newbie). I am trying to make an API that accepts JSON as input and I am using Flask. The API takes POST method, so when a request comes along, it gets the JSON data from the body using
data = requests.get_json()
I expect data to be a string because, if I am not mistaken, JSON is nothing but a formatted string.
So, I do data = json.loads(data) But, my flask app crashes because it says data is a dictionary not a string. Of course, I can fix it by not using json.loads But it just bothers me and I wonder why I get a dictionary not a string.
Here is how I send test-requests, which seriously confuse me
1)
import requests
import pandas as pd
data = pd.read_csv('some.csv')
data = data.iloc[[0]].to_json(orient='records') // get the first row into json
res = requests.post(url, json=data) // I get a string in my Flask app.
import requests
data = {'name':'foo','age':99}
res = requests.post(url, json=data) // I get a dictionary in my Flask app.
const xhr = new XMLHttpRequest();
const json = {'name':'foo','age':99};
xhr.open("POST",url);
xhr.setRequestHeader("Content-Type","application/json");
xhr.send(JSON.stringify(json)); // Though stringified, I get a dictionary in my Flask app. Why?
I am not sure if you can see my confusion. In some cases, I get a dictionary, and in some other cases I get a string. So, I am confused and don't know how to design my API and handle the requests.
Thank you in advance for your attention!
Pandas' DataFrame.to_json returns a string (str). Hence, in this code
data = df.to_json(orient='records')
res = requests.post(url, json=data)
data is actually a str object, and passing it to the json parameter of requests.post will encode that string as JSON again. See
response = requests.post(url, json={"foo": 1})
print(response.request.body)
response = requests.post(url, json='{"foo": 1}')
print(response.request.body)
Will print
b'{"foo": 1}'
b'"{\\"foo\\": 1}"'
What you must do, to send that JSON data correctly, is
data = df.to_json(orient='records')
response = requests.post(url, data=data.encode())
or actually convert the DataFrame to a dict
data = df.to_dict(orient='records')
response = requests.post(url, json=data)
JSON object is nothing but a dictionary in python and flask is framework written python
Accordingly, the json library exposes the dump() method for writing data to files. There is also a dumps() method (pronounced as “dump-s”) for writing to a Python string.
Simple Python objects are translated to JSON according to a fairly intuitive conversion.
Python JSON
dict object
list,tuple array
str string
int,long,float number
True true
False false
None null
so depending upon what is extracted from json python variable behaves accordingly,
like in first case
data = data.iloc[[0]].to_json(orient='records') data variable is nothing but a string,
so this is why res = requests.post(url, json=data) shows such behaviour here
In second Case
data = {'name':'foo','age':99} it's dictionary
so this why
res = requests.post(url, json=data) shows such behaviour
Related
I am making my first API; any advice to improve my process is much appreciated.
I plan on passing JSON-like strings into the HTML request to this FastAPI microservice down there
#app.get("/create/{value}")
def createJSON(value:str):
person_json = value.strip()
fileName = person_json['Value']['0'] + person_json['Value']['1']
with open('%s.JSON','w') as writeFile:
writeFile.write(string)
return "Person has been created"
My HTTP request would look like this:
http://127.0.0.1:8000/create/{"Key":{"0":"name","1":"grad_year","2":"major","3":"quarter","4":"pronoun","5":"hobbies","6":"fun_fact","7":"food","8":"clubs","9":"res"},"Value":{"0":"adfasdfa","1":"adf'asd","2":"asd","3":"fads","4":"fa","5":"sdfa","6":"df","7":"asd","8":"fa","9":"df"}}
However, when doing this. The values passed are strings. Thus rendering the fileName portion of the code useless. How can I convert it to a Python dict? I have tried to use .strip(), but it did not help.
You're on the wrong track, Such a request should be essentially modeled as POST or a PUT request. That would allow you to send JSON in the body of the request and obtain it as a dict in python. You can see here
And even if you want to pass data in a GET request, there are query params
Coming back to the original doubt, you would have to use json.loads() to parse the json data and load it in a python dict then you can dump whatever file you like after that.
I'd recommend using the requests library
import requests
url = 'http://127.0.0.1:8000/create/'
params = dict(
name = 'Josh',
grad_year = '1987',
major = 'computer science',
quarter = '3'
)
resp = requests.get(url=url, params=params)
data = resp.json()
Then see here how to handle the JSON Response Content:
https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content
The dict in the code I posted is different than the JSON you're trying to send through though. I assume you have a specific reason for having a "Key" array with the names than a "Value" array for the values of those specific names. But if not I'd recommend using a dictionary instead that way you can do things like:
fileName = person_json['name'] + person_json['grad-year']
I am trying to access a dictionary that has been returned after making a post to an API, but I am having difficulty formatting the data with JSON.
It seems to be returning the data as a dictionary but I receive the error 'list indices must be integers or slices, not str' which makes me believe that it is just returning a list that looks like a dictionary. I have tried using json.loads() and trying to access the data through lists but I can't seem to get it. The data I am trying to has multiple sub dictionaries/lists.
resp = post(url = endpoint_url, data = data, headers = headers)
data_for_process = resp.json()
print(data_for_process['pages']['keyValuePairs']['key'])
I expected the print statement to return the value for that specific key but I get the error instead.
Any help much appreciated.
you can use the json library of python
import json
import pprint
resp = post(url = endpoint_url, data = data, headers = headers)
resp_json = json.loads(resp.content)
pprint.pprint(resp_json) # to display the dict prettier
I am trying to do the following with requests:
data = {'hello', 'goodbye'}
json_data = json.dumps(data)
headers = {
'Access-Key': self.api_key,
'Access-Signature': signature,
'Access-Nonce': nonce,
'Content-Type': 'application/json',
'Accept': 'text/plain'
}
r = requests.post(url, headers=headers, data=json_data,
files={'file': open('/Users/david/Desktop/a.png', 'rb')})
However, I get the following error:
ValueError: Data must not be a string.
Note that if I remove the files parameter, it works as needed. Why won't requests allow me to send a json-encoded string for data if files is included?
Note that if I change data to be just the normal python dictionary (and not a json-encoded string), the above works. So it seems that the issue is that if files is not json-encoded, then data cannot be json-encoded. However, I need to have my data encoded to match a hash signature that's being created by the API.
When you specify your body to a JSON string, you can no longer attach a file since file uploading requires the MIME type multipart/form-data.
You have two options:
Encapsulate your JSON string as part as the form data (something like json => json.dumps(data))
Encode your file in Base64 and transmit it in the JSON request body. This looks like a lot of work though.
1.Just remove the line
json_data = json.dumps(data)
and change in request as data=data.
2.Remove 'Content-Type': 'application/json' inside headers.
This worked for me.
Alternative solution to this problem is to post data as file.
You can post strings as files. Read more here:
http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file
Here is explained how to post multiple files:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files
removing the following helped me in my case:
'Content-Type': 'application/json'
then the data should be passed as dictionary
If your files are small, you could simply convert the binary (image or anything) to base64 string and send that as JSON to the API. That is much simpler and more straight forward than the suggested solutions. The currently accepted answer claims that is a lot of work, but it's really simple.
Client:
with open('/Users/houmie/Downloads/log.zip','rb') as f:
bytes = f.read()
tb = b64encode(bytes)
tb_str = tb.decode('utf-8')
body = {'logfile': tb_str}
r = requests.post('https://prod/feedback', data=json.dumps(body), headers=headers)
API:
def create(event, context):
data = json.loads(event["body"])
if "logfile" in data:
tb_back = data["logfile"].encode('utf-8')
zip_data = base64.b64decode(tb_back)
I'm hitting the Hacker news API here and want to get the details of each posts that I get through the JSON. I want to send this JSON to my React front-end.
This request is taking a long time. What do I need to do to send the response?
#app.route('/api/posts')
def get_posts():
r = requests.get('https://hacker-news.firebaseio.com/v0/askstories.json?print=pretty')
data = r.text
jsonData = []
for post in data:
r = requests.get('https://hacker-news.firebaseio.com/v0/item/'+post+'.json?print=pretty')
r.text
jsonData.append(r.text)
jsonData = jsonify(jsonData)
print jsonData
return jsonData
You're querying a json API and treating the response as text:
r = requests.get('https://hacker-news.firebaseio.com/v0/askstories.json?print=pretty')
data = r.text
So, r.text would be a string "[1234,1235,1236]" and not a list of integers.
So when you iterate over that in your for post in data what you're doing is getting each character:
for post in data:
print(post)
Would give you:
[
1
2
3
4
,
...etc
So your essentially querying the hacker news API for hundreds of invalid posts, instead of tens of actual ones. You should be treating the json as json— by using the json features built into requests: data = r.json()
That will give you a list of numbers to iterate over— you'd also need to change the bad way you're concatenating your data to make your url string (use .format).
requests has a .json() method that you should use to convert your JSON array string into a python list.
In [1]: import requests
In [2]: r = requests.get('https://hacker-news.firebaseio.com/v0/askstories.json?print=pretty')
In [3]: jsonData = r.json()
In [4]: for data in jsonData[:5]:
... print data
...:
12102489
12100796
12101060
12097110
12094366
As stated in the other answer, for post in data: is going to give you individual characters from the HTTP response. In other words, think about what would for post in "abc": give you.
The page is taking a very long time to load
That's because you are running a new query against all those individual characters.
I'm working on an API wrapper. The spec I'm trying to build to has the following request in it:
curl -H "Content-type:application/json" -X POST -d data='{"name":"Partner13", "email":"example#example.com"}' http://localhost:5000/
This request produces the following response from a little test server I setup to see exatly what headers/params etc are sent as. This little script produces:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data={"name":"Partner13", "email":"example#example.com"}
So that above is the result I want my python script to create when it hits the little test script.
I'm using the python requests module, which is the most beautiful HTTP lib I have ever used. So here is my python code:
uri = "http://localhost:5000/"
headers = {'content-type': 'application/json' }
params = {}
data = {"name":"Partner13", "email":"example#exmaple.com"}
params["data"] = json.dumps(data)
r = requests.post(uri, data=params, headers=headers)
So simple enough stuff. Set the headers, and create a dictionary for the POST parameters. That dictionary has one entry called "data" which is the JSON string of the data I want to send to the server. Then I call the post. However, the result my little test script gives back is:
uri: http://localhost:5000/,
method: POST,
api_key: None,
content_type: application/json,
params: None,
data: data=%7B%22name%22%3A+%22Partner13%22%2C+%22email%22%3A+%22example%40example.com%22%7D
So essentially the json data I wanted to send under the data parameter has been urlendcoded.
Does anyone know how to fix this? I have looked through the requests documentation and cannot seem to find a way to not auto urlencode the send data.
Thanks very much,
Kevin
When creating the object for the data keyword, simply assign a variable the result of json.dumps(data).
Also, because HTTP POST can accept both url parameters as well as data in the body of the request, and because the requests.post function has a keyword argument named "params", it might be better to use a different variable name for readability. The requests docs use the variable name "payload", so thats what I use.
data = {"name":"Partner13", "email":"example#exmaple.com"}
payload = json.dumps(data)
r = requests.post(uri, data=payload, headers=headers)
Requests automatically URL encodes dictionaries passed as data here. John_GG's solution works because rather than posting a dictionary containing the JSON encoded string in the 'data' field it simply passes the JSON encoded string directly: strings are not automatically encoded. I can't say I understand the reason for this behaviour in Requests but regardless, it is what it is. There is no way to toggle this behaviour off that I can find.
Best of luck with it, Kevin.