We Keep Coding

How do I get my JSON decoder to work properly?

I was working on API testing, and I tried everything but it would not print the JSON file into a string. I was wondering if it was the website I was testing the API requests from as I kept getting a 406 error. I even tried taking code from online that shows how to do this but it still would not print and it would give the error listed below. Here I give the code I used and the response Pycharm's console gave me.
import json
import requests
res = requests.get("http://dummy.restapiexample.com/api/v1/employees")
data = json.loads(res.text)
data = json.dumps(data)
print(data)
print(type(data))
Traceback (most recent call last):
File "C:/Users/johnc/PycharmProjects/API_testing/api_testing.py", line 8, in <module>
data = json.loads(res.text)
File "D:\Program Files (x86)\lib\json\__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "D:\Program Files (x86)\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "D:\Program Files (x86)\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

REST API's vary widely in the types of requests they will accept. 406 means that you didn't give enough information for the server to format a response. You should include a user agent because API's are frequently tweaked to deal with the foibles of various http clients and specifically list the format of the output you want. Adding acceptable encodings lets the API compress data. Charset is a good idea. You could even add a language request but most API's don't care.
import json
import requests
headers={"Accept":"application/json",
"User-agent": "Mozilla/5.0",
"Accept-Charset":"utf-8",
"Accept-Encoding":"gzip, deflate",
"Accept-Language":"en-US"} # or your favorite language
res = requests.get("http://dummy.restapiexample.com/api/v1/employees", headers=headers)
data = json.loads(res.text)
data = json.dumps(data)
print(data)
print(type(data))
The thing about REST API's is that they may ignore some or part of the header and return what they please. Its a good idea to form the request properly anyway.

The default Python User-Agent was being probably blocked by the hosting company.
You can setup any string or search for a real device string.
res = requests.get("http://dummy.restapiexample.com/api/v1/employees", headers={"User-Agent": "XY"})

It's you, your connection or a proxy. Things work just fine for me.
>>> import requests
>>> res = requests.get("http://dummy.restapiexample.com/api/v1/employees")
>>> res.raise_for_status() # would raise if status != 200
>>> print(res.json()) # `res.json()` is the canonical way to extract JSON from Requests
{'status': 'success', 'data': [{'id': '1', 'employee_name': 'Tiger Nixon', 'employee_salary': '320800', ...

How to use urllib3 to POST x-www-form-urlencoded data

I am trying to use urllib3 in Python to POST x-www-form-urlencoded data to ServiceNow API. The usual curl command would look like this
curl -d "grant_type=password&client_id=<client_ID>&client_secret=<client_Secret>&username=<username>&password=<password>" https://host.service-now.com/oauth_token.do
So far, I have tried the following:
import urllib3
import urllib.parse
http = urllib3.PoolManager()
data = {"grant_type": "password", "client_id": "<client_ID>", "client_secret": "<client_Secret>", "username": "<username>", "password": "<password>"}
data = urllib.parse.urlencode(data)
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
accesTokenCreate = http.request('POST', "https://host.service-now.com/outh_token.do", headers = headers, fields= data)
print(accesTokenCreate.data)
However, it does not generate the result similar to curl command and gives errors like below:
Traceback (most recent call last):
File "/VisualStudio/Python/ServiceNow.py", line 18, in <module>
accesTokenCreate = http.request('POST', "https://visierdev.service-now.com/outh_token.do", headers = headers, fields= data)
File "/usr/local/homebrew/lib/python3.7/site-packages/urllib3/request.py", line 80, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/usr/local/homebrew/lib/python3.7/site-packages/urllib3/request.py", line 157, in request_encode_body
fields, boundary=multipart_boundary
File "/usr/local/homebrew/lib/python3.7/site-packages/urllib3/filepost.py", line 78, in encode_multipart_formdata
for field in iter_field_objects(fields):
File "/usr/local/homebrew/lib/python3.7/site-packages/urllib3/filepost.py", line 42, in iter_field_objects
yield RequestField.from_tuples(*field)
TypeError: from_tuples() missing 1 required positional argument: 'value'
Could someone please help me understand how to properly use urllib3 to post such data to the ServiceNow API?

According to the urlllib3 documentation, you are not using the request() method properly. Specifically, the fields parameter in your code is not a "parameter of key/value strings AND key/filetuple". It's not suppose to be a URL-encoded string.
To fix your code, simply change the request call's fields parameter to body as in:
accesTokenCreate = http.request(
'POST', "https://host.service-now.com/outh_token.do",
headers=headers, body=data)
Better yet, you can use the request_encode_body() function and pass in the fields directly without urlencode-ing it and let that function call urllib.parse.urlencode() for you (per the same documentation).

Flask Request passing pandas to_json versus curl or Postman

I am calling a flask app that is hosted with AWS Elastic Beanstalk. The app takes in json, creates a pandas data frame from it, does some other processing and passes data into a model. The predictions are returned.
If I run the following, the code executes and returns back the proper response.
import requests
v=pandas_df.iloc[0:5].to_json(orient='records')
headers = {'content-type': 'application/json'}
r = requests.post('http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/', json=v, headers = headers)
r.json() #the predictions
However I can not get the same result using curl or postman.
For curl I have tried to send the json object created by pandas:
pandas_df.iloc[0:5].to_json(orient='records',path_or_buf=jsonexp.json)
curl -X POST "http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/" -H "Content-Type: application/json" -d "jsonexp.json"
but the result is
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>
I have tried passing the string version of v, for the sample of 5 records above
curl -X POST "http://XXXXXXXXX.us-east-1.elasticbeanstalk.com/" -H "Content-Type: application/json" -d '[{"field1":14239302,"field2":29....}]'
but it returns
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
How can I get the same result request through curl?
ADD
Here is a reproducible example. The issue seems to be in how the json is read for creation of a pandas data frame.
Here is the flask app:
from flask import Flask, jsonify, request, render_template, make_response
import pandas as pd
application = Flask(__name__)
#application.route('/', methods=['GET','POST'])
def apicall():
if request.method == 'POST':
f = request.get_json()
print(f)
print(type(f))
f=str(f)
print(type(f))
df=pd.read_json(f,orient='records')
print(df)
return(jsonify(df.to_json(orient='records')))
if __name__ == '__main__':
application.run()
This works perfectly:
r=requests.post('http://127.0.0.1:5000/', json='[{"field1":14239302,"field2":29.90}]', headers = {'content-type': 'application/json'})
r.json()
127.0.0.1 - - [13/Sep/2019 15:28:16] "POST / HTTP/1.1" 500 -
[{"field1":14239302,"field2":29.90}]
<class 'str'>
<class 'str'>
field1 field2
0 14239302 29.9
127.0.0.1 - - [13/Sep/2019 15:34:51] "POST / HTTP/1.1" 200 -
This fails:
!curl -d '[{"field1":14239302,"field2":29.90}]' -H "Content-Type: application/json" -X POST 'http://127.0.0.1:5000/'
with the following
127.0.0.1 - - [13/Sep/2019 15:28:13] "POST / HTTP/1.1" 200 -
{'field1': 14239302, 'field2': 29.9}
<class 'list'>
<class 'str'>
[2019-09-13 15:28:16,851] ERROR in app: Exception on / [POST]
Traceback (most recent call last):
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 2446, in wsgi_app
response = self.full_dispatch_request()
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1951, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1820, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
rv = self.dispatch_request()
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "application.py", line 17, in apicall
df=pd.read_json(f,orient='records')
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 592, in read_json
result = json_reader.read()
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 717, in read
obj = self._get_object_parser(self.data)
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 739, in _get_object_parser
obj = FrameParser(json, **kwargs).parse()
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 849, in parse
self._parse_no_numpy()
File "/home/ubuntu/env_india_flask_eb/lib/python3.6/site-packages/pandas/io/json/_json.py", line 1116, in _parse_no_numpy
loads(json, precise_float=self.precise_float), dtype=None
ValueError: Expected object or value
127.0.0.1 - - [13/Sep/2019 15:28:16] "POST / HTTP/1.1" 500 -
It seems like the issue is with the fact that the input to the flask app is a string for requests and a list for curl?
ADD BASED ON KEVIN's ANSWER:
I tried this function:
from flask import Flask, jsonify, request, render_template, make_response
import pandas as pd
application = Flask(__name__)
#application.route('/', methods=['GET','POST'])
def apicall():
if request.method == 'POST':
f = request.get_json(force=True)
print(f)
print(type(f))
df=pd.read_json(f,orient='records')
return(jsonify(df.to_json(orient='records')))
if __name__ == '__main__':
application.run()
and this input:
s=json.dumps([{'field1':14239302,'field2':29.90}])
!curl -d s -H "Content-Type:application/json" -X POST 'http://127.0.0.1:5000/'
But still receive an error:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>

I guess you are using ipython shell. s variable is not rendered unless you put it in curly brackets. Try
s=json.dumps([{'field1':14239302,'field2':29.90}])
!curl -d '{s}' -H "Content-Type:application/json" -X POST 'http://127.0.0.1:5000/'
s is not valid JSON hence 400 Error.
However the problem's cause is different.
In fact Flask get_json returns str when request is made by requests package or list is returned when using curl which causes 500 error.
This is because requests module escapes quote character and request's body looks this way: b'"[{\\"field1\\":14239302,\\"field2\\":29.90}]"' while body from curl is b'[{"field1": 14239302, "field2": 29.9}]'.
To make it work with curl you will need to escape " quote characters inside dict and put it in quotes like so:
curl -H "Content-Type: application/json" -d '"[{\"field1\":14239302,\"field2\":29.9}]"' -X POST 'http://127.0.0.1:5000/'`

pd.read_json requires a valid JSON string.
You are trying to use string casting to convert to JSON. str(f) is not a reliable way to convert to JSON.
The reason it works with requests is because you got lucky... you can see that it prints valid JSON:
[{"field1":14239302,"field2":29.90}]
However when you run curl, you do not have valid JSON due to the single quotes.
{'field1': 14239302, 'field2': 29.9}
Try using the json module to convert Python objects to a valid JSON string.
import json
json_string = json.dumps([{'field1': 14239302, 'field2': 29.9}])
print(json_string)
# '[{"field1":14239302,"field2":29.90}]'
# see how it prints your json_string with double quotes wrapping the keys

Python requests handle JSON output

I'm trying to query API to get a JSON response, the following is my code.
import requests
query_url = 'https://IP-Address/api/v1/request'
data = {"token":"8f86788b04637676b1f66eed902", "query":"days>50", "type":"json", "size":10}
response = requests.post(query_url, json=data, verify=False)
print(response.json())
I'm getting the following error,
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I also have a option to have CSV as "type" instead of JSON.
I created this Python request from cURL, which works good and outputs a JSON data. The following is the cURL command.
curl -kv -H 'Content-Type: application/json' 'https://IP-Address/api/v1/request' -d '{"token":"64bd230a775656a739b4673eb18", "query":"days>50", "type":"json", "size":10}'
Any suggestions please?

Create and parse multipart HTTP requests in Python

I'm trying to write some python code which can create multipart mime http requests in the client, and then appropriately interpret then on the server. I have, I think, partially succeeded on the client end with this:
from email.mime.multipart import MIMEMultipart, MIMEBase
import httplib
h1 = httplib.HTTPConnection('localhost:8080')
msg = MIMEMultipart()
fp = open('myfile.zip', 'rb')
base = MIMEBase("application", "octet-stream")
base.set_payload(fp.read())
msg.attach(base)
h1.request("POST", "http://localhost:8080/server", msg.as_string())
The only problem with this is that the email library also includes the Content-Type and MIME-Version headers, and I'm not sure how they're going to be related to the HTTP headers included by httplib:
Content-Type: multipart/mixed; boundary="===============2050792481=="
MIME-Version: 1.0
--===============2050792481==
Content-Type: application/octet-stream
MIME-Version: 1.0
This may be the reason that when this request is received by my web.py application, I just get an error message. The web.py POST handler:
class MultipartServer:
def POST(self, collection):
print web.input()
Throws this error:
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 242, in process
return self.handle()
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 233, in handle
return self._delegate(fn, self.fvars, args)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 415, in _delegate
return handle_class(cls)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/application.py", line 390, in handle_class
return tocall(*args)
File "/home/richard/Development/server/webservice.py", line 31, in POST
print web.input()
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/webapi.py", line 279, in input
return storify(out, *requireds, **defaults)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 150, in storify
value = getvalue(value)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 139, in getvalue
return unicodify(x)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 130, in unicodify
if _unicode and isinstance(s, str): return safeunicode(s)
File "/usr/local/lib/python2.6/dist-packages/web.py-0.34-py2.6.egg/web/utils.py", line 326, in safeunicode
return obj.decode(encoding)
File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 137-138: invalid data
My line of code is represented by the error line about half way down:
File "/home/richard/Development/server/webservice.py", line 31, in POST
print web.input()
It's coming along, but I'm not sure where to go from here. Is this a problem with my client code, or a limitation of web.py (perhaps it just can't support multipart requests)? Any hints or suggestions of alternative code libraries would be gratefully received.
EDIT
The error above was caused by the data not being automatically base64 encoded. Adding
encoders.encode_base64(base)
Gets rid of this error, and now the problem is clear. HTTP request isn't being interpreted correctly in the server, presumably because the email library is including what should be the HTTP headers in the body instead:
<Storage {'Content-Type: multipart/mixed': u'',
' boundary': u'"===============1342637378=="\n'
'MIME-Version: 1.0\n\n--===============1342637378==\n'
'Content-Type: application/octet-stream\n'
'MIME-Version: 1.0\n'
'Content-Transfer-Encoding: base64\n'
'\n0fINCs PBk1jAAAAAAAAA.... etc
So something is not right there.
Thanks
Richard

I used this package by Will Holcomb http://pypi.python.org/pypi/MultipartPostHandler/0.1.0 to make multi-part requests with urllib2, it may help you out.

After a bit of exploration, the answer to this question has become clear. The short answer is that although the Content-Disposition is optional in a Mime-encoded message, web.py requires it for each mime-part in order to correctly parse out the HTTP request.
Contrary to other comments on this question, the difference between HTTP and Email is irrelevant, as they are simply transport mechanisms for the Mime message and nothing more. Multipart/related (not multipart/form-data) messages are common in content exchanging webservices, which is the use case here. The code snippets provided are accurate, though, and led me to a slightly briefer solution to the problem.
# open an HTTP connection
h1 = httplib.HTTPConnection('localhost:8080')
# create a mime multipart message of type multipart/related
msg = MIMEMultipart("related")
# create a mime-part containing a zip file, with a Content-Disposition header
# on the section
fp = open('file.zip', 'rb')
base = MIMEBase("application", "zip")
base['Content-Disposition'] = 'file; name="package"; filename="file.zip"'
base.set_payload(fp.read())
encoders.encode_base64(base)
msg.attach(base)
# Here's a rubbish bit: chomp through the header rows, until hitting a newline on
# its own, and read each string on the way as an HTTP header, and reading the rest
# of the message into a new variable
header_mode = True
headers = {}
body = []
for line in msg.as_string().splitlines(True):
if line == "\n" and header_mode == True:
header_mode = False
if header_mode:
(key, value) = line.split(":", 1)
headers[key.strip()] = value.strip()
else:
body.append(line)
body = "".join(body)
# do the request, with the separated headers and body
h1.request("POST", "http://localhost:8080/server", body, headers)
This is picked up perfectly well by web.py, so it's clear that email.mime.multipart is suitable for creating Mime messages to be transported by HTTP, with the exception of its header handling.
My other overall conern is in scalability. Neither this solution nor the others proposed here scale well, as they read the contents of a file into a variable before bundling up in the mime message. A better solution would be one which could serialise on demand as the content is piped out over the HTTP connection. It's not urgent for me to fix that, but I'll come back here with a solution if I get to it.

There is a number of things wrong with your request. As TokenMacGuy suggests, multipart/mixed is unused in HTTP; use multipart/form-data instead. In addition, parts should have a Content-disposition header. A python fragment to do that can be found in the Code Recipes.