We Keep Coding

Create a functioning Response object

For testing purposes I'm trying to create a Response() object in python but it proves harder then it sounds.
i tried this:
from requests.models import Response
the_response = Response()
the_response.code = "expired"
the_response.error_type = "expired"
the_response.status_code = 400
but when I attempted the_response.json() i got an error because the function tries to get len(self.content) and a.content is null.
So I set a._content = "{}" but then I get an encoding error, so I have to change a.encoding, but then it fails to decode the content....
this goes on and on. Is there a simple way to create a Response object that's functional and has an arbitrary status_code and content?

That because the _content attribute on the Response objects (on python3) has to be bytes and not unicodes.
Here is how to do it:
from requests.models import Response
the_response = Response()
the_response.code = "expired"
the_response.error_type = "expired"
the_response.status_code = 400
the_response._content = b'{ "key" : "a" }'
print(the_response.json())

Create a mock object, rather than trying to build a real one:
from unittest.mock import Mock
from requests.models import Response
the_response = Mock(spec=Response)
the_response.json.return_value = {}
the_response.status_code = 400
Providing a spec ensures that the mock will complain if you try to access methods and attributes a real Response doesn't have.

Just use the responses library to do it for you:
import responses
#responses.activate
def test_my_api():
responses.add(responses.GET, 'http://whatever.org',
json={}, status=400)
...
This has the advantage that it intercepts a real request, rather than having to inject a response somewhere.

Another approach by using the requests_mock library, here with the provided fixture:
import requests
def test_response(requests_mock):
requests_mock.register_uri('POST', 'http://test.com/', text='data', headers={
'X-Something': '1',
})
response = requests.request('POST', 'http://test.com/', data='helloworld')
...

python function .json(): No JSON object could be decoded

the function looks like this:
import requests
import json
def parse(s):
r = requests.post('http://166.111.139.15:9000/?properties%3d%7b%22annotators%22%3a%22tokenize%2cssplit%2cpos%2clemma%2cparse%22%2c%22outputFormat%22%3a%22json%22%7d%0a', data=s)
return r.json()
print parse("I am a student")
And when I use it in Django, the web page shows:
"No JSON object could be decoded".Why?

No JSON object could be decoded is an exception message raised at r.json(). If your response is not a valid json object you can still retrieve it with r.text. Even if you are sure your response is always a valid json object, you should still check whether server returned a success code. If there's an internal server error (code 500), you won't get a valid json response!
import requests
def parse(s)
r = requests.post('http://someserver.com', data=s)
if r.status_code !== 200:
return "There was a problem: {} !".format(r.text)
return r.json()

How do I parse a JSON response from Python Requests?

I am trying to parse a response.text that I get when I make a request using the Python Requests library. For example:
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
self.response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
print self.response.text
When I execute this method, the output is:
{"request":"POST /login","result":"success"}
I would like to check whether "result" equals "success", ignoring whatever comes before.

The manual suggests: if self.response.status_code == requests.codes.ok:
If that doesn't work:
if json.loads(self.response.text)['result'] == 'success':
whatever()

Since the output, response, appears to be a dictionary, you should be able to do
result = self.response.json().get('result')
print(result)
and have it print
'success'

If the response is in json you could do something like (python3):
import json
import requests as reqs
# Make the HTTP request.
response = reqs.get('http://demo.ckan.org/api/3/action/group_list')
# Use the json module to load CKAN's response into a dictionary.
response_dict = json.loads(response.text)
for i in response_dict:
print("key: ", i, "val: ", response_dict[i])
To see everything in the response you can use .__dict__:
print(response.__dict__)

import json
def check_user(self):
method = 'POST'
url = 'http://localhost:5000/login'
ck = cookielib.CookieJar()
response = requests.request(method,url,data='username=test1&passwd=pass1', cookies=ck)
#this line converts the response to a python dict which can then be parsed easily
response_native = json.loads(response.text)
return self.response_native.get('result') == 'success'

I found another solution. It is not necessary to use json module. You can create a dict using dict = eval(whatever) and return, in example, dict["result"]. I think it is more elegant. However, the other solutions also work and are correct

Put in the return of your method like this:
return self.response.json()
If you wanna looking for more details, click this following link:
https://www.w3schools.com/python/ref_requests_response.asp
and search for json() method.
Here is an code example:
import requests
url = 'https://www.w3schools.com/python/demopage.js'
x = requests.get(url)
print(x.json())

In some cases, maybe the response would be as expected. So It'd be great if we can built a mechanism to catch and log the exception.
import requests
import sys
url = "https://stackoverflow.com/questions/26106702/how-do-i-parse-a-json-response-from-python-requests"
response = requests.get(url)
try:
json_data = response.json()
except ValueError as exc:
print(f"Exception: {exc}")
# to find out why you have got this exception, you can see the response content and header
print(str(response.content))
print(str(response.headers))
print(sys.exc_info())
else:
if json_data.get('result') == "success":
# do whatever you want
pass

Return a requests.Response object from Flask

I'm trying to build a simple proxy using Flask and requests. The code is as follows:
#app.route('/es/<string:index>/<string:type>/<string:id>',
methods=['GET', 'POST', 'PUT']):
def es(index, type, id):
elasticsearch = find_out_where_elasticsearch_lives()
# also handle some authentication
url = '%s%s%s%s' % (elasticsearch, index, type, id)
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare())
return resp.text
This works, except that it loses the status code from Elasticsearch. I tried returning resp (a requests.models.Response) directly, but this fails with
TypeError: 'Response' object is not callable
Is there another, simple, way to return a requests.models.Response from Flask?

Ok, found it:
If a tuple is returned the items in the tuple can provide extra information. Such tuples have to be in the form (response, status, headers). The status value will override the status code and headers can be a list or dictionary of additional header values.
(Flask docs.)
So
return (resp.text, resp.status_code, resp.headers.items())
seems to do the trick.

Using text or content property of the Response object will not work if the server returns encoded data (such as content-encoding: gzip) and you return the headers unchanged. This happens because text and content have been decoded, so there will be a mismatch between the header-reported encoding and the actual encoding.
According to the documentation:
In the rare case that you’d like to get the raw socket response from the server, you can access r.raw. If you want to do this, make sure you set stream=True in your initial request.
and
Response.raw is a raw stream of bytes – it does not transform the response content.
So, the following works for gzipped data too:
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare(), stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
If you use a shortcut method such as get, it's just:
resp = requests.get(url, stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()

Flask can return an object of type flask.wrappers.Response.
You can create one of these from your requests.models.Response object r like this:
from flask import Response
return Response(
response=r.reason,
status=r.status_code,
headers=dict(r.headers)
)

I ran into the same scenario, except that in my case my requests.models.Response contained an attachment. This is how I got it to work:
return send_file(BytesIO(result.content), mimetype=result.headers['Content-Type'], as_attachment=True)

My use case is to call another API in my own Flask API. I'm just propagating unsuccessful requests.get calls through my Flask response. Here's my successful approach:
headers = {
'Authorization': 'Bearer Muh Token'
}
try:
response = requests.get(
'{domain}/users/{id}'\
.format(domain=USERS_API_URL, id=hit['id']),
headers=headers)
response.raise_for_status()
except HTTPError as err:
logging.error(err)
flask.abort(flask.Response(response=response.content, status=response.status_code, headers=response.headers.items()))

Making a POST call instead of GET using urllib2

There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.
I'm trying to do a simple POST call to a service:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content
I can see the server logs and it says that I'm doing GET calls, when I'm sending the data
argument to urlopen.
The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).

Do it in stages, and modify the object, like this:
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
else:
# handle the error case. connection.read() will still contain data
# if any was returned, but it probably won't be of any use
This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.

This may have been answered before: Python URLLib / URLLib2 POST.
Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.

Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age' : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()
As suggested by #Michael Kent do consider requests, it's great.
EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.

The requests module may ease your pain.
url = 'http://myserver/post_service'
data = dict(name='joe', age='10')
r = requests.post(url, data=data, allow_redirects=True)
print r.content

it should be sending a POST if you provide a data parameter (like you are doing):
from the docs:
"the HTTP request will be a POST instead of a GET when the data parameter is provided"
so.. add some debug output to see what's up from the client side.
you can modify your code to this and try again:
import urllib
import urllib2
url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = opener.open(url, data=data).read()

Try this instead:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content

url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()
May be this can help