The code below sets up _request if the HTTP method is GET, then it has an ifstatement for handling PUT POST and PATCH.
I'm trying to have one single request setup statement for all method types.
Is this possible? It appears to me that there is no way to add data to a prepared request, and if this is true then perhaps I'm stuck with needing two different ways of setting up a request, one way for GET and one way for PUT, PATCH and POST.
def fetch_from_api(self):
s = Session()
headers = { "Authorization" : REST_API_AUTHORIZATION_HEADER}
_request = Request(self.method, self.url_for_api, headers=headers)
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
_request = Request(self.method, self.url_for_api, headers=headers, data=self.postdata)
prepped = _request.prepare()
self.api_response = s.send(prepped)
The question is a little old and hopefully #DukeDougal already has a solution. Maybe this will help others, though.
The first thing I notice in the example is that a Request object is created near the beginning of the method. Then, if the method is "POST", "PATCH", or "PUT", the Request constructor is called again to get another object. In that case, the first object is gone. It was created unnecessarily.
When a data= argument isn't given to the Request constructor, it's the same as specifying data=None. Take advantage of that and call the constructor only once, then a data value won't need to be added to an existing Request (or PreparedRequest) object:
def fetch_from_api(self):
s = Session()
headers = {'Authorization': REST_API_AUTHORIZATION_HEADER}
data = None # Assume no data until method is checked
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
data = self.postdata # Add the data
# Now headers and data are ready, get a Request object
_request = Request(self.method, self.url_for_api, headers=headers, data=data)
prepped = _request.prepare()
self.api_response = s.send(prepped)
If you look at the requests.Request model, it looks like you can set the data attribute if needed:
some_request = Request(method, url, headers=headers)
if # ...we decide we need to add data:
some_request.data = data
Looking at the model, it appears that this would work, because when you prepare the request later on, it looks at the instance's data attribute.
EDIT:
But reading your question a bit more closely, it looks like you want to add data to a prepared_request. I guess you could create your own prepared_request and pass the data in specifically when you call the prepare method, but I don't see how that helps? It seems like you want to just branch and maybe add data or maybe not?
Anyway, the above seems it could potentially simplify your code slightly to the following:
def fetch_from_api(self):
s = Session()
headers = { "Authorization" : REST_API_AUTHORIZATION_HEADER}
_request = Request(self.method, self.url_for_api, headers=headers)
if self.method in ['POST', 'PATCH', 'PUT']:
headers['content-type'] = 'application/x-www-form-urlencoded'
_request.data = self.postdata
prepped = _request.prepare()
self.api_response = s.send(prepped)
(But that doesn't look much simpler to me. What are we trying to achieve? Also, it seems weird to have a method called fetch_from_api that could also be POSTing or PUTing data. As a dev, I would not be expecting that to be the case from the name.)
In the past, I've done stuff like this as a result of having to sign requests: I have to create them in one place and then hand them off to a class that knows how to create signatures, which then hands them back. In other words, you can certainly edit requests before preparing and sending them on their way.
Anyway, I haven't tried any of this, but it's similar to some things I've done in the past with requests, so it looks legit, but I would be concerned about what you are attempting to achieve and whether or not things are being crammed together which maybe should not be.
I am using HTTPforHumans, requests module.
import requests
def pv_request(url, methods, data=None, headers=None, type=None):
try:
if 'POST' in methods:
return requests.post(url=url, headers=headers, data=data).json()
elif 'GET' in methods:
return requests.get(url=url, headers=headers, data=data).json()
elif 'PUT' in methods:
if type == 'music':
return requests.put(url=url, headers=headers, data=data).json()
elif type == 'image':
return requests.put(url=url, headers=headers, data=open(data, 'rb')).json()
except requests.exceptions.ConnectionError:
return None
Might not be in the lines of what you are looking for, but here is my all-in-on purpose request handler.
Related
I have a function like this:
def get_some_data(api_url, **kwargs)
# some logic on generating headers
# some more logic
response = requests.get(api_url, headers, params)
return response
I need to create a fake/mock "api_url", which, when made request to, would generate a valid response.
I understand how to mock the response:
def mock_response(data):
response = requests.Response()
response.status_code = 200
response._content = json.dumps(data)
return response
But i need to make the test call like this:
def test_get_some_data(api_url: some_magic_url_path_that_will_return_mock_response):
Any ideas on how to create an url path returning a response within the scope of the test (only standard Django, Python, pytest, unittest) would be very much appreciated
The documentation is very well written and more than clear on how to mock whatever you want. But, let say you have a service that makes the 3rd party API call:
def foo(url, params):
# some logic on generating headers
# some more logic
response = requests.get(url, headers, params)
return response
In your test you want to mock the return value of this service.
#patch("path_to_service.foo")
def test_api_call_response(self, mock_response):
mock_response.return_value = # Whatever the return value you want it to be
# Here you call the service as usual
response = foo(..., ...)
# Assert your response
Consider an http request using an OAuth token. The access token needs to be included in the header as bearer. However, if the token is expired, another request needs to be made to refresh the token and then try again. So the custom Retry object will look like:
s = requests.Session()
### token is added to the header here
s.headers.update(token_header)
retry = OAuthRetry(
total=2,
read=2,
connect=2,
backoff_factor=1,
status_forcelist=[401],
method_whitelist=frozenset(['GET', 'POST']),
session=s
)
adapter = HTTPAdapter(max_retries=retry)
s.mount('http://', adapter)
s.mount('https://', adapter)
r = s.post(url, data=data)
The Retry class:
class OAuthRetry(Retry):
def increment(self, method, url, *args, **kwargs):
# refresh the token here. This could be by getting a reference to the session or any other way.
return super(OAuthRetry, self).increment(method, url, *args, **kwargs)
The problem is that after the token is refreshed, HTTPConnectionPool is still using the same headers to make the request after calling increment. See: https://github.com/urllib3/urllib3/blob/master/src/urllib3/connectionpool.py#L787.
Although the instance of the pool is passed in increment, changing the headers there will not affect the call since it is using a local copy of the headers.
This seems like a use case that should come up frequently for the request parameters to change in between retries.
Is there a way to change the request headers in between two subsequent retries?
No, in current version of Requests(2.18.4) and urllib3(1.22).
Retrys is finally handled by openurl in urllib3. And by trace the code of the whole function, there is not a interface to change headers between retrys.
And dynamically changing headers should not be considered as a solution. From the doc:
headers – Dictionary of custom headers to send, such as User-Agent, If-None-Match, etc. If None, pool headers are used. If provided, these headers completely replace any pool-specific headers.
headers is a param passed to the function. And there is no guarantee that it will not be copy after passed. Although in current version of urllib3, openurl does not copy headers, any solution based on changing headers is considered hacky, since it's based on the implementation but not the documentation.
One work around
Interrupt a function and edit some verible it's using is very dangerous.
Instead of injecting something into urllib3, one simple solution is that check the response status and try again if needed.
r = s.post(url, data=data)
if r.status_code == 401:
# refresh the token here.
r = s.post(url, data=data)
Why does the original approach not work?
Requests copy the header in prepare_headers before sending it to urllib3. So urllib3 use the copy created before editing when retrying.
Within my unitest when I attempt the following POST; I get the params within the request.base_url. I would like to have it within request.form. How do I achieve that?
self.taskqueue_stub = apiproxy_stub_map.apiproxy.GetStub('taskqueue')
tasks = self.taskqueue_stub.GetTasks("postios")
self.assertEqual(len(tasks), 1)
task = tasks[0]
params = base64.b64decode(task["body"])
headers['Content-Type'] = 'application/json'
response = self.client.post(task["url"], params, headers=headers)
I found a way to pass it as data: request.data, but thats not good enough.
response = self.client.post(task["url"], data=params, headers=headers)
The reason I have to do this, is the way I add the task in my code.
taskqueue.Task(url='/worker',
params={"json_records": jsonified_task_records,
"user": user.key.urlsafe()}
).add(queue_name='postios')
Hence within the /worker view I expect to find the params in request.form.
I ended up using payload instead of params.
params = {"json_records": jsonified_task_records,
"user": user.key.urlsafe()}
taskqueue.Task(url='/worker',
payload=json.dumps(params)
).add(queue_name='postios')
Now I always have it as request.data within my /worker view so that both unit tests and production code can expect to find it there.
Beware that payloads is unicode in this case and nested json structure needs to be converted once again via json.loads.
e.g. /worker view:
jsons = json.loads(request.data)
user_hash = jsons['user']
json_records = json.loads(jsons['json_records']) # jsons['json_records'] is unicode
I'm trying to build a simple proxy using Flask and requests. The code is as follows:
#app.route('/es/<string:index>/<string:type>/<string:id>',
methods=['GET', 'POST', 'PUT']):
def es(index, type, id):
elasticsearch = find_out_where_elasticsearch_lives()
# also handle some authentication
url = '%s%s%s%s' % (elasticsearch, index, type, id)
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare())
return resp.text
This works, except that it loses the status code from Elasticsearch. I tried returning resp (a requests.models.Response) directly, but this fails with
TypeError: 'Response' object is not callable
Is there another, simple, way to return a requests.models.Response from Flask?
Ok, found it:
If a tuple is returned the items in the tuple can provide extra information. Such tuples have to be in the form (response, status, headers). The status value will override the status code and headers can be a list or dictionary of additional header values.
(Flask docs.)
So
return (resp.text, resp.status_code, resp.headers.items())
seems to do the trick.
Using text or content property of the Response object will not work if the server returns encoded data (such as content-encoding: gzip) and you return the headers unchanged. This happens because text and content have been decoded, so there will be a mismatch between the header-reported encoding and the actual encoding.
According to the documentation:
In the rare case that you’d like to get the raw socket response from the server, you can access r.raw. If you want to do this, make sure you set stream=True in your initial request.
and
Response.raw is a raw stream of bytes – it does not transform the response content.
So, the following works for gzipped data too:
esreq = requests.Request(method=request.method, url=url,
headers=request.headers, data=request.data)
resp = requests.Session().send(esreq.prepare(), stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
If you use a shortcut method such as get, it's just:
resp = requests.get(url, stream=True)
return resp.raw.read(), resp.status_code, resp.headers.items()
Flask can return an object of type flask.wrappers.Response.
You can create one of these from your requests.models.Response object r like this:
from flask import Response
return Response(
response=r.reason,
status=r.status_code,
headers=dict(r.headers)
)
I ran into the same scenario, except that in my case my requests.models.Response contained an attachment. This is how I got it to work:
return send_file(BytesIO(result.content), mimetype=result.headers['Content-Type'], as_attachment=True)
My use case is to call another API in my own Flask API. I'm just propagating unsuccessful requests.get calls through my Flask response. Here's my successful approach:
headers = {
'Authorization': 'Bearer Muh Token'
}
try:
response = requests.get(
'{domain}/users/{id}'\
.format(domain=USERS_API_URL, id=hit['id']),
headers=headers)
response.raise_for_status()
except HTTPError as err:
logging.error(err)
flask.abort(flask.Response(response=response.content, status=response.status_code, headers=response.headers.items()))
There's a lot of stuff out there on urllib2 and POST calls, but I'm stuck on a problem.
I'm trying to do a simple POST call to a service:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = urllib2.urlopen(url=url, data=data).read()
print content
I can see the server logs and it says that I'm doing GET calls, when I'm sending the data
argument to urlopen.
The library is raising an 404 error (not found), which is correct for a GET call, POST calls are processed well (I'm also trying with a POST within a HTML form).
Do it in stages, and modify the object, like this:
# make a string with the request type in it:
method = "POST"
# create a handler. you can specify different handlers here (file uploads etc)
# but we go for the default
handler = urllib2.HTTPHandler()
# create an openerdirector instance
opener = urllib2.build_opener(handler)
# build a request
data = urllib.urlencode(dictionary_of_POST_fields_or_None)
request = urllib2.Request(url, data=data)
# add any other information you want
request.add_header("Content-Type",'application/json')
# overload the get method function with a small anonymous function...
request.get_method = lambda: method
# try it; don't forget to catch the result
try:
connection = opener.open(request)
except urllib2.HTTPError,e:
connection = e
# check. Substitute with appropriate HTTP code.
if connection.code == 200:
data = connection.read()
else:
# handle the error case. connection.read() will still contain data
# if any was returned, but it probably won't be of any use
This way allows you to extend to making PUT, DELETE, HEAD and OPTIONS requests too, simply by substituting the value of method or even wrapping it up in a function. Depending on what you're trying to do, you may also need a different HTTP handler, e.g. for multi file upload.
This may have been answered before: Python URLLib / URLLib2 POST.
Your server is likely performing a 302 redirect from http://myserver/post_service to http://myserver/post_service/. When the 302 redirect is performed, the request changes from POST to GET (see Issue 1401). Try changing url to http://myserver/post_service/.
Have a read of the urllib Missing Manual. Pulled from there is the following simple example of a POST request.
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe', 'age' : '10'})
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
print response.read()
As suggested by #Michael Kent do consider requests, it's great.
EDIT: This said, I do not know why passing data to urlopen() does not result in a POST request; It should. I suspect your server is redirecting, or misbehaving.
The requests module may ease your pain.
url = 'http://myserver/post_service'
data = dict(name='joe', age='10')
r = requests.post(url, data=data, allow_redirects=True)
print r.content
it should be sending a POST if you provide a data parameter (like you are doing):
from the docs:
"the HTTP request will be a POST instead of a GET when the data parameter is provided"
so.. add some debug output to see what's up from the client side.
you can modify your code to this and try again:
import urllib
import urllib2
url = 'http://myserver/post_service'
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
content = opener.open(url, data=data).read()
Try this instead:
url = 'http://myserver/post_service'
data = urllib.urlencode({'name' : 'joe',
'age' : '10'})
req = urllib2.Request(url=url,data=data)
content = urllib2.urlopen(req).read()
print content
url="https://myserver/post_service"
data["name"] = "joe"
data["age"] = "20"
data_encoded = urllib2.urlencode(data)
print urllib2.urlopen(url + "?" + data_encoded).read()
May be this can help