Within our server we've got this piece a code calling a function inside my APP like this:
data = urllib.urlencode( dict(api_key="a-key-goes-here") )
headers = {
"User-Agent" : "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.2 Safari/537.36",
"Accept" : "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,text/png,*/*;q=0.5",
"Accept-Language" : "en-US,en;q=0.8",
"Accept-Charset" : "ISO-8859-1,utf-8",
"Content-type": "application/x-www-form-urlencoded; charset=UTF-8"
}
request = urllib2.Request(url, data, headers)
response = urllib2.urlopen(request)
code = response.code
message = response.read()
response.close()
I know that this is not using url_for( neither other ways to call a url trough your APP but this has a reason. Our server is just testing that the call goes correctly but the url is expected to be outside our APP and so is the api key.
So, our server handling url looks like this:
#app.route('/loan_performer', methods=["GET", "POST"])
def loan_performer():
if 'api_key' in request.form and request.form['api_key'] == API_KEY:
ret = dict()
# rate1 return a random number between 3.000 and 4.000 and point1 will be 0
ret['rate_one'] = random.randint(3000, 4000)
ret['point_one'] = 0
# rate2 do it between 3.500 and 4.500, point2 being 0.5
ret['rate_two'] = random.randint(3500, 4500)
ret['point_two'] = 0.5
# rate3 between 4.000 and 5.000 with 1.0
ret['rate_three'] = random.randint(4000, 5000)
ret['point_three'] = 1.0
return json.dumps(ret), 200
else:
return u"Your API Key is invalid.", 403
Our error is as the title says:
We are constantly receiving the error "Bad request (GET and HEAD requests may not contain a request body)" Which is a 404 error handled by Passenger WSGI for Apache. But in other words, for a reason request.form is empty and it shouldn't.
Is there anything I'm doing wrong or any other way to POST data from Flask to outside?
If there is needed more info I'm willing to update, just call it.
Edit 1
I switched to use requests like this:
dat = dict( api_key="a-key-goes-here" )
request = requests.post(url, data=dat)
message = request.text
code = request.status_code
And it refuses to work. This is the logger I made in the handling url function:
(08/02/2014 07:07:20 AM) INFO This is message ImmutableMultiDict([]) ImmutableMultiDict([]) and this code 403
request.args - request.form - request.data , all of them are empty, either using urllib or requests.
Update on Edit 1
After removing "GET" from methods suggested by Martin konecny I got this response:
(08/02/2014 08:35:06 AM) INFO This is message <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>405 Method Not Allowed</title>
<h1>Method Not Allowed</h1>
<p>The method is not allowed for the requested URL.</p>
and this code 405
But in other words, for a reason request.form is empty and it shouldn't.
I don't think you can infer this conclusion. What appears to be happening is you have a GET request with a POST header.
From the Python docs:
data may be a string specifying additional data to send to the server, or None if no such data is needed. Currently HTTP requests are the only ones that use data; the HTTP request will be a POST instead of a GET when the data parameter is provided. data should be a buffer in the standard application/x-www-form-urlencoded format. The urllib.urlencode() function takes a mapping or sequence of 2-tuples and returns a string in this format.
You need to remove the header
"Content-type": "application/x-www-form-urlencoded; charset=UTF-8"
when you are sending requests with an empty data structure, or just remove it alltogether (it will be added automatically by urllib when needed)
Related
I am trying to play around with REST Api. I was practicing to write a server-side and client-side python script, where client will upload a file in the server. The following is server and client side code in python.
# server.py
from flask import Flask, request
app = Flask(__name__)
#app.route("/api/upload", methods=["POST"])
def upload_image():
# Get the image file from the request body
image_file = request.files["image"]
# Save the image file to a specific location
image_file.save("images/image11.jpg")
return "Image uploaded successfully", 200
if __name__ == "__main__":
app.run(debug=True)
# client.py
import requests
# Set the API endpoint URL
url = "http://127.0.0.1:5000/api/upload"
# Set the image file to be uploaded
image_file = "C:\\Users\\user\\Desktop\\Image\\an.jpg"
# Set the request headers
headers = {
"Content-Type": "multipart/form-data"
}
image_file_descriptor = open(image_file, 'rb')
# Requests makes it simple to upload Multipart-encoded files
files = {'image': image_file_descriptor}
# Send the POST request with the image file as a key-value pair
response = requests.post(url, files=files)
# Print the response status code
print(response.status_code)
This code works fine. The image saved into the mentioned directory in the name of image11.jpg and also returned 200. However, if I initiate the POST request in client.py with header like this - requests.post(url, files=files, headers=headers) then it shows following error -
400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
KeyError: 'image'
I tried to remove the "Content-Type": "multipart/form-data" from headers, tried to pass the headers with nothing written inside and writing "Authorization": "Bearer 1514" inside. It was working in both situation.
I also tried to send the request from tools like Postman/Advanced REST client (ARC). It also worked and the file transferred successfully and returned 200.
I have tried find out some solutions out there. Tried to check similar questions on StackOverflow. The following one seems to be a similar one-
Link: Python multipart upload not taking content-type
But failed to find the exact reason. Can anyone help me to figure out the reason.Thanks in advance.
It's hard to get to the bottom of that by reading Flask's code, will just point why Flask complains/errors.
If you add print request.headers in your method, you will see what headers Flask receives.
If you don't set the header "Content-Type": "multipart/form-data", requests will automatically add this one to the request:
Content-Type: multipart/form-data; boundary=072c2c6e071d5dbc3423739ceb4e048b
If you set the header, then the boundary portion is lost, and Flask complains.
If you use web form to upload a file like:
#app.route("/api/upload", methods=["GET", "POST"])
def upload_image():
if request.method == "POST":
.....
else:
return '''
<!doctype html>
<title>Upload new File</title>
<h1>Upload new File</h1>
<form method=post enctype=multipart/form-data>
<input type=file name=image>
<input type=submit value=Upload>
</form>
'''
and upload from your browser, you will see the boundary part also present:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: multipart/form-data; boundary=---------------------------57875131084340261312514205
Content-Length: 85295
In summary: I'm not sure of the details, but Flask/werkzeug requires the boundary in that header, which is populated by the clients (even Python's requests).
I am trying to sign into facebook with python requests.
When I run the following code:
import requests
def get_facebook_cookie():
sign_in_url = "https://www.facebook.com/login.php?login_attempt=1"
#need to post: email , pass
payload = {"email":"xxxx#xxxx.com", "pass":"xxxxx"}
headers = {"accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8", "user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36"}
s = requests.Session()
r1 = s.get(sign_in_url, headers = headers, timeout = 2)
r = s.post(sign_in_url, data = payload, headers = headers, timeout = 2, cookies = r1.cookies)
print r.url
text = r.text.encode("ascii", "ignore")
fbhtml = open("fb.html", "w")
fbhtml.write(text)
fbhtml.close()
return r.headers
print get_facebook_cookie()
( note the url is supposed to redirect to facebook.com-- still does not do that)
facebook returns the following error:
(the email is actually populated in the email box -- so I know it is passing it in atleast)
According to the requests session documentation it handles all of the cookies, so I do not even think I need to be passing it in. However, I have seen other do it elsewhere in order to populate the second request with an empty cookie so I gave it a shot.
The question is, why is facebook telling me I do not have cookies enabled? Is there some extra request header I need to pass in? Would urllib2 be a better choice for something like this?
it looks like according to this answer Login to Facebook using python requests
there is more data that needs to be sent in, in order to successfully sign into facebook.
In my Werkzeug application I am intercepting all error responses and trying to respond with a JSON response if the client expects JSON or return the usual HTML page with 404 or 500:
def handle_error_response(self, environ, start_response, exc):
if ('application/json' in environ.get('CONTENT_TYPE', '')
and exc.get_response().content_type != 'application/json'):
start_response('%s %s' % (exc.code, exc.name),
(('Content-Type', 'application/json'), ))
return (json.dumps({"success": False, "error": exc.description}, ensure_ascii=False), )
# go the regular path
...
In this solution I am relying on Content-Type header containing string'application/json'.
However this doesn't look like a correct solution, because Wikipedia says:
Content-Type The MIME type of the body of the request (used with POST and PUT requests)
Is it a good strategy to check if 'text/html' is inside header Accept and then return HTML response otherwise return JSON response?
Any other more robust solutions?
When Chrome requests an HTML page header
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
is sent, when Ember makes an API request
Accept: application/json, text/javascript, */*; q=0.01
is sent.
Maybe X-Requested-With: XMLHttpRequest should be taken into account?
You should probably add AcceptMixin to your request object.
Once you do this, you can use the accept_mimetypes.accept_json, accept_mimetypes.accept_html and accept_mimetypes.accept_xhtml attributes on your request object. The response's default content type really depends only on what your application is; just try and imagine which would result in less confusion.
This works for us:
if ('text/html' not in environ.get('HTTP_ACCEPT', '')
and 'application/json' not in response.content_type):
# the user agent didn't explicitely request html, so we return json
... # make the JSON response
I.e. if the client expects html -- do not return json. Otherwise return
json response, if the response isn't already json.
I am writing a Python 2.7 script using Requests to automate access to a particular website. The website has a requirement that a Referer header matching the request URL is provided, for "security reasons". The URL is built up from a number of items in a params dict, passed to requests.post().
Is there a way to determine what the URL that Requests will use is, prior to making the request, so that the Referer header can be set to this correct value? Let's assume that I have a lot of parameters:
params = { 'param1' : value1, 'param2' : value2, # ... etc
}
base_url = "http://example.com"
headers = { 'Referer' : url } # but what is 'url' to be?
requests.post(base_url, params=params, headers=headers) # fails as Referer does not match final url
I suppose one workaround is to issue the request and see what the URL is, after the fact. However there are two problems with this - 1. it adds significant overhead to the execution time of the script, as there will be a lot of such requests, and 2. it's not actually a useful workaround because the server actually redirects the request to another URL, so reading it afterwards doesn't give the correct Referer value.
I'd like to note that I have this script working with urllib/urllib2, and I am attempting to write it with Requests to see whether it is possible and perhaps simpler. It's not a complicated process the script has to follow, but it may perhaps be slightly beyond the scope of Requests. That's fine, I'd just like to confirm that this is the case.
I think I found a solution, based on Prepared Requests. The concept is that Session.prepare_request() will do everything to prepare the request except send it, which allows my script to then read the prepared request's url, which now includes the parameters whose order are determined by the dict order. Then it can set the Referer header appropriately and then issue the original request.
params = {'param1' : value1, 'param2' : value2, # ... etc
}
url = "http://example.com"
# Referer must be correct
# To determine correct Referer url, prepare a request without actually sending it
req = requests.Request('POST', url, params=params)
prepped = session.prepare_request(req)
#r = session.send(prepped) # don't actually send it
# add the Referer header by examining the prepared url
headers = { 'Referer': prepped.url }
# now send normally
r = session.post(url, params=params, data=data, headers=headers)
it looks like you've found correctly the prepare_request feature in Requests.
However, if you still wanted to use your initial method, I believe you could use your base_url as your Referer:
base_url = "http://example.com"
headers = { 'Referer' : base_url }
requests.post(base_url, params=params, headers=headers)
I suspect this because your POST has the PARAMS directly attached to the base_url. If, for example, you were on:
http://www.example.com/trying-to-send-upload/
adding some params to this POST, you would then use:
referer = "http://www.example.com/trying-to-send-something/"
headers = { 'Referer' : referer, 'Host' : 'example.com' }
requests.post(referer, params=params, headers=headers)
ADDED
I would check my URL visually by using a simple statement after you've created the URL string:
print(post_url)
If this is good, you should print out the details of the reply from the server you're posting to, as it might also give you some hints as to why your query was rejected:
s = requests.post(referer, params=params, headers=headers)
print(s.status_code)
print(s.text)
Love to hear if this works as well for you.
I am new to Web requests . I saw a piece of code that does HTML PDF conversion like this :
headers = {'content-type': 'text/html', 'accept': 'application/pdf'}
urllib2.Request(url, data=html, headers=headers) # html is a string and it works fine
The url does the pdf conversion and it needs html as input .
Why is 'data' keyword argument so important ? Why can't be clubbed as just another param ?
I would have thought that urllib2.Request(url, params = {'data': html}) where data is just one of the key value pairs . And server does it processing accordingly .
Why do we need 'data' as something seperate from other parameters ?
Is it because we specify 'content-type' in the header and it bound to the data keyword as a convention ?
I am writing an API that makes everything is request like a keyword arguement , for a simple library purpose . So I would like to know when is data required and when is not as well . I do understand params but 'data' is that mandatory or only for post requests where you have a specific content-type to sent to server? What if I have multiple content types now ?
When the data attribute is provided, the request is sent as POST. It is not mandatory, it can be None, if it is none (or not provided) it is sent as GET. This is all described here: http://docs.python.org/2/library/urllib2.html#urllib2.Request
Does request also have the same convention ? I ask so because in
request we have request.get . So request.get(url, data=something)
would be converted to a POST ? And how is this data seen at the server
side any idea ?
request.get(url, data="test") would be sent as a GET request with "test" as the body of the request. This is the raw HTTP request:
GET /headers HTTP/1.1\r\nHost: httpbin.org\r\nContent-Length: 4\r\nAccept-Encoding: gzip, deflate, compress\r\nAccept: */*\r\nUser-Agent: python-requests/2.2.1 CPython/2.7.5 Windows/7\r\n\r\ntest
Formatted:
GET /headers HTTP/1.1
Host: httpbin.org
Content-Length: 4
Accept-Encoding: gzip, deflate, compress
Accept: */*
User-Agent: python-requests/2.2.1 CPython/2.7.5 Windows/7
test
The server will in most cases just ignore it.