I am trying to use the Python requests module to issue Http GET commands to access some REST based APIs. The urls are working fine on a RESTClient but when I use the same url in python, I get a connection error.
The code I am trying to execute is:
payload={"mac":new_mac,"token":token}
userloginurl="http://192.168.1.40:9119/uid"
r=requests.get(userloginurl,params=payload)
print(r.url)
If I test this url using RESTClient, I get a 200 OK status code in the response header along with some more fields.
But this is not working with python requests. The traceback of the error is shown below:
Traceback (most recent call last):
File "getAPids.py", line 34, in <module>
r=requests.get(userloginurl,params=payload)
File "C:\Users\garvitab\python\lib\site-packages\requests\api.py", line 65, in
get
return request('get', url, **kwargs)
File "C:\Users\garvitab\python\lib\site-packages\requests\api.py", line 49, in
request
response = session.request(method=method, url=url, **kwargs)
File "C:\Users\garvitab\python\lib\site-packages\requests\sessions.py", line 4
61, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\garvitab\python\lib\site-packages\requests\sessions.py", line 5
73, in send
r = adapter.send(request, **kwargs)
File "C:\Users\garvitab\python\lib\site-packages\requests\adapters.py", line 4
15, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",
))
I looked up for the cause of the problem. Possibly, the response received is not formatted correctly. Is there a way to handle this problem?
Thanks in advance.
The problem was with the url. This connection was meant to be established over https and I was using http in the python script. Hence the issue.
Have you actually checked, what gets send over the wire? I suppose you might have to convert your dictionary to a JSON string by yourself, or use the json= keyword instead of payload=. See http://docs.python-requests.org/en/latest/user/quickstart/#custom-headers for details. This may do the trick:
import json
payload = json.dumps({"mac":new_mac,"token":token})
userloginurl = "http://192.168.1.40:9119/uid"
r = requests.get(userloginurl, data=payload)
print(r.url)
I was getting the same error and spent hours on it. I found that you cannot call the flask server in a client using "Localhost". It has to be "127.0.0.1"
#server
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello World'
if __name__ == '__main__':
app.run(debug=True)
Now the client causing the error:
#client
import requests
x = requests.get('http://localhost:5000') # change it to 127.0.0.1
print(x.text)
Related
I want to move my bash code which uses a cURL command to a Python 2.7 script.
The cURL working command is:
$ curl --data "vm_id='52e4130d-ffe0-495a-87c0-fc84200252ed'&gpu_ip='10.2.0.22'&gpu_port='8308'&mock_ip='10.254.254.254'&mock_port='8308'" http://rodvr-services:8080/rodvr-assign_gpu
And my Python script contains this:
import requests
import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()
payload = {'vm_id': '52e4130d-ffe0-495a-87c0-fc84200252ed', 'gpu_ip': '10.2.0.22', 'gpu_port': '8308', 'mock_ip': '10.254.254.254', 'mock_port': '8308'}
r = requests.get('http://rodvr-services:8080/rodvr-assign_gpu', params=payload)
When I execute the script, I get the following error:
$ python exec.py
Traceback (most recent call last):
File "exec.py", line 9, in <module>
r = requests.post('http://rodvr-services:8080/rodvr-assign_gpu', params=payload)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 502, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 612, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\n',))
Just in case, I checked what would happen using Python 3, and this is the output:
HTTPConnectionPool(host='rodvr-services', port=8080): Max retries exceeded with url: /rodvr-assign_gpu?mock_ip=10.254.254.254&vm_id=52e4130d-ffe0-495a-87c0-fc84200252ed&gpu_ip=10.2.0.22&mock_port=8308&gpu_port=8308 (Caused by <class 'http.client.BadStatusLine'>:
However, using the urllib2 library, it works:
data = "vm_id='52e4130d-ffe0-495a-87c0-fc84200252ed'&gpu_ip='10.2.0.22'&gpu_port='8308'&mock_ip='10.254.254.254'&mock_port='8308'"
r = urllib2.Request(url='http://rodvr-services:8080/rodvr-assign_gpu', data=data)
f = urllib2.urlopen(r)
print f.read()
Try r = requests.post('http://rodvr-services:8080/rodvr-assign_gpu', data=payload)
This website helps you to convert your curl command to python code.
You can see the code suggested by that website below:
import requests
data = [
('vm_id', '\'52e4130d-ffe0-495a-87c0-fc84200252ed\''),
('gpu_ip', '\'10.2.0.22\''),
('gpu_port', '\'8308\''),
('mock_ip', '\'10.254.254.254\''),
('mock_port', '\'8308\''),
]
requests.post('http://rodvr-services:8080/rodvr-assign_gpu', data=data)
# it is slightly different from your code
Due to my personal problems with my laptop, I can't test your code. hope this works for you.
I'm running a webserver based on the Bottle framework. This framework exposes an endpoint where you can upload a file. Sometimes, if the disk is full, the webserver is supposed to return a code 429 without reading the file. For some reason, when I'm uploading a file, if the server tries to return a 429 status code the following exception is raised:
File "../hooks.py", line 325, in post_video_content
response = requests.post(url, data=fh)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 110, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 473, in send
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(32, 'Broken pipe'))
But if try the exact same post but instead of posting a file I simply post some small amount of data, I actually get a response object back from the request and I can inspect the status code.
To reproduce:
webserver.py
from bottle import Bottle, route, run, request, response, get, post, put, abort
#post('/content')
#authenticate
def content():
response.status = 429
return "failed to write incoming movie to temp file (space might be full temporarily)"
run(host="127.0.0.1", port=8080, debug=True)
send_request.py
import requests
# this works and prints the status code
ret = requests.post("127.0.0.1:8080/content", data="sdfsdfsdfsdf")
print ret.status_code
# this part will throw an exception
try:
with open("~/some_large_video_file.mp4", 'rb') as fh:
ret = requests.post('127.0.0.1:8080/content', data=fh)
except:
print traceback.format_exc()
EDIT - I've found out that it depends on the amount of data. If you try and post something like 10KB of data we get the 429 return code. I'm trying to figure out what the limit is that causes it to start raising an exception.
EDIT2 - So it looks like the magic filesize is somewhere between 117K and 131K. If I try the former it works as intended, I get the response from the request and can access the status code. If I try the former I get the exception raised.
My questions are:
Why does this happen? Is it a bug in bottle? A bug in requests? Why would the size/type of data I'm posting change the response?
Is there any way to get around this? I'm trying to associate a ConnectionError exception with the server being down, and a 429 response code meaning the disk is full. If I'm getting the same exception for both I can't tell the difference between the situations which blocks me from implementing a back-off to wait for disk space to open up
So I cross-posted this to the requests github issue page, and it looks like the issue is in the httplib module that requests is built on top of, so this is a no-fix for now.
see: https://github.com/kennethreitz/requests/issues/4062
I am trying to send a request to an API I have set up on an AWS machine.
The code I use is as follows:
import requests
import json
report_dict = {
"client_name": "Wayne Enterprises",
"client_id": 123,
"report_type": "api_testing",
"timestamp_generated": "2015-07-29T11:00:00Z",
"report_data": {"revenue": 9000.00}
}
report_json = json.dumps(report_dict)
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=False)
Doing this, I get:
Traceback (most recent call last):
File "art2_java_test.py", line 124, in <module>
main()
File "art2_java_test.py", line 9, in main
test_post_good_data()
File "art2_java_test.py", line 29, in test_post_good_data
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=Fal
se)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\adapters
.py", line 428, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
90)
But when I send the request as http instead of https, it (usually) works fine. I've found some evidence that this can have to do with proxy servers, but I am not using one. Are there any other potential reasons for this error? This is a website only available on my company's local network, if that's relevant.
.... https://my-url.com:8080/my-api/reports
...But when I send the request as http instead of https, it (usually) works fine.
My guess is that you are trying the same port 8080 for http and https. But, servers usually listen on a single port either for http or https and not both. This means that if your client is trying to start the TLS handshake needed for https against this port it will get a plain error message back. The client then tries to interpret this error message as TLS and returns some weird error messages, because the response is not TLS at all.
How to escape # character in the password of proxy. So that python can create the request correctly. I have tried \\ but still not able to hit the url correctly.
proxy = {
"http": "http://UserName:PassWord#X.X.X.X:Port_No"
}
Update question:
I am using python requests module for the http request. It split the string (to get host) from first occurrence of # where as it was suppose to split from second #.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='XXXXXXXX#X.X.X.X', port=XXXXX): Max retries exceeded with url: http:/URL (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
You have to do urlencoding like in this post:
Escaping username characters in basic auth URLs
This way the # in the PW becomes %40
You don't mention which library you are using to perform your HTTP requests, so you should consider using requests, not only to solve this problem, but because it is a great library.
Here is how to use a proxy with basic authentication:
import requests
proxy = {'http': 'http://UserName:PassWord#X.X.X.X:Port_No'}
r = requests.get("http://whereever.com", proxies=proxy)
Update
Successfully tested with requests and proxy URLs:
http://UserName:PassWord#127.0.0.1:1234
http://UserName:PassWord##127.0.0.1:1234
http://User#Name:PassWord#1234#127.0.0.1:1234
If, instead, you need to use Python's urllib2 library, you can do this:
import urllib2
handler = urllib2.ProxyHandler({'http': 'http://UserName:PassWord#X.X.X.X:Port_No'})
opener = urllib2.build_opener(handler)
r = opener.open('http://whereever.com')
Note that in neither case is it necessary to escape the #.
A third option is to set environment variables HTTP_PROXY and/or HTTPS_PROXY (in *nix).
I want to debug some python requests using charles proxy.
I need to include the certificate for charles on the call, but is not working
import requests
endpoint_url = 'https://www.httpsnow.org/'
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
print "empexo"
print r
I have added the https address on Charles,
I get on Charles:
SSLHandshake: Remote host closed connection during handshake
and on python the log with error
empexo
Traceback (most recent call last):
File "/Users/iosdev/DopPy/GetCelebs.py", line 15, in <module>
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Process finished with exit code 1
I found this thread while I was troubleshooting a similar issue. In the scenario I ran into the cert argument was being used to define the path to a ".crt" file when the verify argument should have been used instead.
The correct usage ended up looking like:
requests.get(endpoint_url, verify='/path/to/file.crt')
See Requests' documentation for more details: https://2.python-requests.org/en/v1.1.0/user/advanced/#ssl-cert-verification
As an aside, I find employing Request's ability to specify the path to a ".crt" via the REQUESTS_CA_BUNDLE environmental variable more effective when using Charles Proxy for local debugging.
Running something like the following in shell saves having to specify the path to Charles' ".crt" for every Requests call:
REQUESTS_CA_BUNDLE=/path/to/file.crt
export REQUESTS_CA_BUNDLE