How to escape # character in the password of proxy. So that python can create the request correctly. I have tried \\ but still not able to hit the url correctly.
proxy = {
"http": "http://UserName:PassWord#X.X.X.X:Port_No"
}
Update question:
I am using python requests module for the http request. It split the string (to get host) from first occurrence of # where as it was suppose to split from second #.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='XXXXXXXX#X.X.X.X', port=XXXXX): Max retries exceeded with url: http:/URL (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
You have to do urlencoding like in this post:
Escaping username characters in basic auth URLs
This way the # in the PW becomes %40
You don't mention which library you are using to perform your HTTP requests, so you should consider using requests, not only to solve this problem, but because it is a great library.
Here is how to use a proxy with basic authentication:
import requests
proxy = {'http': 'http://UserName:PassWord#X.X.X.X:Port_No'}
r = requests.get("http://whereever.com", proxies=proxy)
Update
Successfully tested with requests and proxy URLs:
http://UserName:PassWord#127.0.0.1:1234
http://UserName:PassWord##127.0.0.1:1234
http://User#Name:PassWord#1234#127.0.0.1:1234
If, instead, you need to use Python's urllib2 library, you can do this:
import urllib2
handler = urllib2.ProxyHandler({'http': 'http://UserName:PassWord#X.X.X.X:Port_No'})
opener = urllib2.build_opener(handler)
r = opener.open('http://whereever.com')
Note that in neither case is it necessary to escape the #.
A third option is to set environment variables HTTP_PROXY and/or HTTPS_PROXY (in *nix).
Related
I want to move my bash code which uses a cURL command to a Python 2.7 script.
The cURL working command is:
$ curl --data "vm_id='52e4130d-ffe0-495a-87c0-fc84200252ed'&gpu_ip='10.2.0.22'&gpu_port='8308'&mock_ip='10.254.254.254'&mock_port='8308'" http://rodvr-services:8080/rodvr-assign_gpu
And my Python script contains this:
import requests
import requests.packages.urllib3
requests.packages.urllib3.disable_warnings()
payload = {'vm_id': '52e4130d-ffe0-495a-87c0-fc84200252ed', 'gpu_ip': '10.2.0.22', 'gpu_port': '8308', 'mock_ip': '10.254.254.254', 'mock_port': '8308'}
r = requests.get('http://rodvr-services:8080/rodvr-assign_gpu', params=payload)
When I execute the script, I get the following error:
$ python exec.py
Traceback (most recent call last):
File "exec.py", line 9, in <module>
r = requests.post('http://rodvr-services:8080/rodvr-assign_gpu', params=payload)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 502, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 612, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('\n',))
Just in case, I checked what would happen using Python 3, and this is the output:
HTTPConnectionPool(host='rodvr-services', port=8080): Max retries exceeded with url: /rodvr-assign_gpu?mock_ip=10.254.254.254&vm_id=52e4130d-ffe0-495a-87c0-fc84200252ed&gpu_ip=10.2.0.22&mock_port=8308&gpu_port=8308 (Caused by <class 'http.client.BadStatusLine'>:
However, using the urllib2 library, it works:
data = "vm_id='52e4130d-ffe0-495a-87c0-fc84200252ed'&gpu_ip='10.2.0.22'&gpu_port='8308'&mock_ip='10.254.254.254'&mock_port='8308'"
r = urllib2.Request(url='http://rodvr-services:8080/rodvr-assign_gpu', data=data)
f = urllib2.urlopen(r)
print f.read()
Try r = requests.post('http://rodvr-services:8080/rodvr-assign_gpu', data=payload)
This website helps you to convert your curl command to python code.
You can see the code suggested by that website below:
import requests
data = [
('vm_id', '\'52e4130d-ffe0-495a-87c0-fc84200252ed\''),
('gpu_ip', '\'10.2.0.22\''),
('gpu_port', '\'8308\''),
('mock_ip', '\'10.254.254.254\''),
('mock_port', '\'8308\''),
]
requests.post('http://rodvr-services:8080/rodvr-assign_gpu', data=data)
# it is slightly different from your code
Due to my personal problems with my laptop, I can't test your code. hope this works for you.
I am trying to send a request to an API I have set up on an AWS machine.
The code I use is as follows:
import requests
import json
report_dict = {
"client_name": "Wayne Enterprises",
"client_id": 123,
"report_type": "api_testing",
"timestamp_generated": "2015-07-29T11:00:00Z",
"report_data": {"revenue": 9000.00}
}
report_json = json.dumps(report_dict)
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=False)
Doing this, I get:
Traceback (most recent call last):
File "art2_java_test.py", line 124, in <module>
main()
File "art2_java_test.py", line 9, in main
test_post_good_data()
File "art2_java_test.py", line 29, in test_post_good_data
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=Fal
se)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\adapters
.py", line 428, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
90)
But when I send the request as http instead of https, it (usually) works fine. I've found some evidence that this can have to do with proxy servers, but I am not using one. Are there any other potential reasons for this error? This is a website only available on my company's local network, if that's relevant.
.... https://my-url.com:8080/my-api/reports
...But when I send the request as http instead of https, it (usually) works fine.
My guess is that you are trying the same port 8080 for http and https. But, servers usually listen on a single port either for http or https and not both. This means that if your client is trying to start the TLS handshake needed for https against this port it will get a plain error message back. The client then tries to interpret this error message as TLS and returns some weird error messages, because the response is not TLS at all.
I want to debug some python requests using charles proxy.
I need to include the certificate for charles on the call, but is not working
import requests
endpoint_url = 'https://www.httpsnow.org/'
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
print "empexo"
print r
I have added the https address on Charles,
I get on Charles:
SSLHandshake: Remote host closed connection during handshake
and on python the log with error
empexo
Traceback (most recent call last):
File "/Users/iosdev/DopPy/GetCelebs.py", line 15, in <module>
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Process finished with exit code 1
I found this thread while I was troubleshooting a similar issue. In the scenario I ran into the cert argument was being used to define the path to a ".crt" file when the verify argument should have been used instead.
The correct usage ended up looking like:
requests.get(endpoint_url, verify='/path/to/file.crt')
See Requests' documentation for more details: https://2.python-requests.org/en/v1.1.0/user/advanced/#ssl-cert-verification
As an aside, I find employing Request's ability to specify the path to a ".crt" via the REQUESTS_CA_BUNDLE environmental variable more effective when using Charles Proxy for local debugging.
Running something like the following in shell saves having to specify the path to Charles' ".crt" for every Requests call:
REQUESTS_CA_BUNDLE=/path/to/file.crt
export REQUESTS_CA_BUNDLE
I am trying to get headers of url using python using http://docs.python-requests.org/en/latest/ this tutorial. I am trying following code in python idle , I am getting following error,
>>> import requests
>>> r = requests.get('https://api.github.com/user')
Traceback (most recent call last):
File "<pyshell#32>", line 1, in <module>
r = requests.get('https://api.github.com/user')
File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\api.py", line 55, in get
return request('get', url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.3.0-py2.7.egg\requests\adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='api.github.com', port=443): Max retries exceeded with url: /user (Caused by <class 'socket.error'>: [Errno 10013] An attempt was made to access a socket in a way forbidden by its access permissions)
Looks like github is denying you access to the requested page. Before attempting to request pages in python try typing them into the browser to see what is returned. When I did this I was returned some JSON stating
{
"message": "Requires authentication",
"documentation_url": "https://developer.github.com/v3"
}
If you want to test your code and find headers of a webpage, try a publicly accessible webpage before delving into APIs.
Authenticated proxy through python-requests returns the following error:
>>> import requests
>>> proxies = {'https': 'http://username:password#proxy.company.com:8080',}
>>> requests.get('https://api.github.com/',proxies=proxies,verify=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='proxy.company.com', port=8080): Max retries exceeded with url: https://api.github.com/ (Caused by <class 'httplib.BadStatusLine'>: )
Authenticated proxy works fine in curl:
$ curl --proxy-user username:password --proxy http://proxy.company.com:8080 -k https://api.github.com/
{
"current_user_url": "https://api.github.com/user",
"authorizations_url": "https://api.github.com/authorizations",
"emails_url": "https://api.github.com/user/emails",
"emojis_url": "https://api.github.com/emojis",
"events_url": "https://api.github.com/events",
"feeds_url": "https://api.github.com/feeds",
"following_url": "https://api.github.com/user/following{/target}",
"gists_url": "https://api.github.com/gists{/gist_id}",
"hub_url": "https://api.github.com/hub",
"issue_search_url": "https://api.github.com/legacy/issues/search/{owner}/{repo}/{state}/{keyword}",
"issues_url": "https://api.github.com/issues",
"keys_url": "https://api.github.com/user/keys",
"notifications_url": "https://api.github.com/notifications",
"organization_repositories_url": "https://api.github.com/orgs/{org}/repos/{?type,page,per_page,sort}",
"organization_url": "https://api.github.com/orgs/{org}",
"public_gists_url": "https://api.github.com/gists/public",
"rate_limit_url": "https://api.github.com/rate_limit",
"repository_url": "https://api.github.com/repos/{owner}/{repo}",
"repository_search_url": "https://api.github.com/legacy/repos/search/{keyword}{?language,start_page}",
"current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
"starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
"starred_gists_url": "https://api.github.com/gists/starred",
"team_url": "https://api.github.com/teams",
"user_url": "https://api.github.com/users/{user}",
"user_organizations_url": "https://api.github.com/user/orgs",
"user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
"user_search_url": "https://api.github.com/legacy/user/search/{keyword}"
}
How do I troubleshoot?
Requests has support for HTTP proxies just since 2.0: https://github.com/kennethreitz/requests/pull/1515
BTW, it does respect proxy environment variables, so you can just set:
export http_proxy="http://username:password#proxy.company.com:8080"
export https_proxy=$http_proxy
More details about the state of proxy support in requests 1.x can be found in this blog post: https://lukasa.co.uk/2013/07/Python_Requests_And_Proxies/
Your proxy is indeed HTTP not HTTPS. To use proxy with HTTPS it needs to use connect protocol, not HTTP, with presumably CONNECT basic auth. Now I do not have a HTTPS proxy with auth available, and do not know how to install such system, but what happens if you have just "https://username:password#proxy.company.com:8080" there instead?
On the other hand, you might want to make sure that requests/urllib3 are up to date. See this bug report.