I'm trying define a proxy handler to use http.client behind a proxy company. I know just how to use or define a proxy handler to urllib.:
http_proxy_full_auth_string = "http://"+"%s:%s#%s:%s" % (http_proxy_user,
http_proxy_passwd,
http_proxy_server,
http_proxy_port)
proxy_handler = urllib.request.ProxyHandler({"http": http_proxy_full_auth_string})
opener = urllib.request.build_opener(proxy_handler)
urllib.request.install_opener(opener)
resp = urllib.request.urlopen(uri).read()
And using http.client...?
P.S: sorry for the low english skills...
This might be old thread but folks may stumble upon it like I did and dont know how to authenticate.
import http.client
import base64
auth_hash = base64.b64encode(b"username:password").decode("utf-8")
conn = http.client.HTTPSConnection("proxy-ip or hostname", port="proxy-port")
conn.set_tunnel(
"example.com",
headers={"Proxy-Authorization": f"Basic {auth_hash}"})
conn.request("GET", "/")
This is how you do it with basic authentication.
See the httplib python 3 documentation
import http.client
conn = http.client.HTTPSConnection("proxy_domain", 8080)
conn.set_tunnel("www.python.org")
conn.request("HEAD","/index.html")
Related
I have a site that has PKI security enabled. Each client used either a card reader to load their certificate, or the certificate is installed in the IE certificate storage on their box.
So my question are:
How can I use either the card reader certificate or the certificate stored on the system to verify the system?
How do I pass the credentials onto the site to say, hey I'm me and I can access the service? They example can be using soft certificates. I can figure out the card reader part later.
I've been searching around, and I haven't come up with anything to help me in this situation. Django has a bunch of modules, but this isn't an option because I'm only concerned of the client side of things. I'm not creating a site to host the service. I need to just access these services.
I have this code working sort of. I just do not know how to handle the redirect I am getting:
import httplib
KEYFILE = r"C:\cert\my.key"
CERTFILE = r"c:\cert\my.pem"
HOSTNAME = 'machine.com'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = KEYFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/arcgis/sharing/rest?f=json')
conn.endheaders()
response = conn.getresponse()
print response.read()
The result of all of this is:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved here.</p>
</body></html>
Any help provided would be great!
Software specs: python 2.7.8, Windows 2012 R2
I created a PKI handler to handle the requests so I can use it work urllib2 library.
import httplib, urllib2
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
#Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host,
key_file=self.key,
cert_file=self.cert,
timeout=timeout)
To use this, you will need to use a cookiejar with the handler.
from cookielib import CookieJar
cookiejar = CookieJay()
handlers = []
handlers.append(HTTPSClientAuthHandler(somekey, somecert))
handlers.append(urllib2.HTTPCookieProcessor(cookiejar))
opener = urllib2.build_opener(*handlers)
... do other urllib2 calls ....
Hope this helps everyone!
Try this code
#!/usr/bin/env python
import httplib
CERTFILE = '/home/robr/mycert'
HOSTNAME = 'localhost'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = CERTFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/ssltest/')
conn.endheaders()
response = conn.getresponse()
print response.read()
I have to work with an API that has multiple services. All of which require the JSESSION cookie from the authentication one below. When I call the next service however, it doesn't keep the cookie and so rejects them.
from suds.client import Client
url = 'http://example/ws/Authenticate?wsdl'
client = Client(url)
result = client.service.connect(username='admin', password='admin')
print client.options.transport.cookiejar
>>> <cookielib.CookieJar[<Cookie JSESSIONID=XXXXXXXXXX for localhost.local/Service/>]>
I believe that the way to get it to keep this cookie is to extract it, then provide it as a custom header in the format: -
url = 'http://example/ws/dostuffnowloggedin?wsdl'
client2 = Client(url, headers= { 'Cookie': 'JSESSIONID=value'})
But I can't figure out how to do it. I've reviewed the SUDS Docs, URL2LIB and Cookiejar python docs, looked over stack & asked on Reddit. This is the first question I've asked on Stack, I've tried to make it meaningful and specific, but if I've commited a faux par, tell me and I'll do my best to correct it.
Try this.
from suds.client import Client
url = 'http://example/ws/Authenticate?wsdl'
client = Client(url)
result = client.service.connect(username='admin', password='admin')
url2='url of second service'
client2=Client(url2)
client2.options.transport.cookiejar=client.options.transport.cookiejar
I want to ignore the certification validation during my request to the server with an internal corporate link.
With python requests library I would do this:
r = requests.get(link, allow_redirects=False,verify=False)
How do I do the same with urllib2 library?
In the meantime urllib2 seems to verify server certificates by default.
The warning, that was shown in the past disappeared for 2.7.9 and I currently ran into this problem in a test environment with a self signed certificate (and Python 2.7.9).
My evil workaround (don't do this in production!):
import urllib2
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
urllib2.urlopen("https://your-test-server.local", context=ctx)
According to docs calling SSLContext constructor directly should work, too. I haven't tried that.
The easiest way:
python 2
import urllib2, ssl
request = urllib2.Request('https://somedomain.co/')
response = urllib2.urlopen(request, context=ssl._create_unverified_context())
python 3
from urllib.request import urlopen
import ssl
response = urlopen('https://somedomain.co', context=ssl._create_unverified_context())
For those who uses an opener, you can achieve the same thing based on Enno Gröper's great answer:
import urllib2, ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx), your_first_handler, your_second_handler[...])
opener.addheaders = [('Referer', 'http://example.org/blah.html')]
content = opener.open("https://localhost/").read()
And then use it as before.
According to build_opener and HTTPSHandler, a HTTPSHandler is added if ssl module exists, here we just specify our own instead of the default one.
According to #Enno Gröper 's post, I've tried the SSLContext constructor and it works well on my machine. code as below:
import ssl
ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
urllib2.urlopen("https://your-test-server.local", context=ctx)
if you need opener, just added this context like:
opener = urllib2.build_opener(urllib2.HTTPSHandler(context=ctx))
NOTE: all above test environment is python 2.7.12. I use PROTOCOL_SSLv23 here since the doc says so, other protocol might also works but depends on your machine and remote server, please check the doc for detail.
A more explicit example, built on Damien's code (calls a test resource at http://httpbin.org/). For python3. Note that if the server redirects to another URL, uri in add_password has to contain the new root URL (it's possible to pass a list of URLs, also).
import ssl
import urllib.parse
import urllib.request
def get_resource(uri, user, passwd=False):
"""
Get the content of the SSL page.
"""
uri = 'https://httpbin.org/basic-auth/user/passwd'
user = 'user'
passwd = 'passwd'
context = ssl.create_default_context()
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
password_mgr = urllib.request.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, uri, user, passwd)
auth_handler = urllib.request.HTTPBasicAuthHandler(password_mgr)
opener = urllib.request.build_opener(auth_handler, urllib.request.HTTPSHandler(context=context))
urllib.request.install_opener(opener)
return urllib.request.urlopen(uri).read()
urllib2 does not verify server certificate by default. Check this documentation.
Edit: As pointed out in below comment, this is not true anymore for newer versions (seems like >= 2.7.9) of Python. Refer the below ANSWER
I have this program that check a website, and I want to know how can I check it via proxy in Python...
this is the code, just for example
while True:
try:
h = urllib.urlopen(website)
break
except:
print '['+time.strftime('%Y/%m/%d %H:%M:%S')+'] '+'ERROR. Trying again in a few seconds...'
time.sleep(5)
By default, urlopen uses the environment variable http_proxy to determine which HTTP proxy to use:
$ export http_proxy='http://myproxy.example.com:1234'
$ python myscript.py # Using http://myproxy.example.com:1234 as a proxy
If you instead want to specify a proxy inside your application, you can give a proxies argument to urlopen:
proxies = {'http': 'http://myproxy.example.com:1234'}
print("Using HTTP proxy %s" % proxies['http'])
urllib.urlopen("http://www.google.com", proxies=proxies)
Edit: If I understand your comments correctly, you want to try several proxies and print each proxy as you try it. How about something like this?
candidate_proxies = ['http://proxy1.example.com:1234',
'http://proxy2.example.com:1234',
'http://proxy3.example.com:1234']
for proxy in candidate_proxies:
print("Trying HTTP proxy %s" % proxy)
try:
result = urllib.urlopen("http://www.google.com", proxies={'http': proxy})
print("Got URL using proxy %s" % proxy)
break
except:
print("Trying next proxy in 5 seconds")
time.sleep(5)
Python 3 is slightly different here. It will try to auto detect proxy settings but if you need specific or manual proxy settings, think about this kind of code:
#!/usr/bin/env python3
import urllib.request
proxy_support = urllib.request.ProxyHandler({'http' : 'http://user:pass#server:port',
'https': 'https://...'})
opener = urllib.request.build_opener(proxy_support)
urllib.request.install_opener(opener)
with urllib.request.urlopen(url) as response:
# ... implement things such as 'html = response.read()'
Refer also to the relevant section in the Python 3 docs
Here example code guide how to use urllib to connect via proxy:
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})
# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
urllib.request.CacheFTPHandler)
# install it
urllib.request.install_opener(opener)
f = urllib.request.urlopen('http://www.google.com/')
"""
For http and https use:
proxies = {'http':'http://proxy-source-ip:proxy-port',
'https':'https://proxy-source-ip:proxy-port'}
more proxies can be added similarly
proxies = {'http':'http://proxy1-source-ip:proxy-port',
'http':'http://proxy2-source-ip:proxy-port'
...
}
usage
filehandle = urllib.urlopen( external_url , proxies=proxies)
Don't use any proxies (in case of links within network)
filehandle = urllib.urlopen(external_url, proxies={})
Use proxies authentication via username and password
proxies = {'http':'http://username:password#proxy-source-ip:proxy-port',
'https':'https://username:password#proxy-source-ip:proxy-port'}
Note: avoid using special characters such as :,# in username and passwords
What's the best way to specify a proxy with username and password for an http connection in python?
This works for me:
import urllib2
proxy = urllib2.ProxyHandler({'http': 'http://
username:password#proxyurl:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)
conn = urllib2.urlopen('http://python.org')
return_str = conn.read()
Use this:
import requests
proxies = {"http":"http://username:password#proxy_ip:proxy_port"}
r = requests.get("http://www.example.com/", proxies=proxies)
print(r.content)
I think it's much simpler than using urllib. I don't understand why people love using urllib so much.
Setting an environment var named http_proxy like this: http://username:password#proxy_url:port
The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don't want to embed the proxy password in the url or the python source code (unless it's just a quick hack).
import urllib2
def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, proxyurl, proxyuser, proxypass)
proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)
return urllib2.build_opener(proxy_handler, proxy_auth_handler)
if __name__ == "__main__":
import sys
if len(sys.argv) > 4:
url_opener = get_proxy_opener(*sys.argv[1:4])
for url in sys.argv[4:]:
print url_opener.open(url).headers
else:
print "Usage:", sys.argv[0], "proxy user pass fetchurls..."
In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.
Or if you want to install it, so that it is always used with urllib2.urlopen (so you don't need to keep a reference to the opener around):
import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()
Here is the method use urllib
import urllib.request
# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})
# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
urllib.request.CacheFTPHandler)
# install it
urllib.request.install_opener(opener)
f = urllib.request.urlopen('http://www.python.org/')
"""