Hi I have written a few simple lines of code. But I seem to be getting a Authentication error. Can anyone please suggest , what credentials are being looked for python here ?
Code:
import urllib2
response = urllib2.urlopen('http://google.com')
html = response.read()
Error
urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required
PS: I do not have acces to IE -->Advanced settings or regedit
As advised I've modified the code :
import urllib2
proxy_support = urllib2.ProxyHandler({'http':r'http://usename:psw#IP:port'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy_support, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://google.com')
html = response.read()
Also I have created two environment variables :
HTTP_PROXY = http://username:password#proxyserver.domain.com
HTTPS_PROXY = https://username:password#proxyserver.domain.com
But still getting the error .
urllib2.HTTPError: HTTP Error 407: Proxy Authentication Required
There are multiple ways to work-around your problem. You may want to try defining an environment variables with the names http_proxy and https_proxy with each set to you proxy URL. Refer to this link for more details.
Alternatively, you may want to explicitly define a ProxyHandler to work with urllib2 while handling requests through the proxy. The link is already present within the comment to your query; however I am including it here for the sake of completeness.
Hope this helps
If your OS is windows and behind ISA proxy, urllib2 does not use any information about proxy; instead "Firewall Client for ISA server" automatically authenticates the user. That means we don't need to set http_proxy and https_proxy system environment variables. Keep it empty in ProxyHandler as following:
proxy = urllib2.ProxyHandler({})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
u = urllib2.urlopen('your-url-goes-here')
data = u.read()
The error code and message seem that the username and password failed to pass the authentications of proxy servers.
The following code:
proxy_handler = urllib2.ProxyHandler({'http': 'usename:psw#IP:port'})
opener = urllib2.build_opener(proxy_handler)
urllib2.install_opener(opener)
response = urllib2.urlopen('http://google.com')
html = response.read()
should also works if the authentication is passed.
Related
I'm working on a webscrape function that's going to be pulling HTML data from internal (non public) servers. I have a connection through a VPN and proxy server so when I ping any public site I get code 200 no problem, but our internals are returning 401.
Heres my code:
http_str = f'http://{username}:{password}#proxy.yourorg.com:80'
proxyDict = {
'http' : http_str,
'https' : https_str,
'ftp' : https_str
}
html_text = requests.get(url, verify=True, proxies=proxyDict, auth=HTTPBasicAuth(user, pwd))
I've tried flushing my DNS server, using different certificate chains (that had a whole new list of problems). I'm using urllib3 on version 1.23 because that seemed to help with SSL errors. I've considered using a requests session but I'm not sure what that would change.
Also, the url's we're trying to access DO NOT require a log in. I'm not sure why its throwing 401 errors but the auth is for the proxy server, I think. Any help or idea are appreciated, along with questions as at this point I'm not even sure what to ask to move this along.
Edit: the proxyDict has a string with the user and pwd passed it for each type, https http fts, etc.
To use HTTP Basic Auth with your proxy, use the http://user:password#host/ syntax in any of the proxy configuration entries. See apidocs.
import requests
proxyDict = {
"http": "http://username:password#proxy.yourorg.com:80",
"https": "http://username:password#proxy.yourorg.com:80"
}
url = 'http://myorg.com/example'
response = requests.get(url, proxies=proxyDict)
If, however, you are accessing internal URLs via VPN (i.e., internal to your organization on your intranet) then you should NOT need the proxy to access them.
Try:
import requests
url = 'http://myorg.com/example'
response = requests.get(url, verify=False)
I have TP Link router (WR841N).I want to login into my TP link router and needs to change primary and secondary DNSusing script.
I tried to login using below script but not succeeded:
import urllib2
import urllib
import cookielib
def main():
userName = 'admin'
pcPassword = 'admin'
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'userName' : userName, 'pcPassword' : pcPassword})
resp = opener.open('http://192.168.0.1/userRpm/LoginRpm.htm', login_data)
print(resp.read())
if __name__ == '__main__':
main()
And then how to change primary and secondary dns using script.
CookieProcessor doesn't set POST header, obviously.
You need to set Content-Type and Content-Length to match your login_data.
I would recommend you to install the opener you built using urllib2.install_opener(), and then use request:
r = urllib2.Request('http://192.168.0.1/userRpm/LoginRpm.htm')
r.add_header("Content-Type", "application/x-www-form-urlencoded")
r.add_header("Content-Length", str(len(login_data)))
r.add_data(login_data)
u = urllib2.urlopen(r)
print u.read()
u.close()
Then you have to continue with filling all other forms to change what you want.
If cookies aren't managed by javascript, you will be able to do it. If they are, perhaps even then if you examine the code carefully and extract cookie results manually from javascript code. I did it before.
But, yeah, SSH or telnet or rlogin would be easier than HTTP. To continue using HTTP, take a look at Requests package, it can be helpful, and make your code smaller. It includes session managing for you.
Adding urlencoded type to content-type might not help if login form has enctype attribute set to something else. (plain text or multipart).
I don't think that'll be a case, but if it is you can still do it with a bit more work.
I am currently getting the following error when trying to use a proxy I got from ProxyMesh:
407 proxy authorization required
I'm using Nokogiri and a rotating set of agents to access a URL.. The code looks like:
url = Nokogiri::HTML(open(address, :proxy => 'http://555.XXX.2.203:XXXXX',
"User-Agent" => "#{aliases[0]}"))
There is a setup needed in my app where I pass in my user and password, but they don't have a page explaining it for Ruby.. Here is the example in Python if anyone can translate??
>>> import requests
>>> auth = requests.auth.HTTPProxyAuth('USERNAME', 'PASSWORD')
>>> proxies = {'http': 'http://aa-aa.proxymesh.com:12345'}
>>> response = requests.get('http://example.com', proxies=proxies, auth=auth)
Here is the page on ProxyMesh with the other languages..
ProxyMesh Explained
It might be that you didn't set up the IP address in the ProxyMesh dashboard. I had the same error as yours and it fixed when I did that.
I have a perl program that retrieves data from the database of my university library and it works well. Now I want to rewrite it in python but encounter the problem
<urlopen error [errno 104] connection reset by peer>
The perl code is:
my $ua = LWP::UserAgent->new;
$ua->cookie_jar( HTTP::Cookies->new() );
$ua->timeout(30);
$ua->env_proxy;
my $response = $ua->get($url);
The python code I wrote is:
cj = CookieJar();
request = urllib2.Request(url); # url: target web page
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj));
opener = urllib2.install_opener(opener);
data = urllib2.urlopen(request);
I use VPN(virtual private network) to log in my university's library at home, and I tried both the perl code and python code. The perl code works as I expected, but the python code always encountered "urlopen error".
I googled for the problem and it seems that the urllib2 fails to load the environmental proxy. But according to the document of urllib2, the urlopen() function works transparently with proxies which do not require authentication. Now I feels quite confusing. Can anybody help me with this problem?
I tried faking the User-Agent headers as Uku Loskit and Mikko Ohtamaa suggested, and solved my problem. The code is as follows:
proxy = "YOUR_PROXY_GOES_HERE"
proxies = {"http":"http://%s" % proxy}
headers={'User-agent' : 'Mozilla/5.0'}
proxy_support = urllib2.ProxyHandler(proxies)
opener = urllib2.build_opener(proxy_support, urllib2.HTTPHandler(debuglevel=1))
urllib2.install_opener(opener)
req = urllib2.Request(url, None, headers)
html = urllib2.urlopen(req).read()
print html
Hope it is useful for someone else!
Firstly, as Steve said, you need response.read(), but that's not your problem
import urllib2
response = urllib2.urlopen('http://python.org/')
html = response.read()
Can you give details of the error? You can get it like this:
try:
urllib2.urlopen(req)
except URLError, e:
print e.code
print e.read()
Source: http://www.voidspace.org.uk/python/articles/urllib2.shtml
(I put this in a comment but it ate my newlines)
You might find that the requests module is a much easier-to-use replacement for urllib2.
Did you try specifying the proxy manually?
proxy = urllib2.ProxyHandler({'http': 'your_proxy_ip'})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)
urllib2.urlopen('http://www.uni-database.com')
if it still fails, try faking your User-Agent headers so as to make it seem that the request is coming from a real browser.
I've been reading about Python's urllib2's ability to open and read directories that are password protected, but even after looking at examples in the docs, and here on StackOverflow, I can't get my script to work.
import urllib2
# Create an OpenerDirector with support for Basic HTTP Authentication...
auth_handler = urllib2.HTTPBasicAuthHandler()
auth_handler.add_password(realm=None,
uri='https://webfiles.duke.edu/',
user='someUserName',
passwd='thisIsntMyRealPassword')
opener = urllib2.build_opener(auth_handler)
# ...and install it globally so it can be used with urlopen.
urllib2.install_opener(opener)
socks = urllib2.urlopen('https://webfiles.duke.edu/?path=/afs/acpub/users/a')
print socks.read()
socks.close()
When I print the contents, it prints the contents of the login screen that the url I'm trying to open will redirect you to. Anyone know why this is?
auth_handler is only for basic HTTP authentication. The site here contains a HTML form, so you'll need to submit your username/password as POST data.
I recommend you using the mechanize module that will simplify the login for you.
Quick example:
import mechanize
browser = mechanize.Browser()
browser.open('https://webfiles.duke.edu/?path=/afs/acpub/users/a')
browser.select_form(nr=0)
browser.form['user'] = 'username'
browser.form['pass'] = 'password'
req = browser.submit()
print req.read()