I'm creating script which will tell us is website is down or up. But I got error HTTPSConnectionPool(host='gooogle.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError("hostname 'gooogle.com' doesn't match 'www.google.com'")))
here is my code:
import requests
website_url = input("Enter a website url: ")
r = requests.get(f'https://{website_url}', timeout=5)
if r.status_code != 200:
print(f"uh no, {website_url} is down")
else:
print(f"Good News, {website_url} is up")
I am using the requests module for this. To be honest, I don't know how to check the website is down or up but still, I write this code if you know how please answer me with details.
First of all, you're referring to https://www.gooogle.com instead of https://www.google.com; which I assume is the webpage that you want to test.
Regarding the reason of your error:
Since you're accessing with the HTTPS protocol, you will check the certificates of the destination server, in order to verify that this one is who is supposed to be.
Apparently, the domain https://www.gooogle.com is offering the same certificate than https://www.google.com, and also it performs a redirect to it, but that domain isn't included in the Subject Alternate Name (SAN) of the Google certificate, so actually you cannot verify the authenticity of the accessed domain.
If you use your code against a site that has correctly set the SSL verification, it should work without problems. On another hand, if you just want to check if a site is UP or DOWN independently on its SSL configuration, you can just pass the flag verify set to False:
requests.get(f'https://{website_url}', verify=False, timeout=5)
Related
I'm trying to connect to one of my internal services at: https://myservice.my-alternative-domain.com through Python Requests. I'm using Python 3.6
I'm using a custom CA bundle to verify the request, and I'm getting the next error:
SSLError: hostname 'myservice.my-domain.com' doesn't match either of 'my-domain.com', 'my-alternative-domain.com'
The SSL certificate that the internal service uses has as CN: my-domain.com, and as SAN (Subject Alternative Names): 'my-domain.com', 'my-alternative-domain.com'
So, I'm trying to access the service through one of the alternative names (this has to be like this and it's not under my control)
I think the error is correct, and that the certificate should have also as SAN:
'*.my-alternative-domain.com'
in order for the request to work.
The only thing that puzzles me is that I can access the service through the browser.
Can somebody confirm the behavior of Python Requests is correct?
This is how I call the service:
response = requests.get('https://myservice.my-alternative-domain.com', params=params, headers=headers, verify=ca_bundle)
Thanks
pass verify as false might work
x=requests.get(-----,verify=false)
I am behind a proxy settings with the environment variables on the proxy settings set properly. The environment variables are working properly when doing pip installs for instance. I retrieve the proxy settings with the getproxies method, which I checked and returns the correct dict.
I am trying the following approach:
import requests
import urllib
r = requests.get('http://www.nu.nl', proxies=urllib.request.getproxies())
The error message I get:
ProxyError: HTTPSConnectionPool(host='www.nu.nl', port=443): Max retries
exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.',
OSError('Tunnel connection failed: 407 authenticationrequired',)))
What am I missing here?
Ok, figured it out. The issue with my environment variables was that the username and password were not specified because I work in a single sign on environment. Therefore the password needs to be properly set. Just substituting my credentials in the proxy definition did not work, so I had to use the urllib opener to fix my issue. Now it works like a charm.
import urllib
username = 'userID' # ex. ID
password = "password" # password
targetUrl = "http://www.example.org/"
proxies = {
'https': 'https://{}:{}#proxyAdress:port'.format(username, password)}
proxy = urllib.request.ProxyHandler(proxies)
opener = urllib.request.build_opener(proxy)
urllib.request.install_opener(opener)
with urllib.request.urlopen(targetUrl) as url:
text = str(url.read())
The HTTP 407 code states that you need to authenticate to the proxy server.
The http response header Proxy-Authenticate will tell you what kind of authentication is required - print out the response headers.
It might also be a problem with the url...try removing the www bit
For reference : https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/407
I'm trying to fetch data from Entsoe's transparency web-API on my server which is behind a proxy. However, when I run the following code I keep getting a bad handshake error. The web-API does connect from the browser.
import requests
url='https://transparency.entsoe.eu/api'
proxies = {'http':'172.26.28.25:8080', 'https':'172.26.28.25:8080'}
requests.get(url, proxies = proxies)
I keep getting error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='transparency.entsoe.eu', port=443): Max retries exceeded with url: /api?... (Caused by SSLError(SSLError("bad handshake: SysCallError(10054, 'WSAECONNRESET')",),))
How can I solve this error?
Kind regards,
Matthijs Toorenburg
If you want to initiate an API call you should supply the request parameters too.
W/o an API token Rejected TP request you'd only get Unauthorized (HTTP: 401).
You should also keep in mind, that the TP allows only a limited number of requests posted from a single IP within a specific timeframe. Id. TP query limits
In other cases, you address the host of the API, without acutally initiating an API call
(you could use API Tester as a toy example).
I've been struggling with my company proxy to make an https request.
import requests
from requests.auth import HTTPProxyAuth
proxy_string = 'http://user:password#url_proxt:port_proxy'
s = requests.Session()
s.proxies = {"http": proxy_string , "https": proxy_string}
s.auth = HTTPProxyAuth(user,password)
r = s.get('http://www.google.com') # OK
print(r.text)
r = s.get('https://www.google.com',proxies={"http": proxy_string , "https": proxy_string}) #OK
print(r.text)
r = s.get('https://www.google.com') # KO
print(r.text)
When KO, I have the following exception :
HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required',)))
I looked online but didn't find someone having this specific issue with HTTPS.
Thank you for your time
Thanks to the amazing help of Lukasa, I solved my issue.
Please see discussion on fix here
or set :
session.trust_env=False
I personally solved the above problem on my system by updating the environment variables http_proxy,https_proxy,socks_proxy,ftp_proxy.
First enter the command on your terminal : printenv
This should show you the environment variables on your system.
In my case intially:
http_proxy=http://proxyserver:port/
I changed it to : http_proxy=http://username:password#proxy:port/
using the command
export http_proxy="http://username:password#proxy:port/"
Similarly for https_proxy,socks_proxy,ftp_proxy
Other way i have resolved is - speak with your corporate IT administrator and find a direct proxy port which connects to external domain (with / without password)
pip install --proxy=http://proxyhost:proxy_port pixiedust
Found from other colleagues using the proxy (proxy_port direct connection) in their eclipse settings (network)
To anyone else that tried the accepted answer's "session.trust_env=False" with no success, there may be a deeper issue that produces a similar error (which is probably not the issue the OP had): There may be a corporate proxy configuration that requires specific headers to be sent upon CONNECT, and python requests doesn't send them ('User-Agent' and 'Host', for example).
I do not have a solution for that at the moment. See https://github.com/psf/requests/issues/5028 for a discussion on the subject.
If I run:
urllib2.urlopen('http://google.com')
even if I use another url, I get the same error.
I'm pretty sure there is no firewall running on my computer or router, and the internet (from a browser) works fine.
The problem, in my case, was that some install at some point defined an environment variable http_proxy on my machine when I had no proxy.
Removing the http_proxy environment variable fixed the problem.
The site's DNS record is such that Python fails the DNS lookup in a peculiar way: it finds the entry, but zero associated IP addresses. (Verify with nslookup.) Hence, 11004, WSANO_DATA.
Prefix the site with 'www.' and try the request again. (Use nslookup to verify that its result is different, too.)
This fails essentially the same way with the Python Requests module:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='...', port=80): Max retries exceeded with url: / (Caused by : [Errno 11004] getaddrinfo failed)
This may not help you if it's a network-level issue but you can get some debugging info by setting debuglevel on httplib. Try this:
import urllib, urllib2, httplib
url = 'http://www.mozillazine.org/atom.xml'
httplib.HTTPConnection.debuglevel = 1
print "urllib"
data = urllib.urlopen(url);
print "urllib2"
request = urllib2.Request(url)
opener = urllib2.build_opener()
feeddata = opener.open(request).read()
Which is copied directly from here, hope that's kosher: http://bytes.com/topic/python/answers/517894-getting-debug-urllib2
You probably need to use a proxy. Check your normal browser settings to find out which. Take a look at opening websites using urllib2 from behind corporate firewall - 11004 getaddrinfo failed for a similar problem with solution.,
To troubleshoot the issue:
let us know on what OS is the script running and what version of Python
In command prompt on that very same machine, do ping google.com and observe if that works (or you get say "could not find host")
If (2) worked, open browser on that machine (try in IE if on Windows) and try opening "google.com" there. If there is a problem, look closely at proxy settings in Internet Options / Connections / LAN Settings
Let us know how it goes either way.
add s to the http i.e urllib2.urlopen('https://google.com')
worked for me