Automatize checking if a link is live using python - python

I need to often manually check if some links are live. To avoid this I would like to write a small script that just returns the https status of all the links so I can immediately see any 404s. Problem is I have no idea what i'm doing I just know it should be possible :D
I tried to do this with a tester page using python's requests and I get the following error
import requests
requests.get('https://exp04.zih.tu-dresden.de/')
HTTPSConnectionPool(host='exp04.zih.tu-dresden.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
I understand the individual words of the error message but now nothign about server/client stuff. How do I return the status code for this page?

import requests
requests.get('https://exp04.zih.tu-dresden.de/', verify=False)
Just ignore the SSL certificate verifying.

Related

Locust endpoint authentication failing unless another endpoint is also called

so I was trying to load test a Kerberos authenticated endpoint using the below locustfile (details removed):
from locust import HttpUser, TaskSet, task
from requests_kerberos import HTTPKerberosAuth
class UserBehaviour(TaskSet):
#task
def method1(self):
self.client.post("/method1", auth=HTTPKerberosAuth(force_preemptive=True), json={})
class FilterAndPrioritiseUser(HttpUser):
tasks = [UserBehaviour]
Then I continually get an error saying SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)'), even though I can manually hit the endpoint fine.
However, if I add another task:
#task(1)
def method2(self):
self.client.get("/endpoint2", verify=False)
Then the results look like this:
Type Name # Requests # Fails
GET /endpoint2 6 6
POST /endpoint1 19 4
Where the errors are:
# fails Method Name Type
6 GET /method2 HTTPError('401 Client Error: Unauthorized for url')
4 POST /method1 SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)')
This makes no sense, as why does this other endpoint being hit and failing for the being unauthorised cause the original endpoint to stop failing after a few retries?
Any help would be very appreciated as I'm very confused!
Turns out I needed to add certs to the default cert file that's used by Python, which I found this using this other post. After doing that, the requests all passed authentication as expected!

Python ERROR SSL: WRONG_VERSION_NUMBER on code that worked few days earlier

Using google-search package (https://pypi.org/project/googlesearch-python/) to open few google search results straight from cmd. Worked fine few days ago.
Now that trying to launch the code I get SSL: WRONG_VERSION_NUMBER Error
Any ideas how to fix? And if ideas what caused the error that would be interesting.
Code:
from googlesearch import search
import webbrowser, sys
searching_for = input(("Input search words: "))
num_results = int(input("How many results : ") or "3")
result = search(searching_for)
for i in result[:num_results]:
webbrowser.open(i)
ERROR
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /search?q=tim+ferriss&num=11&hl=en (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)'))))
I was able to reproduce the error and fix it using the following trick.
try changing the next line of code to this:
result = search(searching_for,proxy="http")
Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy',
It says that you can try using HTTP on the URL. But the search library that you are using, I don't see any options for changing the protocol........

SSL Error while connecting to external websites

I am trying to download Landsat scenes using the library 'landsatxplore' in python using the following code:
from landsatxplore.earthexplorer import EarthExplorer
ee = EarthExplorer(username, password)
ee.download(scene_id='LT51960471995178MPS00', output_dir=r'C:\Users\user1\Desktop\landsat')
ee.logout()
After running the above code I am getting following error:
SSLError: HTTPSConnectionPool(host='ers.cr.usgs.gov', port=443): Max retries exceeded with url: /login/ (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1051)')))
I am facing the same issue while downloading sentinel scenes too while using the Sentinel Python API.
I am working on a system in which proxy is configured. In the environment variables of my system I have added both HTTP_PROXY & HTTPS_PROXY. Even my '.condarc' file located at "C:\Users\user1" looks like:
channels:
- defaults
show_channel_urls: True
allow_other_channels: True
proxy_servers:
http: username:password#xx.xx.xx.xx:8080
https: username:password#xx.xx.xx.xx:8080
ssl_verify: True
Kindly let me know what could be the issue and how to solve this problem.

SSLError( '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),)) in Python while accessing an url [duplicate]

This question already has an answer here:
SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed
(1 answer)
Closed 4 years ago.
I am writing a script that will will update a macro enabled Excel file that is present in a given url.I am using Python3.6 for this work. I decided it to first download in a local copy then update the local copy and after updating push it back to the same url. But when I am writing code to download the file I am getting the error as-
(Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
the code that I am using is-
import requests
url = 'https://sharepoint.amr.ith.intel.com/sites/SKX/patchboard/Shared%20Documents/Forms/AllItems.aspx?RootFolder=%2Fsites%2FSKX%2Fpatchboard%2FShared%20Documents%2FReleaseInfo&FolderCTID=0x0120004C1C8CCA66D8D94FB4D7A0D2F56A8DB7&View={859827EF-6A11-4AD6-BD42-23F385D43AD6}/Copy of Patch_Release_Utilization'
r = requests.get(url)
open('Excel.xlsm', 'wb').write(r.content)
I have tried solution given in-Python requests SSL error - certificate verify failed
,but this is not working for me. How to resolve this problem?? Please help me with the solution if somebody has already tackled it.
EDIT:
I have tried using-
r=request.get(url,verify=False)
After doing this I am getting the warning as - "InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings" and also when I am trying to open the "Excel.xlsm" file so created I am getting the error message as- "Excel cannot open the file "Excel.xlsm" because the file format or file extensionis not valid.Verify that the file has not been corrupted and that the file extension matches the format of trhe file"
NOTE- I am trying to access the macro enabled Excel file(.xlsm) file
You can use the verify=False parameter to ignore verifying the SSL certificate, per the documentation:
r = requests.get(url, verify=False)

SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed while generating SMS from python script

I have my python script similar to below, Scripts works fine in my personal laptop.
import plivo
import sys
auth_id = "XXXXXX"
auth_token = "YYYYYYYYYYYY"
test = plivo.RestClient(auth_id, auth_token)
message_created = test.messages.create(
src='ZZZZZZ',
dst='+NNNNN',
text='Testing!!'
)
However while running the script in our organization PC's its throwing error
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.plivo.com', port=443): **Max retries exceeded with url**: /v1/Account/SXXXXXYW/Message/ (Cau
sed by SSLError(SSLError(1, u'[SSL: **CERTIFICATE_VERIFY_FAILED**] certificate verify failed (_ssl.c:590)'),))
I tried to add ssl._create_default_https_context = ssl._create_unverified_context and PYTHONHTTPSVERIFY=0 but unfortunately nothing works for me. Can anyone one help me how to resolve the error?
Try the solution from https://github.com/locustio/locust/issues/417
How to get rid from “SSL: CERTIFICATE_VERIFY_FAILED” Error
On Windows, Python does not look at the system certificate, it uses its own located at ?\lib\site-packages\certifi\cacert.pem.
The solution to your problem:
download the domain validation certificate as *.crt or *pem file
open the file in editor and copy it's content to clipboard
find your cacert.pem location: from requests.utils import
DEFAULT_CA_BUNDLE_PATH;
print(DEFAULT_CA_BUNDLE_PATH)
edit the cacert.pem file and paste your domain validation
certificate at the end of the file.
Save the file and enjoy requests!

Categories