We Keep Coding

Connection Refused Error when using python requests to get cluster configs using Databricks Clusters API during jobs/workflows

I'm trying to get the cluster configs using the databricks Clusters API when I run a notebook in a "jobs workflow" on Azure Databricks.
Can anyone advise on the best approach for this please?
The approaches recommended here:
Azure Databricks python command to show current cluster config
How to call Cluster API and start cluster from within
Databricks
Notebook?
...work fine when running notebooks "locally" (outside of jobs/workflows), but does not work during jobs/workflows
The first issue that I noticed when running a notebook in jobs/workflows was that "browserHostName" does not exist in the "tags". Instead there is "hostName" (EDIT - do not use "hostName" to resolve this issue, see below):
import requests
jsonNotebookConfigs = dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()
dictNotebookConfigs = json.loads(jsonNotebookConfigs)
try:
strHostName = dictNotebookConfigs['tags']['browserHostName']
except:
strHostName = dictNotebookConfigs['tags']['hostName']
strHostToken = dictNotebookConfigs['extraContext']['api_token']
strClusterId = dictNotebookConfigs['tags']['clusterId']
strUrl = f'https://{strHostName}/api/2.0/clusters/get?cluster_id={strClusterId}'
dictHeader = {'Authorization': f'Bearer {strHostToken}'}
response = requests.get(strUrl,
headers = dictHeader
)
dictResponse = response.json()
The second issue is that when running the code above, I get the following error(s):
ConnectionRefusedError: [Errno 111] Connection refused
...followed by...
ConnectionError: HTTPSConnectionPool(host='xxx', port=443): Max retries exceeded with url: /api/2.0/clusters/get?cluster_id=xxx (Caused by
...followed by...
NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa32582d970>: Failed to establish a new connection: [Errno 111] Connection refused'))
...followed by...
MaxRetryError: HTTPSConnectionPool(host='xxx', port=443): Max retries exceeded with url: /api/2.0/clusters/get?cluster_id=xxx (Caused by
...etc...
Any help to resolve this would be much appreciated, thanks
Edit 2022-10-17
I have found a resolution (the error derived from the first issue that I identified):
The API requires that strHostName should be the workspace instance, i.e.
strHostName = 'adb-{WorkspaceId}.{RandomNumber}.azuredatabricks.net'
It turns out that "hostName" does not actually return the workspace instance during jobs/workflows (which causes the second issue).
Instead, you need to use "orgId" which maps to the WorkspaceId, and hard-code the RandomNumber (I do not see any alternative to this). This will fix the issues.

Python ERROR SSL: WRONG_VERSION_NUMBER on code that worked few days earlier

Using google-search package (https://pypi.org/project/googlesearch-python/) to open few google search results straight from cmd. Worked fine few days ago.
Now that trying to launch the code I get SSL: WRONG_VERSION_NUMBER Error
Any ideas how to fix? And if ideas what caused the error that would be interesting.
Code:
from googlesearch import search
import webbrowser, sys
searching_for = input(("Input search words: "))
num_results = int(input("How many results : ") or "3")
result = search(searching_for)
for i in result[:num_results]:
webbrowser.open(i)
ERROR
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /search?q=tim+ferriss&num=11&hl=en (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)'))))

I was able to reproduce the error and fix it using the following trick.
try changing the next line of code to this:
result = search(searching_for,proxy="http")

Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy',
It says that you can try using HTTP on the URL. But the search library that you are using, I don't see any options for changing the protocol........

Automatize checking if a link is live using python

I need to often manually check if some links are live. To avoid this I would like to write a small script that just returns the https status of all the links so I can immediately see any 404s. Problem is I have no idea what i'm doing I just know it should be possible :D
I tried to do this with a tester page using python's requests and I get the following error
import requests
requests.get('https://exp04.zih.tu-dresden.de/')
HTTPSConnectionPool(host='exp04.zih.tu-dresden.de', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1125)')))
I understand the individual words of the error message but now nothign about server/client stuff. How do I return the status code for this page?

import requests
requests.get('https://exp04.zih.tu-dresden.de/', verify=False)
Just ignore the SSL certificate verifying.

Having 'certificate verify failed' error when registering a model on Azure

Today, I got an error when I wanted to register a new model on Azure work space. This error wasn't happening before! The only change that I have is the model itself but the code is the same:
from azureml.core import Workspace
from azureml.core.compute import AksCompute, ComputeTarget
from azureml.core.webservice import Webservice, AksWebservice
from azureml.core.image import Image
from azureml.core.model import Model
import azureml.core
from azureml.core.workspace import Workspace
from azureml.core.model import Model
ws = Workspace.from_config()
model_reg = Model.register(model_path = "./outputs/model_final",
model_name = "ModelX",
tags = {'area': "XXX", 'type': "deep"},
description = "Model 1",
workspace = ws)
And here is the error that I got:
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='xxx
.blob.core.windows.net', port=443): Max retries exceeded with url: /azureml/Loca
lUpload/190530T093046-fc5a89ca/tfModel.tar.gz?sv=2018-03-28&sr=b&sig=9FFnsaefR9w
PkR%2FvsMlnQYrA2wlnJG7Cu%2BP9%2FePPZEY%3D&st=2019-05-30T08%3A20%3A46Z&se=2019-05
-31T08%3A30%3A46Z&sp=rcw&comp=block&blockid=TURBd01EQXdNREF3TURBd01EQXdNREF3TURB
d01EQXdNREF3TURBd01EQSUzRA%3D%3D (Caused by SSLError(SSLError("bad handshake: Er
ror([('SSL routines', 'tls_process_server_certificate', 'certificate verify fail
ed')],)",),))
I tried to update all the python packages and even creating a new conda env with fresh installation of all packages, but still getting this error.
Any help or similar experience?

Just as reference, there may be two reasons cause this issue.
Network issue. The network environment for running this Python script changed. May now the script run through a proxy like Fiddler which not support or enable SSL/TLS feature, or some IP restrictions in your network cause it like the SO thread SSL Certificate error while doing a request via python.
The target site upgraded SSL version. You can follow the SO thread answer Django paypalrestsdk error - OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')] to check.
Hope it helps.

Access GitHub repository through proxy

I am trying to get repositories from my Github ut I get following error:
HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /tejasshah2227/user/repos (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x0000024E647925F8>, 'Connection to github.com timed out. (connect timeout=15)'))
I guess I am getting this error because I am accessing this through a proxy. But I can't find anything that will help me fix this issu.
Following is my code:
from github import Github
g = Github("userID", "password")
print(g.get_user())
for repo in g.get_user().get_repos():
print(repo.name)
Also, when I printed this:
print(g.get_user())
I get following as the output:
AuthenticatedUser(login=None)
Can anyoe help me with this?