We Keep Coding

Connection Refused Error when using python requests to get cluster configs using Databricks Clusters API during jobs/workflows

I'm trying to get the cluster configs using the databricks Clusters API when I run a notebook in a "jobs workflow" on Azure Databricks.
Can anyone advise on the best approach for this please?
The approaches recommended here:
Azure Databricks python command to show current cluster config
How to call Cluster API and start cluster from within
Databricks
Notebook?
...work fine when running notebooks "locally" (outside of jobs/workflows), but does not work during jobs/workflows
The first issue that I noticed when running a notebook in jobs/workflows was that "browserHostName" does not exist in the "tags". Instead there is "hostName" (EDIT - do not use "hostName" to resolve this issue, see below):
import requests
jsonNotebookConfigs = dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()
dictNotebookConfigs = json.loads(jsonNotebookConfigs)
try:
strHostName = dictNotebookConfigs['tags']['browserHostName']
except:
strHostName = dictNotebookConfigs['tags']['hostName']
strHostToken = dictNotebookConfigs['extraContext']['api_token']
strClusterId = dictNotebookConfigs['tags']['clusterId']
strUrl = f'https://{strHostName}/api/2.0/clusters/get?cluster_id={strClusterId}'
dictHeader = {'Authorization': f'Bearer {strHostToken}'}
response = requests.get(strUrl,
headers = dictHeader
)
dictResponse = response.json()
The second issue is that when running the code above, I get the following error(s):
ConnectionRefusedError: [Errno 111] Connection refused
...followed by...
ConnectionError: HTTPSConnectionPool(host='xxx', port=443): Max retries exceeded with url: /api/2.0/clusters/get?cluster_id=xxx (Caused by
...followed by...
NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa32582d970>: Failed to establish a new connection: [Errno 111] Connection refused'))
...followed by...
MaxRetryError: HTTPSConnectionPool(host='xxx', port=443): Max retries exceeded with url: /api/2.0/clusters/get?cluster_id=xxx (Caused by
...etc...
Any help to resolve this would be much appreciated, thanks
Edit 2022-10-17
I have found a resolution (the error derived from the first issue that I identified):
The API requires that strHostName should be the workspace instance, i.e.
strHostName = 'adb-{WorkspaceId}.{RandomNumber}.azuredatabricks.net'
It turns out that "hostName" does not actually return the workspace instance during jobs/workflows (which causes the second issue).
Instead, you need to use "orgId" which maps to the WorkspaceId, and hard-code the RandomNumber (I do not see any alternative to this). This will fix the issues.

Python ERROR SSL: WRONG_VERSION_NUMBER on code that worked few days earlier

Using google-search package (https://pypi.org/project/googlesearch-python/) to open few google search results straight from cmd. Worked fine few days ago.
Now that trying to launch the code I get SSL: WRONG_VERSION_NUMBER Error
Any ideas how to fix? And if ideas what caused the error that would be interesting.
Code:
from googlesearch import search
import webbrowser, sys
searching_for = input(("Input search words: "))
num_results = int(input("How many results : ") or "3")
result = search(searching_for)
for i in result[:num_results]:
webbrowser.open(i)
ERROR
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='www.google.com', port=443): Max retries exceeded with url: /search?q=tim+ferriss&num=11&hl=en (Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy', SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)'))))

I was able to reproduce the error and fix it using the following trick.
try changing the next line of code to this:
result = search(searching_for,proxy="http")

Caused by ProxyError('Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#https-proxy-error-http-proxy',
It says that you can try using HTTP on the URL. But the search library that you are using, I don't see any options for changing the protocol........

How to proxy for OpenWeatherMap API

i have a program and i want to show the current weather information in the corner. The code works at my own computer but at my work notebook i cant establish connection and i get this error:
requests.exceptions.ProxyError:
HTTPSConnectionPool(host='api.openweathermap.org', port=443): Max
retries exceeded with url:
/data/2.5/weather%5Bhttps://api.openweathermap.org/data/2.5/weather%5D?APPID=abc123
d88&q=Frankfurt&units=metric (Caused by ProxyError('Cannot connect to
proxy.', NewConnectionError('<urllib3.connection.HTTPSConnection
object at 0x000002563074FB48>: Failed to establish a new connection:
[WinError 10060]
I found this as a solution and i sounds promising:
Using an HTTP Proxy
Unforutnately i dont get it how i need to implement it because i dont have knowledge about the internet settings of the company. Something similar to this:
response = requests.get(url, params=params, proxies={"http": "12.34.56.78:1234", "https": "12.34.56.78:1234"})
what i do know is that i need to set this in PyCharm's Terminal to use pip install, and it seems related:
set HTTP_PROXY=12.34.56.78:1234
set HTTPS_PROXY=12.34.56.78:1234
I dont understand much about network settings but with that it works. Would i have to do this for the program as well? The port of the error message (443) is not matching to the port i enter above (1234).
Can you help me here? Would be much apreciated! :-)

I solved it:
i defined this globally:
proxies = {'http':'http://12.34.45.67:1234','https':'http://12.34.45.67:1234'}
And made my request like this:
response = requests.get(url, params=params, proxies=proxies)
I had some typos in the code before, which didnt help also ...

Max retries exceeded with url REST Python

I am performing a REST post request using python code. I am trying to communicate with an external server. I am performing several REST calls but a specific one fails. That specific request is the following:
r = requests.post(url, json=item[1], headers=headers)
I am getting the following error:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='....euapi', port= ...):
Max retries exceeded with url: ...(Caused by NewConnectionError
('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at: ...
Failed to establish a new connection: [Errno 11004] getaddrinfo failed',))
It seems that the server is blocking the connection to me because of the number of connction with it. However, I just perform only one call. Does that make sense? Any idea of how can I overcome this issue?

It seems that your 'url' value is not correct (address cannot be resolved). Please check it carefully
Similar question here

Connection reset by peer with Python-Requests on Heroku

I was wondering if I can get some help with a problem I'm having.
I'm using requests to pull some data from an api for a web app that I'm deploying on Heroku. On my local computer I have no problem connecting to the API and pulling data down, but when I deploy to Heroku it won't work. (The same thing happens when I try testing on Cloud 9 IDE)
Example Code below:
import requests
email = 'example#example.com'
url = 'https://api.example.com/'
auth = ('example', 'example')
headers = {'Accept': 'application/json', 'content-type': 'application/json'}
params = {'emailaddress': email}
r = requests.get(url+'customer.svc/search', params=params, auth=auth, headers=headers)
It's very basic, but I just can not get it to work.
Everytime I try to connect I get the error:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.example.com', port=443): Max retries exceeded with url: /customer.svc/search?emailaddress=example%40example.com (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
I'm new to Heroku, so this might be prety basic. What am I missing?
Edit:
It seems the error only happens when I deploy my code. I've tried the code on several machines and I have no trouble with the request. I wonder if it has something to do with the connection back to my Heroku app?