I am using prometheus with scaphandre (docs) to measure energy consumption in my kubernetes cluster. In the guide the way they use to access prometheus data is to forward the port from the pod to the host, but I've seen it's slower than using a service and I've been having some issues with that too.
I've defined a service to access prometheus data:
# prometheus_service.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: default
annotations:
prometheus.io/scrape: 'true'
prometheus.io/port: '9090'
spec:
selector:
app: prometheus
type: NodePort
ports:
- port: 9090
targetPort: 9090
nodePort: 30030
My problem is that I've been trying to get a query result from the URI: 192.168.131.180:30030/api/v1/query?query=scaph_host_power_microwatts, but it sometimes works fine and other times I get a connection refused error.
More precisely I have tried 3 ways to access it:
By using VSCode Thunder Client extension (a REST API extension that works like POSTMAN or Insomnia) and when I send the GET request 50% of the times I get Connection was refused by the server. and the other 50% I get a proper result.
By directly inputting the query in my browser, in this case I always get a result when accessing the URI (and I can tell it is not cached because the query result has a time attribute which is always different).
By making the query through a python program with the requests library (which is my main goal to go on with my app). In this case it also fails 50% of the time.
If the browser had also failed 50% of the times I would've thought that prometheus fails when no new data is available, but since the browser worked fine, there must be another reason.
For additional info: when I change the previous IP for localhost the same thing happens.
Is there a reason for this error?
PD: If you want the python traceback, I leave it here (sorry I wanted to write it in a kind of "See More" or "Expand Block" to avoid pasting it there but didn't know how):
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 187, in connect
conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe1d7788520>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.131.180', port=30030): Max retries exceeded with url: /api/v1/query?query=scaph_host_power_microwatts (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe1d7788520>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "pruebametrics.py", line 55, in <module>
foo()
File "pruebametrics.py", line 51, in foo
microwatts =requests.get("http://192.168.131.180:30030/api/v1/query", params={'query': 'scaph_host_power_microwatts'}).json()
File "/usr/lib/python3/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.131.180', port=30030): Max retries exceeded with url: /api/v1/query?query=scaph_host_power_microwatts (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe1d7788520>: Failed to establish a new connection: [Errno 111] Connection refused'))
Related
This question already has answers here:
python requests module and connection reuse
(2 answers)
Closed 3 months ago.
im currently working on a python script that executes POST requests a few thousand times to fill dummy data into a database. The POST request sends a string to our backend which fills in this string into the database.
The first 10000 requests are working fine but then a ConnectionError appears.
This is a simplified implementation of my code
sqlfiller.py
import requests
url = "http://localhost:3100/api"
payload = "test"
#40 represents all business-days in two months
for i in range(40):
#600 data entries per day
for i in range(600):
requests.post(url, payload)
As i mentioned, it works fine for a couple thousands POST requests. But sometime this error appears and the program crashes.
Traceback (most recent call last):
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connection.py", line 174, in _new_conn
conn = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\util\connection.py", line 95, in create_connection
raise err
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\util\connection.py", line 85, in create_connection
sock.connect(sa)
OSError: [WinError 10048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Anschluss) nur jeweils einmal verwendet werden
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 398, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1282, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1328, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1277, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 1037, in _send_output
self.send(msg)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\http\client.py", line 975, in send
self.connect()
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connection.py", line 205, in connect
conn = self._new_conn()
^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x0000020309A81250>: Failed to establish a new connection: [WinError 10048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Anschluss) nur jeweils einmal verwendet werden
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\adapters.py", line 489, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\connectionpool.py", line 787, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\urllib3\util\retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=3100): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000020309A81250>: Failed to establish a new connection: [WinError 10048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Anschluss) nur jeweils einmal verwendet werden'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Workbench\sqlfiller.py", line 55, in <module>
requests.post(url, payload)
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\api.py", line 115, in post
return request("post", url, data=data, json=json, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\adriand\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests\adapters.py", line 565, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3100): Max retries exceeded with url: /api (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000020309A81250>: Failed to establish a new connection: [WinError 10048] Normalerweise darf jede Socketadresse (Protokoll, Netzwerkadresse oder Anschluss) nur jeweils einmal verwendet werden'))
This german error line WinError 10048 translates to "Only one usage of each socket address (protocol/network address/port) is normally permitted"
Someone knows whats going on?
Thanks :)
Instead of request.post use httplib.HTTPConnection(url) with concurrency .
You need to change your code,That will definitely help to solve this problem.
Running a docker container InfluxDB database, and I'm trying to use the InfluxDBListener to send locust data to the database.
Here's my code:
#events.init.add_listener
def on_locust_init(environment, **_kwargs):
"""
Hook event that enables starting an influxdb connection
"""
bucket = 'load_bucket'
org = 'some_organization' # not used
token = 'zzzz_etc_etc_etc_==' # not used
# this settings matches the given docker-compose file
influxDBSettings = InfluxDBSettings(
influx_host='localhost',
influx_port=8086,
user='Load Tester',
pwd='some_password',
database=bucket
)
# start listener with the given configuration
InfluxDBListener(env=environment, influxDbSettings=influxDBSettings)
Error in locust log:
[2021-06-02 11:42:35,514] ILGUYL-LT1/ERROR/root: Could not connect to influxdb
Traceback (most recent call last):
File "c:\git\engine-load-tests\venv\lib\site-packages\locust_influxdb_listener\__init__.py", line 55, in __init__
self.influxdb_client.create_database(influxDbSettings.database)
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 746, in create_database
self.query("CREATE DATABASE {0}".format(quote_ident(dbname)),
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 521, in query
response = self.request(
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 378, in request
raise InfluxDBClientError(err_msg, response.status_code)
influxdb.exceptions.InfluxDBClientError: 401: {"code":"unauthorized","message":"Unauthorized"}
The script which I am running is not in a docker container but just running locally.
Confirmed:
Influx is definitely running on localhost:8086. I can go to http://localhost:8086 on my computer to get to InfluxDB.
the bucket load_bucket already exists [noticed that the InfluxDBListener actually tries to create the DB]
user name and password are correct
In another script I wrote, I can do this with no problem:
with InfluxDBClient(url="http://localhost:8086", token=token, org=org) as _client:
with _client.write_api(write_options=WriteOptions(batch_size=500,
flush_interval=10_000,
jitter_interval=2_000,
retry_interval=5_000,
max_retries=5,
max_retry_delay=30_000,
exponential_base=2,
write_type=SYNCHRONOUS)) as _write_client:
After changing influx_host='localhost' to influx_host='influxdb', I get the following error:
Traceback (most recent call last):
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\util\connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "c:\git\engine-load-tests\venv\lib\site-packages\gevent\_socketcommon.py", line 247, in getaddrinfo
addrlist = get_hub().resolver.getaddrinfo(host, port, family, type, proto, flags)
File "c:\git\engine-load-tests\venv\lib\site-packages\gevent\resolver\thread.py", line 63, in getaddrinfo
return self.pool.apply(_socket.getaddrinfo, args, kwargs)
File "c:\git\engine-load-tests\venv\lib\site-packages\gevent\pool.py", line 161, in apply
return self.spawn(func, *args, **kwds).get()
File "src\\gevent\\event.py", line 329, in gevent._gevent_cevent.AsyncResult.get
File "src\\gevent\\event.py", line 359, in gevent._gevent_cevent.AsyncResult.get
File "src\\gevent\\event.py", line 347, in gevent._gevent_cevent.AsyncResult.get
File "src\\gevent\\event.py", line 327, in gevent._gevent_cevent.AsyncResult._raise_exception
File "c:\git\engine-load-tests\venv\lib\site-packages\gevent\_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "c:\git\engine-load-tests\venv\lib\site-packages\gevent\threadpool.py", line 167, in __run_task
thread_result.set(func(*args, **kwargs))
socket.gaierror: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connection.py", line 234, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "C:\Python38\lib\http\client.py", line 1255, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Python38\lib\http\client.py", line 1301, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Python38\lib\http\client.py", line 1250, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Python38\lib\http\client.py", line 1010, in _send_output
self.send(msg)
File "C:\Python38\lib\http\client.py", line 950, in send
self.connect()
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connection.py", line 200, in connect
conn = self._new_conn()
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000221080B8C10>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\git\engine-load-tests\venv\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "c:\git\engine-load-tests\venv\lib\site-packages\urllib3\util\retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='influxdb', port=8086): Max retries exceeded with url: /query?q=CREATE+DATABASE+%22load_bucket%22&db=load_bucket (Caused by NewConnectionError('<urllib3.connection.HTTPConnectio
n object at 0x00000221080B8C10>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\git\engine-load-tests\venv\lib\site-packages\locust_influxdb_listener\__init__.py", line 55, in __init__
self.influxdb_client.create_database(influxDbSettings.database)
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 746, in create_database
self.query("CREATE DATABASE {0}".format(quote_ident(dbname)),
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 521, in query
response = self.request(
File "c:\git\engine-load-tests\venv\lib\site-packages\influxdb\client.py", line 332, in request
response = self._session.request(
File "c:\git\engine-load-tests\venv\lib\site-packages\requests\sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "c:\git\engine-load-tests\venv\lib\site-packages\requests\sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "c:\git\engine-load-tests\venv\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='influxdb', port=8086): Max retries exceeded with url: /query?q=CREATE+DATABASE+%22load_bucket%22&db=load_bucket (Caused by NewConnectionError('<urllib3.connection.HTTPConnec
tion object at 0x00000221080B8C10>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed'))
Here is the influx section of the docker-compose.yml:
$ cat docker-compose.yml
influxdb:
image: influxdb:latest
container_name: influxdb
ports:
- "8083:8083"
- "8086:8086"
- "8090:8090"
env_file:
- 'env.influxdb'
volumes:
# Data persistency
# sudo mkdir -p /srv/docker/influxdb/data
- /srv/docker/influxdb/data:/var/lib/influxdb2
$ cat influxdb.conf
[default]
url = "http://localhost:8086"
token = "<my-token>"
org = "<my org name>"
active = true
$ cat env.influxdb
INFLUXDB_DATA_ENGINE=tsm1
INFLUXDB_REPORTING_DISABLED=false
Influx is definitely running on localhost:8086
Not quite. Assuming the InfluxDB service is called "influxdb" in your docker-compose.yml file, the db is running at influxdb:8086 which, if you expose port 8086, is visible as localhost:8086 on your docker host.
However, the container running your script does not see the exposed port as localhost, so in the python script localhost is the container where thy script runs, not the Docker host machine. You're attempting to connect to the local port 8086, which doesn't have a DB connection, and so the error.
The easiest solution is to replace influx_host='localhost' with influx_host='influxdb' (again with the assumption that influxdb is the name of the InfluxDB service you have in the docker-compose.yml file, edit as needed).
We are trying to call An API using Python Shell in AWS glue. We are making an HTTPS call using python 3.6 using "requests" library. The issue is pertaining to AWS glue.
The Job config is
Python Shell
Python 3 ( Glue Version 1.0 )
This is the error that we get
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
raise err
File "/usr/local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
chunked=chunked,
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.6/site-packages/urllib3/connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/usr/local/lib/python3.6/site-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=#####&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 123, in <module>
runpy.run_path(temp_file_path, run_name='__main__')
File "/usr/local/lib/python3.6/runpy.py", line 263, in run_path
pkg_name=pkg_name, script_name=fname)
File "/usr/local/lib/python3.6/runpy.py", line 96, in _run_module_code
mod_name, mod_spec, pkg_name, script_name)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/tmp/glue-python-scripts-fhh0x612/test.py", line 16, in <module>
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=#####0&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/runscript.py", line 142, in <module>
raise e_type(e_value).with_traceback(new_stack)
File "/tmp/glue-python-scripts-fhh0x612/test.py", line 16, in <module>
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='xyz.com', port=443): Max retries exceeded with url: /api-v2/client/orders/?auth_token=####&start_date=01-04-2021&end_date=02-04-2021&status_filter=all&page_number=1 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f862b3b4160>: Failed to establish a new connection: [Errno 110] Connection timed out',))
Is there an issue with the security group that AWS Glue refers to or some issue with the API call. We are not able to figure out.
Is there more than one connection from glue? like calling S3 , redshift...etc. if more than one then u may have to add other connections in connection properties. Edit job -> add connection.
I have developed a python script which reads streams from multiple cameras and pass it to a server to display information on the frames. But the problem is that the script always fails if even of the ports of the server is not responding. I have tried exception handling but still no luck.
What I want is that if the server port is not responding then the frames must be displayed to the user without any information.
I have used python requests module.
Here is the code snippet:
self.frame = cv2.resize(frame, (self.screen_width, self.screen_height))
try:
payload = {
"frame":frame,
"timestamp": timestamp
}
payload = jsonpickle.encode(payload)
lock.acquire()
response = requests.post(self.frame_target_ip, data=payload, #self.frame_target_ip is the server ip with port number
headers=self.headers)
lock.release()
objects = jsonpickle.decode(response.content)
self.event_data, self.dets = self.parse_objects(objects)
if self.show_bounding_box == True:
draw_frame = self.draw_bounding_boxes(self.frame, self.dets)
self.frame = cv2.cvtColor(draw_frame, cv2.COLOR_BGR2RGB)
except Exception as e:
logging.error("Couldn't reach the frame_target_ip", exc_info=True)
I am using python 3 and windows 10.
Below is error traceback
00:31:52:327-root-INFO--Application Started
00:32:07:682-root-ERROR--Couldn't reach the frame_target_ip
Traceback (most recent call last):
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\connection.py", line 84, in create_connection
raise err
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\connection.py", line 74, in create_connection
sock.connect(sa)
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 677, in urlopen
chunked=chunked,
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 392, in _make_request
conn.request(method, url, **httplib_request_kw)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1252, in request
self._send_request(method, url, body, headers, encode_chunked)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1298, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1247, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1026, in _send_output
self.send(msg)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 966, in send
self.connect()
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 187, in connect
conn = self._new_conn()
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 172, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x00000295D3350E48>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 725, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000295D3350E48>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:/Users/Deepanshu/Desktop/VIDS_1cam/UI_1cam.py", line 221, in set_frame
headers=self.headers)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "C:\Users\Deepanshu\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000295D3350E48>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it'))
I am running a shell script as Ubuntu's systemd service to start at boot-time. The script internally executes a Python script (python_simulator.py) that connects to InfluxDB (through Python's influxdb package).
The Python script fails to start at boot-time and checking the logs suggests that's because of 'ConnectionError' while connecting to InfluxDB. I interpreted it as it's possible that the influxdb service as not started by the time the Python service activates at boot time. So I have tried to add the order dependency in the service by adding "After" and "Wants" as "influxdb.service" which activates the Python service a few seconds after influxdb service. But, I still get the same connection error.
The systemd service (myservice.service) looks like:
[Unit]
Description= Python startup service.
After=influxdb.service
Wants=influxdb.service
[Service]
Type=forking
ExecStart=/bin/bash /home/test_user/Deploy/start.sh
ExecStop=/bin/bash /home/test_user/Deploy/stop.sh
[Install]
WantedBy=multi-user.target
The log file of the Python script (python_simulator.py):
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection
raise err
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 354, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 181, in connect
conn = self._new_conn()
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection: [$
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "python_simulator.py", line 467, in <module>
main(host=args.host, port=args.port)
File "python_simulator.py", line 312, in main
for db_dict in client.get_list_database():
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 570, in get_list_database
return list(self.query("SHOW DATABASES").get_points())
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 416, in query
expected_response_code=expected_response_code
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/influxdb/client.py", line 267, in request
timeout=self._timeout
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/test_user/Deploy/py_venv/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=SHOW+DATABASES (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f817d91b400>: Failed to establish a new connection$
Lastly, blocking tree of daemons shows that myservice.service executes after influxdb.service:
myservice.service +6.872s
└─influxdb.service #11.344s
└─network-online.target #11.337s
└─NetworkManager-wait-online.service #4.706s +6.630s
└─NetworkManager.service #3.940s +674ms
└─dbus.service #3.914s
└─basic.target #3.728s
└─sockets.target #3.728s
└─snapd.socket #3.722s +5ms
└─sysinit.target #3.712s
└─apparmor.service #3.276s +435ms
└─local-fs.target #3.266s
└─run-user-1000.mount #49.841s
└─swap.target #3.160s
└─dev-disk-by\x2duuid-16e1b46a\x2d79fc\x2d4965\x2d9932\x2d8f589e9e7057.swap #3.132s +23ms
└─dev-disk-by\x2duuid-16e1b46a\x2d79fc\x2d4965\x2d9932\x2d8f589e9e7057.device #3.130s
I am not sure why I am still not able to execute the script (python_simulator.py) with influxdb in place. Are there any other dependencies? Are any changes needed in myservice.service? Any help will be appreciated.
Edit 1:
Can the reason be ConnectionRefusedError instead of ConnectionError and that may be because by the time it connects to influx at Port 8086, nothing is listening on it? If so, how can I put that in dependency order?
You may still run into a problem even if you create the service dependency (because it's possible that influxdb takes a moment before it is ready to accept connections, during which time the Python code might start). The solution is to either:
write your Python code to retry connections when they are refused or
configure systemd to restart your service if it fails.
Without seeing your Python code it's hard to suggest what the first solution might look like, but configuring your unit to restart on failure is as simple as adding the following to your [Service] section:
Restart=on-failure
You can delay startup of your Python code until influxdb is ready by adding a small shell script to your systemd unit:
ExecStartPre=/bin/sh -c 'while ! curl -sf http://localhost:8086/ping; do sleep 1; done'
This will loop indefinitely (thus preventing the service from starting) until influxdb is responding successfully to the /ping endpoint.