I have a server built in python, using Tornado and RethinkDB. Its been running for a long time, opening several connections to the DB per user. I never had any major problems by leaving them open -never closing them-. But checking out the log file I found out there's a lot of this warnings:
File "/home/bundleroot/commentserver/app.py", line 30, in <module>
db_connection = r.connect(RDB_HOST,RDB_PORT)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 542, in connect
return conn.reconnect(timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 475, in reconnect
return self._instance.connect(timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 360, in connect
self._socket = SocketWrapper(self, timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 268, in __init__
(self.host, self.port, ex))
rethinkdb.errors.RqlDriverError: Could not connect to localhost:28015. Error: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 244, in __init__
socket.create_connection((self.host, self.port), timeout)
File "/usr/lib/python3.4/socket.py", line 509, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 500, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Should I worry about it? In that case, what are the good practices?
Yes, you need to close the connection to avoid connection leaks. This Flask example, from RethinkDB, does it:
https://github.com/rethinkdb/rethinkdb-example-flask-backbone-todo/blob/master/todo.py#L47-L65
Even if RethinkDB does not enforce a hard limit on number of connections (I'm not sure if it does or not), you can run into the limits of the OS.
Related
I write a test bot for telegram by Pyrogram library. It work on my local but on pythonanywhere bash not work and can not connect:
INFO:pyrogram.crypto.aes:Using TgCrypto
Pyrogram v1.2.9, Copyright (C) 2017-2021 Dan <https://github.com/delivrance>
Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
WARNING:pyrogram.connection.connection:Connection failed! Trying again...
INFO:pyrogram.connection.connection:Disconnected
INFO:pyrogram.session.session:Session stopped
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
Then when I press ctrl+c to stop app:
Traceback (most recent call last):
File "/home/taregh/bot.py", line 37, in <module>
client.run()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/utilities/run.py", line 61, in run
self.start()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/sync.py", line 56, in async_to_sync_wrap
return loop.run_until_complete(coroutine)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
handle._run()
File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/utilities/start.py", line 53, in start
is_authorized = await self.connect()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/auth/connect.py", line 46, in connect
await self.session.start()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/session/session.py", line 127, in start
await self.connection.connect()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/connection.py", line 57, in connect
await self.protocol.connect(self.address)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/transport/tcp/tcp_abridged_o.py", line 40, in connect
await super().connect(address)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/transport/tcp/tcp.py", line 82, in connect
self.socket.connect(address)
File "/usr/local/lib/python3.9/site-packages/socks.py", line 47, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/socks.py", line 780, in connect
super(socksocket, self).connect((dest_addr, dest_port))
KeyboardInterrupt
I installed all requirements. But dose not work at all.
Unable to connect due to network issues: [Errno 111] Connection refused
Error should be quite clear. The connection is being refused.
Pythonanywhere doesn't let you use random ports and does not work with Telegram. Libraries that work via HTTP (Python-Telegram-Bot for example) should work, but Pyrogram does not.
When trying to start Airflow's scheduler with the command airflow scheduler, I'm getting the following error if executor = LocalExecutor is in airflow.cfg:
File "/usr/local/lib/python3.6/site-packages/airflow/executors/local_executor.py", line 92, in run
key, command = self.task_queue.get()
File "<string>", line 2, in get
File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
self._connect()
File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused
Python: 3.6.13
Airflow: 1.10.15 (installed with its constraints)
OS: FreeBSD 12
This is because on FreeBSD, the TCP socket's default accept queue is very short (128), so if there are many connections to a socket, which the server doesn't accept() fast enough, the kernel will send a TCP RST to the excess requests.
Raising the length of the queue helps preventing this:
sysctl kern.ipc.soacceptqueue=1024
In a python script, I have the following:
with smtplib.SMTP_SSL(sender_server, 465, context=context) as server:
server.login(sender_email, sender_password)
server.sendmail(
sender_email, email, message.as_string()
)
On the Windows machine I wrote the script on, everything works as expected and the email is sent and delivered without issue. However, when I try to run the same code on my Linux VPS, a TimeoutError is thrown every time.
Traceback (most recent call last):
File "script.py", line 151, in <module>
with smtplib.SMTP_SSL(sender_server, 465, context=context) as server:
File "/usr/lib/python3.6/smtplib.py", line 1031, in __init__
source_address)
File "/usr/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python3.6/smtplib.py", line 1037, in _get_socket
self.source_address)
File "/usr/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
What could be causing this difference? My Google searches didn't bear much fruit, but maybe I'm just searching for the wrong terms.
TimeoutError: [Errno 110] Connection timed out
This means that access to the remote system is likely blocked somewhere on the way to the system. Given that you have this problem on a "Linux VPS" it is likely that the connection is blocked by the company hosting this VPS or that you need to configure your machine specifically to allow such access. Check with your specific hoster and its documentation.
My environment is Mac OS 10.9.2, python3.3, redis-2.6.9 (64-bit).
I have many threads (nealy 2000) that use the same redis instance to write data, but some threads throw the following exceptions:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 250, in connect
sock = self._connect()
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 268, in _connect
self.socket_timeout)
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 417, in create_connecti
on
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 901, in _bootstrap_i
nner
self.run()
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 858, in run
self._target(*self._args, **self._kwargs)
File "proxypool.py", line 289, in _efficiency_proxy
self.rdb.zadd(db_proxy, time_delay, proxy)
File "/usr/local/lib/python3.3/site-packages/redis/client.py", line 1345, in zadd
return self.execute_command('ZADD', name, *pieces)
File "/usr/local/lib/python3.3/site-packages/redis/client.py", line 464, in execute_command
connection.send_command(*args)
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 334, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 316, in send_packed_command
self.connect()
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 253, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 8 connecting localhost:6379. nodename nor servname provided, or not known.
Is there some way to solve this problem?
I tried to connect to redis server when this exception happens and use ulimit -n to set a larger number, but the two methods don't work.
I know why this happened. It's caused by the max number of connections to redis-server and it's related to the max number of file handlers that the OS sets.
If I set:
$ ulimit -n 1024
it works.
Most likely the reds server is not running. Run it using "redis-server" command.
brew services start redis (on mac)
I also faced this issue.
When you are using the wrong Redis HOST then you also meet that error.
I'm not great with Python but I've inherited a script (from a former co-worker) that is giving the above error. Here's the traceback:
File "/abackup/rob/python/mail/one_and_one/__init__.py", line 119, in open
mailbox = imaplib.IMAP4_SSL(self.host, self.port)
File "/usr/local/lib/python2.6/imaplib.py", line 1137, in __init__
IMAP4.__init__(self, host, port)
File "/usr/local/lib/python2.6/imaplib.py", line 163, in __init__
self.open(host, port)
File "/usr/local/lib/python2.6/imaplib.py", line 1150, in open
self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile)
File "/usr/local/lib/python2.6/ssl.py", line 350, in wrap_socket
suppress_ragged_eofs=suppress_ragged_eofs)
File "/usr/local/lib/python2.6/ssl.py", line 118, in __init__
self.do_handshake()
File "/usr/local/lib/python2.6/ssl.py", line 293, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol
As far as I know, this was working a month ago. I have tried Googling and looked at the only other thread of stackoverflow that references this, but it doesn't seem to apply... the error seems to be occurring within the library, since all I'm calling is the constructor with a valid hostname and port.
It looks like this is related to our firewall or something... which I of course have no control over... sigh...