Why am I getting ConnectionRefusedError with Airflow's LocalExecutor? - python

When trying to start Airflow's scheduler with the command airflow scheduler, I'm getting the following error if executor = LocalExecutor is in airflow.cfg:
File "/usr/local/lib/python3.6/site-packages/airflow/executors/local_executor.py", line 92, in run
key, command = self.task_queue.get()
File "<string>", line 2, in get
File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
self._connect()
File "/usr/local/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
conn = self._Client(self._token.address, authkey=self._authkey)
File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 487, in Client
c = SocketClient(address)
File "/usr/local/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
s.connect(address)
ConnectionRefusedError: [Errno 61] Connection refused
Python: 3.6.13
Airflow: 1.10.15 (installed with its constraints)
OS: FreeBSD 12

This is because on FreeBSD, the TCP socket's default accept queue is very short (128), so if there are many connections to a socket, which the server doesn't accept() fast enough, the kernel will send a TCP RST to the excess requests.
Raising the length of the queue helps preventing this:
sysctl kern.ipc.soacceptqueue=1024

Related

pyrogram [Errno 111] Connection refused on pythonanywhere.com

I write a test bot for telegram by Pyrogram library. It work on my local but on pythonanywhere bash not work and can not connect:
INFO:pyrogram.crypto.aes:Using TgCrypto
Pyrogram v1.2.9, Copyright (C) 2017-2021 Dan <https://github.com/delivrance>
Licensed under the terms of the GNU Lesser General Public License v3 or later (LGPLv3+)
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
WARNING:pyrogram.connection.connection:Connection failed! Trying again...
INFO:pyrogram.connection.connection:Disconnected
INFO:pyrogram.session.session:Session stopped
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
INFO:pyrogram.connection.connection:Connecting...
WARNING:pyrogram.connection.connection:Unable to connect due to network issues: [Errno 111] Connection refused
Then when I press ctrl+c to stop app:
Traceback (most recent call last):
File "/home/taregh/bot.py", line 37, in <module>
client.run()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/utilities/run.py", line 61, in run
self.start()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/sync.py", line 56, in async_to_sync_wrap
return loop.run_until_complete(coroutine)
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
self._run_once()
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
handle._run()
File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run
self._context.run(self._callback, *self._args)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/utilities/start.py", line 53, in start
is_authorized = await self.connect()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/methods/auth/connect.py", line 46, in connect
await self.session.start()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/session/session.py", line 127, in start
await self.connection.connect()
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/connection.py", line 57, in connect
await self.protocol.connect(self.address)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/transport/tcp/tcp_abridged_o.py", line 40, in connect
await super().connect(address)
File "/home/taregh/.local/lib/python3.9/site-packages/pyrogram/connection/transport/tcp/tcp.py", line 82, in connect
self.socket.connect(address)
File "/usr/local/lib/python3.9/site-packages/socks.py", line 47, in wrapper
return function(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/socks.py", line 780, in connect
super(socksocket, self).connect((dest_addr, dest_port))
KeyboardInterrupt
I installed all requirements. But dose not work at all.
Unable to connect due to network issues: [Errno 111] Connection refused
Error should be quite clear. The connection is being refused.
Pythonanywhere doesn't let you use random ports and does not work with Telegram. Libraries that work via HTTP (Python-Telegram-Bot for example) should work, but Pyrogram does not.

Can't invoke nameko shell within Docker container

I'm invoking nameko shell inside my docker container of the example service, but I receive this error. I have setup two containers. My rabbitmq container and my service container. I'm invoking the nameko shell from inside the service container bash. The containers start up correctly and the service container connects successfully. But I can't use the shell.
Error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 138, in _connect
host, port, family, socket.SOCK_STREAM, SOL_TCP)
File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -9] Address family for hostname not supported
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/nameko", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/nameko/cli/main.py", line 112, in main
args.main(args)
File "/usr/local/lib/python3.6/site-packages/nameko/cli/commands.py", line 143, in main
main(args)
File "/usr/local/lib/python3.6/site-packages/nameko/cli/shell.py", line 98, in main
ctx['n'] = make_nameko_helper(config)
File "/usr/local/lib/python3.6/site-packages/nameko/cli/shell.py", line 73, in make_nameko_helper
module.rpc = proxy.start()
File "/usr/local/lib/python3.6/site-packages/nameko/standalone/rpc.py", line 228, in start
self._reply_listener.setup()
File "/usr/local/lib/python3.6/site-packages/nameko/rpc.py", line 260, in setup
self.queue_consumer.register_provider(self)
File "/usr/local/lib/python3.6/site-packages/nameko/standalone/rpc.py", line 123, in register_provider
self._setup_consumer()
File "/usr/local/lib/python3.6/site-packages/nameko/standalone/rpc.py", line 102, in _setup_consumer
channel = self.connection.channel()
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 289, in channel
chan = self.transport.create_channel(self.connection)
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 867, in connection
max_retries=1, reraise_as_library_errors=False
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 445, in _ensure_connection
callback, timeout=timeout
File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 344, in retry_over_time
return fun(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 874, in _connection_factory
self._connection = self._establish_connection()
File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 809, in _establish_connection
conn = self.transport.establish_connection()
File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 314, in connect
self.transport.connect()
File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 78, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 149, in _connect
"failed to resolve broker hostname"))
File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 162, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Dockerfile
FROM python:3-onbuild
CMD ["nameko", "run", "--config", "conf.yml", "helloworld"]
Config file
AMQP_URI: 'pyamqp://guest:guest#rabbitmq'
docker-compose file
version: '2'
services:
echo:
build: ./echo
restart: always
volumes:
- .:/echo/code
depends_on:
- rabbitmq
rabbitmq:
image: "rabbitmq"
ports:
- "15673:15672"
I found out after a while that it was my own stupid mistake. I forgot to add the config file in my nameko shell command. You have to specify the message broker when executing nameko shell. In my case I needed to run nameko shell --config config.yml. That enabled me to connect and test my nameko service.

Connecting to SMTP server failing on Linux but not Windows

In a python script, I have the following:
with smtplib.SMTP_SSL(sender_server, 465, context=context) as server:
server.login(sender_email, sender_password)
server.sendmail(
sender_email, email, message.as_string()
)
On the Windows machine I wrote the script on, everything works as expected and the email is sent and delivered without issue. However, when I try to run the same code on my Linux VPS, a TimeoutError is thrown every time.
Traceback (most recent call last):
File "script.py", line 151, in <module>
with smtplib.SMTP_SSL(sender_server, 465, context=context) as server:
File "/usr/lib/python3.6/smtplib.py", line 1031, in __init__
source_address)
File "/usr/lib/python3.6/smtplib.py", line 251, in __init__
(code, msg) = self.connect(host, port)
File "/usr/lib/python3.6/smtplib.py", line 336, in connect
self.sock = self._get_socket(host, port, self.timeout)
File "/usr/lib/python3.6/smtplib.py", line 1037, in _get_socket
self.source_address)
File "/usr/lib/python3.6/socket.py", line 724, in create_connection
raise err
File "/usr/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
What could be causing this difference? My Google searches didn't bear much fruit, but maybe I'm just searching for the wrong terms.
TimeoutError: [Errno 110] Connection timed out
This means that access to the remote system is likely blocked somewhere on the way to the system. Given that you have this problem on a "Linux VPS" it is likely that the connection is blocked by the company hosting this VPS or that you need to configure your machine specifically to allow such access. Check with your specific hoster and its documentation.

should I close each connection to the DB in rethinkDB?

I have a server built in python, using Tornado and RethinkDB. Its been running for a long time, opening several connections to the DB per user. I never had any major problems by leaving them open -never closing them-. But checking out the log file I found out there's a lot of this warnings:
File "/home/bundleroot/commentserver/app.py", line 30, in <module>
db_connection = r.connect(RDB_HOST,RDB_PORT)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 542, in connect
return conn.reconnect(timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 475, in reconnect
return self._instance.connect(timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 360, in connect
self._socket = SocketWrapper(self, timeout)
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 268, in __init__
(self.host, self.port, ex))
rethinkdb.errors.RqlDriverError: Could not connect to localhost:28015. Error: [Errno 111] Connection refused
Traceback (most recent call last):
File "/home/bundleroot/commentserver/env34/lib/python3.4/site-packages/rethinkdb/net.py", line 244, in __init__
socket.create_connection((self.host, self.port), timeout)
File "/usr/lib/python3.4/socket.py", line 509, in create_connection
raise err
File "/usr/lib/python3.4/socket.py", line 500, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
Should I worry about it? In that case, what are the good practices?
Yes, you need to close the connection to avoid connection leaks. This Flask example, from RethinkDB, does it:
https://github.com/rethinkdb/rethinkdb-example-flask-backbone-todo/blob/master/todo.py#L47-L65
Even if RethinkDB does not enforce a hard limit on number of connections (I'm not sure if it does or not), you can run into the limits of the OS.

Redis Error 8 connecting localhost:6379. nodename nor servname provided, or not known

My environment is Mac OS 10.9.2, python3.3, redis-2.6.9 (64-bit).
I have many threads (nealy 2000) that use the same redis instance to write data, but some threads throw the following exceptions:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 250, in connect
sock = self._connect()
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 268, in _connect
self.socket_timeout)
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/socket.py", line 417, in create_connecti
on
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 901, in _bootstrap_i
nner
self.run()
File "/usr/local/Cellar/python3/3.3.4/Frameworks/Python.framework/Versions/3.3/lib/python3.3/threading.py", line 858, in run
self._target(*self._args, **self._kwargs)
File "proxypool.py", line 289, in _efficiency_proxy
self.rdb.zadd(db_proxy, time_delay, proxy)
File "/usr/local/lib/python3.3/site-packages/redis/client.py", line 1345, in zadd
return self.execute_command('ZADD', name, *pieces)
File "/usr/local/lib/python3.3/site-packages/redis/client.py", line 464, in execute_command
connection.send_command(*args)
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 334, in send_command
self.send_packed_command(self.pack_command(*args))
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 316, in send_packed_command
self.connect()
File "/usr/local/lib/python3.3/site-packages/redis/connection.py", line 253, in connect
raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 8 connecting localhost:6379. nodename nor servname provided, or not known.
Is there some way to solve this problem?
I tried to connect to redis server when this exception happens and use ulimit -n to set a larger number, but the two methods don't work.
I know why this happened. It's caused by the max number of connections to redis-server and it's related to the max number of file handlers that the OS sets.
If I set:
$ ulimit -n 1024
it works.
Most likely the reds server is not running. Run it using "redis-server" command.
brew services start redis (on mac)
I also faced this issue.
When you are using the wrong Redis HOST then you also meet that error.

Categories