I'm facing an issue related to using an active I/O connection in the SigTerm handler using gunicorn eventlet server.
server.py
def exit_with_grace(*args):
conn = get_redis_connection()
conn.set('exited_gracefully', True)
signal.signal(signal.SIGTERM, exit_with_grace)
I also tried to fire up the celery task (using amqp broker) but all my ideas failed. When I start server in debug mode using python server.py it works perfectly. Gunicorn+ eventlet does not allow to connect to redis in sigterm handler, resulting with an following error:
Traceback (most recent call last):
File "/project/handlers/socketio/redis_context_backend.py", line 256, in publish_pattern
return conn.publish(pattern, serialized)
File "/project/venv/lib/python3.6/site-packages/redis/client.py", line 3098, in publish
return self.execute_command('PUBLISH', channel, message)
File "/project/venv/lib/python3.6/site-packages/redis/client.py", line 898, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/project/venv/lib/python3.6/site-packages/redis/connection.py", line 1192, in get_connection
connection.connect()
File "/project/venv/lib/python3.6/site-packages/redis/connection.py", line 559, in connect
sock = self._connect()
File "/project/venv/lib/python3.6/site-packages/redis/connection.py", line 603, in _connect
sock.connect(socket_address)
File "/project/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 250, in connect
self._trampoline(fd, write=True)
File "/project/venv/lib/python3.6/site-packages/eventlet/greenio/base.py", line 210, in _trampoline
mark_as_closed=self._mark_as_closed)
File "/project/venv/lib/python3.6/site-packages/eventlet/hubs/__init__.py", line 142, in trampoline
assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
Gunicorn command:
gunicorn --worker-class eventlet -w 1 server:ws --reload -b localhost:5001
Related
Celery connection breaks when long running task is executed.
I am using amqp as broker and redis as backend.
Celery is running on remote host, the command I am using for celery worker is as follows
pipenv run celery worker -A src.celery_app -l info -P gevent --without-mingle --without-heartbeat --without-gossip -Q queue1 -n worker1
The error log is as follows:-
[2020-09-09 14:26:46,820: CRITICAL/MainProcess] Couldn't ack 1, reason:ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None)
Traceback (most recent call last):
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\kombu\message.py", line 131, in ack_log_error
self.ack(multiple=multiple)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\kombu\message.py", line 126, in ack
self.channel.basic_ack(self.delivery_tag, multiple=multiple)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\channel.py", line 1394, in basic_ack
spec.Basic.Ack, argsig, (delivery_tag, multiple),
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\abstract_channel.py", line 59, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\method_framing.py", line 189, in write_frame
write(view[:offset])
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\amqp\transport.py", line 305, in write
self._write(s)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\gevent\_socket3.py", line 534, in sendall
return _socketcommon._sendall(self, data_memory, flags)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\gevent\_socketcommon.py", line 392, in _sendall
timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\gevent\_socketcommon.py", line 321, in __send_chunk
data_sent += socket.send(chunk, flags)
File "c:\users\g-us01.test\.virtualenvs\celery-z5n-38vt\lib\site-packages\gevent\_socket3.py", line 515, in send
return self._sock.send(data, flags)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
Unable to understand why this error is occurring.
I'm trying to run some code with APScheduler with MongoDB as a jobstore. I've downloaded pymongo and I tested it, so it does work. APScheduler will work when I instantiate the scheduler, but when I run scheduler.start() it throws the following error:
Traceback (most recent call last):
File "aps_ro.py", line 56, in <module>
scheduler.start()
File "/usr/lib/python3.8/site-packages/apscheduler/schedulers/background.py", line 33, in start
BaseScheduler.start(self, *args, **kwargs)
File "/usr/lib/python3.8/site-packages/apscheduler/schedulers/base.py", line 158, in start
store.start(self, alias)
File "/usr/lib/python3.8/site-packages/apscheduler/jobstores/mongodb.py", line 57, in start
self.collection.ensure_index('next_run_time', sparse=True)
File "/usr/lib/python3.8/site-packages/pymongo/collection.py", line 2028, in ensure_index
self.__create_index(keys, kwargs, session=None)
File "/usr/lib/python3.8/site-packages/pymongo/collection.py", line 1881, in __create_index
with self._socket_for_writes(session) as sock_info:
File "/usr/lib/python3.8/site-packages/pymongo/collection.py", line 195, in _socket_for_writes
return self.__database.client._socket_for_writes(session)
File "/usr/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1266, in _socket_for_writes
server = self._select_server(writable_server_selector, session)
File "/usr/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1253, in _select_server
server = topology.select_server(server_selector)
File "/usr/lib/python3.8/site-packages/pymongo/topology.py", line 233, in select_server
return random.choice(self.select_servers(selector,
File "/usr/lib/python3.8/site-packages/pymongo/topology.py", line 192, in select_servers
server_descriptions = self._select_servers_loop(
File "/usr/lib/python3.8/site-packages/pymongo/topology.py", line 208, in _select_servers_loop
raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused
My actual code sample is as follows:
jobstores = {
'mongo': {'type': 'mongodb'},
'default': SQLAlchemyJobStore(url='sqlite:///jobs.sqlite')
}
executors = {...}
job_defaults = {...}
scheduler = BackgroundScheduler()
scheduler.configure(jobstores=jobstores, executors=executors, job_defaults=job_defaults, timezone=utc)
#scheduler.scheduled_job(trigger='cron', day_of_week=4, hour=22, minute=21, second=0)
def tester():
print("hello")
scheduler.start()
By default, the mongodb python driver will connect to a database on host localhost and port 27017.
So you will need to either:
make sure mongodb server is running on the same machine as the python code and is running, on the default port (you can test this easily by typing mongo at your favourite shell and checking you get a > prompt and not an error).
supply a connection string or connection arguments if mongodb is running on a different server or a different port.
In Python3 I have a program to test the web framework bottle:
from bottle import route, run
#route('/')
def index():
return "olá pessoas"
if __name__ == '__main__':
run()
I am running the programs inside a virtualenv, in the command line. But there is this error:
(live-de-python) reinaldo#reinaldo-Inspiron-5567:~/Documentos/Code/live-de-python/repo$ python3 basic_bottle.py
Bottle v0.12.13 server starting up (using WSGIRefServer())...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.
Traceback (most recent call last):
File "basic_bottle.py", line 11, in <module>
run()
File "/home/reinaldo/Documentos/Code/live-de-python/lib/python3.6/site-packages/bottle.py", line 3127, in run
server.run(app)
File "/home/reinaldo/Documentos/Code/live-de-python/lib/python3.6/site-packages/bottle.py", line 2781, in run
srv = make_server(self.host, self.port, app, server_cls, handler_cls)
File "/usr/lib/python3.6/wsgiref/simple_server.py", line 153, in make_server
server = server_class((host, port), handler_class)
File "/usr/lib/python3.6/socketserver.py", line 453, in __init__
self.server_bind()
File "/usr/lib/python3.6/wsgiref/simple_server.py", line 50, in server_bind
HTTPServer.server_bind(self)
File "/usr/lib/python3.6/http/server.py", line 136, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/lib/python3.6/socketserver.py", line 467, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
I'm using Ubuntu. Does the error say that port 8080 is already in use? But I have no other program open
Please, does anyone know what it can be?
The port 8080 is already used by an other process. You probably launched the script and forgot to end it. Check with
$ ps -aux | grep basic_bottle
Or simply change the port with run(port=8090).
I am going through the setting of the following components on CentOS server. I get supervisord task to get the web site up and running, but I am blocked on setting the supervisor for celery. It seems that it recognizes the tasks, but when I try to execute the tasks, it won't connect to them. My redis is up and running on port 6380
Django==1.10.3
amqp==1.4.9
billiard==3.3.0.23
celery==3.1.25
kombu==3.0.37
pytz==2016.10
my celeryd.ini
[program:celeryd]
command=/root/myproject/myprojectenv/bin/celery worker -A mb --loglevel=INFO
environment=PATH="/root/myproject/myprojectenv/bin/",VIRTUAL_ENV="/root/myproject/myprojectenv",PYTHONPATH="/root/myproject/myprojectenv/lib/python2.7:/root/myproject/myprojectenv/lib/python2.7/site-packages"
directory=/home/.../myapp/
user=nobody
numprocs=1
stdout_logfile=/home/.../myapp/log_celery/worker.log
sterr_logfile=/home/.../myapp/log_celery/worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 1200
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; Set Celery priority higher than default (999)
; so, if rabbitmq(redis) is supervised, it will start first.
priority=1000
The process starts and when I go to the project folder and do:
>python manage.py celery status
celery#ssd-1v: OK
1 node online.
When I open the log file of celery I see that the tasks are loaded.
[tasks]
. mb.tasks.add
. mb.tasks.update_search_index
. orders.tasks.order_created
my mb/tasks.py
from mb.celeryapp import app
import django
django.setup()
#app.task
def add(x, y):
print(x+y)
return x + y
my mb/celeryapp.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
from django.conf import settings
# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mb.settings")
app = Celery('mb', broker='redis://localhost:6380/', backend='redis://localhost:6380/')
app.conf.broker_url = 'redis://localhost:6380/0'
app.conf.result_backend = 'redis://localhost:6380/'
app.conf.timezone = 'Europe/Sofia'
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
my mb/settings.py:
...
WSGI_APPLICATION = 'mb.wsgi.application'
BROKER_URL = 'redis://localhost:6380/0'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
...
when I run:
python manage.py shell
>>> from mb.tasks import add
>>> add.name
'mb.tasks.add'
>>> result=add.delay(1,1)
>>> result.ready()
False
>>> result.status
'PENDING'
And as mentioned earlier I do not see any change in the log anymore.
If I try to run from the command line:
/root/myproject/myprojectenv/bin/celery worker -A mb --loglevel=INFO
Running a worker with superuser privileges when the
worker accepts messages serialized with pickle is a very bad idea!
If you really want to continue then you have to set the C_FORCE_ROOT
environment variable (but please think about this before you do).
User information: uid=0 euid=0 gid=0 egid=0
But I suppose that's normal since I run it after with user nobody. Interesting thing is that the command just celery status (without python manage.py celery status) gives an error on connection, probably because it is looking for different port for redis, but the process of supervisord starts normally... and when I call 'celery worker -A mb' it says it's ok. Any ideas?
(myprojectenv) [root#ssd-1v]# celery status
Traceback (most recent call last):
File "/root/myproject/myprojectenv/bin/celery", line 11, in <module>
sys.exit(main())
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/__main__.py", line 3
0, in main
main()
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
81, in main
cmd.execute_from_commandline(argv)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
793, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
11, in execute_from_commandline
return self.handle_argv(self.prog_name, argv[1:])
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
785, in handle_argv
return self.execute(command, argv)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
717, in execute
).run_from_argv(self.prog_name, argv[1:], command=argv[0])
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
15, in run_from_argv
sys.argv if argv is None else argv, command)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 3
77, in handle_argv
return self(*args, **options)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/base.py", line 2
74, in __call__
ret = self.run(*args, **kwargs)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
473, in run
replies = I.run('ping', **kwargs)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
325, in run
return self.do_call_method(args, **kwargs)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/bin/celery.py", line
347, in do_call_method
return getattr(i, method)(*args)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 100, in ping
return self._request('ping')
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 71, in _request
timeout=self.timeout, reply=True,
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/celery/app/control.py", line 316, in broadcast
limit, callback, channel=channel,
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/pidbox.py", line 283, in _broadcast
chan = channel or self.connection.default_channel
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 771, in default_channel
self.connection
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 756, in connection
self._connection = self._establish_connection()
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/connection.py", line 711, in _establish_connection
conn = self.transport.establish_connection()
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
conn = self.Connection(**opts)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/root/myproject/myprojectenv/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
socket.error: [Errno 111] Connection refused
Any help will be highly appreciated.
UPDATE:
when I run
$:python manage.py shell
>>from mb.tasks import add
>>add
<#task: mb.tasks.add of mb:0x**2b3f6d0**>
the 0x2b3f6d0is different from what celery claims to be its memory space in its log, namely:
[config]
- ** ---------- .> app: mb:0x3495bd0
- ** ---------- .> transport: redis://localhost:6380/0
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 1 (prefork)
Ok, the answer in this case was that the gunicorn file was actually starting the project from the common python library, instead of the virtual env
I have a celery worker consisting of a task to run puppet agent on a remote machine which is a blocking call. I also have other http api flows in other tasks that can benefit from eventlet
#app.task(soft_time_limit=600)
def run_puppet_agent(hostname):
try:
env.host_string = self.host
env.user = self.username
env.password = self.password
return sudo('puppet agent -t')
except SoftTimeLimitExceeded:
raise Exception('Puppet agent TIMED OUT AFTER 600 seconds')
except Exception as e:
#run_puppet_agent.retry(args=[hostname], countdown=20)
LOG.info('')
raise Exception('Puppet agent failed with error message %s' % e.message)
When I run the worker as
celery multi start 2 -A phantom.celery.manage -P eventlet -c 15 --loglevel=INFO
It gives me an exception trace like follows:
Traceback (most recent call last):
File "/home/uruddarraju/Phantom/phantom/tasks/fabric_tasks.py", line 19, in run_puppet_agent
return sudo(command, quiet=False)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/network.py", line 639, in host_prompting_wrapper
return func(*args, **kwargs)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/operations.py", line 1095, in sudo
stderr=stderr, timeout=timeout, shell_escape=shell_escape,
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/operations.py", line 911, in _run_command
stderr=stderr, timeout=timeout)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/operations.py", line 795, in _execute
worker.raise_if_needed()
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/thread_handling.py", line 12, in wrapper
callable(*args, **kwargs)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/fabric/io.py", line 231, in input_loop
r, w, x = select([sys.stdin], [], [], 0.0)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/eventlet/green/select.py", line 79, in select
listeners.append(hub.add(hub.READ, k, on_read))
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 52, in add
self.register(fileno, new=True)
File "/home/uruddarraju/Phantom/tools/virtualenv/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 45, in register
self.poll.register(fileno, mask)
IOError: [Errno 1] Operation not permitted
I have read: Why does select.select() work with disk files but not epoll()? and see that there is a problem having eventlet working with fabric or such blocking calls. Is there a way I can tell celery not to monkey patch this particular task ?
This error was mitigated when I used gevent instead. I hope it enables the same performance for me.