Python Pyramid Shutdown Event - python

I'm new-ish to Python and Pyramid so apologies if I'm trying to do the wrong thing here.
I'm currently running a Pyramid application inside of a Docker container, with the following entrypoint:
pipenv run pserve development.ini --reload
This serves my application correctly and I can then edit code directly inside the container. This all works fine. I then attempted to register this service to an instance of Netflix's Eureka Service Registry so I could then proxy to this service with a gateway (such as Netflix Zuul). I used Eureka's REST API to achieve this and again, this all worked fine.
However, when I go to shutdown the Pyramid service, I would like to send an additional HTTP request to Eureka to DELETE the registered service - This is ideal so I don't have to wait for expiry on Eureka and there will never be a window where Zuul might be proxying requests to a downed service.
The problem is I cannot reliably find a way to run a shutdown event in Pyramid. Basically, when i stop the Docker container, the service receives exit code 137 (which I believe is the result of a kill -9) and nothing ever happens. I've attempted using atexit as well as signal event such as SIGKILL, SIGTERM, SIGINT, etc and nothing ever happens. I've also tried running pserve without a --reload flag but that still doesn't work.
Is there anyway for me to reliably get this DELETE event to send right before the server and docker container shuts down?
This is the development.ini file I'm using:
[app:main]
use = egg:my-app
pyramid.reload_templates = true
pyramid.includes =
pyramid_debugtoolbar
pyramid_redis_sessions
pyramid_tm
debugtoolbar.hosts = 0.0.0.0/0
sqlalchemy.url = mysql://root:root#mysql/keyblade
my-app.secret = secretkey
redis.sessions.secret = secretkey
redis.sessions.host = redis
redis.sessions.port = 6379
[server:main]
use = egg:waitress#main
listen = 0.0.0.0:8000
# Logging Configuration
[loggers]
keys = root, debug, sqlalchemy.engine.base.Engine
[logger_debug]
level = DEBUG
handlers =
qualname = debug
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console
[logger_sqlalchemy.engine.base.Engine]
level = INFO
handlers =
qualname = sqlalchemy.engine.base.Engine
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s

There is no shutdown protocol/api for a WSGI application (there technically isn't one for startup either despite people using/hoping that application-creation is close to the same time the server starts handling requests). You may be able to find a WSGI server that provides some hooks (for example gunicorn provides http://docs.gunicorn.org/en/stable/settings.html#worker-exit), but the better approach is to have your upstream handle your servers disappearing via health checks. Expecting that you'll be able to send a DELETE reliably when things go wrong is very unlikely to be a robust solution.

However, when I go to shutdown the Pyramid service, I would like to send an additional HTTP request to Eureka to DELETE the registered service - This is ideal so I don't have to wait for expiry on Eureka and there will never be a window where Zuul might be proxying requests to a downed service.
This is a web server specific and Pyramid cannot provide abstractions for it, as "your mileage may vary". Web server workers itself cannot know when they killed, as it is externally forced.
I would take an approach where you have an external process to monitor to web server and then perform clean up actions when it detects the web server is no longer running. The definition of no longer running could be "no a single process alive". Then you just have a background scheduled job (cron) to check for this condition. Or even better, have it on another monitoring instance that sits on a different server and can act in the situation in the server itself goes down.

Related

APScheduler does not run scheduled tasks: Flask + uWSGI

I have an application on a Flask and uWSGI with a jobstore in a SQLite. I start the scheduler along with the application, and add new tasks through add_task when some url is visited.
I see that the tasks are saved correctly in the jobstore, I can view them through the API, but it does not execute at the appointed time.
A few important data:
uwsgi.ini
processes = 1
enable-threads = true
__init__.py
scheduler = APScheduler()
scheduler.init_app(app)
with app.app_context():
scheduler.start()
main.py
scheduler.add_job(
id='{}{}'.format(test.id, g.user.id),
func = pay_day,
args = [test.id, g.user.id],
trigger ='interval',
minutes=test.timer
)
in service.py
def pay_day(tid, uid):
with scheduler.app.app_context():
*some code here*
Interesting behavior: if you create a task by going to the URL and restart the application after that, the task will be executed. But if the application is running and one of the users creates a task by going to the URL, then this task will not be completed until the application is restarted.
I don't get any errors or exceptions, even in the scheduler logs.
I already have no idea how to make it work and what I did wrong. I need a hint.
uWSGI employs some tricks which disable the Global Interpreter Lock and with it, the use of threads which are vital to the operation of APScheduler. To fix this, you need to re-enable the GIL using the --enable-threads switch. See the uWSGI documentation for more details.
I know that you had enable-threads = true in uwsgi.ini, but try the to enable it using the command line.

How to use the logging module in Python with gunicorn

I have a flask-based app. When I run it locally, I run it from the command line, but when I deploy it, I start it with gunicorn with multiple workers.
I want to use the logging module to log to a file. The docs I've found for this are https://docs.python.org/3/library/logging.html and https://docs.python.org/3/howto/logging-cookbook.html .
I am confused over the correct way to use logging when my app may be launched with gunicorn. The docs address threading but assume I have control of the master process. Points of confusion:
Will logger = logging.getLogger('myapp') return the same logger object in different gunicorn worker threads?
If I am attaching a logging FileHandler to my logger in order to log to a file, how can I avoid doing this multiple times in the different workers?
My understanding - which may be wrong - is that if I just call logger.setLevel(logging.DEBUG), this will send messages via the root logger which may have a higher default logging level and may ignore debug messages, and so I also need to call logging.basicConfig(logging.DEBUG) in order for my debug messages to get through. But the docs say not to call logging.basicConfig() from a thread. How can I correctly set the root logging level when using gunicorn? Or do I not need to?
This is my typical Flask/Gunicorn setup. Note gunicorn is ran via supervisor.
wsgi_web.py. Note ProxyFix to get a client's real IP address from Nginx.
from werkzeug.contrib.fixers import ProxyFix
from app import create_app
import logging
gunicorn_logger = logging.getLogger('gunicorn.error')
application = create_app(logger_override=gunicorn_logger)
application.wsgi_app = ProxyFix(application.wsgi_app, num_proxies=1)
Edit February 2020, for newer versions of werkzeug use the following and adjust the parameters to ProxyFix as necessary:
from werkzeug.middleware.proxy_fix import ProxyFix
from app import create_app
import logging
gunicorn_logger = logging.getLogger('gunicorn.error')
application = create_app(logger_override=gunicorn_logger)
application.wsgi_app = ProxyFix(application.wsgi_app, x_for=1, x_host=1)
Flask application factory create_app
def create_app(logger_override=None):
app = Flask(__name__)
if logger_override:
# working solely with the flask logger
app.logger.handlers = logger_override.handlers
app.logger.setLevel(logger_override.level)
# OR, working with multiple loggers
# for logger in (app.logger, logging.getLogger('sqlalchemy')):
# logger.handlers = logger_override.handlers
# logger.setLevel(logger_override.level)
# more
return app
Gunicorn command (4th line) within supervisor conf, note the --log-level parameter has been set to info in this instance. Note X-REAL-IP passed to access --access-logformat
[program:web]
directory = /home/paul/www/example
environment = APP_SETTINGS="app.config.ProductionConfig"
command = /home/paul/.virtualenvs/example/bin/gunicorn wsgi_web:application -b localhost:8000 --workers 3 --worker-class gevent --keep-alive 10 --log-level info --access-logfile /home/paul/www/logs/admin.gunicorn.access.log --error-logfile /home/paul/www/logs/admin.gunicorn.error.log --access-logformat '%%({X-REAL-IP}i)s %%(l)s %%(u)s %%(t)s "%%(r)s" %%(s)s %%(b)s "%%(f)s" "%%(a)s"'
user = paul
autostart=true
autorestart=true
Each worker is an isolated process with its own memory so you can't really share the same logger across different workers.
Your code runs inside these workers because the master process only cares about managing the workers.
The master process is a simple loop that listens for various process
signals and reacts accordingly. It manages the list of running workers
by listening for signals like TTIN, TTOU, and CHLD. TTIN and TTOU tell
the master to increase or decrease the number of running workers.
In Gunicorn itself, there are two main run modes
Sync
Async
So this is different from threading, this is multiprocessing.
However since Gunicorn 19, a threads option can be used to process requests in
multiple threads. Using threads assumes use of the gthread worker.
With this in mind, the logging code will be written once and will be invoked multiple times each time a new worker is created. You can use Singelton pattern to ensure the same logger instance is being used inside the same worker.
For configuring the logger itself, you just need to follow the normal process of setting the root logger levels and the different loggers levels.
basicConfig() won't affect the root handler if it's already setup:
This function does nothing if the root logger already has handlers configured for it.
To set the level on root explicitly do
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(name)
Then the level can be set on the handler or logger level.
handler = logging.handlers.TimedRotatingFileHandler(log_path, when='midnight', backupCount=30)
handler.setLevel(min_level)
You can check this similar answer for logging related details
Set logging levels
More Resources :
http://docs.gunicorn.org/en/stable/design.html

Why cherrypy server logs error messages even though it seems to work?

I'm using cherrypy to start a server, that its computation engine is Apache Spark. It logs this weird result:
[06/Dec/2017:13:25:42] ENGINE Bus STARTING
INFO:cherrypy.error:[06/Dec/2017:13:25:42] ENGINE Bus STARTING
[06/Dec/2017:13:25:42] ENGINE Started monitor thread 'Autoreloader'.
INFO:cherrypy.error:[06/Dec/2017:13:25:42] ENGINE Started monitor thread 'Autoreloader'.
[06/Dec/2017:13:25:42] ENGINE Serving on http://0.0.0.0:5432
INFO:cherrypy.error:[06/Dec/2017:13:25:42] ENGINE Serving on http://0.0.0.0:5432
[06/Dec/2017:13:25:42] ENGINE Bus STARTED
INFO:cherrypy.error:[06/Dec/2017:13:25:42] ENGINE Bus STARTED
My question is, what is this INFO:cherrypy.error: it logs?
This is how I run the server:
def run_server(app):
# Enable WSGI access logging via Paste
app_logged = TransLogger(app)
# Mount the WSGI callable object (app) on the root directory
cherrypy.tree.graft(app_logged, '/')
# Set the configuration of the web server
cherrypy.config.update({
'engine.autoreload.on': True,
'log.screen': True,
'server.socket_port': 5432,
'server.socket_host': '0.0.0.0'
})
# Start the CherryPy WSGI web server
cherrypy.engine.start()
cherrypy.engine.block()
There's absolutely nothing wrong with what you're seeing in the log file. I see the same bus and serving statements when I run Cherrypy. I guess Cherrypy haven't really used the term 'error' too effectively there, like some people call the HTTP status codes 'error codes', when in fact code 200 means everything is all good.
I think for your case, the listeners (for activities logging) are essentially wired with the function _buslog in cherrypy/__ init__.py , and they eventually will call the function error() in cherrypy/_cplogging.py
and according to the description there:
""" Write the given ``msg`` to the error log.
This is not just for errors! Applications may call this at any time
to log application-specific information.
If ``traceback`` is True, the traceback of the current exception
(if any) will be appended to ``msg``.
"""
So , yeah, this is not just for errors...

How to set up logging with Cherrypy?

I am trying to set up logging with Cherrypy on my Openshift python 3.3 app. The 'appserver.log' file only updates until the actual server starts then nothing gets added to the log file. I have read and followed (as far as I know) the documentation at the below links. Still no logging.
CherryPy server errors log
http://docs.cherrypy.org/dev/refman/_cplogging.html
My python code snippet:
def run_cherrypy_server(app, ip, port=8080):
from cherrypy import wsgiserver
from cherrypy import config
# log.screen: Set this to True to have both “error” and “access” messages printed to stdout.
# log.access_file: Set this to an absolute filename where you want “access” messages written.
# log.error_file: Set this to an absolute filename where you want “error” messages written.
appserver_error_log = os.path.join(os.environ['OPENSHIFT_HOMEDIR'], 'python', 'logs','appserver_error.log')
appserver_access_log = os.path.join(os.environ['OPENSHIFT_HOMEDIR'], 'python', 'logs','appserver_access.log')
config.update({
'log.screen': True,
'log.error_file': appserver_error_log,
'log.access_file': appserver_access_log
})
server = wsgiserver.CherryPyWSGIServer(
(ip, port), app, server_name='www.cherrypy.example')
server.start()
The 'appserver_error.log' and 'appserver_access.log' files actually get created in the proper Openshift python directory. However, no logging information in both of the files appserver_error.log and appserver_access.log.
Everything runs fine but no logging.
Any ideas what I am doing wrong?
The WSGI server itself does not do any logging. The CherryPy engine (which controls process startup and shutdown) writes to the "error" log, and only CherryPy applications (that use CherryPy's Request and Response objects) write to the access log. If you're passing your own WSGI application, you'll have to do your own logging.

Bottle equivalent of engine.restart()

I am trying to transfer from Cherrypy to Bottle & Gevent(server).
After I run:
application=bottle.default_app() #bottle
WSGIServer(('', port), application, spawn=None).serve_forever() #gevent
I want to restart the server just as if the reloader reloaded the server (but only when I tell the server to).
So I want to access a page with credential request and only after correct authentication will it restart.
Here is my functional example in Cherrypy:
#expose
def reloadMe(self, u=None, p=None):
if u=="username" and p=="password":
engine.restart()
raise HTTPRedirect('/')
More simply I am asking how do I reload this script so that my edits to the source file are implemented but only when I retrieve a "restart" page.
I literally only need the Bottlepy equivalent of
engine.restart() #cherrypy
Does no one know how to do this?
You can write a small shell script to restart gevent wsgi server.
then using this code, you can call the script.
#get('/restartmyserver')
def handler():
http_auth_data = bottle.request.auth() # returns a tuple (username,password) only basic auth.
if http_auth_data[0] == user and http_auth_data[1] == password:
os.system("your_shell_script_to_restart_gevent_wsgi")
bottle.redirect('/')
let me know if you need more info.

Categories