I have a python 3 flask server running through WSGI. I have a config file that is imported by my api code that handles environment variables, and I set up sentry in this file. This is my code which is setup exactly as described in the sentry docs https://docs.sentry.io/platforms/python/logging/
if sentry_dsn is not None:
sentry_logging = LoggingIntegration(
level=logging.INFO,
event_level=logging.CRITICAL,
)
LOG.debug(f"Initialising sentry for environment {sentry_environment}")
sentry_sdk.init(
sentry_dsn,
environment=sentry_environment,
integrations=[sentry_logging],
)
else:
LOG.warn("Sentry key not set up")
The problem is this does not send any events to sentry, in exception logs or even uncaught exceptions. I know that the DSN is correct because if I set up sentry like this, all uncaught exceptions as well as error and exception logs are sent to my sentry project:
if sentry_dsn is not None:
LOG.debug("Initialising sentry")
sentry_sdk.init(sentry_dsn, environment=sentry_environment)
else:
LOG.warn("Sentry key not set up")
I've tried the setup with the debug=True setting in the sentry init and logs confirm that sentry intialises and sets up integrations. But when an event occurs that it should report, there is no log or anything recorded by sentry.
Any help would be appreciated
Turns out that flask was intercepting the exceptions because the default sentry integrations exclude Flask. This meant that nothing was reaching sentry. I added the flask integration according to sentry docs (https://docs.sentry.io/platforms/python/flask/) and sentry messages were sent as I expected.
Related
I am trying to setup AI similar to how it is being done here
I am using Python 3.9.13 and following packages: opencensus==0.11.0, opencensus-ext-azure==1.1.7, opencensus-context==0.1.3
My code looks something like this:
import logging
import time
from opencensus.ext.azure.log_exporter import AzureLogHandler
# create the logger
app_insights_logger = logging.getLogger(__name__)
# set the handler
app_insights_logger.addHandler(AzureLogHandler(
connection_string='InstrumentationKey=00000000-0000-0000-0000-000000000000')
)
# set the logging level
app_insights_logger.setLevel(logging.INFO)
# this prints 'logging level = 20'
print('logging level = ',app_insights_logger.getEffectiveLevel())
# try to log an exception
try:
result = 1 / 0
except Exception:
app_insights_logger.exception('Captured a math exception.')
app_insights_logger.handlers[0].flush()
time.sleep(5)
However the exception does not get logged, I tried adding the explicit flush as mentioned in this post
Additionally, I tried adding the instrumentation key as mentioned in the docs, when that didn't work I tried with the entire connection string(the one with the ingestion key)
So,
How can I debug if my app is indeed sending requests to Azure ?
How can I check on the Azure portal if it is a permission issue ?
You can set the severity level before logging any kind of telemetry information to Application Insights.
Note: By default, root logger can be configured with warning severity. if you want to add other severity information you have to set like (logger.setLevel(logging.INFO)).
I am using below code to log the telemetry information in Application Insights
import logging
from logging import Logger
from opencensus.ext.azure.log_exporter import AzureLogHandler
AI_conn_string= '<Your AI Connection string>'
handler = AzureLogHandler(connection_string=AI_conn_string)
logger = logging.getLogger()
logger.addHandler(handler)
#by default root logger can be configured with warning severity.
logger.warning('python console app warning log in AI ')
# setting severity for information level logging.
logger.setLevel(logging.INFO)
logger.info('Test Information log')
logger.info('python console app information log in AI')
try:
logger.warning('python console app Try block warning log in AI')
result = 1 / 0
except Exception:
logger.setLevel(logging.ERROR)
logger.exception('python console app error log in AI')
Results
Warning and Information log
Error log in AI
How can I debug if my app is indeed sending requests to Azure ?
We cannot debug the information whether the telemetry data send to Application Insights or not. But we can see the process like below
How can I check on the Azure portal if it is a permission issue ?
Instrumentation key and connection string has the permission to access the Application Insights resource.
I have a tornado application and a custom logger method. My code to build and use the custom logger is the following:
def create_logger():
"""
This function creates the logger functionality to be used throughout the Python application
:return: bool - true if successful
"""
# Configuring the logger
filename = "PythonLogger.log"
# Change the current working directory to the logs folder, so that the logs files is written in it.
os.chdir(os.path.normpath(os.path.normpath(os.path.dirname(os.path.abspath(__file__)) + os.sep + os.pardir + os.sep + os.pardir + os.sep + 'logs')))
# Create the logs file
logging.basicConfig(filename=filename, format='%(asctime)s %(message)s', filemode='w')
# Creating the logger
logger = logging.getLogger()
# Setting the threshold of logger to DEBUG
logger.setLevel(logging.NOTSET)
logger.log(0, 'El logger está inicializado')
return True
def log_info_message(msg):
"""
Utility for message logging with code 20
:param msg:
:return:
"""
return logging.getLogger().log(20, msg)
In the code, I initialize the logger and already write a message to it before the Tornado application initialization:
if __name__ == '__main__':
# Logger initialization
create_logger()
# First log message
log_info_message('Initiating Python application')
# Starting Tornado
tornado.options.parse_command_line()
# Specifying what app exactly is being started
server = tornado.httpserver.HTTPServer(test.app)
server.listen(options.port)
try:
if 'Windows_NT' not in os.environ.values():
server.start(0)
tornado.ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
Then let's say my method get of HTTP request is as follows (only interesting lines):
class API(tornado.web.RequestHandler):
def get(self):
self.write('Get request ')
logging.getLogger("tornado.access").log(20, 'Hola')
logging.getLogger("tornado.application").log(20, '1')
logging.getLogger("tornado.general").log(20, '2')
log_info_message('Received a GET request at: ' + datetime.datetime.now().strftime("%d-%b-%Y (%H:%M:%S.%f)"))
What I see is a difference between local testing and testing on server.
A) On local, I can see log message at first script running, and log messages of requests (after initializing Tornado app) in my log file and the Tornado logs.
B) On server, I only see the first message, not my log messages when Get requests are accepted and also see Tornado's loggers when there's an error, but even don't see the messages produced by Tornado's loggers. I guess that means that somehow Tornado is re-initializing the logger and making mine and his 3 ones write in some other file (somehow that does not affect when errors happens??).
I am aware that Tornado uses its own 3 logging functions, but somehow I would like to use mine as well, at the same time as keeping the Tornado's ones and writing them all into the same file. Basically reproduce that local behaviour on server but also keeping it when some error happens, of course.
How could I achieve this?
Thanks in advance!
P.S.: if I add a name to the logger, let's say logging.getLogger('Example') and changed log_info_message function to return logging.getLogger('Example').log(20, msg), Tornado's logger would fail and raise error. So that option destroys its own loggers...
It seems the only problem was that, on the server side, tornado was setting the the mininum level for a log message to be written on log file higher (minimum of 40 was required). So that logging.getLogger().log(20, msg) would not write on the logging file but logging.getLogger().log(40, msg) would.
I would like to understand why, so if anybody knows, your knowledge would be more than welcome. For the time being that solution is working though.
tornado.log defines options that can be used to customise logging via command line (check tornado.options) - one of them is logging that defines the log level used. You are likely using this on the server and setting it to error.
When debugging logging I suggest you create a RequestHandler that will log or return the structure of the existing loggers by inspecting the root logger. When you see the structure it is much easier to understand why it works the way it works.
I'm new-ish to Python and Pyramid so apologies if I'm trying to do the wrong thing here.
I'm currently running a Pyramid application inside of a Docker container, with the following entrypoint:
pipenv run pserve development.ini --reload
This serves my application correctly and I can then edit code directly inside the container. This all works fine. I then attempted to register this service to an instance of Netflix's Eureka Service Registry so I could then proxy to this service with a gateway (such as Netflix Zuul). I used Eureka's REST API to achieve this and again, this all worked fine.
However, when I go to shutdown the Pyramid service, I would like to send an additional HTTP request to Eureka to DELETE the registered service - This is ideal so I don't have to wait for expiry on Eureka and there will never be a window where Zuul might be proxying requests to a downed service.
The problem is I cannot reliably find a way to run a shutdown event in Pyramid. Basically, when i stop the Docker container, the service receives exit code 137 (which I believe is the result of a kill -9) and nothing ever happens. I've attempted using atexit as well as signal event such as SIGKILL, SIGTERM, SIGINT, etc and nothing ever happens. I've also tried running pserve without a --reload flag but that still doesn't work.
Is there anyway for me to reliably get this DELETE event to send right before the server and docker container shuts down?
This is the development.ini file I'm using:
[app:main]
use = egg:my-app
pyramid.reload_templates = true
pyramid.includes =
pyramid_debugtoolbar
pyramid_redis_sessions
pyramid_tm
debugtoolbar.hosts = 0.0.0.0/0
sqlalchemy.url = mysql://root:root#mysql/keyblade
my-app.secret = secretkey
redis.sessions.secret = secretkey
redis.sessions.host = redis
redis.sessions.port = 6379
[server:main]
use = egg:waitress#main
listen = 0.0.0.0:8000
# Logging Configuration
[loggers]
keys = root, debug, sqlalchemy.engine.base.Engine
[logger_debug]
level = DEBUG
handlers =
qualname = debug
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console
[logger_sqlalchemy.engine.base.Engine]
level = INFO
handlers =
qualname = sqlalchemy.engine.base.Engine
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
There is no shutdown protocol/api for a WSGI application (there technically isn't one for startup either despite people using/hoping that application-creation is close to the same time the server starts handling requests). You may be able to find a WSGI server that provides some hooks (for example gunicorn provides http://docs.gunicorn.org/en/stable/settings.html#worker-exit), but the better approach is to have your upstream handle your servers disappearing via health checks. Expecting that you'll be able to send a DELETE reliably when things go wrong is very unlikely to be a robust solution.
However, when I go to shutdown the Pyramid service, I would like to send an additional HTTP request to Eureka to DELETE the registered service - This is ideal so I don't have to wait for expiry on Eureka and there will never be a window where Zuul might be proxying requests to a downed service.
This is a web server specific and Pyramid cannot provide abstractions for it, as "your mileage may vary". Web server workers itself cannot know when they killed, as it is externally forced.
I would take an approach where you have an external process to monitor to web server and then perform clean up actions when it detects the web server is no longer running. The definition of no longer running could be "no a single process alive". Then you just have a background scheduled job (cron) to check for this condition. Or even better, have it on another monitoring instance that sits on a different server and can act in the situation in the server itself goes down.
I'd like to add kind of application logging in a REST API I'm developing with the fantastic Python EVE framework. The goal is to provide auditing features to the application, so that application logging is my first choice for it.
I've followed steps detailed in this section from the official documentation, but I haven't been lucky. Probably I'm missing something:
from eve import Eve
import logging
def log_every_get(resource, request, payload):
# custom INFO-level message is sent to the log file
app.logger.info('We just answered to a GET request!')
app = Eve()
app.on_post_GET += log_every_get
if __name__ == "__main__":
# enable logging to 'app.log' file
handler = logging.FileHandler('<chrootedfolder>/app.log')
# set a custom log format, and add request
# metadata to each log line
handler.setFormatter(logging.Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(filename)s:%(lineno)d] -- ip: %(clientip)s, '
'url: %(url)s, method:%(method)s'))
# the default log level is set to WARNING, so
# we have to explictly set the logging level
# to INFO to get our custom message logged.
app.logger.setLevel(logging.INFO)
# append the handler to the default application logger
app.logger.addHandler(handler)
app.run()
To give you some more info about the environment where I'm running my REST API: it's running under UWSGI (chrooted to a specific folder) which is behind NGINX (acting as a proxy).
Hope you can help somehow guys.
Thanks in advance.
I'm having an issue logging to Sentry from within a celery task. Errors in tasks work fine. However, when I try to manually log an event, it gets logged to the celery logs, but not to the sentry server.
The code I'm using is:
#task
def myWorker():
logger = logging.getLogger('celery.task')
logger.addHandler(SentryHandler())
logger.warn("Some condition happened", exc_info=True, extra={ 'extra': 'data' })
I've found some posts on here and around the net on this, but they all seem to be very out of date