I'm trying to use logging module to write my logs directly to MongoDB. I figured out there's a way to use logging module along with log4mongo module of MongoDB. Below is the sample code I wrote -
import logging
from log4mongo.handlers import MongoHandler
logger = logging.getLogger('test')
logger.addHandler(MongoHandler(host='localhost', database_name='logs', collection='sample',
reuse=True))
logger.warning("This is a sample log message", extra={"ID":1})
logger.warning("This is a sample extra log message", extra={"ID":1})
The issue with this is, it creates two documents inside the sample collection with "ID":1. However, what I really wish is to write both the logs to the same document along with the field "ID:":1.
Is there a way to achieve this in Python + MongoDB? BufferedMongoHandler doesn't serve my purpose as it performs timed insertion of logs which is not what I require.
Related
Maybe someone here have a good idea on how to solve my issue. I have a REST API project driven by FastAPI. Every incomming request comes with a hash in the header. I am looking for a simple solution to write this hash as an extra parameter to the logs. I want to avoid adding it every time per hand. I first come up with the solution to write a Middleware which writes the hash in a Logger Object and then later use the loggerObject.log() function which adds the hash automatically. But this only works for my own log messages. Log messages from for example exceptions or from libraries I use dont have the extra parameter.
I solved a similar problem using the structlog library. It allows you to set global context variables that will be outputed to every log line.
Note that this only applies to logs outputted by structlog, special configuration is required in order to output the stdlib logging logs with these context variables.
Here is a demo FastAPI application I wrote that generates a request-id for every incoming http request, and adds the request ID to every log line outputted during that HTTP request.
In this example I have configured the stdlib logging logs to be processed by structlog, so them to have the request-id parameter added to them.
https://gitlab.com/sagbot/structlog-demo
Reference files in demo:
demo/route.py - A bunch of demo routes. note the /long route outputs both stdlib logs and structlog logs, but both of them are outputted with the request-id property
demo/configure-logging.py - This func configures both structlog and stdlib logging. Note that the "Special configuration" required to show the context variables in stdlib logs is in this func, where I added the custom formatters in logging.config.dictConfig(...) func
demo/middlewares/request_logging/middleware.py - This is the middleware that generates and adds the request-id to the logs. You can alter this middleware to not generate the value, but extract it from the hash header of the incomming request, as you wanted
In python 2.7, the app engine sdk was doing the work in the background to nest all logs with the parent request to have a correlation in Google StackDriver.
As of the transition to python 3, it is through the usage of google cloud logging or structured logging, and from all the different references I could found, it's important to have the same trace id in the 'sub' logs for stack driver to make a match with the 'request' log.
And still as you can see below, it still appear as different logs.
For context, I even tried this on an empty django project deployed on app engine.
Got the same result, even when following the example in the documentation:
https://cloud.google.com/run/docs/logging#writing_structured_logs
Trying to log to the stdout is giving the same result.
Edit:
After the initial request, all other request will be nested under the initial request when using the stdout.
But, the highest severity of the 'child' logs is not taken by the 'parent' log, therefore the filters won't pick up the actual log. See below:
Thanks for the question!
It looks like you're logging the trace correctly, but your logName indicates that you're not using a stdout or stderr. If you use one of these for your logs, they will correlate properly, like this:
StackDriver Logs Screenshot
You can see that the logName ends with stdout. An stdout or stderr will correlate. You can create this as shown here in the tutorial:
# Build structured log messages as an object.
global_log_fields = {}
# Add log correlation to nest all log messages
# beneath request log in Log Viewer.
trace_header = request.headers.get('X-Cloud-Trace-Context')
if trace_header and PROJECT:
trace = trace_header.split('/')
global_log_fields['logging.googleapis.com/trace'] = (
f"projects/{PROJECT}/traces/{trace[0]}")
# Complete a structured log entry.
entry = dict(severity='NOTICE',
message='This is the default display field.',
# Log viewer accesses 'component' as jsonPayload.component'.
component='arbitrary-property',
**global_log_fields)
print(json.dumps(entry))
EDIT:
To filter out the stdout and only see the Request logs in the stackdriver UI, you can de-select the stdout from the filter. Logs Filter
For a sample using the python client API, please see this article and attached sample Flask app. Combining correlated Log Lines in Google Stackdriver
I was able to achieve this kind of logging structure on Google Cloud Logging Console:
I was using the Django Framework. I wrote Django middleware which integrates Google Cloud Logging API.
"Trace" needs to be added to every log object which points to its parent log object.
please check manage nesting of logs in Google Stackdriver with Django
Please check django-google-stackdriver-nested-logging log_middleware.py source on Github.
I have created a Python Azure Functions app. In this application I want to check log details like DEBUG, INFO etc. I have written some code for logging purpose, but I am not able to get any log after executing my azure function application.
I have written basic code for logging purpose as below but after executing Azure Functions I am not able to see the log on console.
import logging
import azure.functions as func
data = "Hello"
logging.basicConfig(level=logging.DEBUG)
logging.debug(data)
Is there any other solution or workaround for the above problem?
Most likely the root logger got messed with by azure and basicConfig only creates a root logger with some sane defaults. To work around this create your own independent logger.
import logging
logger = logging.getLogger('akshay')
logger.setLevel(logging.DEBUG)
sh = logging.StreamHandler()
sh.setLevel(logging.DEBUG)
logger.addHandler(sh)
logger.debug('Hello')
# Outputs 'Hello'
This GitHub issue explains the solution:
As for the original issue noted here, the desired behavior can be achieved with a combination of two steps:
Setting the root logger in user code to use the debug level (it is set to to info by default).
Configuring the host.json to log Debug messages: [...]
Specifically, this is the code that you need in your script:
import logging
logging.Logger.root.level = 10
logging.debug("Debug message here")
probably I don't quite understand how logging really works in Python. I'm trying to debug a Flask+SQLAlchemy (but without flask_sqlalchemy) app which mysteriously hangs on some queries only if run from within Apache, so I need to have proper logging to get meaningful information. The Flask application by default comes with a nice logger+handler, but how do I get SQLAlchemy to use the same logger?
The "Configuring Logging" section in the SQLAlchemy just explains how to turn on logging in general, but not how to "connect" SQLAlchemy's logging output to an already existing logger.
I've been looking at Flask + sqlalchemy advanced logging for a while with a blank, expressionless face. I have no idea if the answer to my question is even in there.
EDIT: Thanks to the answer given I now know that I can have two loggers use the same handler. Now of course my apache error log is littered with hundreds of lines of echoed SQL calls. I'd like to log only error messages to the httpd log and divert all lower-level stuff to a separate logfile. See the code below. However, I still get every debug message into the http log. Why?
if app.config['DEBUG']:
# Make logger accept all log levels
app.logger.setLevel(logging.DEBUG)
for h in app.logger.handlers:
# restrict logging to /var/log/httpd/error_log to errors only
h.setLevel(logging.ERROR)
if app.config['LOGFILE']:
# configure debug logging only if logfile is set
debug_handler = logging.FileHandler(app.config['LOGFILE'])
debug_handler.setLevel(logging.DEBUG)
app.logger.addHandler(debug_handler)
# get logger for SQLAlchemy
sq_log = logging.getLogger('sqlalchemy.engine')
sq_log.setLevel(logging.DEBUG)
# remove any preconfigured handlers there might be
for h in sq_log.handlers:
sq_log.removeHandler(h)
h.close()
# Now, SQLAlchemy should not have any handlers at all. Let's add one
# for the logfile
sq_log.addHandler(debug_handler)
You cannot make SQLAlchemy and Flask use the same logger, but you can make them writing to one place by add a common Handler. And maybe this article is helpful: https://www.electricmonk.nl/log/2017/08/06/understanding-pythons-logging-module/
By the way, if you want to get all logs in one single request, you can set a uniq name for current thread before request, and add the threadName in you logging's formatter.
Answer to my question at EDIT: I still had "echo=True" set on the create_engine, so what I saw was all the additional output on stderr. echo=False stops that but still logs to debug level DEBUG.
Clear all corresponding handlers created by SqlAlchemy:
logging.getLogger("sqlalchemy.engine.Engine").handlers.clear()
The code above should be called after engine created.
I am a newbie using python and I wanted to ask for your help in showing me how can I print messages from Python into robot framework console.
There are several ways for a python function to send information to the robot logs or to the console. These are all documented in the Robot framework user's guide, in the section titled Logging information.
The cleanest way is to use the logging API, which gives specialized functions for various kinds of logging. For example, to send information to the console you would use logger.console(message).
Using the logging API
Here is a library file that uses this method:
# ExampleKeywords.py
from robot.api import logger
def write_to_console(s):
logger.console(s)
You can use this library in the following manner:
*** Settings ***
| Library | ExampleKeywords.py
*** Test Cases ***
| Use a custom keyword to write to the console
| | Write to console | Hello, world
This will appear in the console only, and will not show up in the logs. If you want the information to show up in the logs you can use logger methods info, warn, debug, or trace. To log an error you would simply throw an exception.
Calling built-in keywords
There are other ways for your custom keywords to send information to the logs. For example, you can get a reference to the BuiltIn library, and directly call the log or log to console keywords like this:
from robot.libraries.BuiltIn import BuiltIn
def write_to_console(s):
BuiltIn().log_to_console("Hello, world")
Using print statements
Finally, you can write information to the logs (but not only to the console) using print statements. You can prefix the string with *<level>* to affect the log level. For example, to print a warning you can do:
print "*WARN* Danger Will Robinson"
Summary
Using the API is arguably the best method to log information from your keywords. However, this is a fairly new API only available since Robot Framework 2.6, so if you are using an older version of Robot you may need to use one of the other techniques.
What it sounds like you're asking for is a way to write a library such that your testcase can print to the console during test execution.
This is pretty easy, you can find it in the RF Manual under Logging Information. The short version is you can log a "WARN" that will appear in both the log and on screen with a print statement, like so:
print "*WARN* This text will show up on the console."
This command can be used to print any message or content on robot console.
from robot.libraries.BuiltIn import BuiltIn
BuiltIn().log_to_console("Message that needs to be printed")