We Keep Coding

Best practices for logging in Python in shared libraries

In a nutshell, I write ETL pipelines. They are usually described in high-level scripts. In them, I use different internal libraries (we manage them) that provide utility functions, tooling or internal data structure.
What are the common best practices about logging when dealing with multiple packages import from different repositories?
My questions are:
1) Should I put logs in libraries? Or only in top-level scripts?
On one hand, It could be useful to display some information in some library functions/classes. On the other hand, it imposes the library client usage of a particular logger.
I checked a few open-source projects and it seems that there are no logs at all.
2) If we indeed put logs in all shared libraries, what is the best practices in Python to pass a unique logger to everything?
I want my logging format and strategy to be consistent in each library call as everything is run as part "as a whole". Should I init my logger in the main script and pass the same logger in every object I create? It seems redundant to me. I saw another pattern where all classes that need logging would inherit from a logging class. It seems to me that it might overkill and complicates the overall architecture.
I read in another stackoverflow that Actually every logger is a child of the parent's package logger. How to apply that when the packages come different repositories?
thanks

Add a logger with no handlers (or with just the null handler) to the library and do all the internal logging with that. Give it a name that is related to the library. When you do that any app that uses the lib can get the logger and add a handler to access the logs as needed.
An example would be the requests library which does something similar to that.
import logging
import requests
r = logging.getLogger('requests')
r.addHandler(logging.StreamHandler())
r.setLevel(logging.DEBUG)
requests.get('http://stackoverflow.com')
will print
Starting new HTTP connection (1): stackoverflow.com
http://stackoverflow.com:80 "GET / HTTP/1.1" 301 143
Starting new HTTPS connection (1): stackoverflow.com
https://stackoverflow.com:443 "GET / HTTP/1.1" 200 23886

How to make SQLAlchemy and Flask use the same logger?

probably I don't quite understand how logging really works in Python. I'm trying to debug a Flask+SQLAlchemy (but without flask_sqlalchemy) app which mysteriously hangs on some queries only if run from within Apache, so I need to have proper logging to get meaningful information. The Flask application by default comes with a nice logger+handler, but how do I get SQLAlchemy to use the same logger?
The "Configuring Logging" section in the SQLAlchemy just explains how to turn on logging in general, but not how to "connect" SQLAlchemy's logging output to an already existing logger.
I've been looking at Flask + sqlalchemy advanced logging for a while with a blank, expressionless face. I have no idea if the answer to my question is even in there.
EDIT: Thanks to the answer given I now know that I can have two loggers use the same handler. Now of course my apache error log is littered with hundreds of lines of echoed SQL calls. I'd like to log only error messages to the httpd log and divert all lower-level stuff to a separate logfile. See the code below. However, I still get every debug message into the http log. Why?
if app.config['DEBUG']:
# Make logger accept all log levels
app.logger.setLevel(logging.DEBUG)
for h in app.logger.handlers:
# restrict logging to /var/log/httpd/error_log to errors only
h.setLevel(logging.ERROR)
if app.config['LOGFILE']:
# configure debug logging only if logfile is set
debug_handler = logging.FileHandler(app.config['LOGFILE'])
debug_handler.setLevel(logging.DEBUG)
app.logger.addHandler(debug_handler)
# get logger for SQLAlchemy
sq_log = logging.getLogger('sqlalchemy.engine')
sq_log.setLevel(logging.DEBUG)
# remove any preconfigured handlers there might be
for h in sq_log.handlers:
sq_log.removeHandler(h)
h.close()
# Now, SQLAlchemy should not have any handlers at all. Let's add one
# for the logfile
sq_log.addHandler(debug_handler)

You cannot make SQLAlchemy and Flask use the same logger, but you can make them writing to one place by add a common Handler. And maybe this article is helpful: https://www.electricmonk.nl/log/2017/08/06/understanding-pythons-logging-module/
By the way, if you want to get all logs in one single request, you can set a uniq name for current thread before request, and add the threadName in you logging's formatter.

Answer to my question at EDIT: I still had "echo=True" set on the create_engine, so what I saw was all the additional output on stderr. echo=False stops that but still logs to debug level DEBUG.

Clear all corresponding handlers created by SqlAlchemy:
logging.getLogger("sqlalchemy.engine.Engine").handlers.clear()
The code above should be called after engine created.

Accessing Request from __init__.py in Pyramid

When setting up a Pyramid app and adding settings to the Configurator, I'm having issues understanding how to access information from request, like request.session and such. I'm completely new at using Pyramid and I've searched all over the place for information on this but found nothing.
What I want to do is access information in the request object when sending out exception emails on production. I can't access the request object, since it's not global in the __init__.py file when creating the app. This is what I've got now:
import logging
import logging.handlers
from logging import Formatter
config.include('pyramid_exclog')
logger = logging.getLogger()
gm = logging.handlers.SMTPHandler(('localhost', 25), 'email#email.com', ['email#email.com'], 'Error')
gm.setLevel(logging.ERROR)
logger.addHandler(gm)
This works fine, but I want to include information about the logged in user when sending out the exception emails, stored in session. How can I access that information from __init__.py?

Attempting to make request a global variable, or somehow store a pointer to "current" request globally (if that's what you're going to try with subscribing to NewRequest event) is not a terribly good idea - a Pyramid application can have more than one thread of execution, so more than one request can be active within a single process at the same time. So the approach may appear to work during development, when the application runs in a single thread mode and just one user accesses it, but produce really funny results when deployed to a production server.
Pyramid has pyramid.threadlocal.get_current_request() function which returns thread-local request variable, however, the docs state that:
This function should be used extremely sparingly, usually only in unit
testing code. it’s almost always usually a mistake to use
get_current_request outside a testing context because its usage makes
it possible to write code that can be neither easily tested nor
scripted.
which suggests that the whole approach is not "pyramidic" (same as pythonic, but for Pyramid :)
Possible other solutions include:
look at exlog.extra_info parameter which should include environ and params attributes of the request into the log message
registering exception views would allow completely custom processing of exceptions
Using WSGI middleware, such as WebError#error_catcher or Paste#error_catcher to send emails when an exception occurs
if you want to log not only exceptions but possibly other non-fatal information, maybe just writing a wrapper function would be enough:
if int(request.POST['donation_amount']) >= 1000000:
send_email("Wake up, we're rich!", authenticated_userid(request))

How do I configure Warming Requests avoiding errors 404 on app's log?

I have enabled Warming Requests to my app, adding the following directive in app.yaml.
inbound_services:
- warmup
Looking at app's log I see several entries of this kind:
1.
01-05 02:49PM 50.037 /_ah/warmup 404 300ms 280cpu_ms 1kb
See details
0.1.0.3 - - [05/Jan/2011:05:49:50 -0800] "GET /_ah/warmup HTTP/1.1" 404 1188
2.
I 01-05 02:49PM 50.336
This request caused a new process
to be started for your application,
and thus caused your application code
to be loaded for the first time.
This request may thus take longerand
use more CPU than a typical request
for your application.
This makes sense because the Warming Requests documentation says:
This causes the App Engine
infrastructure to issue GET requests
to /_ah/warmup. You can implement
handlers in this directory to perform
application-specific tasks, such as
pre-caching application data.
AFAIK ah is a reserved URL, i.e. script handler and static file handler paths will never match these paths!
Should I simply add the ah/warmup route associating it to an empty web handler for example? Is this correct?

Urls starting with /_ah/ work just fine, despite what the documentation might lead you to believe.
So, yes, just map a handler to /_ah/warmup to make the warmup requests work. I'm not sure how much benefit you'll get from using an empty handler, though. Usually you'd want to import all of your important modules and do any cache warmups your app needs to be responsive.

Environment variables

I use the module mechanize in order to log in a site. When I import twill.commands without any other apparent use, some debug messages [0] are displayed [1]. When I delete it, these messages disappear.
How can I see what is changed in the environment in order to emulate it and remove this dependency?
[0] Using the logging module.
[1] More specifically, I am interested in a Following HTTP-EQUIV=REFRESH message.
UPDATE: It turned out that there is a bug in twill.commands which was creating an error when trying to follow the HTTP-EQUIV=REFRESH header. After removing the import twill.commands and the ugly work around it, everything works smoothly.

My guess - without digging in the libraries - is that twill is instantiating a logger, and mechanize is doing the Right Thing for a library, logging if logging has been turned on, not if not.
To enable the logging of mechanize configure a logging.basicConfig root in your application code.

twill uses mechanize internally, you can log into a web site directly with twill.
To follow http-equiv redirection, just use the go command.
go <url> -- visit the given URL. The Python function returns the final URL visited, after all redirects.
To debug http-equiv redirects, enable the relevant debug level.
debug <what> <level> -- turn on or off debugging/tracing for
various functions. The first argument is either 'http' to show HTTP headers, 'equiv-refresh' to test HTTP EQUIV-REFRESH headers, or 'commands' to show twill commands. The second argument is '0' for off, '1' for on.