We Keep Coding

Common logging module in Python

I am new to python and just trying to learn and find better ways to write code. I want to create a custom class for logging and use package logging inside it. I want the function in this class to be reusable and do my logging from other scripts rather than writing custom code in each and every scripts. Is there a good link you guys can share? Is this the right way to handle logging? I just want to avoid writing the same code in every script if I can reuse it from one module.
I would highly appreciate any reply.

You can build a custom class that utilizes the built in python logging library. There isn't really any right way to handle logging as the library allows you to use 5 standard levels indicating the severity of events (DEBUG, INFO, WARNING, ERROR, and CRITICAL). The way you use these levels are application specific. Here's another good explanation of the package.

It's indeed a good idea to keep all your logging configuration (formatters, level, handlers) in one place.
create a class wrapping a custom logger with your configuration
expose methods for logging with different levels
import this class wherever you want
create an instance of this class to log where you want
To make sure all you custom logging objects have the same config, you should make logging class own the configuration.
I don't think there's any links I can share for the whole thing but you can find links for the individual details I mentioned easily enough.

Custom Processing on Python logging messages in a generic way

I am trying to figure out the best approach to apply some custom processing on Python logging messages with minimal impact to our codebase.
The problem is this: we have many different projects logging a lot of things, and among these can be found some AWS keys. As a security requirement, we need to strip out all AWS keys from the logs, and there are multiple ways to go about this:
The naive approach would be to go in each and every project, and modify each logging call to manually strip out keys. This is the least preferred approach as it would be the most manual.
Implement a different module that provides the same function as the logging module (like info, error, ...) and each function definition would first apply a regex to filter out AWS keys, and then call the actual logging method behind the scenes. Then each project can be modified to something like import custom_logging_module as logging and none of the logging calls need to be modified. The drawback of this approach though is that it looks like every logging call comes from this module in the log, so you can't track where your messages originate from.
Not sure in what form yet, but it sounds like it would be possible to implement a custom Logger or LogRecord and register it when initializing the logging. This wouldn't have the problems of the previous approach.
I have done some research on approach #3 but couldn't really find a way to do this. Does anyone have experience applying some custom processing on logging messages that would apply to this use case?

You could use a custom LogRecord class to achieve this, as long as you could identify keys in text unambiguously. For example:
import logging
import re
KEY = 'PK_SOME_PUBLIC_KEY'
SECRET_KEY = 'SK_SOME_PRIVATE_KEY'
class StrippingLogRecord(logging.LogRecord):
pattern = re.compile(r'\b[PS]K_\w+\b', re.I)
def getMessage(self):
message = super(StrippingLogRecord, self).getMessage()
message = self.pattern.sub('-- key redacted --', message)
return message
if hasattr(logging, 'setLogRecordFactory'):
# 3.x has this
logging.setLogRecordFactory(StrippingLogRecord)
else:
# 2.x needs monkey-patching
logging.LogRecord = StrippingLogRecord
logging.basicConfig(level=logging.DEBUG)
logging.debug('Message with a %s', KEY)
logging.debug('Message with a %s', SECRET_KEY)
In my example I've assumed you could use a simple regex to spot keys, but a more sophisticated alternative method could be used if that's not workable.
Note that the above code should be run before any of the code which logs keys.

extending python logging module

I want to extend python(2.7)'s logging module (and specifically the logger class).
Currently I have:
class MyLogger(logging.Logger):
def __init__(self, name):
logging.Logger.__init__(self, name)
Is this the right way to initialize MyLogger?
Will I be able to use Logger.manager (logging.Logger.manager)?
Is it possible to "get" a logger (I only know logging.getLogger(name) - which is not available since I'm extending the Logger itself, and I know static methods aren't popular in python as they are in Java, for example)?
Where can I learn more about extending classes? The documentation in python.org is very poor and did not help me.
My goal is to be able to start a logger with standard configurations and handlers based on the caller module's name, and to set the entire system loggers to the same level with a short, readable, call.

Seems like my approach was wrong altogether.
I prefer the way stated in python.org:
Using configuration files for the cleans up code and allows to propagate changes easily.
A configuration file is loaded like so:
# view example on python.org site (logging for multiple modules)
logging.config.fileConfig('logging.conf')
As for batch abilities, since we keep logging.Logger.manager.loggerDict and logging.getLogger, batch operations can use simple loops to create changes (like setting loggers to a single level) throughout the system.

What is the most Pythonic way to pass objects between imported libraries of code?

I have a program which imports two modules, one we will call operations (which is just a series of functions) and the other we call tracking (which is a class). The program tracking module monitors a series of messages, has some error state flags, and so forth. The program sorts information in tracking by severity and relevant parties, then dumps this to different files at the end.
I create a single instance of the tracking class with myTrack = tracking.Tracking(). (Yes, this is global state and therefore bad, but it is pretty handy)
Unforunately, I would like to use my tracking object within the operations module, just to track errors and warnings. It looks like I can pass myTrack to functions in the operations module as an argument, modifying each and every one of the functions.
However, is there a "better" or "more Pythonic" way to do this? I suspect there is something with namespaces which I have failed to grasp.

There are a lot of ways you could refactor this, but one place you might start is to add a track() function to your operations module. Have it do nothing by default.
def track(message): # use the right signature though!
pass
In your operations module you would then call your track() function anywhere you might want to track something.
Assuming your Tracking object has a method called track() that does the actual tracking, then in your main module, you can do this:
myTrack = tracking.Tracking()
operations.track = myTrack.track
This replaces the module's (do-nothing) track function with the method from your Tracking instance.
Basically, you have provided a "hook" which anyone can use to add tracking to your operations module, and then used that hook yourself.
Yes, this is more "global state," but it's module-global, which is not really global.

Your tracking module (recording details about a series of events for later analysis or display) sounds suspiciously like the standard library's logging module. So you may want to investigate that a little more closely.
If you decide to head down that path, then the operations module would just log events to a specific logger (e.g. "myapp.operations"), your tracking module would provide a logging handler that did what you wanted, and your main module would hook the two together (by registering the tracking handler with the "myapp.operations" logger).
You can also set up something like that yourself, but really, if you want to track events in a Python program... just use logging.

I am not sure I understand your problem correctly, but I think you are looking for a way to make the functions in one module automatically aware of the state an object in another module without explicitly passing that object every time you call a function.
The basic problem is that at some level you have to pass the object and have it available to all the functions you need. Modules are simply not meant to work like that.
I think a better idea will be to define an Operations class that contains all the functions you need as methods as well as holding an instance of Tracking. You can just pass in your Tracking object and create an Operations instance, and use that to call whatever function that you need.
myTrack = tracking.Tracking()
myOperation=operations.Operations(myTrack)
myOperation.doSomething()

Method logging in Python

I'd like something equivalent to
calling method: $METHOD_NAME
args: $ARGS
output: $OUTPUT
to be automatically logged to a file (via the logging module, possibly) for every (user-defined) method call. The best solution I can come up with is to write a decorator that will do this, and then add it to every function. Is there a better way?
Thanks

You could look at the trace module in the standard library, which
allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line.
You can also log to disk:
import sys
import trace
# create a Trace object, telling it what to ignore, and whether to
# do tracing or line-counting or both.
tracer = trace.Trace(
ignoredirs=[sys.prefix, sys.exec_prefix],
trace=0,
count=1)
# run the new command using the given tracer
tracer.run('main()')
# make a report, placing output in /tmp
r = tracer.results()
r.write_results(show_missing=True, coverdir="/tmp")

One approach that might simplify things a bit would be to use a metaclass to automatically apply your decorator for you. It'd cut down on the typing at the expense of requiring you to delve into the arcane and mysterious world of metaclass programming.

It depends how exactly are you going to use it.
Most generic approach would be to follow stdlib's 'profile' module path and therefore have control over each call, but its somewhat slow.
If you know which modules you need to track before giving them control, I'd go with iterating over all their members and wrapping with tracking decorator. This way tracked code stays clean and it doesn't take too much coding to implement.

A decorator would be a simple approach for a smaller project, however with decorators you have to be careful about passing arguments to make sure that they don't get mangled on their way through. A metaclass would probably be more of the "right" way to do it without having to worry about adding decorators to every new method.