We Keep Coding

werkzeug: disable bash colors when logging to file

In a Flask application, I use a RotatingFileLogger to log werkzeug access logs to a file like shown in this question:
file_handler_access_log = RotatingFileHandler("access.log",
backupCount=5,
encoding='utf-8')
formatter = logging.Formatter('%(asctime)s %(module)s %(levelname)s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')
file_handler_access_log.setFormatter(formatter)
werkzeug_logger.addHandler(file_handler_access_log)
werkzeug_logger.setLevel(logging.DEBUG)
In the access.log file, the request looks like this:
2020-10-07 09:43:51 _internal INFO: 127.0.0.1 - - [07/Oct/2020 09:43:51] "[37mGET /api/foo HTTP/1.1[0m" 200 -
I want to get rid of the color codes like [37m in the log file.
The werkzeug documentation states:
The development server can optionally highlight the request logs in
different colors based on the status code. Install Click to enable
this feature.
Click is a Flask dependency, so I cannot uninstall it. How can I disable the colored logging?

OK, so what you are hitting is
if click:
color = click.style
if code[0] == "1": # 1xx - Informational
msg = color(msg, bold=True)
...
self.log("info", '"%s" %s %s', msg, code, size)
Source: https://github.com/pallets/werkzeug/blob/ef545f0d0bf28cbad02066b4cb7471bea50a93ee/src/werkzeug/serving.py
Not easy to prevent the behavior. The second option is to remove color codes from messages. I would try to use log Filter to update the message, something like
import logging
import click
class RemoveColorFilter(logging.Filter):
def filter(self, record):
if record and record.msg and isinstance(record.msg, str):
record.msg = click.unstyle(record.msg)
return True
remove_color_filter = RemoveColorFilter()
file_handler_access_log.addFilter(remove_color_filter)
The above suggestion was inspired by the following answer https://stackoverflow.com/a/60692906/4183498.
I didn't test the proposed solution.

I went with what i think is a more straightforward solution, which is to simply throw away all the styling arguments - something like this in your application startup
old_color = click.style
def new_color(text, fg=None, bg=None, bold=None, dim=None, underline=None, blink=None, reverse=None, reset=True):
return old_color(text)
# replace flask styling with non-colorized styling
click.style = new_color

This is a similar idea to #jhodges, but even simpler - it just removes all of the "click" module's styling altogether.
import click
click.style = lambda text, *args, **kwargs: text
EDIT: In the end, I wound up going with a filter to remove escape sequences from log messages headed for a file handler. Because werkzeug puts escape sequences in the args list for the log message, those need to be stripped as well. This is the basic outline:
class NoEscape(logging.Filter):
def __init__(self):
self.regex = re.compile(r'(\x9B|\x1B\[)[0-?]*[ -\/]*[#-~]')
def strip_esc(self, s):
try: # string-like
return self.regex.sub('',s)
except: # non-string-like
return s
def filter(self, record: logging.LogRecord) -> int:
record.msg = self.strip_esc(record.msg)
if type(record.args) is tuple:
record.args = tuple(map(self.strip_esc, record.args))
return 1
Hope this helps!

I came across this question myself, but the filter solutions presented in other answers modified the record for all cases it was emitted, whereas I wanted to only remove styling for file handlers and leave it for console handlers.
I ended up subclassing Formatter instead of Filter:
import logging
import click
class AntiColorFormatter(logging.Formatter):
def format(self, record: logging.LogRecord) -> str:
return click.unstyle(super().format(record))
which then lends itself to something like:
logger = logging.getLogger()
FMT = '{asctime} {levelname} {message}'
handler = logging.StreamHandler()
# regular console output gets normal styling
handler.setFormatter(logging.Formatter(FMT, style='{'))
logger.addHandler(handler)
handler = logging.FileHandler('log.log', encoding='utf8')
# file handler gets styling stripped
handler.setFormatter(AntiColorFormatter(FMT, style='{'))
logger.addHandler(handler)

Thanks to #jhodges answer, that worked for me with a minor change:
# workaround to suppress color logging in werkzeug
try:
import click
old_color = click.style
def _color(text, fg=None, bg=None, bold=None, dim=None, underline=None, blink=None, reverse=None, reset=True):
return click.unstyle(text)
click.style = _color
except:
pass
Calling click.style(text) still produces escape sequences in some cases which I would like to avoid completely, so I used unstyle that works perfectly here.

What's the worst level log that I just logged?

I've added logs to a Python 2 application using the logging module.
Now I want to add a closing statement at the end, dependent on the worst thing logged.
If the worst thing logged had the INFO level or lower, print "SUCCESS!"
If the worst thing logged had the WARNING level, write "SUCCESS!, with warnings. Please check the logs"
If the worst thing logged had the ERROR level, write "FAILURE".
Is there a way to get this information from the logger? Some built in method I'm missing, like logging.getWorseLevelLogSoFar?
My current plan is to replace all log calls (logging.info et al) with calls to wrapper functions in a class that also keeps track of that information.
I also considered somehow releasing the log file, reading and parsing it, then appending to it. This seems worse than my current plan.
Are there other options? This doesn't seem like a unique problem.
I'm using the root logger and would prefer to continue using it, but can change to a named logger if that's necessary for the solution.

As you said yourself, I think writing a wrapper function would be the neatest and fastest approach. The problem would be that you need a global variable, if you're not working within a class
global worst_log_lvl = logging.NOTSET
def write_log(logger, lvl, msg):
logger.log(lvl, msg)
if lvl > worst_log_lvl:
global worst_log_lvl
worst_log_lvl = lvl
or make worst_log_lvl a member of a custom class, where you emulate the signature of logging.logger, that you use instead of the actual logger
class CustomLoggerWrapper(object):
def __init__(self):
# setup of your custom logger
self.worst_log_lvl = logging.NOTSET
def debug(self):
pass
# repeat for other functions like info() etc.

As you're only using the root logger, you could attach a filter to it which keeps track of the level:
import argparse
import logging
import random
LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
class LevelTrackingFilter(logging.Filter):
def __init__(self):
self.level = logging.NOTSET
def filter(self, record):
self.level = max(self.level, record.levelno)
return True
def main():
parser = argparse.ArgumentParser()
parser.add_argument('maxlevel', metavar='MAXLEVEL', default='WARNING',
choices=LEVELS,
nargs='?', help='Set maximum level to log')
options = parser.parse_args()
maxlevel = getattr(logging, options.maxlevel)
logger = logging.getLogger()
logger.addHandler(logging.NullHandler()) # needs Python 2.7
filt = LevelTrackingFilter()
logger.addFilter(filt)
for i in range(100):
level = getattr(logging, random.choice(LEVELS))
if level > maxlevel:
continue
logger.log(level, 'message')
if filt.level <= logging.INFO:
print('SUCCESS!')
elif filt.level == logging.WARNING:
print('SUCCESS, with warnings. Please check the logs.')
else:
print('FAILURE')
if __name__ == '__main__':
main()

There's a "good" way to get this done automatically by using context filters.
TL;DR I've built a package that has the following contextfilter baked in. You can install it with pip install ofunctions.logger_utils then use it with:
from ofunctions import logger_utils
logger = logger_utils.logger_get_logger(log_file='somepath', console=True)
logger.error("Oh no!")
logger.info("Anyway...")
# Now get the worst called loglevel (result is equivalent to logging.ERROR level in this case)
worst_level = logger_utils.get_worst_logger_level(logger)
Here's the long solution which explains what happens under the hood:
Let's built a contextfilter class that can be injected into logging:
class ContextFilterWorstLevel(logging.Filter):
"""
This class records the worst loglevel that was called by logger
Allows to change default logging output or record events
"""
def __init__(self):
self._worst_level = logging.INFO
if sys.version_info[0] < 3:
super(logging.Filter, self).__init__()
else:
super().__init__()
#property
def worst_level(self):
"""
Returns worst log level called
"""
return self._worst_level
#worst_level.setter
def worst_level(self, value):
# type: (int) -> None
if isinstance(value, int):
self._worst_level = value
def filter(self, record):
# type: (str) -> bool
"""
A filter can change the default log output
This one simply records the worst log level called
"""
# Examples
# record.msg = f'{record.msg}'.encode('ascii', errors='backslashreplace')
# When using this filter, something can be added to logging.Formatter like '%(something)s'
# record.something = 'value'
if record.levelno > self.worst_level:
self.worst_level = record.levelno
return True
Now inject this filter into you logger instance
logger = logging.getLogger()
logger.addFilter(ContextFilterWorstLevel())
logger.warning("One does not simply inject a filter into logging")
Now we can iter over present filters and extract the worst called loglevel like this:
for flt in logger.filters:
if isinstance(flt, ContextFilterWorstLevel):
print(flt.worst_level)

python- split() doesn't work inside __init__

I am writing code to serve a html file using wsgi.When I write a straight forward function I get no error like :
from wsgiref.simple_server import make_server
import os
...
...
def app(environ, start_response):
path_info = environ["PATH_INFO"]
resource = path_info.split("/")[1] #I get no error here the split works totally fine.
Now when I try to put the code inside a class I get error NoneType has no attribute split.
Perhaps the environ inside __init__ doesn't get initialised , that's why it split returns nothing. Following is the file in which my class Candy resides :
import os
class Candy:
def __init__(self):
#self.environ = environ
#self.start = start_response
self.status = "200 OK"
self.headers = []
def __call__(self , environ , start_response):
self.environ = environ
self.start = start_response
#headers = []
def content_type(path):
if path.endswith(".css"):
return "text/css"
else:
return "text/html"
def app(self):
path_info = self.environ["PATH_INFO"]
resource = path_info.split("/")[1]
#headers = []
self.headers.append(("Content-Type", content_type(resource)))
if not resource:
resource = "login.html"
resp_file = os.path.join("static", resource)
try:
with open(resp_file, "r") as f:
resp_file = f.read()
except Exception:
self.start("404 Not Found", self.headers)
return ["404 Not Found"]
self.start("200 0K", self.headers)
return [resp_file]
Following is the server.py file where I invoke my make_server :
from wsgiref.simple_server import make_server
from candy import Candy
#from app import candy_request
candy_class = Candy()
httpd = make_server('localhost', 8000, candy_class.app)
print "Serving HTTP on port 8000..."
# Respond to requests until process is killed
httpd.serve_forever()
# Alternative: serve one request, then exit
#httpd.handle_request()
Any help ? How to get this error sorted and am I right in my assumption?

To explain what you're doing wrong here, let's start with simple concepts - what a WSGI application is.
WSGI application is just a callable that receives a request environment, and a callback function that starts a response (sends status line and headers back to user). Then, this callable must return one or more strings, that constitute the response body.
In the simplest form, that you have working it's just
def app(environ, start_response):
start_response("200 OK", [("Content-Type", "text/plain")])
return "hello, world"
make_server('localhost', 8000, app).serve_forever()
Whenever a request comes, app function gets called, it starts the response and returns a string (or it could return an iterable of multiple strings, e.g. ["hello, ", "world"])
Now, if you want it to be a class, it works like this:
class MyApp(object):
def __init__(self):
pass
def __call__(self, environ, start_response):
start_response("200 OK", [("Content-Type", "text/plain")])
return "something"
app = MyApp()
make_server("localhost", 8000, app).serve_forever()
In this case, the callable is app, and it's actually __call__ method of Caddy class instance.
When request comes, app.__call__ gets called (__call__ is the magic method that turns your class instance in a callable), and otherwise it works exactly the same as the app function from the first example. Except that you have a class instance (with self), so you can do some pre-configuration in the __init__ method. Without doing anything in __init__ it's useless. E.g., a more realistic example would be this:
class MyApp(object):
def __init__(self):
self.favorite_color = "blue"
def __call__(self, environ, start_response):
start_response("200 OK", [("Content-Type", "text/plain")])
return "my favorite color is {}".format(self.favorite_color)
...
Then, there's another thing. Sometimes you want a streaming response, generated over time. Maybe it's big, or maybe it takes a while. That's why WSGI applications can return an iterable, rather than just a string.
def app(environ, start_response):
start_response("200 OK", [("Content-Type", "text/plain")]))
yield "This was a triumph\n"
time.sleep(1)
yield "I'm making a note here\n"
time.sleep(1)
yield "HUGE SUCCESS\n"
make_server("localhost", 8000, app).serve_forever()
This function returns a generator that returns text, piece by piece. Although your browser may not always show it like this, but try running curl http://localhost:8000/.
Now, the same with classes would be:
class MyApp(object):
def __init__(self, environ, start_response):
self.environ = environ
self.start = start_response
def __iter__(self):
self.start("200 OK", [("Content-Type", "text/plain")]))
yield "This was a triumph\n"
time.sleep(1)
yield "I'm making a note here\n"
time.sleep(1)
yield "HUGE SUCCESS\n"
make_server("localhost", 8000, MyApp).serve_forever()
Here, you pass MyApp (the class) as a application callable - which it is. When request comes it gets called, it's like someone had written MyApp(environ, start_response) somewhere, so __init__ starts and creates an instance for this specific request. Then, as the instance is iterated, __iter__ starts to produce a response. After it's done, the instance is discarded.
Basically, that's it. Classes here are only convenience closures that hold data. If you don't need them, don't use classes, use plain functions - flat is better than nested.
Now, about your code.
What your code uses for callable is Candy().app. This doesn't work because it doesn't even made to receive environ and start_response it will be passed. It should probably fail with 500 error, saying something like app() takes 1 positional arguments but 3 were given.
I assume the code in your question is modified after you got that NoneType has no attribute split issue, and you had passed something to the __init__ when creating candy_instance = Candy() when your __init__ still had 2 arguments (3 with self). Not even sure what exactly it was - it should've failed earlier.
Basically, you passed the wrong objects to make_server and your class was a mix of two different ideas.
I suggest to check my examples above (and read PEP-333), decide what you actually need, and structure your Candy class like that.
If you just need to return something on every request, and you don't have a persistent state - you don't need a class at all.
If you need a persistent state (config, or, maybe, a database connection) - use a class instance, with __call__ method, and make that __call__ return the response.
If you need to respond in chunks, use either a generator function, or a class with __iter__ method. Or a class with __call__ that yields (just like a function).
Hope this helps.

Writing Python output to either screen or filename

I am writing some code that will output a log to either the screen, or a file, but not both.
I thought the easiest way to do this would be to write a class:
class WriteLog:
"write to screen or to file"
def __init__(self, stdout, filename):
self.stdout = stdout
self.logfile = file(filename, 'a')
def write(self, text):
self.stdout.write(text)
self.logfile.write(text)
def close(self):
self.stdout.close()
self.logfile.close()
And then call it something like this:
output = WriteLog(sys.stdout, 'log.txt')
However, I'm not sure how to allow for switching between the two, i.e. there should be an option within the class that will set WriteLog to either use stdout, or filename. Once that option has been set I just use WriteLog without any need for if statements etc.
Any ideas? Most of the solutions I see online are trying to output to both simultaneously.
Thanks.

Maybe something like this? It uses the symbolic name 'stdout' or 'stderr' in the constructor, or a real filename. The usage of if is limited to the constructor. By the way, I think you're trying to prematurely optimize (which is the root of all evil): you're trying to save time on if's while in real life, the program will spend much more time in file I/O; making the potential waste on your if's negligible.
import sys
class WriteLog:
def __init__(self, output):
self.output = output
if output == 'stdout':
self.logfile = sys.stdout
elif output == 'stderr':
self.logfile = sys.stderr
else:
self.logfile = open(output, 'a')
def write(self, text):
self.logfile.write(text)
def close(self):
if self.output != 'stdout' and self.output != 'stderr':
self.logfile.close()
def __del__(self):
self.close()
if __name__ == '__main__':
a = WriteLog('stdout')
a.write('This goes to stdout\n')
b = WriteLog('stderr')
b.write('This goes to stderr\n')
c = WriteLog('/tmp/logfile')
c.write('This goes to /tmp/logfile\n')

I'm not an expert in it, but try to use the logging library, and maybe you can have logger with 2 handlers, one for file and one for stream and then add/remove handlers dynamically.

I like the suggestion about using the logging library. But if you want to hack out something yourself, maybe passing in the file handle is worth considering.
import sys
class WriteLog:
"write to screen or to file"
def __init__(self, output):
self.output = output
def write(self, text):
self.output.write(text)
def close(self):
self.output.close()
logger = WriteLog(file('c:/temp/log.txt','a' ))
logger.write("I write to the log file.\n")
logger.close()
sysout = WriteLog(sys.stdout)
sysout.write("I write to the screen.\n")

You can utilize the logging library to do something similar to this. The following function will set up a logging object at the INFO level.
def setup_logging(file_name, log_to_file=False, log_to_console=False ):
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# Create Console handler
if log_to_file:
console_log = logging.StreamHandler()
console_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
console_log.setFormatter(formatter)
logger.addHandler(console_log)
# Log file
if log_to_console:
file_log = logging.FileHandler('%s.log' % (file_name), 'a', encoding='UTF-8')
file_log.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(levelname)-8s - %(name)-12s - %(message)s')
file_log.setFormatter(formatter)
logger.addHandler(file_log)
return logger
You pass it the name of your log, and where you wish to log (either to the file, the console or both). Then you can utilize this function in your code block like this:
logger = setup_logging("mylog.log", log_to_file=True, log_to_console=False)
logger.info('Message')
This example will log to a file named mylog.log (in the current directory) and have output like this:
2014-11-05 17:20:29,933 - INFO - root - Message
This function has areas for improvement (if you wish to add more functionality). Right now it logs to both the console and file at log level INFO on the .setLevel(logging.INFO) lines. This could be set dynamically if you wish.
Additionally, as it is now, you can easily add standard logging lines (logger.debug('Message'), logger.critcal('DANGER!')) without modifying a class. In these examples, the debug messages won't print (because it is set to INFO) and the critical ones will.

How to start Bottle as a daemon from another script?

I would like to use BottlePy as a daemon started from another script and I have issues turning a standalone script (webserver.py) into a class. The standalone version of my webserver below works fine:
import bottle
#bottle.get('/hello')
def hello():
return 'Hello World'
#bottle.error(404)
def error404(error):
return 'error 404'
bottle.run(host='localhost', port=8080)
My intent is now to start it from the main script below as
from webserver import WebServer
from multiprocessing import Process
def start_web_server():
# initialize the webserver class
WebServer()
# mainscript.py operations
p = Process(target=start_web_server)
p.daemon = True
p.start()
# more operations
where WebServer() would be in the naively now-modified webserver.py:
import bottle
class WebServer():
def __init__(self):
bottle.run(host='localhost', port=8080)
#bottle.get('/hello')
def hello(self):
return 'Hello World'
#bottle.error(404)
def error404(self, error):
return 'error 404'
What works: the whole thing starts and the webserver is listening
What does not work: upon calling http://localhost:8080/hello
127.0.0.1 - - [11/Dec/2013 10:16:23] "GET /hello HTTP/1.1" 500 746
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\bottle.py", line 764, in _handle
return route.call(**args)
File "C:\Python27\lib\site-packages\bottle.py", line 1575, in wrapper
rv = callback(*a, **ka)
TypeError: hello() takes exactly 1 argument (0 given)
My questions are:
what kind of parameters am I expected to pass to hello() and error404()?
what should I do in order to parametrize #bottle.get('/hello')? I would like to have something like #bottle.get(hello_url) but where should hello_url = '/hello' be initialized? (self.hello_url is unknown to #bottle.get)
EDIT: while preparing a fork of this question to handle question 2 (about the parametrization) I had an epiphany and tried the obvious solution which works (code below). I am not that used to classes yet so I did not have the reflex to add the variable in the scope of the class.
# new code with the path as a parameter
class WebServer():
myurl = '/hello'
def __init__(self):
bottle.run(host='localhost', port=8080, debug=True)
#bottle.get(myurl)
def hello():
return 'Hello World'
#bottle.error(404)
def error404(error):
return 'error 404'

what kind of parameters am I expected to pass to hello() and error404()?
The short answer: none. Just remove self and they should start to work.
#bottle.get('/hello')
def hello():
return 'Hello World'
#bottle.error(404)
def error404(error):
return 'error 404'
what should I do in order to parametrize #bottle.get('/hello')? I
would like to have something like #bottle.get(hello_url) but where
should hello_url = '/hello' be initialized? (self.hello_url is unknown
to #bottle.get)
I can interpret this a couple of different ways, so I'm not sure how to help you. But since this is a completely separate question (with potentially a much larger scope), please consider asking it in a new, separate SO question.