My project uses these python libraries:
requests==2.18.4
requests-toolbelt==0.8.0
and calls this at the main.py
requests_toolbelt.adapters.appengine.monkeypatch()
App engine is logging Error message inside google cloud logs when using the requests library:
AppEnginePlatformWarning: urllib3 is using URLFetch on Google App
Engine sandbox instead of sockets. To use sockets directly instead of
URLFetch see
https://urllib3.readthedocs.io/en/latest/reference/urllib3.contrib.html.
AppEnginePlatformWarning: URLFetch does not support granular timeout
settings, reverting to total or default URLFetch timeout.
but this is actually just Warning and my code works as expected. But this log is problematic for me because I cannot know if there is really error or not in the logs when I see the statistics. That's why I have to log it as warning somehow.
Here is one stackoverflow answer. This answer states that if this warning is shown on GAE standard environment then the code will work without problem. So it's really warning for me. How to log it as such?
AppEnginePlatformWarning: urllib3 is using URLFetch on Google App Engine sandbox instead of sockets
You can do this using the logging.captureWarnings function.
From the docs:
This function is used to turn the capture of warnings by logging on
and off.
If capture is True, warnings issued by the warnings module will be
redirected to the logging system. Specifically, a warning will be
formatted using warnings.formatwarning() and the resulting string
logged to a logger named 'py.warnings' with a severity of WARNING.
If capture is False, the redirection of warnings to the logging system
will stop, and warnings will be redirected to their original
destinations (i.e. those in effect before captureWarnings(True) was
called).
Executing logging.captureWarnings(True) in appengine_config.py causes these warnings to be logged as warnings for me.
See also the docs for the warnings module.
Edit:
This question includes this code fragment for suppressing the message altogether:
# Use the App Engine Requests adapter. This makes sure that Requests uses
# URLFetch.
requests_toolbelt.adapters.appengine.monkeypatch()
# squelch warning
requests.packages.urllib3.disable_warnings(
requests.packages.urllib3.contrib.appengine.AppEnginePlatformWarning
)
Related
I'm making a project and using a library from the requirements of the project. The library implements logging and automatically logs to a file, when I call its functions.
Now, I'm implementing logging by myself, and only want my own messages to be logged to the file and not the library messages.
One solution I thought of would be switching the logging file each time I call a function from the library and then removing that file, but that seems overly complicated and clutterly. What can I do in order to not log the messages from the library?
P.S.:
I'm using the logging library and I initalize it as:
logging.basicConfig(level = logging.INFO,filename = loggingFile,format = "%(message)s")
, which means, all messages from myself and from the library will get logged in loggingFile
Libraries should not directly output anything to logs - that should be done only by handlers configured by an application. A library that logs output is an anti-pattern - if a library definitely does that, I'd log a bug against that library's issue tracker.
On the other hand, it might be that the library is only outputting stuff because you have configured output via your basicConfig() call. If you need more than basic configuration, don't usebasicConfig() - use the other APIs provided.
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.
Is there a way to log waitress-serve output into a file?
The current command I use is:
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app
The application we used was not written with waitress in mind earlier, so we choose to serve it with command line to avoid change (for now at least).
TLDR waitress-serve doesn't provide a way to do it. See the 'how do i get it to log' section.
Background
Per the documentation for the command-line usage of waitress-serve, no - there's no way to setup logging. See arguments docs.
waitress-serve is just an executable to make running your server more convenient. It's source-code is here runner.py. If you read it, you can see it actually is basically just calling from waitress import serve; serve(**args) for you. (That code clip is not literally what it's doing, but in spirit yes).
The documentation for waitress says that it doesn't log http traffic. That's not it's job. But it will log it's own errors or stacktraces. logging docs. If you read the waitress source trying to find when it logs stuff, you'll notice it doesn't seem to log http traffic anywhere github log search. It primarily logs stuff to do with the socket layer.
Waitress does say that if you want to log http traffic, then you need another component. In particular, it points you to pastedeploy docs which is some middle-ware that can log http traffic for you.
The documentation from waitress is actually kind of helpful answering you question, though not direct and explicit. It says
The WSGI design is modular.
per the logging doc
I.e. waitress won't log http traffic for you. You'll need another WSGI component to do that, and because WSGI is modular, you can probably choose a few things.
If you want some background on how this works, there's a pretty good post here leftasexercise.com
OK, how do I get it to log?
Use tee
Basically, if you just want to log the same stuff that is output from waitress-serve then you don't need anything special.
waitress-serve --listen=localhost:8080 --threads=1 my_app_api:app | tee -a waitress-serve.log
Python logging
But if you're actually looking for logging coming from python's standard logger (say you app is making logger calls or you want to log http traffic) then, you can set that up in your python application code. E.g. edit your applications soure-code and get it to setup logging to a file
import logging
logging.basicConfig(filename='app.log', encoding='utf-8', level=logging.DEBUG)
PasteDeploy middleware for http logs
Or if your looking for apache type http logging then you can use something like PasteDeploy to do it. Note, PasteDeploy is another python dependency so you'll need to install it. E.g.
pip install PasteDeploy
Then you need to setup a .ini file that tells PasteDeploy how to start your server and then also tell it to use TransLogger to create apache type http logs. This is explained more detail here logging with pastedeploy The ini file is specific to each app, but from your question is sounds like the ini file should look like:
[app:wsgiapp]
use = my_app_api:app
[server:main]
use = egg:waitress#main
host = 127.0.0.1
port = 8080
[filter:translogger]
use = egg:Paste#translogger
setup_console_handler = False
[pipeline:main]
pipeline = translogger
app
You'll still need to edit the source-code of your app to get PasteDeploy to load the app with your configuration file:
from paste.deploy import loadapp
wsgi_app = loadapp('config:/path/to/config.ini')
Webframework-dependent roll-your-own http logging
Even if you want to log http traffic, you don't necessarily need something like PasteDeploy. For example, if you are using flask as the web-framework, you can write your own http logs using after_request decorator:
#app.after_request
def after_request(response):
timestamp = strftime('[%Y-%b-%d %H:%M]')
logger.error('%s %s %s %s %s %s', timestamp, request.remote_addr, request.method, request.scheme, request.full_path, response.status)
return response
See the full gist at https://gist.github.com/alexaleluia12/e40f1dfa4ce598c2e958611f67d28966
I'm building a website using Flask and I'm now in the process of adding some logging to it for which I found these docs. The basic example is as follows:
if not app.debug:
import logging
from themodule import TheHandlerYouWant
file_handler = TheHandlerYouWant(...)
file_handler.setLevel(logging.WARNING)
app.logger.addHandler(file_handler)
after which you can log using app.logger.error('An error occurred'). This works fine, but apart from the fact that I do not see any advantage over the regular python logging module I also see a major downside: if I want to log outside of a request context (when for example running some code with a cron job) I get errors because I'm using app outside of the request context.
So my main question; why would I use the Flask logger at all? What is the reason that it was ever built?
The Flask logger uses the "generic" Python logging, it's a logging.getLogger(name) like any other.
The logger exists so that Flask app and views can log things that happen during execution. For example, it will log tracebacks on 500 errors during debug mode. The configuration example is there to show how to enable these logs, which are still useful in production, when you are not in debug mode.
Having an internal logger is not unique to Flask, it's the standard way to use logging in Python. Every module defines it's own logger, but the configuration of the logging is only handled by the last link in the chain: your project.
You can also use app.logger for your own messages, although it's not required. You could also create a separate logger for your own messages. In the end, it's all Python logging.
I use the module mechanize in order to log in a site. When I import twill.commands without any other apparent use, some debug messages [0] are displayed [1]. When I delete it, these messages disappear.
How can I see what is changed in the environment in order to emulate it and remove this dependency?
[0] Using the logging module.
[1] More specifically, I am interested in a Following HTTP-EQUIV=REFRESH message.
UPDATE: It turned out that there is a bug in twill.commands which was creating an error when trying to follow the HTTP-EQUIV=REFRESH header. After removing the import twill.commands and the ugly work around it, everything works smoothly.
My guess - without digging in the libraries - is that twill is instantiating a logger, and mechanize is doing the Right Thing for a library, logging if logging has been turned on, not if not.
To enable the logging of mechanize configure a logging.basicConfig root in your application code.
twill uses mechanize internally, you can log into a web site directly with twill.
To follow http-equiv redirection, just use the go command.
go <url> -- visit the given URL. The Python function returns the final URL visited, after all redirects.
To debug http-equiv redirects, enable the relevant debug level.
debug <what> <level> -- turn on or off debugging/tracing for
various functions. The first argument is either 'http' to show HTTP headers, 'equiv-refresh' to test HTTP EQUIV-REFRESH headers, or 'commands' to show twill commands. The second argument is '0' for off, '1' for on.