I believe this question was raised lots of times already but I have a specific use case where I can't solve the issue with many of the methods described on the web.
In one of my projects, I am using joblib library, and it shows DeprecationWarning because it uses imp library somewhere internally:
from sklearn.externals.joblib import Parallel, delayed
def main():
xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
print(sum(xs))
if __name__ == '__main__':
main()
I am trying to filter out warning with interpreter option -W but it doesn't help:
$ python -W ignore example.py
[...]/lib/python3.7/site-packages/sklearn/externals/joblib/externals/cloudpickle/cloudpickle.py:47:
DeprecationWarning: the imp module is deprecated in favour of importlib;
see the module's documentation for alternative uses import imp
55
Also, I was trying an explicit filtering using warnings module but it is not helping also:
import warnings
warnings.simplefilter('ignore', category=DeprecationWarning)
from sklearn.externals.joblib import Parallel, delayed
def main():
xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
print(sum(xs))
if __name__ == '__main__':
main()
I had similar issues with matplotlib module and some other third-party libraries. Probably there are some other ways (i.e., env vars) but I don't understand why these solutions don't work.
Could anyone explain how the warnings system actually works in Python? It is possible that third-party libraries intentionally override client's warnings settings? I would say that this question is one of the most obscure topics for me.
As requested, here's the answer as a separate post:
The trick is to use "with" warnings while importing sklearn (or the dependency that uses sklearn, in my case it was the hdbscan package):
with warnings.catch_warnings():
# filter sklearn\externals\joblib\parallel.py:268:
# DeprecationWarning: check_pickle is deprecated
warnings.simplefilter("ignore", category=DeprecationWarning)
import hdbscan
This will disable DeprecationWarning only for this module (because warnings-modification is attached to the with-block).
It is important to put this statement at the first position in your code where the module is imported, otherwise it won't work. E.g. if I was loading hdbscan in __init__.py, and the above code block appears in some sub-class that also loads hdbscan, I would still get the DeprecationWarning because Python ignores any subsequent import statement if the module/package is already loaded.
Therefore, it is important to check which modules/packages uses joblib\parallel.py and where those, from a linear code-perspective, are earliest loaded to the python object heap.
[EDIT]
As #devforfu points out in comments, the above solution does not work (anymore). I've looked again into this and since Python 3.7 DeprecationWarning is once again shown by default when triggered directly by code in __main__.. Furthermore, ignore warnings doesn't seem to work when a Dependency explicitly loads a depreciated module of some other package.
This is what appears to happen in my hdbscan example, which loads the depreciated modules sklearn.external.six and sklearn.externals.joblib.
Here's how to finally solve this annoying issue:
make sure you've explicitly installed the standalone packages that are depreciated, e.g. conda install -c conda-forge joblib six
create a fake import that will override the dependency import, e.g.:
try:
sys.modules['sklearn.externals.six'] = __import__('six')
sys.modules['sklearn.externals.joblib'] = __import__('joblib')
import hdbscan
except ImportError:
import hdbscan
If there's no import error, standalone six and joblib will be used. Otherwise, e.g. if a user hasn't installed six or joblib, the program will still work (because it loads both modules from sklearn.externals), but it will show the depreciation warning.
Interesting enough, that even following #Alex's advice, I still have warnings output, like this:
import warnings
with warnings.catch_warnings():
warnings.simplefilter('ignore', category=DeprecationWarning)
from sklearn.externals.joblib import Parallel, delayed
def main():
xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
print(sum(xs))
if __name__ == '__main__':
main()
# $ python -W ignore example.py
# [...]
# DeprecationWarning: the imp module is deprecated in favour of importlib;
# see the module's documentation for alternative uses
# import imp
# 55
So eventually, I decided to do it in a very hacky way and disable all warnings because I am bit tired of looking for a proper way to deal with them. (Not only for this library, but for many others that seem to be very eager about bombarding you with non-suppressible warnings).
import warnings
def noop(*args, **kargs): pass
warnings.warn = noop
from sklearn.externals.joblib import Parallel, delayed
def main():
xs = Parallel()(delayed(lambda x: x**2)(i) for i in range(1, 6))
print(sum(xs))
if __name__ == '__main__':
main()
If I use #Alex's advice wrongly or some of you have a better solution, I would be glad to accept it as an answer.
Update 1
Ok, it seems that it is pretty hard to influence the warnings, raised somewhere internally in the package. So probably the most simple thing would be to just replace warnings.warn with noop, or maybe somehow import the internal dependencies in advance and suppress them with a context manager.
Update 2
Some time ago I found one more possible way to deal with warnings. You can redirect them into logging. In case if no logger is explicitly configured, these warnings are essentially suppressed. It works for Jupyter and some libraries I've tested.
import logging
logging.captureWarnings(True)
Related
I am working with code that throws a lot of (for me at the moment) useless warnings using the warnings library. Reading (/scanning) the documentation I only found a way to disable warnings for single functions. But I don't want to change so much of the code.
Is there a flag like python -no-warning foo.py?
What would you recommend?
Look at the Temporarily Suppressing Warnings section of the Python docs:
If you are using code that you know will raise a warning, such as a deprecated function, but do not want to see the warning, then it is possible to suppress the warning using the catch_warnings context manager:
import warnings
def fxn():
warnings.warn("deprecated", DeprecationWarning)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fxn()
I don't condone it, but you could just suppress all warnings with this:
import warnings
warnings.filterwarnings("ignore")
Ex:
>>> import warnings
>>> def f():
... print('before')
... warnings.warn('you are warned!')
... print('after')
...
>>> f()
before
<stdin>:3: UserWarning: you are warned!
after
>>> warnings.filterwarnings("ignore")
>>> f()
before
after
There's the -W option.
python -W ignore foo.py
You can also define an environment variable (new feature in 2010 - i.e. python 2.7)
export PYTHONWARNINGS="ignore"
Test like this: Default
$ export PYTHONWARNINGS="default"
$ python
>>> import warnings
>>> warnings.warn('my warning')
__main__:1: UserWarning: my warning
>>>
Ignore warnings
$ export PYTHONWARNINGS="ignore"
$ python
>>> import warnings
>>> warnings.warn('my warning')
>>>
For deprecation warnings have a look at how-to-ignore-deprecation-warnings-in-python
Copied here...
From documentation of the warnings module:
#!/usr/bin/env python -W ignore::DeprecationWarning
If you're on Windows: pass -W ignore::DeprecationWarning as an argument to Python. Better though to resolve the issue, by casting to int.
(Note that in Python 3.2, deprecation warnings are ignored by default.)
Or:
import warnings
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=DeprecationWarning)
import md5, sha
yourcode()
Now you still get all the other DeprecationWarnings, but not the ones caused by:
import md5, sha
Not to make it complicated, just use these two lines
import warnings
warnings.filterwarnings('ignore')
If you don't want something complicated, then:
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
This is an old question but there is some newer guidance in PEP 565 that to turn off all warnings if you're writing a python application you should use:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
The reason this is recommended is that it turns off all warnings by default but crucially allows them to be switched back on via python -W on the command line or PYTHONWARNINGS.
When all else fails use this: https://github.com/polvoazul/shutup
pip install shutup
then add to the top of your code:
import shutup; shutup.please()
Disclaimer: I am the owner of that repository. I wrote it after the 5th time I needed this and couldn't find anything simple that just worked.
If you know what are the useless warnings you usually encounter, you can filter them by message.
import warnings
#ignore by message
warnings.filterwarnings("ignore", message="divide by zero encountered in divide")
##part of the message is also okay
warnings.filterwarnings("ignore", message="divide by zero encountered")
warnings.filterwarnings("ignore", message="invalid value encountered")
import sys
if not sys.warnoptions:
import warnings
warnings.simplefilter("ignore")
Change ignore to default when working on the file or adding new functionality to re-enable warnings.
I realise this is only applicable to a niche of the situations, but within a numpy context I really like using np.errstate:
np.sqrt(-1)
__main__:1: RuntimeWarning: invalid value encountered in sqrt
nan
However, using np.errstate:
with np.errstate(invalid='ignore'):
np.sqrt(-1)
nan
The best part being you can apply this to very specific lines of code only.
More pythonic way to ignore WARNINGS
Since 'warning.filterwarnings()' is not suppressing all the warnings, i will suggest you to use the following method:
import logging
for name in logging.Logger.manager.loggerDict.keys():
logging.getLogger(name).setLevel(logging.CRITICAL)
#rest of the code starts here...
OR,
If you want to suppress only a specific set of warnings, then you can filter like this:
import logging
for name in logging.Logger.manager.loggerDict.keys():
if ('boto' in name) or ('urllib3' in name) or ('s3transfer' in name) or ('boto3' in name) or ('botocore' in name) or ('nose' in name):
logging.getLogger(name).setLevel(logging.CRITICAL)
#rest of the code starts here...
warnings are output via stderr and the simple solution is to append '2> /dev/null' to the CLI. this makes a lot of sense to many users such as those with centos 6 that are stuck with python 2.6 dependencies (like yum) and various modules are being pushed to the edge of extinction in their coverage.
this is especially true for cryptography involving SNI et cetera. one can update 2.6 for HTTPS handling using the proc at:
https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2
the warning is still in place, but everything you want is back-ported. the re-direct of stderr will leave you with clean terminal/shell output although the stdout content itself does not change.
responding to FriendFX. sentence one (1) responds directly to the problem with an universal solution. sentence two (2) takes into account the cited anchor re 'disable warnings' which is python 2.6 specific and notes that RHEL/centos 6 users cannot directly do without 2.6. although no specific warnings were cited, para two (2) answers the 2.6 question I most frequently get re the short-comings in the cryptography module and how one can "modernize" (i.e., upgrade, backport, fix) python's HTTPS/TLS performance. para three (3) merely explains the outcome of using the re-direct and upgrading the module/dependencies.
I am working with code that throws a lot of (for me at the moment) useless warnings using the warnings library. Reading (/scanning) the documentation I only found a way to disable warnings for single functions. But I don't want to change so much of the code.
Is there a flag like python -no-warning foo.py?
What would you recommend?
Look at the Temporarily Suppressing Warnings section of the Python docs:
If you are using code that you know will raise a warning, such as a deprecated function, but do not want to see the warning, then it is possible to suppress the warning using the catch_warnings context manager:
import warnings
def fxn():
warnings.warn("deprecated", DeprecationWarning)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fxn()
I don't condone it, but you could just suppress all warnings with this:
import warnings
warnings.filterwarnings("ignore")
Ex:
>>> import warnings
>>> def f():
... print('before')
... warnings.warn('you are warned!')
... print('after')
...
>>> f()
before
<stdin>:3: UserWarning: you are warned!
after
>>> warnings.filterwarnings("ignore")
>>> f()
before
after
There's the -W option.
python -W ignore foo.py
You can also define an environment variable (new feature in 2010 - i.e. python 2.7)
export PYTHONWARNINGS="ignore"
Test like this: Default
$ export PYTHONWARNINGS="default"
$ python
>>> import warnings
>>> warnings.warn('my warning')
__main__:1: UserWarning: my warning
>>>
Ignore warnings
$ export PYTHONWARNINGS="ignore"
$ python
>>> import warnings
>>> warnings.warn('my warning')
>>>
For deprecation warnings have a look at how-to-ignore-deprecation-warnings-in-python
Copied here...
From documentation of the warnings module:
#!/usr/bin/env python -W ignore::DeprecationWarning
If you're on Windows: pass -W ignore::DeprecationWarning as an argument to Python. Better though to resolve the issue, by casting to int.
(Note that in Python 3.2, deprecation warnings are ignored by default.)
Or:
import warnings
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=DeprecationWarning)
import md5, sha
yourcode()
Now you still get all the other DeprecationWarnings, but not the ones caused by:
import md5, sha
Not to make it complicated, just use these two lines
import warnings
warnings.filterwarnings('ignore')
If you don't want something complicated, then:
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)
This is an old question but there is some newer guidance in PEP 565 that to turn off all warnings if you're writing a python application you should use:
import sys
import warnings
if not sys.warnoptions:
warnings.simplefilter("ignore")
The reason this is recommended is that it turns off all warnings by default but crucially allows them to be switched back on via python -W on the command line or PYTHONWARNINGS.
When all else fails use this: https://github.com/polvoazul/shutup
pip install shutup
then add to the top of your code:
import shutup; shutup.please()
Disclaimer: I am the owner of that repository. I wrote it after the 5th time I needed this and couldn't find anything simple that just worked.
If you know what are the useless warnings you usually encounter, you can filter them by message.
import warnings
#ignore by message
warnings.filterwarnings("ignore", message="divide by zero encountered in divide")
##part of the message is also okay
warnings.filterwarnings("ignore", message="divide by zero encountered")
warnings.filterwarnings("ignore", message="invalid value encountered")
import sys
if not sys.warnoptions:
import warnings
warnings.simplefilter("ignore")
Change ignore to default when working on the file or adding new functionality to re-enable warnings.
I realise this is only applicable to a niche of the situations, but within a numpy context I really like using np.errstate:
np.sqrt(-1)
__main__:1: RuntimeWarning: invalid value encountered in sqrt
nan
However, using np.errstate:
with np.errstate(invalid='ignore'):
np.sqrt(-1)
nan
The best part being you can apply this to very specific lines of code only.
More pythonic way to ignore WARNINGS
Since 'warning.filterwarnings()' is not suppressing all the warnings, i will suggest you to use the following method:
import logging
for name in logging.Logger.manager.loggerDict.keys():
logging.getLogger(name).setLevel(logging.CRITICAL)
#rest of the code starts here...
OR,
If you want to suppress only a specific set of warnings, then you can filter like this:
import logging
for name in logging.Logger.manager.loggerDict.keys():
if ('boto' in name) or ('urllib3' in name) or ('s3transfer' in name) or ('boto3' in name) or ('botocore' in name) or ('nose' in name):
logging.getLogger(name).setLevel(logging.CRITICAL)
#rest of the code starts here...
warnings are output via stderr and the simple solution is to append '2> /dev/null' to the CLI. this makes a lot of sense to many users such as those with centos 6 that are stuck with python 2.6 dependencies (like yum) and various modules are being pushed to the edge of extinction in their coverage.
this is especially true for cryptography involving SNI et cetera. one can update 2.6 for HTTPS handling using the proc at:
https://urllib3.readthedocs.io/en/latest/user-guide.html#ssl-py2
the warning is still in place, but everything you want is back-ported. the re-direct of stderr will leave you with clean terminal/shell output although the stdout content itself does not change.
responding to FriendFX. sentence one (1) responds directly to the problem with an universal solution. sentence two (2) takes into account the cited anchor re 'disable warnings' which is python 2.6 specific and notes that RHEL/centos 6 users cannot directly do without 2.6. although no specific warnings were cited, para two (2) answers the 2.6 question I most frequently get re the short-comings in the cryptography module and how one can "modernize" (i.e., upgrade, backport, fix) python's HTTPS/TLS performance. para three (3) merely explains the outcome of using the re-direct and upgrading the module/dependencies.
In my Django application, when I import one third party library, I get this warning in the console:
the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
If however I do the import inside Python shell, then everything is ok. I want to achive the same behaviour in Django. This is what I've tried based on answers in other OS threads:
import warnings
from django.utils.deprecation import RemovedInDjango110Warning
warnings.filterwarnings(action="ignore", category=RemovedInDjango110Warning)
The above code results in another error message, that says that RemovedInDjango110Warning does not exists. I also tried this:
import warnings
def fxn():
warnings.warn("deprecated", DeprecationWarning)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
fxn()
from third_party_lib import some_module
But still I get the very same error message. So, it seems like all previous answers to this problem got outdated. And we need some fresh fix. Thanks!
I also tried this:
import warnings
with warnings.catch_warnings():
warnings.filterwarnings("ignore",category=DeprecationWarning)
from third_party_lib import some_module
But it has no effect.
There are a couple of issues with the code you've tried. If you want to filter PendingDeprecationWarning, then you should use PendingDeprecationWarning in your code. Your code is using DeprecationWarning and RemovedInDjango110Warning, which are different warnings. Secondly, the fxn() function in the docs is an example function that creates a warning. It doesn't make sense to include it in your code.
You can either filter all pending deprecation warnings
import warnings
warnings.simplefilter("ignore", category=PendingDeprecationWarning)
However this might hide pending deprecations in your own code that you should fix. A better approach would be to use a context manager, to filter out warnings when importing the third-party lib.
with warnings.catch_warnings():
warnings.simplefilter("ignore", category=PendingDeprecationWarning)
from third_party_lib import some_module
I am working on a large Python program which makes use of a multitude of modules depending on command-line options, in particular, numpy. We have recently found a need to run this on a small embedded module which precludes the use of numpy. From our perspective, this is easy enough (just don't use the problematic command line options.)
However, following PEP 8, our import numpy is at the beginning of each module that might need it, and the program will crash due to numpy not being installed. The straightforward solution is to move import numpy from the top of the file to the functions that need it. The question is, "How bad is this"?
(An alternative solution is to wrap import numpy in a try .. except. Is this better?)
Here is a best practice pattern to check if a module is installed and make code branch depending on it.
# GOOD
import pkg_resources
try:
pkg_resources.get_distribution('numpy')
except pkg_resources.DistributionNotFound:
HAS_NUMPY = False
else:
HAS_NUMPY = True
# You can also import numpy here unless you want to import it inside the function
Do this in every module imports having soft dependency to numpy. More information in Plone CMS coding conventions.
Another idiom which I've seen is to import the module as None if unavailable:
try:
import numpy as np
except ImportError:
np = None
Or, as in the other answer, you can use the pkg_resources.get_distribution above, rather than try/except (see the blog post linked to from the plone docs).
In that way, before using numpy you can hide numpy's use in an if block:
if np:
# do something with numpy
else:
# do something in vanilla python
The key is to ensure your CI tests have both environments - with and without numpy (and if you are testing coverage this should count both block as covered).
I am trying to set up a regression test suite and incorporate the system configuration at runtime. The following code finds most of the modules used by the system (and their versions):
import pkg_resources
import sys
for m in sys.modules:
try:
ml.append(m+'-'+pkg_resources.get_distribution(m).version)
except:
pass
print "# modules used:",sort(ml)
which returns:
# modules used: ['IPython-4.0.0', 'PIL-1.1.7', 'bottleneck-1.0.0',
'decorator-4.0.2', 'ipykernel-4.0.3', 'ipython_genutils-0.1.0',
'jupyter_client-4.0.0', 'jupyter_core-4.0.4', 'matplotlib-1.4.3',
'numexpr-2.4.3', 'numpy-1.9.2', 'pandas-0.16.2', 'pexpect-3.3',
'pickleshare-0.5', 'pyparsing-2.0.3', 'pysal-1.10.0', 'pytz-2015.4',
'scipy-0.16.0', 'simplegeneric-0.8.1', 'six-1.9.0', 'tornado-4.2.1',
'traitlets-4.0.0', 'xlsxwriter-0.7.3']
But there are not all of them. A quick check of the following:
ml = ['sklearn-'+sklearn.__version__,
'osgeo-'+osgeo.__version__]
print "# package versions:",sort(ml)
returns:
# package versions: ['osgeo-2.0.0' 'sklearn-0.16.1']
which are installed and used in the main scripts. I should also add that some of the reported modules like tornado and xlsxwriter and not called or used directly, while some of the ones which are imported directly into the code were not found.
Does anyone know a trick to report jut the ones which are called from the scripts, or at least one that does not miss important ones?