I need to support both python 3.8 and versions lower that 3.8, but the package I need to import into my stub (*.pyi) file had different name in <3.8
import sys
if sys.version_info.minor < 8:
import xyz
else:
import zyx
In general this should do the job, but when I run flake8 with *.pyi files config (flake8 --config flake8-pyi.ini) I get this:
Y002 If test must be a simple comparison against sys.platform or sys.version_info
Any ideas what could be done about that?
Thanks in advance!
From the description of flake8-pyi (a flake8 plugin, not part of flake8 itself):
Y002: If test must be a simple comparison against sys.platform or sys.version_info. Stub files support simple conditionals to indicate differences between Python versions or platforms, but type checkers only understand a limited subset of Python syntax, and this warning triggers on conditionals that type checkers will probably not understand.
The fix is to change your condition to:
if sys.version_info < (3, 8):
note that your code would break for 2.8 (yes, some people do this!) and 4.0 so you should be careful with eliding parts of comparisons ;) -- I've written a flake8 plugin which helps lint against conditions that might be problematic: flake8-2020
Related
I am working to incorporate the symspellpy package for spell checking and correcting large amounts of data. However, the package suggests using pkg_resources.resource_filename, which is no longer supported. Can you please provide guidance on how to access the necessary resources using the currently preferred method?
dictionary_path = pkg_resources.resource_filename("symspellpy", "frequency_dictionary_en_82_765.txt")
bigram_path = pkg_resources.resource_filename("symspellpy", "frequency_bigramdictionary_en_243_342.txt")
The replacement is the importlib_resources.files function.
It is integrated in standard library from Python 3.9, as importlib.resources.files
If you just need to support Python 3.9 or newer, it's straightforward
import importlib.resources
importlib.resources.files(...)
Otherwise, if you want to support Python 3.8 and older, here's how you do it:
add importlib_resources>=1.3; python_version < '3.9' to your dependencies (requirements.txt, setup.cfg, setup.py or pyproject.toml, depending how the project is organised)
In your code, adapt as
import sys
if sys.version_info >= (3, 9):
import importlib.resources as importlib_resources
else:
import importlib_resources
importlib_resources.files(...)
See https://importlib-resources.readthedocs.io/en/latest/migration.html
As I'm getting further into TensorFlow and the finer points of Python, I've noticed these 3 statements at the top of many (perhaps most) .py files in the TensorFlow repository:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
Here are two examples:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_deep.py#L25
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/word2vec/word2vec_basic.py#L17
I should clarify that I have no interest or need to support Python 2 or substantially older versions of Python 3 in any projects I'm undertaking currently.
After reading about from __future__ import xyz statements, for example from these sources:
What is __future__ in Python used for and how/when to use it, and how it works
https://docs.python.org/2/library/future.html
https://docs.python.org/3/library/future.html
http://python-future.org/imports.html
I'm able to understand most of this documentation but I'm left with the following questions:
If only using Python 3, is there any need for these statements or can they safely be removed entirely in all cases? More specifically, if I put the following statement in the beginning of a program:
# check Python version is at least 3.6, if not, show an error and bail
if sys.version_info.major < 3 or sys.version_info.minor < 6:
print("ERROR: currently running Python version " + sys.version + ", at least version 3.6 is required")
return
# end if
Can the above statements be removed with no possibility of ill effects?
Would these statements matter in an older version of Python 3 vs a newer version of Python 3, ex. Python 3.1 vs Python 3.6?
Since TensorFlow requires Python version 3.5 or later, if these statements only matter if Python 2 (or possibly an older version of Python 3? see the previous question) is being used, why are these included in the TensorFlow codebase?
If removing these could ever cause a problem even when using a recent version of Python (ex. 3.5 or later), what would be an example to demonstrate such a problem?
-- EDIT --
user2357112 just pointed out that on Ubuntu TensorFlow supports Python 2.7 or Python 3.4:
I was honestly not aware of this as I'd been using the Windows version of TensorFlow, which requires at least Python 3.5:
So, I guess my question is specific to either a Windows TensorFlow install, or a TensorFlow install on a different OS using a recent version of 3.x.
We've been long-time fans of pylint. Its static analysis has become a critical part of all our python projects and has saved tons of time chasing obscure bugs. But after upgrading from 1.3 -> 1.4, almost all compiled c extensions result in E1101(no-member) errors.
Projects that previously run perfectly clean through pylint 1.3 now complain about almost every C extension member with E1101. We've been forced to disable E1101 errors, but this materially detracts from the usefulness of pylint.
For example, this perfectly valid use of the lxml package
r"""valid.py: demonstrate pylint 1.4 error"""
from lxml import etree
print etree.Element('mydoc')
Run this through pylint, and it reports:
$ pylint -rn valid.py
No config file found, using default configuration
************* Module valid
E: 3, 6: Module 'lxml.etree' has no 'Element' member (no-member)
But it is perfectly valid:
$ python valid.py
<Element mydoc at 7fddf67b1ba8>
Here's where it gets really weird. A very small handful of C extensions seem to work just fine through pylint, e.g.:
r"""valid2.py: this one works fine"""
import sqlite3
print sqlite3.version
$ pylint -rn valid2.py
No config file found, using default configuration
My question is, has anyone else witnessed this? And if so, would you be willing to share your workaround/solution?
We've experimented with trying to create plugins to suppress these warnings
(http://docs.pylint.org/plugins.html#enter-plugin), but we're having difficulty making heads or tails of the docs -- and the astroid base class is uber-complex and has defied our attempts to grok it.
For real bonus points (and our eternal gratitude) we'd love to understand what changed in pylint. We'd be happy to fix the code (or at least publish a best practice document for C extension authors) that would satisfy pylint.
Platform details
$ pylint --version
No config file found, using default configuration
pylint 1.4.0,
astroid 1.3.2, common 0.63.2
Python 2.7.5 (default, Jul 1 2013, 18:09:11)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)]
Shortly after posting my question, I found the answer. The change was in fact done on purpose as a security measure. Pylint imports modules to effectively identify valid methods and attributes. It was decided that importing c extensions that are not part of the python stdlib is a security risk and could introduce malicious code.
This was done in the release of Astroid 1.3.1 https://mail.python.org/pipermail/code-quality/2014-November/000394.html
Only C extensions from trusted sources (the standard library) are
loaded into the examining Python process to build an AST from the live
module.
There are four solutions if you want to use pylint on projects that import non-stdlib c extensions.
1) Disable safety using the --unsafe-load-any-extension=y command line option. This feature is undocumented and classified as a hidden option (https://mail.python.org/pipermail/code-quality/2014-November/000439.html).
2) Disable safety using the pylint.rc setting unsafe-load-any-extensions=yes. This is recommended over option 1 and includes full documentation in the default pylint.rc file (created with --generate-rcfile).
3) Specifically list packages or modules names that you trust to be loaded by pylint in the pylint.rc file using the extension-pkg-whitelist= option.
4) Create a plugin to manipulate the AST (I have no idea how to effect this -- but it's regularly discussed on on the pylint mailing list).
We opted for Option 3. We added the following line to our project pylint.rc file:
extension-pkg-whitelist=lxml
#user590028, thanks a lot for your answer! I just ran into this same problem with the libraries win32api, win32evtlog, win32file, win32gui, and win32process, and your solution worked.
I used another method I think is worth posting here, which is to call pylint and pass the whitelisted packages as a parameter:
pylint --extension-pkg-whitelist=win32api,win32evtlog,win32file,win32gui,win32process myfile.py
For those of you using VS Code, it's a bit tricky to find where to put the command as I couldn't find my executable.
In VS Code;
click on File > Preferences > Settings.
Scroll down to "Python Configurations" in the left window
scroll down to "Python Linting: Mypy Args" in the right window
click on "Edit in settings.json" link
edit the json to include:
"--extension-pkg-whitelist="
I had to do all this because PyLint isn't executable from my Windows command line...
If you're using VS Code for Mac, this is what you need to do in order to edit the settings.json file:
Click on Code (i.e. the Visual Studio Code tab which is on the left of the 'File' tab) -> Preferences - > Settings
Scroll down to Extensions and click on Python in the list.
Click on any of the Edit in settings.json links. This opens up settings.json for editing.
Add the line "python.linting.pylintArgs": ["----extension-pkg-whitelist=1xml"].
For a Python script I need a specific Python version. Now my installation of Python 2.6 contains both python26 and python2.6
Which one should I put in the shebang?
Option 1:
#!/usr/bin/env python2.6
Option 2:
#!/usr/bin/env python26
EDIT: Yes, there is a reason not to use plain python. In some of our environments in the university python is linked to python2.4 and my code uses quite some 2.6 features.
You can't always guarantee that the shebang will be used (or even that the user will have that version).
You shouldn't really limit to a specific version exactly. It's best to require at least a given version (if your code works on Python 2.6, why wouldn't it work on Python 2.7? I might not have Python 2.6 installed in a few months time.)
I would stick with the /usr/bin/env python shebang and instead dynamically detect the version. Believe it or not, the "normal" way of doing this is:
import sys
ver = sys.version[:3]
That will give you a 3-character string such as "2.6" or "2.7". I would just check that the first character = '2' (assuming you want to prevent Python 3 from running your scripts, since it's largely incompatible) and the third character >= '6'.
Edit: See Petr's comment -- use sys.version_info[0:2] instead (gives you a pair like (2, 6) or (2, 7).
Just checked on my Linux system there is only python2.6 not python26 so the former looks better.
Just to clarify, I would use conditional imports instead, in my case I need OrderedDict which is python 2.7+ only;
try:
from collections import OrderedDict
except ImportError:
print("Python 2.7+ is needed for this script.")
sys.exit(1)
Why don't you just use /usr/bin/python instead? Is there any reason for not doing that?
If you don't have it already, you can create a link to it using this command:
ln -s /usr/bin/python26 /usr/bin/python
This ensures compatibility if you ever upgrade your python in the future.
I have Python code that uses the "with" keyword (new in 2.6) and I want to check if the interpreter version is at least 2.6, so I use this code:
import sys
if sys.version < '2.6':
raise Exception( "python 2.6 required" )
However, the 2.4 interpreter chokes on the with keyword (later in the script) because it doesn't recognize the syntax, and it does this before it evaluates my check.
Is there something in Python analogous to Perl's BEGIN{} block?
Take a look here:
How can I check for Python version in a program that uses new language features?
Perhaps someone has a better answer, but my first thought would be to have a separate script to perform the check, then import the "real" script once the check has passed. Python won't check the syntax until the import happens.
import sys
if sys.version < '2.6':
raise Exception( "python 2.6 required" )
import myscript # runs myscript