I am writing a module to let me write code in python 3, but still run it in 2. It looks surprisingly easy actually... anything else I should add? From my (limited) flailing on the interactive interpreter, the future imports do not affect python 3 and are viewed as redundant.
# _2or3.py
'''
Common usage:
from __future__ import print_function, nested_scopes, division, absolute_import, unicode_literals
from _2or3 import *
'''
import sys
if sys.version[0] == '2':
range = xrange
input = raw_input
Obviously there are some things you cannot do that you would normally be able to do in 3 (like dictionary compressions), and there are a few gotchas between the languages (like bytecodes. It looks like you should NEVER use bytes)
Any comments would be appreciated.
Check out six, that already does this, and loads more. It also has methods that helps you do binary and Unicode in both versions. Not all techniques you need to do can be done this way, though, especially if you need to support Python 2.5 or earlier. I tried to cover most of them in the book, but I'm sure I've missed out on some.
Related
I am working on a project that used to use python 2, but does not anymore.
I see many lines of code like these :
from __future__ import absolute_import
Can I safely remove them all ?
From what I read, it seems it was only used in python 2 to mimic Python 3 behavior
CF linked question :
What does from __future__ import absolute_import actually do?
As well this source :
Using the absolute_import line in Python 2 changes the search strategy
to mimic the Python 3 search strategy
The only feature that requires using the future statement is
annotations (see PEP 563).
All historical features enabled by the future statement are still
recognized by Python 3. The list includes absolute_import, division,
generators, generator_stop, unicode_literals, print_function,
nested_scopes and with_statement. They are all redundant because they
are always enabled, and only kept for backwards compatibility.
https://docs.python.org/3/reference/simple_stmts.html#future-statements
So, yes, you can safely remove it for Python 3 code.
I've been fascinated by the __future__ module - in particular, its ability to change the way statements are parsed in python.
What's most interesting is how doing something like
from __future__ import print_function
Enables you to use print (and not print_function, like you would expect any other normal import to do).
I have read What is __future__ in Python used for and how/when to use it, and how it works thoroughly and in particular came across a particular line:
A future statement is a directive to the compiler that a particular
module should be compiled using syntax or semantics that will be
available in a specified future release of Python.
I would love to know the intricacies of what exactly makes this possible. In particular, how something like
from __future__ import division
Can enable true division on python2, while
from __future__ import barry_as_FLUFL
Can enable the <> syntax on python3 (what I find most funny is that you have to import a feature from "__future__" for backward compatibility).
Anyway, to summarise, I would like to know how the directive is understood and executed by the compiler when __future__ or its artefacts are imported.
from __future__ import print_function tells the parser to not treat print as a keyword (leaving it as a name instead). That way the compiler treats it as the function and not a statement.
To track this, the compiler struct has a c_future field that holds a PyFutureFeatures object that tracks which future directives have been enabled. Various parts of the parser and compiler check the flags and alter behaviour.
This is mostly handled in the future.c source file, which has a future_parse() function that checks for import from AST objects with the module parameter set to __future__, and sets flags based on what is found.
For example, for the barry_as_FLUFL 'feature', the parser refuses != as syntax but accepts <> instead:
if (type == NOTEQUAL) {
if (!(ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
strcmp(str, "!=")) {
PyObject_FREE(str);
err_ret->error = E_SYNTAX;
break;
}
else if ((ps->p_flags & CO_FUTURE_BARRY_AS_BDFL) &&
strcmp(str, "<>")) {
PyObject_FREE(str);
err_ret->text = "with Barry as BDFL, use '<>' "
"instead of '!='";
err_ret->error = E_SYNTAX;
break;
}
}
You can find the other examples by grepping for the FUTURE_* flags listed in compile.h.
Note that there is a __future__ Python module, but it is not directly involved in the parsing and compilation of code; it is merely there to give Python code easy access to metadata about directives (including bitfield values to pass to the flags argument of the compile() function), nothing more.
I was recently tasked with maintaining a bunch of code that uses from module import * fairly heavily.
This codebase has gotten big enough that import conflicts/naming ambiguity/"where the heck did this function come from, there are like eight imported modules that have one with the same name?!"ism have become more and more common.
Moving forward, I've been using explicit members (i.e. import module ... module.object.function() to make the maintenance work I do more readable.
But I was wondering: is there an IDE or utility which robustly parses Python code and refactors * import statements into module import statements, and then prepends the full module path onto all references to members of that module?
We're not using metaprogramming/reflection/inspect/monkeypatching heavily, so if aforementened IDE/util behaves poorly with such things, that is OK.
Not a perfect solution, but what I usually do is this:
Open Pydev
Remove all * imports
Use the optimize imports command (ctrl+shift+o) to re-add all the imports
Roughly solves the problem :)
If you want to build a solution yourself, try http://docs.python.org/library/modulefinder.html
Here are the other related tools mentioned:
working with AST directly, which is very low-level for your use.
working with modulefinder which may have a lot of the boilerplate code you are looking for,
rope, a refactoring library (#Lucas Graf),
the bicycle repair man, a refactoring libary
the logilab-astng library used in pylint
More about pylint
pylint is a very good tool built on top of ast that is already able to tell you where in your code there are from somemodule import * statements, as well as telling you which imports are not necessary.
example:
# next is what's on line 32
from re import *
this will complain:
W: 32,0: Wildcard import re
W: 32,0: Unused import finditer from wildcard import
W: 32,0: Unused import LOCALE from wildcard import
... # this is a long list ...
Towards a solution?
Note that in the above output pylint gives you the line numbers. it might be some effort, but a refactoring tool can look at those particular warnings, get the line number, import the module and look at the __all__ list, or using a sandboxed execfile() statement to see the module's global names (would modulefinder help with that? maybe...). With the list of global names from __all__ and the names that pylint complains about, you can have two set() and proceed to get the difference. Replace the line featuring wildcard imports with specific imports.
I wrote some refactoring tools to do just that. Star Namer will go through all of your wildcard * imports for a script and replace them with the actual functions to be imported.
Usage: ./star_namer.py module_filename script_filename
Once you've converted all of your star imports to actual names you can use from_to_import.py to fix them. This is how it works:
Running your script through pylint and counting up all of the currently undefined words.
Removing all of the from modname import lines from the script.
Running the script through pylint again and comparing the difference in undefined words.
Going through the json output of pylint (in reverse order), it determines the exact position of replacements to be made and inserts the modname. in the correct place.
I thought this approach would be a little more robust, by offloading the syntax processing to an advanced utility, that's designed for it, instead of trying to grep through all the text myself with regex expressions.
Usage: from_to_import.py script_name modname
It will show you what changes are to be made before making them. Press y to save. The main issues I've found so far are text alignment issues caused by inserting the modname. text which makes comments misaligned and it doesn't deal with aliased function names well (from ... import quickrun as qrun)
Full documentation here: https://github.com/SurpriseDog/Star-Wrangler
Today I tried using pyPdf 1.12 in a script I was writing that targets Python 2.6. When running my script, and even importing pyPdf, I get complaints about deprecated functionality (md5->hashsum, sets). I'd like to contribute a patch to make this work cleanly in 2.6, but I imagine the author does not want to break compatibility for older versions (2.5 and earlier).
Searching Google and Stack Overflow have so far turned up nothing. I feel like I have seen try/except blocks around import statements before that accomplish something similar, but can't find any examples. Is there a generally accepted best practice for supporting multiple Python versions?
There are two ways to do this:
(1) Just like you described: Try something and work around the exception for old versions. For example, you could try to import the json module and import a userland implementation if this fails:
try:
import json
except ImportError:
import myutils.myjson as json
This is an example from Django (they use this technique often):
try:
reversed
except NameError:
from django.utils.itercompat import reversed # Python 2.3 fallback
If the iterator reversed is available, they use it. Otherwise, they import their own implementation from the utils package.
(2) Explicitely compare the version of the Python interpreter:
import sys
if sys.version_info < (2, 6, 0):
# Do stuff for old version...
else:
# Do 2.6+ stuff
sys.version_info is a tuple that can easily be compared with similar version tuples.
You can certainly do
try:
import v26
except ImportError:
import v25
Dive Into Python—Using Exceptions for Other Purposes
Multiple versions of Python are supported here. You can a) conditionally use the newer version, which takes a little work, or b) turn off the warnings, which should really be the default (and is on newer Pythons).
How do you gracefully handle failed future feature imports? If a user is running using Python 2.5 and the first statement in my module is:
from __future__ import print_function
Compiling this module for Python 2.5 will fail with a:
File "__init__.py", line 1
from __future__ import print_function
SyntaxError: future feature print_function is not defined
I'd like to inform the user that they need to rerun the program with Python >= 2.6 and maybe provide some instructions on how to do so. However, to quote PEP 236:
The only lines that can appear before
a future_statement are:
The module docstring (if any).
Comments.
Blank lines.
Other future_statements.
So I can't do something like:
import __future__
if hasattr(__future__, 'print_function'):
from __future__ import print_function
else:
raise ImportError('Python >= 2.6 is required')
Because it yields:
File "__init__.py", line 4
from __future__ import print_function
SyntaxError: from __future__ imports must occur at the beginning of the file
This snippet from the PEP seems to give hope of doing it inline:
Q: I want to wrap future_statements
in try/except blocks, so I can use
different code depending on which
version of Python I'm running. Why
can't I?
A: Sorry! try/except is a runtime
feature; future_statements are
primarily compile-time gimmicks, and
your try/except happens long after the
compiler is done. That is, by the
time you do try/except, the semantics
in effect for the module are already a
done deal. Since the try/except
wouldn't accomplish what it looks
like it should accomplish, it's simply
not allowed. We also want to keep
these special statements very easy to
find and to recognize.
Note that you can import __future__
directly, and use the information in
it, along with sys.version_info, to
figure out where the release you're
running under stands in relation to a
given feature's status.
Ideas?
"I'd like to inform the user that they need to rerun the program with Python >= 2.6 and maybe provide some instructions on how to do so."
Isn't that what a README file is for?
Here's your alternative. A "wrapper": a little blob of Python that checks the environment before running your target aop.
File: appwrapper.py
import sys
major, minor, micro, releaselevel, serial = sys.version_info
if (major,minor) <= (2,5):
# provide advice on getting version 2.6 or higher.
sys.exit(2)
import app
app.main()
What "direct import" means. You can examine the contents of __future__. You're still bound by the fact the a from __future__ import print_function is information to the compiler, but you can poke around before importing the module that does the real work.
import __future__, sys
if hasattr(__future__, 'print_function'):
# Could also check sys.version_info >= __future__. print_function.optional
import app
app.main()
else:
print "instructions for upgrading"
A rather hacky but simple method I've used before is to exploit the fact that byte literals were introduced in Python 2.6 and use something like this near the start of the file:
b'This module needs Python 2.6 or later. Please do xxx.'
This is harmless in Python 2.6 or later, but a SyntaxError in any earlier versions. Anyone trying to compile your file will still get an error, but they also get whatever message you want to give.
You might think that as you will have to have this line after your from __future__ import print_function then it will be the import that generates the SyntaxError and you won't get to see the useful error message, but strangely enough the later error takes precedence. I suspect that as the error from the import isn't really a syntax error in itself it isn't raised on the first compilation pass, and so real syntax errors get raised first (but I'm guessing).
This might not meet you criteria for being 'graceful', and it is very Python 2.6 specific, but it is quick and easy to do.
Just put a comment on the same line with the "from __future__ import ...", like this:
from __future__ import print_function, division # We require Python 2.6 or later
Since Python displays the line containing the error, if you try to run the module with Python 2.5 you'll get a nice, descriptive error:
from __future__ import print_function, division # We require Python 2.6 or later
SyntaxError: future feature print_function is not defined