In a python project I would like to globber imports into a single file called common_imports.py in order to reduce number of import statements in python files.
Instead of writing
file1.py
import foo
import bar
import baz
[...]
file2.py
import foo
import bar
import baz
[...]
I would like to write
file1.py
from common_imports import *
[...]
file2.py
from common_imports import *
[...]
common_imports.py
import foo
import bar
import baz
However, this gives me a lot of pylint false positives.
I can disable pylint warnings in the common_imports.py file by adding a pylint disable comment. I can disable wildcard imports. Unfortunately, I can disable unused imports only globally but not specific for all imports from common_imports.py.
Somebody has an idea howto get pylint on the track?
Summarising my comments above into a proper answer:
TL;DR:
While the reusable code motive is commendable, it's not fit for purpose here. Listen to the linter, and save your hard-earned respect among your colleagues. :-)
Pythonic Viewpoint:
Don't
Why? Python convention, in all its organisational glory and documented structure, states that if you use a library in a module, import it in the module. Plain and simple.
Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.
-- PEP8 - Imports
At a lower level, the sys.modules dict, which tracks imports, will only import a library if it hasn’t been imported already. So from an efficiency point of view, there is no gain.
Maintainer's Viewpoint:
Don't
Why? If (when) the code is changed / optimised in a module, thus alleviating the need for a specific import … "remind me where I look to find where that library is imported? Oh ya, here. But this other module needs that import, but not this new library I’m using to optimise this code. Where should I import that? Ugh!!!"
You've lost the hard-earned respect of following maintainers.
Related
My project structure like this:
/project
main.py
/a_module
__init__.py
/sub_module
__init__.py
some_file.py
main.py
from a_module import main_api
a_module/__init__.py
from sub_module import sub_api
sub_module/__init__.py
from some_file import detail_api
In a_module/__init__.py gives Unable to import 'sub_module' error.
Why I cannot import 'sub_module'?
When I change to the relative path solve the error.
from .sub_module import sub_api
But I don't understand, does __init__.py design for public the API of the module? Why don't treat sub_module as a module instead of a directory? it's such a bad design to me...
__init__.py is executed when you import the package that contains it. But it's not your problem. Your problem is that module imports are always absolute unless explicitly relative. That means that they must chain from some directory in sys.path. By default this includes the working directory, so when you run main.py from within project, it can find a_module, and nothing else.
from sub_module import sub_api
In a_module/__init__.py doesn't work though, because imports are always absolute unless explicitly relative. So that import says "starting from some sys.path root, find a top level package named sub_module and import sub_api from it". Since no such module exists you get an error. from .sub_module import sub_api works because you opted into relative imports, so it doesn't start over from sys.path.
For an example of why you would do this, I'll give you something that broke in our own code back in the Python 2 days before absolute import by default was the law (from __future__ import absolute_import enabled the Py3 behavior, which is how we fixed it, but despite what the docs say, it was never enabled by default in Py2, the only enabled by default behavior was relative imports). Our layout was:
teamnamespace/
module.py
math/
mathrelatedsubmodule.py
othermathsubmodule.py
Now, we innocently thought hey, we'll put all our packages under a single shared top level namespace, and subpackages cover broad categories within them, and since we had a lot of additional utilities for basic mathematics, we put them under teamnamespace.math. Problem was, for the non-math modules, like teamnamespace.module, when they did:
import math # or
from math import ceil
it defaulted to relative lookup, and imported teamnamespace.math as math (a thoroughly useless import, since it was a namespace package only, all the functionality was in the sub-modules), not the built-in math module. In fact, without the Python 3 behavior, there was no reasonable way to get the built-in math module from a module under teamnamespace. Whereas with the Python 3 behavior, you can get either one or both (by aliasing one or the other with as, with no ambiguity:
# Gets built-in
import math
# Gets teamnamespace.math
from . import math
I'm building a dependency graph in python3 using the ast module. How do I know what file(s) will be imported if that import statement were to be executed?
Not a complete answer, but here are some bits you should be aware of:
Imports might happen in conditional or try-catch blocks. So depending on a setting of an environment variable, module A might or might not import module B.
There's a wide variety of import syntax: import A, from A import B, from A import *, from . import A, from .. import A, from ..A import B as well as their versions with A replaced with sub-modules.
Imports can happen in any executable context - the top-level of the file, in a function, in a class definition etc.
eval can evaluate code with imports. Up to you if you consider such code to be a dependency.
The standard library modulefinder module might help.
As suggested in a comment: the other answers are valid, but one of the fundamental problems is that your examples only work for 'simple' scripts or files: A lot of more complex code will use things like dynamic imports: consider the following:
path, task_name = "module.function".rsplit(".", 1);
module = importlib.import_module(path);
real_func = getattr(module, task_name);
real_func();
The actual original string could be obfuscated, or pulled from a DB, or a file or...
There are alternatives to importlib, but this is on top of the exec type stuff you might see in #horia's good answer.
I have a following project structure:
project
|----app.py
|----package
|---__init__.py
|---module.py
|---module2.py
|---module3.py
|---....
My __init__.py file currently is empty. In module.py I have a definition of a class:
class UsefulClass:
...
And in other modules similar definitions as well. My app.py looks like this:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from package.module import UsefulClass
from package.module2 import UsefulClass2
...
usefulclass = UsefulClass()
usefulclass2 = UsefulClass2()
....
My question is: how can I replace this from package.module... import UsefulClass statements? Even now, I have only 4 modules defined and this imports starting to look ugly. Can I import them in __init__.py file and then just use import package in app.py? I have tried that and it gives me an error.
I am looking for a clean and elegant solution.
In Python 3:
package/__init__.py:
from .foo import bar
package/foo.py:
bar=0
app1.py:
import package
print(package.bar)
app2.py:
from package import bar
print(bar)
Either way, this prints 0, just as you want.
In Python 2, just change from .foo import bar to from foo import bar.
(In fact, the 2.x code often works in Python 3, but it's not correct, and in some edge cases it will fail. For example, if you have a bar.py at the same level as the app, you'll end up with bar being that module, instead of 0.)
In real life, you probably want to specify a __all__ from each package and module that you might ever from foo import … (if for no other reason than to allow to test things at the interactive interpreter with from foo import *).
It sounds like you're saying you already tried this, and got an error. Without knowing exactly what you tried, and what the error was, and which Python version you're using, I have no idea what in particular you might have gotten wrong, but presumably you got something wrong.
The .foo specifies a package-relative import. Saying from .foo import bar means "from the foo module in the same package as me, import bar". If you leave off the dot, you're saying "from the foo module in the standard module path, import bar".
The tutorial section on Intra-package References (and surrounding sections) gives a very brief explanation. The reference docs for import and the import system in general give most of the details, but the original proposal in PEP 328 explains the details, and the rationale behind the design, a lot more simply.
The reason you need to leave off the dot in 2.x is that 2.x didn't have any way to distinguish relative and absolute imports. There's only from foo import bar, which means "from the foo module of the same package as me, or, if there is no such module, the one in the standard module path, import bar".
I was recently tasked with maintaining a bunch of code that uses from module import * fairly heavily.
This codebase has gotten big enough that import conflicts/naming ambiguity/"where the heck did this function come from, there are like eight imported modules that have one with the same name?!"ism have become more and more common.
Moving forward, I've been using explicit members (i.e. import module ... module.object.function() to make the maintenance work I do more readable.
But I was wondering: is there an IDE or utility which robustly parses Python code and refactors * import statements into module import statements, and then prepends the full module path onto all references to members of that module?
We're not using metaprogramming/reflection/inspect/monkeypatching heavily, so if aforementened IDE/util behaves poorly with such things, that is OK.
Not a perfect solution, but what I usually do is this:
Open Pydev
Remove all * imports
Use the optimize imports command (ctrl+shift+o) to re-add all the imports
Roughly solves the problem :)
If you want to build a solution yourself, try http://docs.python.org/library/modulefinder.html
Here are the other related tools mentioned:
working with AST directly, which is very low-level for your use.
working with modulefinder which may have a lot of the boilerplate code you are looking for,
rope, a refactoring library (#Lucas Graf),
the bicycle repair man, a refactoring libary
the logilab-astng library used in pylint
More about pylint
pylint is a very good tool built on top of ast that is already able to tell you where in your code there are from somemodule import * statements, as well as telling you which imports are not necessary.
example:
# next is what's on line 32
from re import *
this will complain:
W: 32,0: Wildcard import re
W: 32,0: Unused import finditer from wildcard import
W: 32,0: Unused import LOCALE from wildcard import
... # this is a long list ...
Towards a solution?
Note that in the above output pylint gives you the line numbers. it might be some effort, but a refactoring tool can look at those particular warnings, get the line number, import the module and look at the __all__ list, or using a sandboxed execfile() statement to see the module's global names (would modulefinder help with that? maybe...). With the list of global names from __all__ and the names that pylint complains about, you can have two set() and proceed to get the difference. Replace the line featuring wildcard imports with specific imports.
I wrote some refactoring tools to do just that. Star Namer will go through all of your wildcard * imports for a script and replace them with the actual functions to be imported.
Usage: ./star_namer.py module_filename script_filename
Once you've converted all of your star imports to actual names you can use from_to_import.py to fix them. This is how it works:
Running your script through pylint and counting up all of the currently undefined words.
Removing all of the from modname import lines from the script.
Running the script through pylint again and comparing the difference in undefined words.
Going through the json output of pylint (in reverse order), it determines the exact position of replacements to be made and inserts the modname. in the correct place.
I thought this approach would be a little more robust, by offloading the syntax processing to an advanced utility, that's designed for it, instead of trying to grep through all the text myself with regex expressions.
Usage: from_to_import.py script_name modname
It will show you what changes are to be made before making them. Press y to save. The main issues I've found so far are text alignment issues caused by inserting the modname. text which makes comments misaligned and it doesn't deal with aliased function names well (from ... import quickrun as qrun)
Full documentation here: https://github.com/SurpriseDog/Star-Wrangler
all.
I'd think that this could be answered easily, but it isn't. As long as I've been searching for an answer, I keep thinking that I'm overlooking something simple.
I have a python workspace with the following package structure:
MyTestProject
/src
/TestProjectNamespace
__init__.py
Module_A.py
Module_B.py
SecondTestProject
/src
/SecondTestProjectNamespace
__init__.py
Module_1.py
Module_2.py
...
Module_10.py
Note that MyTestProjectNamespace has a reference to SecondTestProjectNamespace.
In MyTestProjectNamespace, I need to import everything in SecondTestProjectNamespace. I could import one module at a time with the following statement(s):
from SecondTestProjectNamespace.Module_A import *
from SecondTestProjectNamespace.Module_B import *
...but this isn't practical if the SecondTestProject has 50 modules in it.
Does Python support a way to import everything in a namespace / package? Any help would be appreciated.
Thanks in advance.
Yes, you can roll this using pkgutil.
Here's an example that lists all packages under twisted (except tests), and imports them:
# -*- Mode: Python -*-
# vi:si:et:sw=4:sts=4:ts=4
import pkgutil
import twisted
for importer, modname, ispkg in pkgutil.walk_packages(
path=twisted.__path__,
prefix=twisted.__name__+'.',
onerror=lambda x: None):
# skip tests
if modname.find('test') > -1:
continue
print(modname)
# gloss over import errors
try:
__import__(modname)
except:
print 'Failed importing', modname
pass
# show that we actually imported all these, by showing one subpackage is imported
print twisted.python
I have to agree with the other posters that star imports are a bad idea.
No. It is possible to set up SecondTestProject to automatically import everything in its submodules, by putting code in __init__.py to do the from ... import * you mention. It's also possible to automate this to some extent using the __import__ function and/or the imp module. But there is no quick and easy way to take a package that isn't set up this way and make it work this way.
It's probably not a good idea anyway. If you have 50 modules, importing everything from all of them into your global namespace is going to cause a proliferation of names, and very likely conflicts among those names.
As other had put it - it might not be a good idea. But there are ways of keeping your namespaces and therefore avoiding naming conflicts - and having all the modules/sub-packages in a module available to the package user with a single import.
Let's suppose I have a package named "pack", within it a module named "a.py" defining some "b" variable. All I want to do is :
>>> import pack
>>> pack.a.b
1
One way of doing this is to put in pack/__init__.py a line that says
import a - thus in your case you'd need fifty such lines, and keep them up to date.
Not that bad.
However, the documentation at http://docs.python.org/tutorial/modules.html#importing-from-a-package - says that if you have a string list named __all__ in your __init__.py file, all module/sub-package names in that list are imported when one does from pack import *
That alone would half-work - but would require users of your package to perform the not-recommended "from x import *" form.
But -- you can do the "... import *" inside __init__.py itself, after defining the __all__ variable - so all you have to do is to keep the __all__ up to date:
With the TestProjectNamespace/__init__.py being like this:
__all__ = ["Module_A", "Module_B", ...]
from TestProjectNamespace import *
your users would have
TestProjectNamespace.Module_A (and others) available upon import of TestProjectNamespace.
And, of course - you could automate the creation of __all__ - it is just a variable, after all - but I would not recommend that.
Does Python support a way to import everything in a namespace / package?
No. A package is not a super-module -- it's a collection of modules grouped together.
At least part of the reason is that it's not trivial to determine what 'everything' means inside a folder: there are problems like network drives, soft links, hard links, ...