Goal
I want to be able to import (on the __init__.py) all functions from every single file inside my package.
Usage
For example in this folder structure.
manage.py
- scripts/
-- __init__.py
-- tests.py
-- deploy.py
I am currently doing the following:
manage.py:
from scripts import *
script/init.py:
from .tests import *
from .deploy import *
But, every time I add another file to the package I have to add an import line on script/__init__.py, which is kind of annoying.
You can do it, manually, but you shouldn't.
Why you really do not want to do this:
You'll end up with a namespace where understanding what is what and from where it came from will be extremely hard, with difficulty increasing as the size of the overall project does. Appart from being completely unintuitive for Python, think of anybody else that might view your code or even worse, think about yourself re-reading it after 1 month and not remembering what's going on. You don't need that in your life.
In addition to that, any functions you expose to the importer that might overlap with other functions in other modules are going to get shaddowed by the most recent one imported. As an example, think of two scripts that contain the same function foo() and watch what happens.
>>> from scrpt1 import *
>>> foo()
Script 1
>>> from scrpt2 import *
>>> foo()
Script 2
Don't need that in your life either. Especially when it is so easy to bypass by being explicit.
Here are some related lines from the text contained in import this:
Explicit is better than implicit.
Be explicit about the place where your functions are defined in. Don't "spaghetti" your code. You'll want to hit yourself in the future if you opt in for a mesh of all stuff in one place.
Special cases aren't special enough to break the rules.
Really self explanatory.
Namespaces are one honking great idea -- let's do more of those!
"more of those!", not less; don't miss out on how wonderful namespaces are. Python is based on them; segregating your code in different namespaces is the foundation of organizing code.
importlib allows you to import any Python module from a string name. You can automate it with going through the list of files in the path.
It's more pythonic to use __all__. Check here for more details.
Related
I'm building a small Python package. Following https://packaging.python.org/tutorials/packaging-projects/, I have a file structure like so:
pkg/
- src/
- pkg/
- __init__.py
- file.py
whith ClassA, ClassB, etc. defined in file.py
I've installed this package on my system. I can now do things like import pkg.file
in an interpreter, which is great. However, it gives me access to whatever not starting with _ in file.py; including all the imports, global variables, etc. that live in this file. I'm happy with pkg.file.ClassA; less so with, for instance, pkg.file.itertools, or pkg.file.PI. It just doesn't feel very clean.
What would be the best practice here? Modifying my import statements in file.py as import itertools as _itertools? Some pythonic trickery in the init file? I thought of adding from file import ClassA, ClassB to it, but it doesn't seem very DRY to me. Additionally, file.py is susceptible to being broken into two or more files in the near future.
Alright, so I came up with a two-stage process:
setting __all__ = ['ClassA', 'ClassB'] at the top level in file.py;
adding from .file import * in init.py.
This way, on ìmport pkg I have directly access to my classes in the namespace pkg. As a side effect, I'm quite happy with this flattening of the hierarchy!
pkg.file.whatever are still accessible, a way to circumvent that would be great (for cleanness if anything else) but I can live with it.
I was recently tasked with maintaining a bunch of code that uses from module import * fairly heavily.
This codebase has gotten big enough that import conflicts/naming ambiguity/"where the heck did this function come from, there are like eight imported modules that have one with the same name?!"ism have become more and more common.
Moving forward, I've been using explicit members (i.e. import module ... module.object.function() to make the maintenance work I do more readable.
But I was wondering: is there an IDE or utility which robustly parses Python code and refactors * import statements into module import statements, and then prepends the full module path onto all references to members of that module?
We're not using metaprogramming/reflection/inspect/monkeypatching heavily, so if aforementened IDE/util behaves poorly with such things, that is OK.
Not a perfect solution, but what I usually do is this:
Open Pydev
Remove all * imports
Use the optimize imports command (ctrl+shift+o) to re-add all the imports
Roughly solves the problem :)
If you want to build a solution yourself, try http://docs.python.org/library/modulefinder.html
Here are the other related tools mentioned:
working with AST directly, which is very low-level for your use.
working with modulefinder which may have a lot of the boilerplate code you are looking for,
rope, a refactoring library (#Lucas Graf),
the bicycle repair man, a refactoring libary
the logilab-astng library used in pylint
More about pylint
pylint is a very good tool built on top of ast that is already able to tell you where in your code there are from somemodule import * statements, as well as telling you which imports are not necessary.
example:
# next is what's on line 32
from re import *
this will complain:
W: 32,0: Wildcard import re
W: 32,0: Unused import finditer from wildcard import
W: 32,0: Unused import LOCALE from wildcard import
... # this is a long list ...
Towards a solution?
Note that in the above output pylint gives you the line numbers. it might be some effort, but a refactoring tool can look at those particular warnings, get the line number, import the module and look at the __all__ list, or using a sandboxed execfile() statement to see the module's global names (would modulefinder help with that? maybe...). With the list of global names from __all__ and the names that pylint complains about, you can have two set() and proceed to get the difference. Replace the line featuring wildcard imports with specific imports.
I wrote some refactoring tools to do just that. Star Namer will go through all of your wildcard * imports for a script and replace them with the actual functions to be imported.
Usage: ./star_namer.py module_filename script_filename
Once you've converted all of your star imports to actual names you can use from_to_import.py to fix them. This is how it works:
Running your script through pylint and counting up all of the currently undefined words.
Removing all of the from modname import lines from the script.
Running the script through pylint again and comparing the difference in undefined words.
Going through the json output of pylint (in reverse order), it determines the exact position of replacements to be made and inserts the modname. in the correct place.
I thought this approach would be a little more robust, by offloading the syntax processing to an advanced utility, that's designed for it, instead of trying to grep through all the text myself with regex expressions.
Usage: from_to_import.py script_name modname
It will show you what changes are to be made before making them. Press y to save. The main issues I've found so far are text alignment issues caused by inserting the modname. text which makes comments misaligned and it doesn't deal with aliased function names well (from ... import quickrun as qrun)
Full documentation here: https://github.com/SurpriseDog/Star-Wrangler
Is it possible to import a test.py file in whole project instead of importing it in each file?
for example calling it some where in start.py
No.
However, suppose you have three modules, a.py, b.py, and c.py. Further, assume that c.py imports b, and b.py imports a. Then in c.py, you can refer to items in a using
b.a.foo
since b imported a into its namespace.
even if it is ... (which in programming is almost always the case) a better question is why would you want to.
It would be difficult to do, and it adds nothing... in fact it can severely hinder development, and you may experience very strange bugs that become very hard to track down
[edit based on OP comment]
your use case sounds like you just want it to be in builtins
start.py
import __builtin__
import my_weird_test_class
__builtin__.tester = my_weird_test_class
from main_entry import main
main()
now in any file you can use
tester.do_something()
without importing tester or whatever
but as I said this tends to be a very bad idea... it is much better to explicitly import it in all your files ...
I'm teaching myself Python (I have experience in other languages).
I found a way to import a "module". In PHP, this would just be named an include file. But I guess Python names it a module. I'm looking for a simple, best-practices approach. I can get fancy later. But right now, I'm trying to keep it simple while not developing bad habits. Here is what I did:
I created a blank file named __init__.py, which I stored in Documents (the folder on the Mac)
I created a file named myModuleFile.py, which I stored in Documents
In myModuleFile.py, I created a function:
def myFunction()
print("hello world")
I created another file: myMainFile.py, which I stored in Documents
In this file, I typed the following:
import myModuleFile.py
myModuleFile.myFunction()
This successfully printed out "hello world" to the console when I ran it on the terminal.
Is this a best-practices way to do this for my simple current workflow?
I'm not sure the dot notation means I'm onto something good or something bad. It throws an error if I try to use myFunction() instead of myModuleFile.myFunction(). I kind of think it would be good. If there were a second imported module, it would know to call myFunction() from myModuleFile rather than the other one. So the dot notation makes everybody know exactly which file you are trying to call the function from.
I think there is some advanced stuff using sys or some sort of exotic configuration stuff. But I'm hoping my simple little way of doing things is ok for now.
Thanks for any clarification on this.
For your import you don't need the ".py" extension
You can use:
import myModuleFile
myModuleFile.myFunction()
Or
from myModuleFile import myFunction
myFunction()
Last syntax is common if you import several functions or globals of your module.
Besides to use the "main" function, I'd put this on your module:
from myModuleFile import myFunction
if __name__ == '__main__':
myFunction()
Otherwise the main code could be executed in imports or other cases.
I'd use just one module for myModuleFile.py and myMainFile.py, using the previous pattern let you know if your module is called from command line or as import.
Lastly, I'd change the name of your files to avoid the CamelCase, that is, I'd replace myModuleFile.py by my_module.py. Python loves the lowercase ;-)
You only need to have init.py if you are creating a package (a package in a simple sense is a subdirectory which has one or more modules in it, but I think it may be more complex than you need right now).
If you have just one folder which has MyModule.py and MyMainFile.py - you don't need the init.py.
In MyMainFile.py you can write :
import myModuleFile
and then use
myModuleFile.MyFunction()
The reason for including the module name is that you may reuse the same function name in more than one module and you need a way of saying which module your program is using.
Module Aliases
If you want to you can do this :
import myModuleFile as MyM
and then use
MyM.MyFunction()
Here you have created MyM as an alias for myModuleFile, and created less typing.
Here Lies Dragons
You will sometimes see one other forms of IMport, which can be dangerous, especially for the beginner.
from myModuleFile import MyFunction
if you do this you can use :
MyFunction()
but this has a problem if you have used the same function name in MyMainFile, or in any other library you have used, as you now can't get to any other definition of the name MyFunction. This is often termed Contaminating the namespace - and should really be avoided unless you are absolutely certain it is safe.
there is a final form which I will show for completeness :
from myModuleFile import *
While you will now be able to access every function defined in myModuleFile without using myModuleFile in front of it, you have also now prevented your MyMainFile from using any function in any library which matches any name defined in myModuleFile.
Using this form is generally not considered to be a good idea.
I hope this helps.
I was recently tasked with maintaining a bunch of code that uses from module import * fairly heavily.
This codebase has gotten big enough that import conflicts/naming ambiguity/"where the heck did this function come from, there are like eight imported modules that have one with the same name?!"ism have become more and more common.
Moving forward, I've been using explicit members (i.e. import module ... module.object.function() to make the maintenance work I do more readable.
But I was wondering: is there an IDE or utility which robustly parses Python code and refactors * import statements into module import statements, and then prepends the full module path onto all references to members of that module?
We're not using metaprogramming/reflection/inspect/monkeypatching heavily, so if aforementened IDE/util behaves poorly with such things, that is OK.
Not a perfect solution, but what I usually do is this:
Open Pydev
Remove all * imports
Use the optimize imports command (ctrl+shift+o) to re-add all the imports
Roughly solves the problem :)
If you want to build a solution yourself, try http://docs.python.org/library/modulefinder.html
Here are the other related tools mentioned:
working with AST directly, which is very low-level for your use.
working with modulefinder which may have a lot of the boilerplate code you are looking for,
rope, a refactoring library (#Lucas Graf),
the bicycle repair man, a refactoring libary
the logilab-astng library used in pylint
More about pylint
pylint is a very good tool built on top of ast that is already able to tell you where in your code there are from somemodule import * statements, as well as telling you which imports are not necessary.
example:
# next is what's on line 32
from re import *
this will complain:
W: 32,0: Wildcard import re
W: 32,0: Unused import finditer from wildcard import
W: 32,0: Unused import LOCALE from wildcard import
... # this is a long list ...
Towards a solution?
Note that in the above output pylint gives you the line numbers. it might be some effort, but a refactoring tool can look at those particular warnings, get the line number, import the module and look at the __all__ list, or using a sandboxed execfile() statement to see the module's global names (would modulefinder help with that? maybe...). With the list of global names from __all__ and the names that pylint complains about, you can have two set() and proceed to get the difference. Replace the line featuring wildcard imports with specific imports.
I wrote some refactoring tools to do just that. Star Namer will go through all of your wildcard * imports for a script and replace them with the actual functions to be imported.
Usage: ./star_namer.py module_filename script_filename
Once you've converted all of your star imports to actual names you can use from_to_import.py to fix them. This is how it works:
Running your script through pylint and counting up all of the currently undefined words.
Removing all of the from modname import lines from the script.
Running the script through pylint again and comparing the difference in undefined words.
Going through the json output of pylint (in reverse order), it determines the exact position of replacements to be made and inserts the modname. in the correct place.
I thought this approach would be a little more robust, by offloading the syntax processing to an advanced utility, that's designed for it, instead of trying to grep through all the text myself with regex expressions.
Usage: from_to_import.py script_name modname
It will show you what changes are to be made before making them. Press y to save. The main issues I've found so far are text alignment issues caused by inserting the modname. text which makes comments misaligned and it doesn't deal with aliased function names well (from ... import quickrun as qrun)
Full documentation here: https://github.com/SurpriseDog/Star-Wrangler