iPython has this excellent feature of being able to auto-complete modules and this works extremely well in there.
For the longest time, I've had a shell function that allows me to change directories to where a given module is located. It does this by trying to import the name of the module from the first argument and looking into its __file__.
What I would like to do is to find out the importable modules that are available so I can write an auto-complete function for this little helper.
I know that for a give module, I can do something like dir(module_name) and that would give me what I need for that module but I am not sure what to do to find out what I can import, just like iPython when you do something like:
import [TAB]
Or
import St[TAB]
Which would autocomplete all the importable modules that start with St.
I know how to write the auto-completion functionality, I am interesting in just finding the way of knowing what can I import.
EDIT:
Digging through the IPython code I managed to find the exact piece that does the work of getting all of the current modules for the given environment:
from IPython.core.completerlib import module_completion
for module in module_completion('import '):
print module
The reason I'm printing the results is because I am actually using this for ZSH completion so I will need to deal with the output.
As a starting point, you could iterate through the paths in sys.path looking for python modules, as that is exactly what import does. Other than that, Python won’t have an index or something for all modules, as the imports are resolved on run-time, when it’s requested. You could easily provide an index for the built-in modules though and maybe have a cache or something for previously imported modules.
Related
I have a package animals with several thousand modules dog_1.py, cat_3.py, etc. Each module contains functions color(), size(), etc, and these functions often depend on other functions in the package outside the module. In other words, the bark() function in dog_2.py might depend on 5 other functions from different modules within the same animals package. At the bottom of each module, I have if __name__ == '__main__': because I need to be able to run each module as a stand-alone script but I don't want the module to execute if it is imported into another module. I could accomplish what I need by adding a bunch of import statements to each module to satsify the dependencies, but I thought there must be a better way to do this.
How is this type of problem typically handled?
Essentially, I need to be able to run each module separately as a script, and each module has (possibly) hundreds of dependencies within the same package.
I tried adding to the __init__.py a statement: __all__ = ['dog_1.py', 'cat_3.py', ...]. This seems like a good idea. But the problem I run into is that when I include from animals import * at the top of a given module, the import of that current module causes an error. I thought it was possible to import the module to itself, but for some reason, it is not working. If I remove the current module from the __all__ list, then it seems to work fine.
I also tried installing the package locally by creating an outer directory and adding .cfg and setup.py files. I thought it might fix the problem, but I wasn't able to get anywhere.
I feel like I might be just going about this completely wrongly. It seems like it would be an easy problem to handle: Make a bunch of program files in one directory and have them all share the functions between each other. Any help is appreciated. Thank you.
I've an unusual question, and I don't find the answer because no one is doing like that :p
I want to use PyLint in order to resolve errors before running the script, especially calling methods from other modules and heritage.
Thing is, my python scripts are not organized the same way than the import.
For exemple I have one module in src/project/module/file.py and the import is like that : progName.app.module
It's because the scripts are precompiled (with operations done on it such as Macros replacement)
But because of that, PyLint is unable to find the folder for the right import.
I can't modify the directories hierarchy, so I have to find a way to tell PyLint that it has to find progName.app.module in src/project/module
If someone ever had this issue...
Thanks !
There are possibly multiple solutions to this. First, you can try to write a failure import hook, using astroid's API, which might look as this: https://github.com/PyCQA/astroid/blob/master/astroid/brain/brain_six.py#L273. After defining it, you can use the same API for loading a plugin, as in: pylint your_project --load-plugins=qualified.name.for.your.plugin.
This might work most of the time, but if it does not, a normal transform could be used (you can take a look at most of the files in that directory, since they are used just for that, for modifying the AST before running the code).
Alternatively, you could try to play with --init-hook="...", in which you can modify the paths before the analysis, but it does not guarantee that your custom locations would be picked by pylint.
In matlab, it's easy to find the path to a '.m' file by 'which XX.m' and also convenient to view the code by 'open XXX.m'.
In Python, is there any similar command?
If you've already imported the module (or can do so without any harm), the inspect module is probably what you want. For example, getsourcefile will give you the path to the file the module was loaded from, while getsource will give you the source code.
Of course these don't always work, because some Python modules can be extension modules written in C, or cached .pyc files that someone installed without the .py file, or bundled up in a .zip file instead of a flat directory, or anything else you can think up and write a loader for… But in those cases, nothing could work; when something reasonable could work, inspect will.
There's nothing quite like the open function because Matlab is a GUI environment, whereas Python is just a language that can run in a wide variety of different GUIs or none at all, but once you know the path, presumably you can figure out how to open it in your IDE session or in your favorite text editor or whatever.
If you can't actually import the module (maybe the reason you're asking is because import XX is failing and you want to find the code to fix the problem…), but want to know which module would be imported, that's not quite as easy in Python 2.7 as in 3.4, but the imp module is often good enough—in particular, imp.find_module('XX') will usually get you what you want (as long as you pay attention to the explanation about packages in the docs).
I write a package P. At the root of P there is a module m0.
Somewhere inside P there are modules m1,m2,... that need to import from m0
Of course I can write in each of these modules:
from P.m0 import ...
However if I change the name of P I have to revisit all the places and rewrite such statements.
I could also use relative imports but if I move a module at a different level in the package hierarchy I have to go fix the number of dots.
There are some other reasons too, but bottom line, I really want to say import from the module m0 that is located at the root of my package, what is the best way to express this?
That's not possible.
However, if you perform major refactoring where you move around modules between subpackages having to update some relative imports is not a huge problem.
Same applies for renaming the top-level package name if you do not use relative imports - that could even be done really fast with search-and-replace over all your files.
If you are willing to modify your question a tiny bit, you can get away with this.
IF the only entrypoints to your package are controlled; e.g. you only test your code by doing something like invoking testsuite package/.../module.py which will
THEN you can make sure that the first thing you do is import firstthing, and in package/firstthing.py you have:
import sys
import os.path
packageDir = os.path.split(__name__)[0]
sys.path[:] = sys.path+[packageDir] # or maybe you want it first...
The main caveat being that you will not be able to run python files without going through your entrypoints. I always want to do this for every project I write in python (to make relative imports work nicely), but I personally find this so inconvenient that I just give up.
There is also a second alternative. It is not that unreasonable to specify that your package requires another package in the python path. This package could be a utility package which performs a major hack. For example if the name of the package was "x", you could do import x which would use the inspect module to perform reflection on the interpreter stack, letting you figure out which module you were importing it from. Then you could do a sort of "backwards os.walk" by going up parent directories until you found the root of your package (by checking for some special indicator file, or manifest, or something). Then the code would programatically perform the above modification of the python path via sys.path. It's the same as the above, but you have the liberty to do things like run any python file without having to go through an awful entrypoint.
If you have extreme control over the shell environment, you can also just augment the $PYTHONPATH to include your package directory, but this is extremely fragile in many ways, and rather inelegant.
I'm trying to import a few libraries into my program (which is a google AppEngine application).
Basically, I'm supposed to put all libraries in the root folder, but I've just created another folder called lib and placed them within that folder. (I've created the __init__.py)
Imports regularly work fine by using the import lib.module or from lib import module, but what happens is that when I try to import a complete package, for instance a folder named pack1 with various modules in it, by calling from lib.pack1 import *, I get this error in one of the modules who has accessed another module statically, i.e. from pack1.mod2 import sth.
What is the easy and clean way to overcome this? Without modifying the libraries themselves.
Edit: Using Python 2.7.
Edit: Error: when using import lib.pack1, I get ImportError: No module named pack1.mod1.
I think that instead of from pack1.mod2 you actually want to say from lib.pack1.mod2.
Edit: and, specifying what version of Python this is would help, since importation semantics have improved gradually over the years!
Edit: Aha! Thank you for your comment; I now understand. You are trying to rename libraries without going inside of them and fixing the fact that their name is now different. The problem is that what you are doing is, unfortunately, impossible. If all libraries used relative imports inside, then you might have some chance of doing it; but, alas, relative imports are both (a) recent and (b) not widely used.
So, if you want to use library p, then you are going to have to put it in your root directory, not inside of lib/p because that creates a library with a different name: lib.p, which is going to badly surprise the library and break it.
But I have two more thoughts.
First, if you are trying to do this to organize your files, and not because you need the import names to be different, then (a) create lib like you are doing but (b) do not put an __init__.py inside! Instead, add the lib directory to your PYTHONPATH or, inside of your program, to sys.path. (Does the GAE let you do something like this? Does it have a PYTHONPATH?)
Second, I am lying when I say this is not possible. Strictly speaking, you could probably do this by adding an entry to sys.metapath that intercepts all module lookups and tries grabbing them from inside of lib if they exist there. But — yuck.