I'm trying to import a few libraries into my program (which is a google AppEngine application).
Basically, I'm supposed to put all libraries in the root folder, but I've just created another folder called lib and placed them within that folder. (I've created the __init__.py)
Imports regularly work fine by using the import lib.module or from lib import module, but what happens is that when I try to import a complete package, for instance a folder named pack1 with various modules in it, by calling from lib.pack1 import *, I get this error in one of the modules who has accessed another module statically, i.e. from pack1.mod2 import sth.
What is the easy and clean way to overcome this? Without modifying the libraries themselves.
Edit: Using Python 2.7.
Edit: Error: when using import lib.pack1, I get ImportError: No module named pack1.mod1.
I think that instead of from pack1.mod2 you actually want to say from lib.pack1.mod2.
Edit: and, specifying what version of Python this is would help, since importation semantics have improved gradually over the years!
Edit: Aha! Thank you for your comment; I now understand. You are trying to rename libraries without going inside of them and fixing the fact that their name is now different. The problem is that what you are doing is, unfortunately, impossible. If all libraries used relative imports inside, then you might have some chance of doing it; but, alas, relative imports are both (a) recent and (b) not widely used.
So, if you want to use library p, then you are going to have to put it in your root directory, not inside of lib/p because that creates a library with a different name: lib.p, which is going to badly surprise the library and break it.
But I have two more thoughts.
First, if you are trying to do this to organize your files, and not because you need the import names to be different, then (a) create lib like you are doing but (b) do not put an __init__.py inside! Instead, add the lib directory to your PYTHONPATH or, inside of your program, to sys.path. (Does the GAE let you do something like this? Does it have a PYTHONPATH?)
Second, I am lying when I say this is not possible. Strictly speaking, you could probably do this by adding an entry to sys.metapath that intercepts all module lookups and tries grabbing them from inside of lib if they exist there. But — yuck.
Related
I have a package animals with several thousand modules dog_1.py, cat_3.py, etc. Each module contains functions color(), size(), etc, and these functions often depend on other functions in the package outside the module. In other words, the bark() function in dog_2.py might depend on 5 other functions from different modules within the same animals package. At the bottom of each module, I have if __name__ == '__main__': because I need to be able to run each module as a stand-alone script but I don't want the module to execute if it is imported into another module. I could accomplish what I need by adding a bunch of import statements to each module to satsify the dependencies, but I thought there must be a better way to do this.
How is this type of problem typically handled?
Essentially, I need to be able to run each module separately as a script, and each module has (possibly) hundreds of dependencies within the same package.
I tried adding to the __init__.py a statement: __all__ = ['dog_1.py', 'cat_3.py', ...]. This seems like a good idea. But the problem I run into is that when I include from animals import * at the top of a given module, the import of that current module causes an error. I thought it was possible to import the module to itself, but for some reason, it is not working. If I remove the current module from the __all__ list, then it seems to work fine.
I also tried installing the package locally by creating an outer directory and adding .cfg and setup.py files. I thought it might fix the problem, but I wasn't able to get anywhere.
I feel like I might be just going about this completely wrongly. It seems like it would be an easy problem to handle: Make a bunch of program files in one directory and have them all share the functions between each other. Any help is appreciated. Thank you.
iPython has this excellent feature of being able to auto-complete modules and this works extremely well in there.
For the longest time, I've had a shell function that allows me to change directories to where a given module is located. It does this by trying to import the name of the module from the first argument and looking into its __file__.
What I would like to do is to find out the importable modules that are available so I can write an auto-complete function for this little helper.
I know that for a give module, I can do something like dir(module_name) and that would give me what I need for that module but I am not sure what to do to find out what I can import, just like iPython when you do something like:
import [TAB]
Or
import St[TAB]
Which would autocomplete all the importable modules that start with St.
I know how to write the auto-completion functionality, I am interesting in just finding the way of knowing what can I import.
EDIT:
Digging through the IPython code I managed to find the exact piece that does the work of getting all of the current modules for the given environment:
from IPython.core.completerlib import module_completion
for module in module_completion('import '):
print module
The reason I'm printing the results is because I am actually using this for ZSH completion so I will need to deal with the output.
As a starting point, you could iterate through the paths in sys.path looking for python modules, as that is exactly what import does. Other than that, Python won’t have an index or something for all modules, as the imports are resolved on run-time, when it’s requested. You could easily provide an index for the built-in modules though and maybe have a cache or something for previously imported modules.
I write a package P. At the root of P there is a module m0.
Somewhere inside P there are modules m1,m2,... that need to import from m0
Of course I can write in each of these modules:
from P.m0 import ...
However if I change the name of P I have to revisit all the places and rewrite such statements.
I could also use relative imports but if I move a module at a different level in the package hierarchy I have to go fix the number of dots.
There are some other reasons too, but bottom line, I really want to say import from the module m0 that is located at the root of my package, what is the best way to express this?
That's not possible.
However, if you perform major refactoring where you move around modules between subpackages having to update some relative imports is not a huge problem.
Same applies for renaming the top-level package name if you do not use relative imports - that could even be done really fast with search-and-replace over all your files.
If you are willing to modify your question a tiny bit, you can get away with this.
IF the only entrypoints to your package are controlled; e.g. you only test your code by doing something like invoking testsuite package/.../module.py which will
THEN you can make sure that the first thing you do is import firstthing, and in package/firstthing.py you have:
import sys
import os.path
packageDir = os.path.split(__name__)[0]
sys.path[:] = sys.path+[packageDir] # or maybe you want it first...
The main caveat being that you will not be able to run python files without going through your entrypoints. I always want to do this for every project I write in python (to make relative imports work nicely), but I personally find this so inconvenient that I just give up.
There is also a second alternative. It is not that unreasonable to specify that your package requires another package in the python path. This package could be a utility package which performs a major hack. For example if the name of the package was "x", you could do import x which would use the inspect module to perform reflection on the interpreter stack, letting you figure out which module you were importing it from. Then you could do a sort of "backwards os.walk" by going up parent directories until you found the root of your package (by checking for some special indicator file, or manifest, or something). Then the code would programatically perform the above modification of the python path via sys.path. It's the same as the above, but you have the liberty to do things like run any python file without having to go through an awful entrypoint.
If you have extreme control over the shell environment, you can also just augment the $PYTHONPATH to include your package directory, but this is extremely fragile in many ways, and rather inelegant.
I am working on a project wherein I need to use a third party module in different project files(.py files). The situation is like this.
I have a file "abc.py" which imports third party module "common.py". There are couple of other files which also import "common.py". All these files are also imported in main project file "main.py".
It seems redundant to import same module in your project multiple times in different files since "main.py" is also importing all the project files.
I am also not sure how the size of the project gets affected by multiple import statements.
Can someone pls help me in making things bit simpler.
Importing only ever loads a module once. Any imports after that simply add it to the current namespace.
Just import things in the files you need them to be available and let Python do the heavy-lifting of figuring out loading the modules.
Yes, you are right, this behavior really exists in Python. Namely, if user code tries to import the same module in different ways, for example - import a and import A.a (where a.py file is located into A package and the first import is done from within the A package while the other import comes as from outside).
This can easily happen in real life, especially for multi-level packaged Python projects.
I have experienced a side-effect of such behavior, namely command isinstance does not work when an object is checked against a class that is defined in module that was imported in such way.
The solution I can think about is to redefine the __builtin__. __ import__ function to perform its work more intelligently.
I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it? I'd like to eventually be able to share python programs.
About the import statement:
(a good writeup is at http://effbot.org/zone/import-confusion.htm and the python tutorial goes into detail at http://docs.python.org/tutorial/modules.html )
There are two normal ways to import code into a python program.
Modules
Packages
A module is simply a file that ends in .py. In order for python, it must exist on the search path (as defined in sys.path). The search path usually consists of the same directory of the .py that is being run, as well as the python system directories.
Given the following directory structure:
myprogram/main.py
myprogram/rss.py
From main.py, you can "import" the rss classes by running:
import rss
rss.rss_class()
#alternativly you can use:
from rss import rss_class
rss_class()
Packages provide a more structured way to contain larger python programs. They are simply a directory which contains an __init__.py as well as other python files.
As long as the package directory is on sys.path, then it can be used exactly the same as above.
To find your current path, run this:
import sys
print(sys.path)
I don't really like answering so late, but I'm not entirely satisfied with the existing answers.
I'm just beginning Python, and I'd like to use an external RSS class. Where do I put that class and how do I import it?
You put it in a python file, and give the python file an extension of .py . Then you can import a module representing that file, and access the class. Supposing you want to import it, you must put the python file somewhere in your import search path-- you can see this at run-time with sys.path, and possibly the most significant thing to know is that the site-packages (install-specific) and current directory ('') are generally in the import search path. When you have a single homogeneous project, you generally put it in the same directory as your other modules and let them import each other from the same directory.
I'd like to eventually be able to share python programs.
After you have it set up as a standalone file, you can get it set up for distribution using distutils. That way you don't have to worry about where, exactly, it should be installed-- distutils will worry for you. There are many other additional means of distribution as well, many OS-specific-- distutils works for modules, but if you want to distribute a proper program that users are meant to run, other options exist, such as using py2exe for Windows.
As for the modules/packages distinction, well, here it goes. If you've got a whole bunch of classes that you want divided up so that you don't have one big mess of a python file, you can separate it into multiple python files in a directory, and give the directory an __init__.py . The important thing to note is that from Python, there's no difference between a package and any other module. A package is a module, it's just a different way of representing one on the filesystem. Similarly, a module is not just a .py file-- if that were the case, sys would not be a module, since it has no .py file. It's built-in to the interpreter. There are infinitely many ways to represent modules on the filesystem, since you can add import hooks that can create ways other than directories and .py files to represent modules. One could, hypothetically, create an import hook that used spidermonkey to load Javascript files as Python modules.
from [module] import [classname]
Where the module is somewhere on your python path.
About modules and packages:
a module is a file ending with .py. You can put your class in such a file. As said by Andy, it needs to be in your python path (PYTHONPATH). Usually you will put the additional module in the same directory as your script is though which can be directly imported.
a package is a directory containing an __init__.py (can be empty) and contains module files. You can then import a la from <package>.<module> import <class>. Again this needs to be on your python path.
You can find more in the documenation.
If you want to store your RSS file in a different place use sys.append("") and pout the module in that directory and use
import or from import *
The first file, where you have created the class, is "first.py"
first.py:
class Example:
...
You create the second file, where you want to use the class contained in the "first.py", which is "second.py"
myprogram/first.py
myprogram/second.py
Then in the second file, to call the class contained in the first file, you simply type:
second.py:
from first import Example
...