I have several project directories and want to have libraries/modules that are specific to them. For instance, I might have a directory structure like such:
myproject/
mymodules/
__init__.py
myfunctions.py
myreports/
mycode.py
Assuming there is a function called add in myfunctions.py, I can call it from mycode.py with the most naive routine:
execfile('../mymodules/myfunctions.py')
add(1,2)
But to be more sophisticated about it, I can also do
import sys
sys.path.append('../mymodules')
import myfunctions
myfunctions.add(1,2)
Is this the most idiomatic way to do this? There is also some mention about modifying the PYTHONPATH (os.environ['PYTHONPATH']?), but is this or other things I should look into?
Also, I have seen import statements contained within class statements, and in other instances, defined at the top of a Python file which contains the class definition. Is there a right/preferred way to do this?
Don't mess around with execfile or sys.path.append unless there is some very good reason for it. Rather, just arrange your code into proper python packages and do your importing as you would any other library.
If your mymodules is in fact a part of one large project, then set your package up like so:
myproject/
__init__.py
mymodules/
__init__.py
myfunctions.py
myreports/
__init__.py
myreportscode.py
And then you can import mymodules from anywhere in your code like this:
from myproject.mymodules import myfunctions
myfunctions.add(1, 2)
If your mymodules code is used by a number of separate and distinct projects, then just make it into a package in its own right and install it into whatever environment it needs to be used in.
Related
I want to ask you something that came to my mind doing some stuff.
I have the following structure:
src
- __init__.py
- class1.py
+ folder2
- __init__.py
- class2.py
I class2.py I want to import class1 to use it. Obviously, I cannot use
from src.class1 import Class1
cause it will produce an error. A workaround that works to me is to define the following in the __init__.py inside folder2:
import sys
sys.path.append('src')
My question is if this option is valid and a good idea to use or maybe there are better solutions.
Another question. Imagine that the project structure is:
src
- __init__.py
- class1.py
+ folder2
- __init__.py
- class2.py
+ errorsFolder
- __init__.py
- errors.py
In class1:
from errorsFolder.errors import Errors
this works fine. But if I try to do in class2 which is at the same level than errorsFolder:
from src.errorsFolder.errors import Errors
It fails (ImportError: No module named src.errorsFolder.errors)
Thank you in advance!
Despite the fact that it's slightly shocking to have to import a "parent" module in your package, your workaround depends on the current directory you're running your application, which is bad.
import sys
sys.path.append('src')
should be
import sys,os
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),os.pardir))
to add the parent directory of the directory of your current module, regardless of the current directory you're running your application from (your module may be imported by several applications, which don't all require to be run in the same directory)
One correct way to solve this is to set the environment variable PYTHONPATH to the path which contains src. Then import src.class1 will always work, regardless of which directory you start in.
from ..class1 import Class1 should work (at least it does here in a similar layout, using python 2.7.x).
As a general rule: messing with sys.path is usually a very bad idea, specially if this allows a same module to be imported from two different paths (which would be the case with your files layout).
Also, you may want to think twice about your current layout. Python is not Java and doesn't require (nor even encourage) a "one class per module" approach. If both classes need to work together, they might be better in a same module, or at least in modules at the same level in the package tree (note that you can use the top-level package's __init__ as a facade to provide direct access to objects defined in submodules / subpackages). NB : I'm not saying that your current layout is necessarily wrong, just that it might not be the simplest one.
There is also a solution that does not involve the use of the environment variable PYTHONPATH.
In src/ create setup.py with these contents:
from setuptools import setup
setup()
Now you can do pip install -e /path/to/src and import to your hearts content.
-e, --editable <path/url> Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
No, Its not good. Python takes modules in two ways:
Python looks for its modules and packages in $PYTHONPATH.
Refer: https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH
To find out what is included in $PYTHONPATH, run the following code in python (3):
import sys
print(sys.path)
all folder containing init.py are marked as python package.(if they are subdirectories under PYTHONPATH)
By help of these two ways you can full-fill the need to create a python project.
Problem
I struggle to find a nice way to import modules in chain when your second import is from outside the folder. More clearly, here is my folder organisation:
main/
__init__.py
a/
__init__.py
foo.py
bar.py
b/
__init__.py
stuff.py
In stuff.py -- typically a big piece of code -- I want to do something like this:
from a.foo import thing
but in foo.py -- typically some kind of library -- I have already something like this:
from bar import other_thing
When I run stuff.py, I have an import error saying that foo.py does not know the module bar. Of course, when I run foo.py alone, I have no error.
Possible solutions so far
A simple workaround is to put:
from a.bar import other_thing
instead in foo.py. But it feels strange to have to specify the module a since both files live in the same folder. When you develop something in foo.py, you should not have to care about this.
I can also add something like:
sys.path.insert(1, '../a')
in stuff.py. But in this case in stuff.py I can completely ignore the module structure and just import:
from foo import thing
which does not feel right either.
This is purely a question of style, but I am sure that there is a nice solution for this, probably using the __init__.py files. I can't find it though. Any idea?
I am using Python 3.4 by the way
You can use package relative imports in foo.py.
from .bar import other_thing
See Intra-Package References for details.
Note that if you run stuff.py directly, it is a top-level script (__main__) and is not a member of the package. If some other module imports b.stuff, you get a different copy. Personally, I put top-level scripts in a different bin directory so they are not mistaken for package modules.
I have a project with the following file structure:
root/
run.py
bot/
__init__.py
my_discord_bot.py
dice/
__init__.py
dice.py
# dice files
help/
__init__.py
help.py
# help files
parser/
__init__.py
parser.py
# other parser files
The program is run from within the root directory by calling python run.py. run.py imports bot.my_discord_bot and then makes use of a class defined there.
The file bot/my_discord_bot.py has the following import statements:
import dice.dice as d
import help.help as h
import parser.parser as p
On Linux, all three import statements execute correctly. On Windows, the first two seem to execute fine, but then on the third I'm told:
ImportError: No module named 'parser.parser'; 'parser' is not a package
Why does it break on the third import statement, and why does it only break on Windows?
Edit: clarifies how the program is run
Make sure that your parser is not shadowing a built-in or third-party package/module/library.
I am not 100% sure about the specifics of how this name conflict would be resolved, but it seems like you can potentially a). have your module overridden by the existing module (which seems like it might be happening in your Windows case), or b). override the existing module, which could cause bugs down the road. It seems like b is what commonly trips people up.
If you think this might be happening with one of your modules (which seems fairly likely with a name like parser), try renaming your module.
See this very nice article for more details and more common Python "import traps".
Put run.py outside root folder, so you'll have run.py next to root folder, then create __init__.py inside root folder, and change imports to:
import root.parser.parser as p
Or just rename your parser module.
Anyway you should be careful with naming, because you can simply mess your own stuff someday.
I have a directory structure that looks like this:
foo/
__init__.py
db.py
database.db
main.py
Now, in db.py, I read the contents of the database:
open('database.db').read()
This works as long as I am running db.py directly. However, when I try to import the foo package in main, and then run db.py, it fails with a file not found error.
How can I avoid this? (Preferably without making any changes to the modules in the foo package)
(Note: I am aware of a solution like this Relative paths in Python but that would entail a lot of modifications to my foo package)
EDIT 1: Is it not possible to add something to __init__.py so that python can find my files?
EDIT 2: As another small example, consider what happens when I start to have nested packages:
foo/
baz/
__init__.py
mod.py
modules.json
__init__.py
db.py
database.py
main.py
Now, if main.py uses the foo package, and db.py requires the baz package, then I need to start making more complicated changes in the modules of the baz as well as the foo package, so that the file reads succeed.
This kind of a situation is happening when I need to use a git module that has several nested submodules. Obviously, it's not very convenient to make so many changes.
One solution can be to mask the open() function with a function of your own, in db.py , and in that custom open() function, you can open the file after appending the directory of __file__ . Example -
def open(file,mode='r',buffering=-1):
import __builtin__ #use import builtins for Python 3.x
import os.path
filePath = os.path.join(os.path.dirname(__file__),file)
return __builtin__.open(filePath,mode,buffering)
__builtin__.open() (or for Python 3.x - builtins.open() ) points to the built-in open() function.
I have used buffering =-1 as it seems to be the default according to documentation , you can leave out that argument altogether, if its not needed.
If possible, you should not mask (maybe use a different name like - open_file() ) , but seems like you do not want to do that.
Though I still think masking should be done only when there is no other solution. Also, please note you should not be import db.py as from db import * , that may cause similar masking of open() in the other .py file where its imported (Ideally, it is not recommended to do from <module> import * in any situtation) .
Use of this module will to resolve this import error issue, the error that you mentioned is issue about path. so the error was python interpreter coulnt find the path where your modules has specified. if you use this module it would work fine.
sys.path
A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default
import sys
sys.path.append('..\Desktop\New folder')# folder destination
from foo import db
I did some websearch, but all I found was frustration.
I have a project in a directory (lets call it) "projectdir", in which I have "main.py".
In projectdir I have a subdirectory called "otherstuff", In which I have "foo.py".
How do I import foo.py, so I can use its contents in main.py, without doing much of the work that python designers/implementors should have, and without relying on boilerplate files?
Or is that impossible in python?
You need to put a __init__.py file in otherstuff subdirectory, to mark it as a package. After, you can import your module using:
import subdirectory.foo
or
from subdirectory import foo
The __init__.py file can be empty. There is no other "clean" way to achieve that in python.
you need to include __init__.py in your otherstuff directory. This is to tell python to search there for imports.
The python documentation explains how the module/package import works. And is def worth the time reading it, despite its kind of long length