I have some problems with my project structure.
Because of historical reasons project looks like this:
Source/
__init__.py
module1/
__init__.py
script1.py
module2/
__init__.py
script2.py
and in production it deploys like this
server/
__init__.py
module1/
__init__.py
script1.py
module2/
__init__.py
script2.py
The problem is that script2.py has such imports:
from server.module1.script1 import something
Is it possible to say python to search server.*** not in server/*** but in Source/***?
Right now I made this with symlinks, but it looks ugly
You can always add keys to the sys.modules dictionary; these act as aliases for the module:
import sys
try:
import server
except ImportError:
import Source
sys.modules['server'] = Source
Once server is an entry in sys.modules, any sub-modules and packages will be found as well as the server entry will be used as a starting point for further imports.
Related
I want import file from main folder to subsubfoler and use functions from there.
start.py
database.py
cogs/
__init__.py
Utility/
__init__.py
utility.py
Python can't import from database.py to utility.py and I don't know how can it works.
I used:
from ...database import Database
from database import Database
from ... import Database
How can I solved this problem? In future I add some folders in cogs.
The problem
Python needs to know where to look for importable modules, and it searches sys.path for them. This can be manipulated from within python, or by setting the environment variable PYTHONPATH (examples are to follow). By default python adds the entrypoint's directory to sys.path at the first position (sys.path is a list), so while running scripts from the base directory would usually work with absolute imports nested under, running from a subdirectory and trying to relatively import upwards - will not. From python's docs regarding modules:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.
This means that start.py will work if it import cogs (absolute), but neither start.py nor utility.py will work if they from . [or .., or ...] import cogs (relative).
How to make it work
Consider the following directory structure (unchanged from the question, just placed under /tmp/kacper/):
/tmp/kacper/
├── start.py
├── database.py
└── cogs
├── __init__.py
└── Utility
├── __init__.py
└── utility.py
__init__.py files are empty, other file contents are as follows;
/tmp/kacper/database.py:
def Database():
print("Function Database in file database.py")
/tmp/kacper/start.py:
#!/usr/bin/env python
from cogs.Utility.utility import Database
print("Done importing, now to start.py's code:")
Database()
print("Done with start.py's code")
/tmp/kacper/cogs/Utility/utility.py:
#!/usr/bin/env python
if __name__ == "__main__":
import sys
import os
if 'PYTHONPATH' not in os.environ:
print("No PYTHONPATH in env, adding project's root")
this_file_dir = os.path.dirname(os.path.abspath(__file__))
base = os.path.join(this_file_dir, "..", "..")
sys.path.append(base)
else:
print("Found PYTHONPATH in env")
else:
print("utility.py is imported, not touching sys.path")
from database import Database
if __name__ == "__main__":
print("In second main of utility.py")
Database()
print("Done second main of utility.py")
Running
Now, as start.py resides in the base of the project, it just works:
$ /tmp/kacper/start.py
utility.py is imported, not touching sys.path
Done importing, now to start.py's code:
Function Database in file database.py
Done with start.py's code
But when directly calling utility.py, we have to either specify PYTHONPATH:
$ PYTHONPATH="/tmp/kacper" /tmp/kacper/cogs/Utility/utility.py
Found PYTHONPATH in env
In second main of utility.py
Function Database in file database.py
Done second main of utility.py
or manipulate sys.path from within the script:
$ /tmp/kacper/cogs/Utility/utility.py
No PYTHONPATH in env, adding project's root
In second main of utility.py
Function Database in file database.py
Done second main of utility.py
If PYTHONPATH is not set correctly (or there's no internal handling of sys.path), this will fail; for example:
$ PYTHONPATH='/tmp' /tmp/kacper/cogs/Utility/utility.py
Found PYTHONPATH in env
Traceback (most recent call last):
File "/tmp/kacper/cogs/Utility/utility.py", line 18, in <module>
from database import Database
ModuleNotFoundError: No module named 'database'
Adding more stuff under cogs
Once python knows where to import from, this should be fairly simple. Consider the following directory structure:
/tmp/kacper/
├── database.py
├── start.py
└── cogs
├── __init__.py
├── Utility
│ ├── __init__.py
│ └── utility.py
└── newCogsFolder
├── __init__.py
├── newCogsFunctionFile.py
└── new_cogs_subfolder
├── __init__.py
└── new_cogs_subfunction_file.py
/tmp/kacper/cogs/newCogsFolder/newCogsFunctionFile.py:
def newCogsFunction():
print("new cogs function")
/tmp/kacper/cogs/newCogsFolder/new_cogs_subfolder/new_cogs_subfunction_file.py:
def new_cogs_subfunction():
print("new cogs sub function")
Then you can add to utility.py (after the first if clause):
from cogs.newCogsFolder.new_cogs_subfolder.new_cogs_subfunction_file import new_cogs_subfunction
from cogs.newCogsFolder.newCogsFunctionFile import newCogsFunction
and to start.py you can mix and match (if it makes sense):
from cogs.Utility.utility import newCogsFunction
from cogs.newCogsFolder.new_cogs_subfolder.new_cogs_subfunction_file import new_cogs_subfunction
Notes:
It usually is not a good idea to execute code during imports, and it is an even worse idea to print during them. The above code is just an example to demo the different cases.
For convenience, the files start.py and utility.py are set as executables in the above example. The result would be no different with omitting the #!/usr/bin/env python line, and running python /tmp/kacper/start.py or python /tmp/kacper/cogs/Utility/utility.py.
In the section Adding more stuff under cogs; the directory, file, and function names intentionally don't follow PEP8 - to show that naming schemes don't matter, as long as python can import them. Hint: don't use hyphens (-) in such names.
In the section Adding more stuff under cogs; While showing that it is possible for start.py to mix between direct imports and importing from utility.py, this is a pattern that is best avoided. For clarity, maintainability, and continuity - one method should be chosen, according to the specific use-case.
I've had similar problems with imports in the past and thus I've create a new import library that gives the programmer more control over their imports: https://github.com/ronny-rentner/ultraimport
It allows you to do file system based imports that will always work, no matter how you run your code, no matter what is in your sys.path and it does not care about init files.
With ultraimport, your utility.py could look like this:
import ultraimport
Database = ultraimport('__dir__/../../database.py', 'Database')
my project has folders that are structured like this:
main
-folder 1
-file1.py
-folder 2
-file2.py
both file1 and file2 having classes.
when I try from main.folder1.file1 import class1 it fails, saying "No module named main". What am I doing wrong and how should I import it?
you have to first create the module by including __init__.py
in the root directory which is same in the hierarchy of main folder and also create an __init__.py in other sub folders to make them accessible as modules.
Here is an example structure from the official documentation.Note how at each level there is __init__.py you have to include similarly.
package/
__init__.py
subpackage1/
__init__.py
moduleX.py
moduleY.py
subpackage2/
__init__.py
moduleZ.py
moduleA.py
your structure can be like below:
main/
__init__.py
folder1/
__init__.py
file1.py
folder2/
__init__.py
file2.py
Then you can append the path to the module at the top level like below.
You can have it in the __init__.py in the root directory.
sys.path.append(path.dirname(path.dirname(path.abspath(file))))
Then try accessing like from folder1.file1 import class1.
This should resolve your problem.
To further understand your problem read about modules and how to include relative imports by referring documentation.
In python you have to declare each folder as a module by adding a file named __init__.py in each directory including the root. This file can be empty.
you can do this if there are one level above.
import sys
sys.path.insert(0,'../folder1')
sys.path.insert(0,'../folder2')
import function1
import function2
....
Once you run the first 3 lines, you are adding those folders to the working directory. You can then import the functions within those files as if they are in the file already.
If the folders are in the same level, do this,
import sys
sys.path.insert(0,'folder1')
sys.path.insert(0,'folder2')
import function1
import function2
....
I have a project structured like so:
/proj
main.py
config/
__init__.py
setup.py
gui/
__init__.py
app.py
The program is run by calling python main.py.
In main.py I have from gui import App.
In gui/__init__.py I have from .app import App.
Lastly, in app.py I have from ..config import configure; this throws a ValueError that reads attempted relative import beyond top-level package. To resolve this, I can alter the strucure:
/proj
main.py
proj/
config/
__init__.py
setup.py
gui/
__init__.py
app.py
But then I have to change the import statments to include proj (i.e. from proj.gui import App). Is it possible to use relative imports in the former case with the current structure? If not, how can I import what I need from the config subpackage?
As strange as it sounds, you can do :
from config import configure
in app.py, for your former case (current structure), because when you run python main.py, it interprets config as the next level package, and .. won't work.
(Btw, I think there is lots of room for Python imports to improve. The current PEP 328 is confusing at best).
I'm working on a Python application consisting of a core and multiple independent modules using the core. I'm having difficulty setting up relative imports of packages.
app
|- __init__.py
|- core
|- __init__.py
|- corefile.py
|- module1
|- __init__.py
|- main.py
The __init__.py files are empty. I'm running Python 2.7.1.
main.py
from .core import *
Running python main.py results in ValueError: Attempted relative import in non-package.
Similar questions: Ultimate answer to relative python imports, How to do relative imports in Python?, Relative imports in Python
Thanks for the help.
In short, you can only use relative imports from packages that are, themselves, imported.
For example, if you had:
$ cat run.py
from app.module1 import main
main.main()
$ python run.py
Then you could use a relative import in app/module1/main.py (although it would need to be from ..core import foo, because core/ is one level above main.py).
import sys
abs_filepath = '/home/n/Documents/IMPORTANT/deep_learning/drori_2018/ final_proj/Ryans_branch/StackGAN/'
# insert your absolute filepath above as abs_filepath = '/path/to/targ/dir'
sys.path.append(abs_filepath)
Please correct it if there are problems with doing the import this way
Other Answers:
Also please see here for a thorough answer about what's going on.
stuff/
__init__.py
mylib.py
Foo/
__init__.py
main.py
foo/
__init__.py
script.py
script.py wants to import mylib.py
This is just an example, but really I just want to do a relative import of a module in a parent directory. I've tried various things and get this error...
Attempted relative import beyond toplevel package
I read somewhere that the script from where the program starts shouldn't in the package, and I tried modifying the structure for that like so...
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
__init__.py
script.py
but got same error.
How can I accomplish this? Is this even an adequate approach?
Edit: In Python 2
After fiddling with it a bit more, I realized how to set it up, and for the sake of specificity I won't use foo bar names. My project directory is set up as...
tools/
core/
object_editor/
# files that need to use ntlib.py
editor.py # see example at bottom
__init__.py
state_editor/
# files that need to use ntlib.py
__init__.py
ntlib.py
__init__.py # core is the top level package
LICENSE
state_editor.py # equivalent to main.py for the state editor
object_editor.py # equivalent to main.py for the object editor
A line in object_editor.py looks like...
from core.object_editor import editor
A line in editor.py looks like...
from .. import ntlib
or alternatively
from core import ntlib
The key is that in the example I gave in the question, the "main" script was being run from within the package. Once I moved it out, created a specific package (core), and moved the library I wanted the editors to share (ntlib) into that package, everything was hunky-dory.
though as long "stuff" is not in your python PATH you got no choice than adding the path.
If you know the level of your script.py from stuff you can do for example:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..'))
I'm running Python 3.4.2 on Windows 7 and tore my hair out over this.
When running either of these:
python -m unittest
python -m unittest discover
...I would get the 'Attempted relative import beyond toplevel package' error.
For me, the solution was dropping the ".." in my [test_stock.py].
The line was:
from ..stock import Stock
Changed it to:
from stock import Stock
.. and it works.
Folder structure:
C:\
|
+-- stock_alerter
|
+-- __init__.py
+-- stock.py
|
\-- tests
|
+-- __init__.py
\-- test_stock.py
From the PEP it appears that you cannot use a relative import to import a file that is not packaged.
So you would need to add a __init__.py to stuff and change your imports to something like from .mylib import *
However, the PEP seems to make no allowance to keep mylib packaged up in a module. So you might be required to change how you call your library functions.
Another alternative is to move mylib into a subpackage and import it as from .libpackage import mylib
If you're on Linux or perhaps a similar *nix, you can hack this with symlinks.
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
script.py
mylib.py -> ../mylib.py
foo2/
script2.py
mylib.py -> ../mylib.py
This is likely not a good pattern to follow.
In my case I opted for it because I had multiple executables dependent on the same library that needed to be put into separate directories.
Implementation of new executable tests shouldn't require the test writer to have a deep understanding of python imports.
tests/
common/
commonlib.py
test1/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py
test2/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py