I want to have this structure for my project:
requirements.txt
README.md
.gitignore
project/
__init__.py
project.py
core/
__init__.py
base.py
engines/
__init__.py
engine1.py
engine2.py
utils/
__init__.py
refine_data.py
whatever.py
The application is run from project/project.py. However, I constantly get import errors when using relative or absolute imports.
Both engines need to import from project.core.base, the utils need to import from project.core.base as well, and project.py (the main file ran) needs to be able to import from engines.
Absolute imports don't work:
# engines/engine1.py
from project.core.base import MyBaseClass
which gives the error:
ImportError: No module named project.core.base
But if I try a relative import instead
# engines/engine1.py
from ..core.base import MyBaseClass
I get:
ValueError: Attempted relative import beyond toplevel package
I've seen other projects on Github structured similarly, but this seems to cause all sorts of problems. How do I get this to work?
Take a look at your sys.path. It's likely that the top project directory is in the python path, and it sees your sub-packages (ie. utils, engines, etc.) as separate packages, which is why it's giving you an error that you're trying to import from outside your package when doing relative imports, and absolute imports don't work because it can't find the top project directory because it's not under any of the python paths.
The directory above the top project directory is what needs to be added to the python path.
Ex.
/path/is/here/project/core/...
# Add this to the PYTHONPATH
/path/is/here
Try to use these imports:
engine1.py:
from core import base
refine_data.py:
from core import base
project.py
from engines import engine1
if you use pycharm mark project directory as sources root and then try to run project.py. If you don't use pycharm you can run project.py by going to project directory and running command:
python project.py
Related
I've a directory structure like this below:
ParentPackage/
childPackage/
__init__.py
subpackageOne/
__init__.py
moduleA.py
moduleB.py
test/
__init__.py
test.py
I'm trying to import a method "methodX" from moduleA.py to my test.py residing inside test folder.
below is the code which I've tried
import os
import sys
sys.path.append(os.path.realpath('..'))
from ..subpackageOne.moduleAimport methodX
but getting error "ImportError: attempted relative import with no known parent package".
In addition to this moduleA.py is importing moduleB.py and while running the test I am getting No module named 'moduleB'.
There are already lot of question has been asked on this topic, however suggestions provided for those didn't work for me.
Based on your architecture, you can do that:
import os
import sys
# Assuming your package is on a drive called C:.
# That is what I mean by absolute path, it needs to refer the path from the disk
# so it does not depend on where you are
sys.path.append("C:/path/to/childPackage")
# Absolute import is just importing wihtout using dots at the beginning.
# In this case, python will look for installed packages in your environment
# and then any folder that is in your path.
from subpackageOne.moduleA import methodX
If you prefer, you may want to add the parentPackage in your path so you can use any childPackage in case you have several.
sys.path.append("C:/path/to/parentPackage")
from childPackage.subpackageOne.moduleA import methodX
Note: I see you have __init__ files in your architecture which suggests you may want to create an installable package that you can build after (using a setup.py or setup.cfg file). In this case, you could install your package in your virtual environment and then you would not need to add the directory to your path. setuptools Doc or some tutorial
I met with a problem with absolute import when using conda env. Here is the structure of my project.
project/
package_1/
__init__.py
file_1.py
subpackage_1/
run.py
In package_1.subpackage_1.run.py there is an absolute import import package_1.file_1. However, when I ran python package_1/subpackage_1/run.py in package folder, I got an error:
ModuleNotFoundError: No module named 'package_1'. I tried to print sys.path. project.package_1.subpackage_1 is in sys.path, but the folder from where I ran the command, project is not. I tried to add project in PATH or PYTHONPATH, but it doesn't work in conda env. Does anyone know how to fix this? Thanks!!!
One of the ways to do this is to add the directory to your sys.path with this code at the top of run.py
import sys
import os
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'package_1'))
And then change the line in run.py
import package_1.file_1
to
import file_1
Now python can import file1 directly since its directory is on the path.
Summary
You can accomplish what you want with relative imports, or with absolute imports if you restructure your project. Modifying your sys.path or PYTHONPATH should not be your go-to solution. If you really want global availability, you could install your local package with conda.
Option 1: Relative Imports
If you want to be able to run a file inside a sub-module directly (i.e. python package_1/subpackage_1/run.py) then you should consider using relative imports.
Example:
project/
package_1/
__init__.py
file_1.py
subpackage_1/
__init__.py
run.py
# run.py
import ..file_1
Option 2: Absolute Imports
If you want to use absolute imports, then your entry point (the script that you run) should be in the top level (package_1) instead of inside a sub-package.
Example:
project/
package_1/
__init__.py
run.py
file_1.py
subpackage_1/
__init__.py
stuff.py
# run.py
import package_1.subpackage_1.stuff
stuff.run()
# stuff.py
import package_1.file_1
Option 3: Installing your local package with conda
Once you configure your package correctly you should be able to simply run
conda install .
Which will install your local package as if it were a published package. This is likely overkill for your needs.
Why not modify PYTHONPATH or sys.path?
If you rely on having your local package path on PYTHONPATH, you every time you move the project or copy it onto a new computer.
Appending entries to sys.path in code often accomplishes a similar effect to what you can do with relative imports, but later import statements lose semantics.
My apps are organized like this:
apps/
code/
libglobal/
funglobal.py
tests/
project/
liblocal/
funlocal.py
main.py
In main.py I have:
import liblocal.funlocal
In funlocal.py I try to import funglobal.py with:
from ....code.libglobal import funglobal
When I run
python3 -B tests/project/main.py
I get an error:
from ....code.libglobal import funglobal
ValueError: attempted relative import beyond top-level package
I have read a lot of information about relative imports with python3 and still don't find how to solve this error without changing the apps organization radically. Any solution?
As the script being executed has its __name__ set as __main__ and defines itself to be on the top level of the package, it refuses to recognize scripts in sibling directories.
You can fix this with a sys.path hack:
import sys, os
sys.path.insert(0, os.path.abspath('../..'))
or an interseting alternative with setuptools is presented in this answer.
Have you a __init__.py script in each folder ?
If no, you should create an empty script named __init__.py in each folder.
I have the following directory structure:
app/
bin/
script1.py
script2.py
lib/
module1/
__init__.py
module1a.py
module1b.py
__init__.py
module2.py
Dockerfile
My problem is that I want to execute script1.py and script2.py, but inside those scripts, I want to import the modules in lib/.
I run my scripts from the root app/ directory (i.e. adjacent to Dockerfile) by simply executing python bin/script1.py. When I import modules into my scripts using from lib.module1 import module1a, I get ImportError: No module named lib.module1. When I try to import using relative imports, such as from ..lib.module1 import module1a, I get ValueError: Attempted relative import in non-package.
When I simply fire up the interpreter and run import lib.module1 or something, I have no issues.
How can I get this to work?
In general, you need __init__.py under app and bin, then you can do a relative import, but that expects a package
If you would structure your python code as python package (egg/wheel) then you could also define an entry point, that would become your /bin/ file post install.
here is an example of a package - https://python-packaging.readthedocs.io/en/latest/minimal.html
and this blog explains entry points quite well - https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
if so, that way you could just do python setup.py install on your package and then have those entry points available within your PATH, as part of that you would start to structure your code in a way that would not create import issues.
You can add to the Python path at runtime in script1.py:
import sys
sys.path.insert(0, '/path/to/your/app/')
import lib.module1.module1a
you have to add current dir to python path.
use export in terminal
or sys.path.insert in your python script both are ok
I have a parent package that has 2 child packages. It looks like this
backend
__init__.py
conf.py
db.py
connections.py
/api
__init__.py
register.py
api.py
/scheduled
__init__.py
helpers.py
All the __init__.py files are empty.
The code in backend/connections.py and backend/conf.py is being used by modules in both packages api and scheduled.
in register.py i have code like
from backend.conf import *
from backend.connections import *
Now when i do python register.py
i get this error
ImportError: No module named backend.conf
Also when i changed from backend.conf import * to from ..conf import * or from .. import conf i get this error
ValueError: Attempted relative import in non-package
What i understand by the above error is that python is not treating the above folders as packages. But i have __init__.py in all the folders. What is wrong?
When you run python register.py, your backend/register.py file is used as the __main__ module of the program, rather than as a module within the backend package. Further more, the Python import path will not automatically include the directory containing the backend directory, which is probably the cause of your problems.
One option that might work is to run your program as python -m backend.register from the top level directory of your project (or set PYTHONPATH so this module can be found). This will search for the script on the normal import path, and then run it as the main program.