I'm writing a little package and I'm trying to include a demo script within it as an example. However, I can't seem to import the package cleanly from within as though I was outside of it.
With a directory structure like:
trainer/
__init__.py
helper.py
trainer.py
[...more files...]
demo.py
In demo.py I can't do from .. import trainer as it complains "Attempted relative import in non-package", despite the __init__.py. If I move the demo up a directory and import trainer it works fine, but I was trying to keep it together with the package.
The hack-looking import __init__ as trainer works, but eeeew.
Importing the various bits from all over the module directly also works, but makes for a messy example. Am I wholly misguided in my attempt or is there a better solution?
If you're trying to run demo.py as python demo.py, the problem that you're having is likely the same as here.
What's happening is that Python's relative import mechanism works by using the __name__ of the current module. When you execute a module directly, the __name__ gets set "__main__" regardless what the actual module name is. Thus, relative (in-package) imports don't work.
To remedy this, you can do the following:
Execute demo.py as a module within a package, like so: python -m trainer.demo. This should fix the error, but you'll still be importing the trainer.py module instead of the package.
Now add from __future__ import absolute_import to demo.py, which will cause your imports to be absolute-only by default, meaning that relative imports have to explicit (as in, from . import (...)). This is force import trainer to import the entire top-level package, instead of the module.
The way you organize the files, demo.py becomes part of the package, which might or might not be what you want. You can organize your files a little differently, moving demo.py outside of the trainer directory:
TopDir/
demo.py
trainer/
__init__.py
helper.py
trainer.py
[... more files ...]
Then, demo.py can do something like:
from trainer import trainer, helper
Related
I apologize for the millionth post about this topic.
I thought I had a good grip of the whole absolute/relative import mechanism - I even replied to a couple of questions about it myself - but I'm having a problem with it and I can't figure out how to solve it.
I'm using Python 3.8.0, this is my directory structure:
project_folder
scripts/
main.py
models/
__init__.py
subfolder00/
subfolder01/
some_script.py --> contains def for some_function
I need to import some_function from some_script.py when running main.py, so I tried:
1) relative import
# in main.py
from ..models.subfolder00.subfolder01.somescript import some_function
but when I run (from the scripts/ folder)
python main.py
this fails with error:
ImportError: attempted relative import with no known parent package
This was expected, because I'm running main.py directly as a script, so its _name_ is set to _main_ and relative imports are bound to fail.
However, I was expecting it to work when running (always from within the scripts folder):
python -m main
but I'm getting always the same error.
2) absolute import
I tried changing the import in main.py to:
# in main.py
from models.subfolder00.subfolder01.somescript import some_function
and running, this time from the main project folder:
python scripts/main.py
so that - I was assuming - the starting point for the absolute import would be the project folder itself, from which it could get to models/....
But now I'm getting the error:
ModuleNotFoundError: No module named 'models'
Why didn't it work when using the -m option in the case of relative import, and it's not working when using absolute ones either? Which is the correct way to do this?
I think quite likely you missed python's official doc ( that even come offline )
https://docs.python.org/3/tutorial/modules.html
you'll need a dummy __init__.py within your module, at same level of some_script.py
I think your "absolute" import may not have been absolute in the truest sense.
Prior to running the python scripts/main.py command, you would have needed to setup PYTHONPATH environment variable to include the path to project_folder.
Alternatively I do something like this in main.py:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),'..','models','subfolder00','subfolder01'))
from somescript import some_function
Maybe it is a little pedantic, but it makes sense to me.
I have a python project structured like this:
repo_dir/
----project_package/
--------__init__.py
--------process.py
--------config.py
----tests/
--------test_process.py
__init__.py is empty
config.py looks like this:
name = 'brian'
USAGE
I use the library by running python process.py from the project/project/ directory, or by specifying the python file path absolutely. I'm running Python 2.7 on Amazon EC2 Linux.
When process.py looks like below, everything works fine and process.py prints brian.
import config
print config.name
When process.py looks like below, I get the error ImportError: No module named project.config.
import project.config
print config.name
When process.py looks like below, I get the error ImportError: No module named project. This makes sense as the same behavior from the previous example should be expected.
from project import config
print config.name
If I add these lines to process.py to include the library root in sys.path, all configurations above, work fine.
import os
import sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
MY CONFUSION
Many resources suggest setting up python libraries to import modules using project.module_name, but it doesn't seem like sys.path appending is standard, and seems weird that I need it. I can see that the sys.path append added my library root as a path in sys, but I thought that's what the __init__.py in my library root was supposed to do. What gives? What am I missing? I know Python importing creates lots of headaches so I've tried to simplify this as much as possible to wrap my head around it. I'm going crazy and it's Friday before a holiday. I'm bummed. Please help!!
QUESTIONS
How should I set up my libraries? How should I import packages? Where should I have __init__.py files? Do I need to append my library root to sys.path in every project? Why is this so confusing?
Your project setup is alright. I renamed the directories just for clarity
in this example, but the structure is the same as yours:
repo_dir/
project_package/
__init__.py
process.py
config.py
# Declare your project in a setup.py file, so that
# it will be installable, both by users and by you.
setup.py
When you have a module that wants to import from another module in
the same project, the best approach is to use relative imports. For example:
# In process.py
from .config import name
...
While working on the code on your dev box, do your work in a Python virtualenv,
and pip install your project in "editable" mode.
# From the root of your repo:
pip install -e .
With that approach, you'll never need to muck around with sys.path -- which
is almost always the wrong approach.
I think the problem is how you're running your script. If you want the script to be living in a package (the inner project folder), you should run it with python -m project.process, rather than by filename. Then you can make absolute or explicit relative imports to get config from process.
An absolute import would be from project import config or import project.config.
An explicit relative import would be from . import config.
Python 2 also allows implicit relative imports, but they're a really bad misfeature that you should never use. With implicit relative imports, internal package modules can shadow top-level modules. For instance, a project/json.py file would hide the standard library's json module from all the other modules in the package. You can tell Python you want to forbid implicit relative imports by putting from __future__ import absolute_import at the top of the file. It's the standard behavior in Python 3.
I know this question has been asked many times in the similiar fashion but I want to understand the import mechanism of Python in the simplest example.
Suppose I have the following directory structure:
.\Project\moduleOne.py
.\Project\moduleTwo.py
Basically I import a function from moduleTwo while being in moduleOne:
from moduleTwo import myFunction
Everything works as intended, I can use myFunction. However, if I change the import statement, like below:
from .moduleTwo import MyFunction
I receive the following error:
ValueError: Attempted relative import in non-package
Even if I have a __init__.py file to make a project directory a package I still receive the same error.
Any help would be really appreciated. Thanks!
I imagine that the current directory when you do this is .\Project, and that you are running
python3 moduleOne.py
You are therefore importing moduleTwo as an independent module rather than as a submodule of a package.
If you were, for example, to ascend the directory hierarchy so that your current directory is the Project directory's parent, and execute
python3 -m Project.moduleOne
you will find that the relative import from .moduleTwo executes perfectly well.
I'm starting a project in python, the code structure now as below:
project/
__init__.py
a.py
b.py
mainA.py
utilities/
__init__.py
mainB.py
c.py
The __init__ files are all blank.
I want to run utilities/mainB.py as a program(using something like python main.py), and mainB needs to import a.py and b.py. So I tried from .. import a and some other approaches, but the import failed. The error information is:
ValueError: Attempted relative import in non-package
So here comes the questions:
how to fix mainB.py so it can be run as a main program?
mainA.py can be run as main program now, it also imports a.py and b.py(using import a and import b).
I think the code structure may become more complex. Say, if mainA.py has to import a module from project/some/directory, how can I do that?
See this previous question. You have two options. One is to use the __package__ attribute as described in PEP 366 to set the relative name of your modules. The other is to execute your scripts as modules (using the -m flag to the interpreter) instead of running them directly as scripts.
You could use Python's built-in module-running functionality (python -m <module>).
python -m project.utilities.mainB
This allows you to write mainB normally as part of the package, so relative and absolute imports will both work correctly.
For an in-depth discussion of this functionality, see PEP-338.
You should add 'project' dir in PYTHON_PATH and then, in mainB.py:
from project import a
First off all: I'm sorry, I know there has been lots of question about relative imports, but I just didn't find a solution. If possible I would like to use the following directory layout:
myClass/
__init__.py
test/
demo.py
benchmark.py
specs.py
src/
__init__.py
myClass.py
Now my questions are:
How do the test files from within the package properly import myClass.py?
How would you import the package from outside, assuming you take myClass as submodule in libs/myClass or include/myClass?
So far I couldn't find an elegant solution for this. From what I understand Guido's Decision it should be possible to do from ..src import myClass but this will error:
ValueError: Attempted relative import in non-package
Which looks as it doesn't treat myClass as packages. Reading the docs:
The __init__.py files are required to make Python treat the directories as containing packages;
It seems I'm missing something that specifies where the scripts of the package are, should I use .pth ?
ValueError: Attempted relative import in non-package
Means you attempt to use relative import in the module which is not package. Its problem with the file which has this from ... import statement, and not the file which you are trying to import.
So if you are doing relative imports in your tests, for example, you should make your tests to be part of your package. This means
Adding __init__.py to test/
Running them from some outside script, like nosetests
If you run something as python myClass/test/demo.py, relative imports will not work too since you are running demo module not as package. Relative imports require that the module which uses them is being imported itself either as package module, from myClass.test.demo import blabla, or with relative import.
After hours of searching last night I found the answer to relative imports in python!! Or an easy solution at the very least. The best way to fix this is to have the modules called from another module. So say you want demo.py to import myClass.py. In the myClass folder at the root of the sub-packages they need to have a file that calls the other two. From what I gather the working directory is always considered __main__ so if you test the import from demo.py with the demo.py script, you will receive that error. To illustrate:
Folder hierarchy:
myClass/
main.py #arbitrary name, can be anything
test/
__init__.py
demo.py
src/
__init__.py
myClass.py
myClass.py:
def randomMaths(x):
a = x * 2
y = x * a
return y
demo.py:
from ..src import myClass
def printer():
print(myClass.randomMaths(42))
main.py:
import test.demo
demo.printer()
If you run demo.py in the interpreter, you will generate an error, but running main.py will not. It's a little convoluted, but it works :D
Intra-package-references describes how to myClass from test/*. To import the package from outside, you should add its path to PYTHONPATH environment variable before running the importer application, or to sys.path list in the code before importing it.
Why from ..src import myClass fails: probably, src is not a python package, you cannot import from there. You should add it to python path as described above.