beyond top level package error in relative import - python

It seems there are already quite some questions here about relative import in python 3, but after going through many of them I still didn't find the answer for my issue.
so here is the question.
I have a package shown below
package/
__init__.py
A/
__init__.py
foo.py
test_A/
__init__.py
test.py
and I have a single line in test.py:
from ..A import foo
now, I am in the folder of package, and I run
python -m test_A.test
I got message
"ValueError: attempted relative import beyond top-level package"
but if I am in the parent folder of package, e.g., I run:
cd ..
python -m package.test_A.test
everything is fine.
Now my question is:
when I am in the folder of package, and I run the module inside the test_A sub-package as test_A.test, based on my understanding, ..A goes up only one level, which is still within the package folder, why it gives message saying beyond top-level package. What is exactly the reason that causes this error message?

EDIT: There are better/more coherent answers to this question in other questions:
Sibling package imports
Relative imports for the billionth time
Why doesn't it work? It's because python doesn't record where a package was loaded from. So when you do python -m test_A.test, it basically just discards the knowledge that test_A.test is actually stored in package (i.e. package is not considered a package). Attempting from ..A import foo is trying to access information it doesn't have any more (i.e. sibling directories of a loaded location). It's conceptually similar to allowing from ..os import path in a file in math. This would be bad because you want the packages to be distinct. If they need to use something from another package, then they should refer to them globally with from os import path and let python work out where that is with $PATH and $PYTHONPATH.
When you use python -m package.test_A.test, then using from ..A import foo resolves just fine because it kept track of what's in package and you're just accessing a child directory of a loaded location.
Why doesn't python consider the current working directory to be a package? NO CLUE, but gosh it would be useful.

import sys
sys.path.append("..") # Adds higher directory to python modules path.
Try this.
Worked for me.

Assumption:
If you are in the package directory, A and test_A are separate packages.
Conclusion:
..A imports are only allowed within a package.
Further notes:
Making the relative imports only available within packages is useful if you want to force that packages can be placed on any path located on sys.path.
EDIT:
Am I the only one who thinks that this is insane!? Why in the world is the current working directory not considered to be a package? – Multihunter
The current working directory is usually located in sys.path. So, all files there are importable. This is behavior since Python 2 when packages did not yet exist. Making the running directory a package would allow imports of modules as "import .A" and as "import A" which then would be two different modules. Maybe this is an inconsistency to consider.

None of these solutions worked for me in 3.6, with a folder structure like:
package1/
subpackage1/
module1.py
package2/
subpackage2/
module2.py
My goal was to import from module1 into module2. What finally worked for me was, oddly enough:
import sys
sys.path.append(".")
Note the single dot as opposed to the two-dot solutions mentioned so far.
Edit: The following helped clarify this for me:
import os
print (os.getcwd())
In my case, the working directory was (unexpectedly) the root of the project.

This is very tricky in Python.
I'll first comment on why you're having that problem and then I will mention two possible solutions.
What's going on?
You must take this paragraph from the Python documentation into consideration:
Note that relative imports are based on the name of the current
module. Since the name of the main module is always "main",
modules intended for use as the main module of a Python application
must always use absolute imports.
And also the following from PEP 328:
Relative imports use a module's name attribute to determine that
module's position in the package hierarchy. If the module's name does
not contain any package information (e.g. it is set to 'main')
then relative imports are resolved as if the module were a top level
module, regardless of where the module is actually located on the file
system.
Relative imports work from the filename (__name__ attribute), which can take two values:
It's the filename, preceded by the folder strucutre, separated by dots.
For eg: package.test_A.test
Here Python knows the parent directories: before test comes test_A and then package.
So you can use the dot notation for relative import.
# package.test_A/test.py
from ..A import foo
You can then have like a root file in the root directory which calls test.py:
# root.py
from package.test_A import test
When you run the module (test.py) directly, it becomes the entry point to the program , so __name__ == __main__. The filename has no indication of the directory structure, so Python doesn't know how to go up in the directory. For Python, test.py becomes the top-level script, there is nothing above it. That's why you cannot use relative import.
Possible Solutions
A) One way to solve this is to have a root file (in the root directory) which calls the modules/packages, like this:
root.py imports test.py. (entry point, __name__ == __main__).
test.py (relative) imports foo.py.
foo.py says the module has been imported.
The output is:
package.A.foo has been imported
Module's name is: package.test_A.test
B) If you want to execute the code as a module and not as a top-level script, you can try this from the command line:
python -m package.test_A.test
Any suggestions are welcomed.
You should also check: Relative imports for the billionth time , specially BrenBarn's answer.

from package.A import foo
I think it's clearer than
import sys
sys.path.append("..")

As the most popular answer suggests, basically its because your PYTHONPATH or sys.path includes . but not your path to your package. And the relative import is relative to your current working directory, not the file where the import happens; oddly.
You could fix this by first changing your relative import to absolute and then either starting it with:
PYTHONPATH=/path/to/package python -m test_A.test
OR forcing the python path when called this way, because:
With python -m test_A.test you're executing test_A/test.py with __name__ == '__main__' and __file__ == '/absolute/path/to/test_A/test.py'
That means that in test.py you could use your absolute import semi-protected in the main case condition and also do some one-time Python path manipulation:
from os import path
…
def main():
…
if __name__ == '__main__':
import sys
sys.path.append(path.join(path.dirname(__file__), '..'))
from A import foo
exit(main())

This is actually a lot simpler than what other answers make it out to be.
TL;DR: Import A directly instead of attempting a relative import.
The current working directory is not a package, unless you import the folder package from a different folder. So the behavior of your package will work fine if you intend it to be imported by other applications. What's not working is the tests...
Without changing anything in your directory structure, all that needs to be changed is how test.py imports foo.py.
from A import foo
Now running python -m test_A.test from the package directory will run without an ImportError.
Why does that work?
Your current working directory is not a package, but it is added to the path. Therefore you can import folder A and its contents directly. It is the same reason you can import any other package that you have installed... they're all included in your path.

Edit: 2020-05-08: Is seems the website I quoted is no longer controlled by the person who wrote the advice, so I'm removing the link to the site. Thanks for letting me know baxx.
If someone's still struggling a bit after the great answers already provided, I found advice on a website that no longer is available.
Essential quote from the site I mentioned:
"The same can be specified programmatically in this way:
import sys
sys.path.append('..')
Of course the code above must be written before the other import
statement.
It's pretty obvious that it has to be this way, thinking on it after the fact. I was trying to use the sys.path.append('..') in my tests, but ran into the issue posted by OP. By adding the import and sys.path defintion before my other imports, I was able to solve the problem.

Just remove .. in test.py
For me pytest works fine with that
Example:
from A import foo

if you have an __init__.py in an upper folder, you can initialize the import as
import file/path as alias in that init file. Then you can use it on lower scripts as:
import alias

In my case, I had to change to this:
Solution 1(more better which depend on current py file path. Easy to deploy)
Use pathlib.Path.parents make code cleaner
import sys
import os
import pathlib
target_path = pathlib.Path(os.path.abspath(__file__)).parents[3]
sys.path.append(target_path)
from utils import MultiFileAllowed
Solution 2
import sys
import os
sys.path.append(os.getcwd())
from utils import MultiFileAllowed

In my humble opinion, I understand this question in this way:
[CASE 1] When you start an absolute-import like
python -m test_A.test
or
import test_A.test
or
from test_A import test
you're actually setting the import-anchor to be test_A, in other word, top-level package is test_A . So, when we have test.py do from ..A import xxx, you are escaping from the anchor, and Python does not allow this.
[CASE 2] When you do
python -m package.test_A.test
or
from package.test_A import test
your anchor becomes package, so package/test_A/test.py doing from ..A import xxx does not escape the anchor(still inside package folder), and Python happily accepts this.
In short:
Absolute-import changes current anchor (=redefines what is the top-level package);
Relative-import does not change the anchor but confines to it.
Furthermore, we can use full-qualified module name(FQMN) to inspect this problem.
Check FQMN in each case:
[CASE2] test.__name__ = package.test_A.test
[CASE1] test.__name__ = test_A.test
So, for CASE2, an from .. import xxx will result in a new module with FQMN=package.xxx, which is acceptable.
While for CASE1, the .. from within from .. import xxx will jump out of the starting node(anchor) of test_A, and this is NOT allowed by Python.
[2022-07-19] I think this "relative-import" limitation is quite an ugly design, totally against (one of) Python's motto "Simple is better than complex".

Not sure in python 2.x but in python 3.6, assuming you are trying to run the whole suite, you just have to use -t
-t, --top-level-directory directory
Top level directory of project (defaults to start directory)
So, on a structure like
project_root
|
|----- my_module
| \
| \_____ my_class.py
|
\ tests
\___ test_my_func.py
One could for example use:
python3 unittest discover -s /full_path/project_root/tests -t /full_path/project_root/
And still import the my_module.my_class without major dramas.

Having
package/
__init__.py
A/
__init__.py
foo.py
test_A/
__init__.py
test.py
in A/__init__.py import foo:
from .foo import foo
when importing A/ from test_A/
import sys, os
sys.path.append(os.path.abspath('../A'))
# then import foo
import foo

Related

Python, absolute/relative import not working as expected

I apologize for the millionth post about this topic.
I thought I had a good grip of the whole absolute/relative import mechanism - I even replied to a couple of questions about it myself - but I'm having a problem with it and I can't figure out how to solve it.
I'm using Python 3.8.0, this is my directory structure:
project_folder
scripts/
main.py
models/
__init__.py
subfolder00/
subfolder01/
some_script.py --> contains def for some_function
I need to import some_function from some_script.py when running main.py, so I tried:
1) relative import
# in main.py
from ..models.subfolder00.subfolder01.somescript import some_function
but when I run (from the scripts/ folder)
python main.py
this fails with error:
ImportError: attempted relative import with no known parent package
This was expected, because I'm running main.py directly as a script, so its _name_ is set to _main_ and relative imports are bound to fail.
However, I was expecting it to work when running (always from within the scripts folder):
python -m main
but I'm getting always the same error.
2) absolute import
I tried changing the import in main.py to:
# in main.py
from models.subfolder00.subfolder01.somescript import some_function
and running, this time from the main project folder:
python scripts/main.py
so that - I was assuming - the starting point for the absolute import would be the project folder itself, from which it could get to models/....
But now I'm getting the error:
ModuleNotFoundError: No module named 'models'
Why didn't it work when using the -m option in the case of relative import, and it's not working when using absolute ones either? Which is the correct way to do this?
I think quite likely you missed python's official doc ( that even come offline )
https://docs.python.org/3/tutorial/modules.html
you'll need a dummy __init__.py within your module, at same level of some_script.py
I think your "absolute" import may not have been absolute in the truest sense.
Prior to running the python scripts/main.py command, you would have needed to setup PYTHONPATH environment variable to include the path to project_folder.
Alternatively I do something like this in main.py:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),'..','models','subfolder00','subfolder01'))
from somescript import some_function
Maybe it is a little pedantic, but it makes sense to me.

Can someone please explain why python relative imports aren't intuitive? [duplicate]

It seems there are already quite some questions here about relative import in python 3, but after going through many of them I still didn't find the answer for my issue.
so here is the question.
I have a package shown below
package/
__init__.py
A/
__init__.py
foo.py
test_A/
__init__.py
test.py
and I have a single line in test.py:
from ..A import foo
now, I am in the folder of package, and I run
python -m test_A.test
I got message
"ValueError: attempted relative import beyond top-level package"
but if I am in the parent folder of package, e.g., I run:
cd ..
python -m package.test_A.test
everything is fine.
Now my question is:
when I am in the folder of package, and I run the module inside the test_A sub-package as test_A.test, based on my understanding, ..A goes up only one level, which is still within the package folder, why it gives message saying beyond top-level package. What is exactly the reason that causes this error message?
EDIT: There are better/more coherent answers to this question in other questions:
Sibling package imports
Relative imports for the billionth time
Why doesn't it work? It's because python doesn't record where a package was loaded from. So when you do python -m test_A.test, it basically just discards the knowledge that test_A.test is actually stored in package (i.e. package is not considered a package). Attempting from ..A import foo is trying to access information it doesn't have any more (i.e. sibling directories of a loaded location). It's conceptually similar to allowing from ..os import path in a file in math. This would be bad because you want the packages to be distinct. If they need to use something from another package, then they should refer to them globally with from os import path and let python work out where that is with $PATH and $PYTHONPATH.
When you use python -m package.test_A.test, then using from ..A import foo resolves just fine because it kept track of what's in package and you're just accessing a child directory of a loaded location.
Why doesn't python consider the current working directory to be a package? NO CLUE, but gosh it would be useful.
import sys
sys.path.append("..") # Adds higher directory to python modules path.
Try this.
Worked for me.
Assumption:
If you are in the package directory, A and test_A are separate packages.
Conclusion:
..A imports are only allowed within a package.
Further notes:
Making the relative imports only available within packages is useful if you want to force that packages can be placed on any path located on sys.path.
EDIT:
Am I the only one who thinks that this is insane!? Why in the world is the current working directory not considered to be a package? – Multihunter
The current working directory is usually located in sys.path. So, all files there are importable. This is behavior since Python 2 when packages did not yet exist. Making the running directory a package would allow imports of modules as "import .A" and as "import A" which then would be two different modules. Maybe this is an inconsistency to consider.
None of these solutions worked for me in 3.6, with a folder structure like:
package1/
subpackage1/
module1.py
package2/
subpackage2/
module2.py
My goal was to import from module1 into module2. What finally worked for me was, oddly enough:
import sys
sys.path.append(".")
Note the single dot as opposed to the two-dot solutions mentioned so far.
Edit: The following helped clarify this for me:
import os
print (os.getcwd())
In my case, the working directory was (unexpectedly) the root of the project.
This is very tricky in Python.
I'll first comment on why you're having that problem and then I will mention two possible solutions.
What's going on?
You must take this paragraph from the Python documentation into consideration:
Note that relative imports are based on the name of the current
module. Since the name of the main module is always "main",
modules intended for use as the main module of a Python application
must always use absolute imports.
And also the following from PEP 328:
Relative imports use a module's name attribute to determine that
module's position in the package hierarchy. If the module's name does
not contain any package information (e.g. it is set to 'main')
then relative imports are resolved as if the module were a top level
module, regardless of where the module is actually located on the file
system.
Relative imports work from the filename (__name__ attribute), which can take two values:
It's the filename, preceded by the folder strucutre, separated by dots.
For eg: package.test_A.test
Here Python knows the parent directories: before test comes test_A and then package.
So you can use the dot notation for relative import.
# package.test_A/test.py
from ..A import foo
You can then have like a root file in the root directory which calls test.py:
# root.py
from package.test_A import test
When you run the module (test.py) directly, it becomes the entry point to the program , so __name__ == __main__. The filename has no indication of the directory structure, so Python doesn't know how to go up in the directory. For Python, test.py becomes the top-level script, there is nothing above it. That's why you cannot use relative import.
Possible Solutions
A) One way to solve this is to have a root file (in the root directory) which calls the modules/packages, like this:
root.py imports test.py. (entry point, __name__ == __main__).
test.py (relative) imports foo.py.
foo.py says the module has been imported.
The output is:
package.A.foo has been imported
Module's name is: package.test_A.test
B) If you want to execute the code as a module and not as a top-level script, you can try this from the command line:
python -m package.test_A.test
Any suggestions are welcomed.
You should also check: Relative imports for the billionth time , specially BrenBarn's answer.
from package.A import foo
I think it's clearer than
import sys
sys.path.append("..")
As the most popular answer suggests, basically its because your PYTHONPATH or sys.path includes . but not your path to your package. And the relative import is relative to your current working directory, not the file where the import happens; oddly.
You could fix this by first changing your relative import to absolute and then either starting it with:
PYTHONPATH=/path/to/package python -m test_A.test
OR forcing the python path when called this way, because:
With python -m test_A.test you're executing test_A/test.py with __name__ == '__main__' and __file__ == '/absolute/path/to/test_A/test.py'
That means that in test.py you could use your absolute import semi-protected in the main case condition and also do some one-time Python path manipulation:
from os import path
…
def main():
…
if __name__ == '__main__':
import sys
sys.path.append(path.join(path.dirname(__file__), '..'))
from A import foo
exit(main())
This is actually a lot simpler than what other answers make it out to be.
TL;DR: Import A directly instead of attempting a relative import.
The current working directory is not a package, unless you import the folder package from a different folder. So the behavior of your package will work fine if you intend it to be imported by other applications. What's not working is the tests...
Without changing anything in your directory structure, all that needs to be changed is how test.py imports foo.py.
from A import foo
Now running python -m test_A.test from the package directory will run without an ImportError.
Why does that work?
Your current working directory is not a package, but it is added to the path. Therefore you can import folder A and its contents directly. It is the same reason you can import any other package that you have installed... they're all included in your path.
Edit: 2020-05-08: Is seems the website I quoted is no longer controlled by the person who wrote the advice, so I'm removing the link to the site. Thanks for letting me know baxx.
If someone's still struggling a bit after the great answers already provided, I found advice on a website that no longer is available.
Essential quote from the site I mentioned:
"The same can be specified programmatically in this way:
import sys
sys.path.append('..')
Of course the code above must be written before the other import
statement.
It's pretty obvious that it has to be this way, thinking on it after the fact. I was trying to use the sys.path.append('..') in my tests, but ran into the issue posted by OP. By adding the import and sys.path defintion before my other imports, I was able to solve the problem.
Just remove .. in test.py
For me pytest works fine with that
Example:
from A import foo
if you have an __init__.py in an upper folder, you can initialize the import as
import file/path as alias in that init file. Then you can use it on lower scripts as:
import alias
In my case, I had to change to this:
Solution 1(more better which depend on current py file path. Easy to deploy)
Use pathlib.Path.parents make code cleaner
import sys
import os
import pathlib
target_path = pathlib.Path(os.path.abspath(__file__)).parents[3]
sys.path.append(target_path)
from utils import MultiFileAllowed
Solution 2
import sys
import os
sys.path.append(os.getcwd())
from utils import MultiFileAllowed
In my humble opinion, I understand this question in this way:
[CASE 1] When you start an absolute-import like
python -m test_A.test
or
import test_A.test
or
from test_A import test
you're actually setting the import-anchor to be test_A, in other word, top-level package is test_A . So, when we have test.py do from ..A import xxx, you are escaping from the anchor, and Python does not allow this.
[CASE 2] When you do
python -m package.test_A.test
or
from package.test_A import test
your anchor becomes package, so package/test_A/test.py doing from ..A import xxx does not escape the anchor(still inside package folder), and Python happily accepts this.
In short:
Absolute-import changes current anchor (=redefines what is the top-level package);
Relative-import does not change the anchor but confines to it.
Furthermore, we can use full-qualified module name(FQMN) to inspect this problem.
Check FQMN in each case:
[CASE2] test.__name__ = package.test_A.test
[CASE1] test.__name__ = test_A.test
So, for CASE2, an from .. import xxx will result in a new module with FQMN=package.xxx, which is acceptable.
While for CASE1, the .. from within from .. import xxx will jump out of the starting node(anchor) of test_A, and this is NOT allowed by Python.
[2022-07-19] I think this "relative-import" limitation is quite an ugly design, totally against (one of) Python's motto "Simple is better than complex".
Not sure in python 2.x but in python 3.6, assuming you are trying to run the whole suite, you just have to use -t
-t, --top-level-directory directory
Top level directory of project (defaults to start directory)
So, on a structure like
project_root
|
|----- my_module
| \
| \_____ my_class.py
|
\ tests
\___ test_my_func.py
One could for example use:
python3 unittest discover -s /full_path/project_root/tests -t /full_path/project_root/
And still import the my_module.my_class without major dramas.
Having
package/
__init__.py
A/
__init__.py
foo.py
test_A/
__init__.py
test.py
in A/__init__.py import foo:
from .foo import foo
when importing A/ from test_A/
import sys, os
sys.path.append(os.path.abspath('../A'))
# then import foo
import foo

Pythonic way to make subdirectories available to other subdirectories in my project [duplicate]

I'm trying to follow PEP 328, with the following directory structure:
pkg/
__init__.py
components/
core.py
__init__.py
tests/
core_test.py
__init__.py
In core_test.py I have the following import statement
from ..components.core import GameLoopEvents
However, when I run, I get the following error:
tests$ python core_test.py
Traceback (most recent call last):
File "core_test.py", line 3, in <module>
from ..components.core import GameLoopEvents
ValueError: Attempted relative import in non-package
Searching around I found "relative path not working even with __init__.py" and "Import a module from a relative path" but they didn't help.
Is there anything I'm missing here?
To elaborate on Ignacio Vazquez-Abrams's answer:
The Python import mechanism works relative to the __name__ of the current file. When you execute a file directly, it doesn't have its usual name, but has "__main__" as its name instead. So relative imports don't work.
You can, as Igancio suggested, execute it using the -m option. If you have a part of your package that is meant to be run as a script, you can also use the __package__ attribute to tell that file what name it's supposed to have in the package hierarchy.
See http://www.python.org/dev/peps/pep-0366/ for details.
Yes. You're not using it as a package.
python -m pkg.tests.core_test
It depends on how you want to launch your script.
If you want to launch your UnitTest from the command line in a classic way, that is:
python tests/core_test.py
Then, since in this case 'components' and 'tests' are siblings folders, you can import the relative module either using the insert or the append method of the sys.path module.
Something like:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from components.core import GameLoopEvents
Otherwise, you can launch your script with the '-m' argument (note that in this case, we are talking about a package, and thus you must not give the '.py' extension), that is:
python -m pkg.tests.core_test
In such a case, you can simply use the relative import as you were doing:
from ..components.core import GameLoopEvents
You can finally mix the two approaches, so that your script will work no matter how it is called.
For example:
if __name__ == '__main__':
if __package__ is None:
import sys
from os import path
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
from components.core import GameLoopEvents
else:
from ..components.core import GameLoopEvents
You can use import components.core directly if you append the current directory to sys.path:
if __name__ == '__main__' and __package__ is None:
from os import sys, path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
In core_test.py, do the following:
import sys
sys.path.append('../components')
from core import GameLoopEvents
Issue is with your testing method,
you tried python core_test.py
then you will get this error
ValueError: Attempted relative import in non-package
Reason: you are testing your packaging from non-package source.
so test your module from package source.
if this is your project structure,
pkg/
__init__.py
components/
core.py
__init__.py
tests/
core_test.py
__init__.py
cd pkg
python -m tests.core_test # dont use .py
or from outside pkg/
python -m pkg.tests.core_test
single . if you want to import from folder in same directory .
for each step back add one more.
hi/
hello.py
how.py
in how.py
from .hi import hello
incase if you want to import how from hello.py
from .. import how
If your use case is for running tests, and it seams that it is, then you can do the following. Instead of running your test script as python core_test.py use a testing framework such as pytest. Then on the command line you can enter
$$ py.test
That will run the tests in your directory. This gets around the issue of __name__ being __main__ that was pointed out by #BrenBarn. Next, put an empty __init__.py file into your test directory, this will make the test directory part of your package. Then you will be able to do
from ..components.core import GameLoopEvents
However, if you run your test script as a main program then things will fail once again. So just use the test runner. Maybe this also works with other test runners such as nosetests but i haven't checked it. Hope this helps.
My quick-fix is to add the directory to the path:
import sys
sys.path.insert(0, '../components/')
As Paolo said, we have 2 invocation methods:
1) python -m tests.core_test
2) python tests/core_test.py
One difference between them is sys.path[0] string. Since the interpret will search sys.path when doing import, we can do with tests/core_test.py:
if __name__ == '__main__':
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from components import core
<other stuff>
And more after this, we can run core_test.py with other methods:
cd tests
python core_test.py
python -m core_test
...
Note, py36 tested only.
As you have already marked everything as a module, there's no need to use the relative reference if you launch as python module.
Instead of
from ..components.core import GameLoopEvents
simply
from pkg.components.core import GameLoopEvents
When you run from the parent of pkg, use the following
python -m pkg.tests.core_test
Old thread. I found out that adding an __all__= ['submodule', ...] to the
__init__.py file and then using the from <CURRENT_MODULE> import * in the target works fine.
You can use from pkg.components.core import GameLoopEvents, for example I use pycharm, the below is my project structure image, I just import from the root package, then it works:
This approach worked for me and is less cluttered than some solutions:
try:
from ..components.core import GameLoopEvents
except ValueError:
from components.core import GameLoopEvents
The parent directory is in my PYTHONPATH, and there are __init__.py files in the parent directory and this directory.
The above always worked in python 2, but python 3 sometimes hit an ImportError or ModuleNotFoundError (the latter is new in python 3.6 and a subclass of ImportError), so the following tweak works for me in both python 2 and 3:
try:
from ..components.core import GameLoopEvents
except ( ValueError, ImportError):
from components.core import GameLoopEvents
Try this
import components
from components import *
If someone is looking for a workaround, I stumbled upon one. Here's a bit of context. I wanted to test out one of the methods I've in a file. When I run it from within
if __name__ == "__main__":
it always complained of the relative imports. I tried to apply the above solutions, but failed to work, since there were many nested files, each with multiple imports.
Here's what I did. I just created a launcher, an external program that would import necessary methods and call them. Though, not a great solution, it works.
Here's one way which will piss off everyone but work pretty well. In tests run:
ln -s ../components components
Then just import components like you normally would.
For me only this worked: I had to explicitly set the value of package to the parent directory, and add the parent directory to sys.path
from os import path
import sys
if __package__ is None:
sys.path.append( path.dirname( path.dirname( path.abspath(__file__) ) ) )
__package__= "myparent"
from .subdir import something # the . can now be resolved
I can now directly run my script with python myscript.py.
python <main module>.py does not work with relative import
The problem is relative import does not work when you run a __main__ module from the command line
python <main_module>.py
It is clearly stated in PEP 338.
The release of 2.5b1 showed a surprising (although obvious in retrospect) interaction between this PEP and PEP 328 - explicit relative imports don't work from a main module. This is due to the fact that relative imports rely on __name__ to determine the current module's position in the package hierarchy. In a main module, the value of __name__ is always '__main__', so explicit relative imports will always fail (as they only work for a module inside a package).
Cause
Python Bug Tracker Issue1510172: Absolute/relative import not working?
The issue isn't actually unique to the -m switch. The problem is that relative imports are based on __name__, and in the main module, __name__ always has the value __main__. Hence, relative imports currently can't work properly from the main module of an application, because the main module doesn't know where it really fits in the Python module namespace (this is at least fixable in theory for the main modules executed through the -m switch, but directly executed files and the interactive interpreter are completely out of luck).
To understand further, see Relative imports in Python 3 for the detailed explanation and how to get it over.
I've had similar issues and as a software engineer, I think some of the suggested solutions here are not ideal. If you want relative imports, you should not have try/except and then sometimes do an absolute import. Also, to run a program, you should not have to change sys.path.
Furthermore, the program should always work, independent of your current working directory and independent of how you start it.
Thus, I've created a new, experimental import library: ultraimport
It allows file system based imports, no matter how you run your code.
From the original question, you would change your core_test.py to something like
import ultraimport
GameLoopEvents = ultraimport('__dir__/../components/core.py', 'GameLoopEvents')
print(GameLoopEvents)
and it would always find it, no matter how you run your tests.
$ python -m tests.core_test
<class 'core.GameLoopEvents'>
python ./tests/core_test.py
<class 'core.GameLoopEvents'>
I've also put this example into the examples folder in the git repo.
As the library is experimental, I am interested in feedback. It works for me but it not widely tested, yet.
If your project structure would look like this:
project
|
| --- module1
| |
| file1.py
|
|-----module2
| |
| file2.py
and you are going import file1.py from within file2.py,
you can do this in file2.py:
import sys
sys.path.append('.')
import file2
I still don't know why and how, but it worked for me.
This is very confusing and if you are using IDE like Pycharm, it's little more confusing.
What worked for me:
Make Pycharm project settings (if you are running python from a VE or from Python directory)
There is nothing wrong with the way you defined. Sometime it works with:
from folder1.file1 import class
if it does not work, use:
import folder1.file1
Your environment variable should be correctly mentioned in system or provide it in your command line argument.
Because your code contains if __name__ == "__main__", which doesn't be imported as a package, you'd better use sys.path.append() to solve the problem.

something wrong with my pythonpath

I know this is a dumb question but i'm stumped. My directory structure used to look like this:
-src
|
-module.py
-program.py
when this what my directory structure, I referenced module from program and all was well.
I've since changed my directory structure to this:
-src
|
-__init.py
-module.py
|
-programDir
|
-__init.py
-program.py
now, of course, I can't reach the module from program. How can I reference src as a package. I tried to create an
__init__.py
file in the src directory, but no luck.
Moar deets:
import statements i've tried in program.py:
import module
and
from src import module
the first one worked when the other module and program were in the same directory.
error i'm getting:
ImportError: No module named module
and just for the record: No, my module and program are not called module OR program
update: I've tried this in my program.py file:
from ...src import module
and
from ..src import module
both are giving me:
ValueError: Attempted relative import in non-package
For starters, I recommend reading the entry Modifying Python's Search Path in the docs.
It might be frowned upon by some, but if you wish to modify the PYTHONPATH from within your program, according to the documentation's standard modules entry you can use the sys.path.append method:
import sys
sys.path.append('..')
import module
Couldn't you use PEP 328 to solve this?
If you run program.py directly, with python program.py or with #!, then module.py's directory should be in the PYTHONPATH for import module to work. This can be achieved using a helper shell script that's kept in programDir, for instance, and looks something like:
#!/bin/bash
script_dir=`dirname $0`
# Add the script's parent directory to the PYTHONPATH
export PYTHONPATH=$PYTHONPATH:$script_dir/..
python $script_dir/program.py
Another, probably better, way would be to have program.py export a "main()" function, and create a helper python script at src/program that looks like:
#!/usr/bin/env python
from programDir.program import main
main()
In this case, you can use relative imports in src/programDir/program.py, so this should work:
from .. import module
The first one worked because Python's sys.path's first entry is '' which means it will look for module names in the current working directory from which you've executed the Python interpreter.
The issue you seem to have is that the directory located at src is not set on your PYTHONPATH. So, you can do is set the PYTHONPATH environment variable explicitly.
Here's an example using bash:
export PYTHONPATH=PATH_TO_SRC:${PYTHONPATH}
then run your program as normal
Another approach is that you can explicitly set sys.path by appending to it upon execution of your program.
So, in your program.py, you would have:
if __name__ == '__main__':
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
your_main_function()
Lastly, for serious python development, you should consider virtualenv and virtualenvwrapper as it will take care of most of these things for you.
You need to add __init__.py to /programDir to interpret the directory as a package. Once a package, you can import the package's contents.
So, in your case, if /src is on the PYTHONPATH, from module.py you can import program.py with from programDir import program.
If you use program as part of a package, in another python module, such as
import src.programDir.program as p
p.some_method()
you can use relative import in program.py, assuming you are creating a package with src (__init__.py in both src and programDir)
from .. import module
If not, for example you are calling program.py from the command line, you must add the directory containing src to your search path either by modifying sys.path or the PYTHONPATH env var, before importing.

python packaging for relative imports

First off all: I'm sorry, I know there has been lots of question about relative imports, but I just didn't find a solution. If possible I would like to use the following directory layout:
myClass/
__init__.py
test/
demo.py
benchmark.py
specs.py
src/
__init__.py
myClass.py
Now my questions are:
How do the test files from within the package properly import myClass.py?
How would you import the package from outside, assuming you take myClass as submodule in libs/myClass or include/myClass?
So far I couldn't find an elegant solution for this. From what I understand Guido's Decision it should be possible to do from ..src import myClass but this will error:
ValueError: Attempted relative import in non-package
Which looks as it doesn't treat myClass as packages. Reading the docs:
The __init__.py files are required to make Python treat the directories as containing packages;
It seems I'm missing something that specifies where the scripts of the package are, should I use .pth ?
ValueError: Attempted relative import in non-package
Means you attempt to use relative import in the module which is not package. Its problem with the file which has this from ... import statement, and not the file which you are trying to import.
So if you are doing relative imports in your tests, for example, you should make your tests to be part of your package. This means
Adding __init__.py to test/
Running them from some outside script, like nosetests
If you run something as python myClass/test/demo.py, relative imports will not work too since you are running demo module not as package. Relative imports require that the module which uses them is being imported itself either as package module, from myClass.test.demo import blabla, or with relative import.
After hours of searching last night I found the answer to relative imports in python!! Or an easy solution at the very least. The best way to fix this is to have the modules called from another module. So say you want demo.py to import myClass.py. In the myClass folder at the root of the sub-packages they need to have a file that calls the other two. From what I gather the working directory is always considered __main__ so if you test the import from demo.py with the demo.py script, you will receive that error. To illustrate:
Folder hierarchy:
myClass/
main.py #arbitrary name, can be anything
test/
__init__.py
demo.py
src/
__init__.py
myClass.py
myClass.py:
def randomMaths(x):
a = x * 2
y = x * a
return y
demo.py:
from ..src import myClass
def printer():
print(myClass.randomMaths(42))
main.py:
import test.demo
demo.printer()
If you run demo.py in the interpreter, you will generate an error, but running main.py will not. It's a little convoluted, but it works :D
Intra-package-references describes how to myClass from test/*. To import the package from outside, you should add its path to PYTHONPATH environment variable before running the importer application, or to sys.path list in the code before importing it.
Why from ..src import myClass fails: probably, src is not a python package, you cannot import from there. You should add it to python path as described above.

Categories