I have started learning python very recently. I do have a typescript and C# background. I was trying to understand the packages and imports statements. Following is the very basic folder structure I have created and I have few questions on this
app/
|-> pkg1/
|-> fib.py
|-> __init__.py
|- >pkg2/
|-> mul.py
|-> __init__.py
|-> app.py
ext/
|-> ext.py
|-> __init__.py
fib.py
------
from pkg2.mul import mul
// from ..pkg2.mul import mul -> Error: attempted relative import beyond top-level package
// but this should work as per docs https://docs.python.org/3/tutorial/modules.html#intra-package-references
// can someone tell why do I get this error?
def fib(n):
return mul(n) * 2
mul.py
------
def mul(n):
return n * 10
app.py
------
from pkg1.fib import fib
print(fib(1))
I have read Python treats any file as entry file.
Does python even know app.py is the entry point? No I guess.
Why does it work when I have given pkg2.mul in fib.py
Why does it work when I have given pkg1.fib in app.py
What if I want to include ext.py inside mul.py and app.py
As per folder structure, if app.py imports pkg1.fib (relative to app.py) is valid then fib.py must import mul in different way correct? why it is working for pkg2.mul? Is it actually Valid? pkg2/ is not relative to fib.py.
Am I missing something? In typescript we use relative imports and C# we use namespaces and include files but here I am pretty confused. Any help is greatly appreciated.
Your relative import attempt did not work because pkg1 and pkg2 are separate packages. Relative imports only work within one package.
Does python even know app.py is the entry point?
No. It's just some .py file sitting around in a directory.
Why does it work when I have given pkg2.mul in fib.py
Coincidence. You are in the app/ directory, and the first element of sys.path is the working directory. It likely won't work if you cd to some other directory.
Why does it work when I have given pkg1.fib in app.py
Same as above.
What if I want to include ext.py inside mul.py and app.py
You have to package ext and include it as a dependency of app.
As per folder structure, if app.py imports pkg1.fib (relative to app.py) is valid then fib.py must import mul in different way correct? Why it is working for pkg2.mul? Is it actually valid? pkg2/ is not relative to fib.py.
It's not importing relative to app.py. They are both importing from cwd (check the first element of sys.path displayed by using python -m site)
The basic issue is that your app is not package (because it is missing an __init__.py file) which means when you go two levels up to the app directory you aren't in a package any more, and thus the error stating that you are attempting to import beyond the top-level package.
See this section on intra-package references for more info on how you could restructure your package to allow what you are attempting to do. You also check out this section explaining the module search path for a little more context.
Related
I'm new to pytest and python. My project structure is :
projroot/
|-src/
|-a.py
|-test/
|-test_a.py
and test_a.py is:
from ..src import a
def test_a():
pass
Then run pytestunder projroot:
projroot>pytest
And always there is the error info: E ImportError: attempted relative import with no known parent package.
Python verison: 3.8 on windows 10 x64.
I read a lot articles and blogs and official suggestions, but still failed to figure out. Help me, please!
I found I must add __init__.py in test and projroot. __init__.py in src is not necessary.
And pytest could run correctly in 3 folders:
# parent folder of project root
C:\>pytest projroot
# project root
C:\>cd projroot
C:\projroot>pytest
# test folder
C:\projroot>cd test
C:\projroot\test>pytest
By including a setup.py file in root directory and init.py in every folder worked for me and for importing a package inside any file just give that package location from parent directory
setup.py
from setuptools import setup, find_packages
setup(name="PACKAGENAME", packages=find_packages())
projroot/
|-__init__.py
|-setup.py
|-src/
|-a.py
|-__init__.py
|-test/
|-test_a.py
|-__init__.py
from src import a
def test_a():
pass
Reference
https://docs.pytest.org/en/6.2.x/goodpractices.html
Adding __init__.py to project root actually messed up my pytest discovery and also pylint (causing Unable to import 'angles' pylint(import-error)). I had to remove it from project root and only add it inside my tests folder, which solved both my pytest discovery and pylint warning.
With __init__.py in project root, i'll have ModuleNotFoundError: No module named 'angles'
Below is my folder structure with both pytest discovery and pylint (without warning) working :
powerfulpython
├── angles.py
└── tests
├── __init__.py
└── test_angles.py
In angles.py I have a Class Angle which I try to absolute import using from angles import Angle inside test_angles.py.
Before I found out that the issue was having an unnecessary __init__.py in project root. I also tried
editing python.testing.cwd
setting '--rootdir' of pytest
Downgrading Python extension to 2020.9.114305 (following https://github.com/microsoft/vscode-python/issues/14579)
None of them worked.
Before i deleted the __init__.py in project root, when testing pytest, doing python -m pytest worked because the project root is prepended to sys.path automatically so the absolute import was working correctly from root and the import statement can find angles under root. Doing pytest (without python -m) fails because the project root is not prepended to sys.path. The latter can be temporarily solved by adding the root directory manually to $PYTHONPATH. However this is an unsustainable solution as I don't want to add every single new project i want to test as a PYTHONPATH edit to my ~/.zshrc.
Another hack I tried which fixed both pytest and test discovery was inserting sys.path.insert(0,'/Users/hanqi/Documents/Python/powerfulpython') in the code but this is way too ugly. I didn't want to add lines to code just for testing then have to delete them later.
After all these, I deleted __init__.py in root and all's good.
No need for any of the above list of 3 items.
Many solutions I went through deal with finding the tests folder, but that was not my problem. My tests were found correctly in their folders, just that the from angles import Angle was breaking which causes test discovery to fail.
Investigating how does __init__.py in project root break things:
I printed sys.path and realized with it, ~/Documents/Python got prepended to sys.path and without it, ~/Documents/Python/powerfulpython got prepended. The latter path is exactly my project root.
I'm guessing this has something to do with https://docs.pytest.org/en/6.2.x/pythonpath.html (--import-mode=prepend (default)).
I was expecting to see the __init__.py inside tests to cause ~/Documents/Python/powerfulpython to be prepended and the __init__.py in project root cause ~/Documents/Python to be prepended, but strangely i only saw the latter when I left both __init__.py inside.
I also tried deleting the __init__.py inside tests (but let __init__.py in root stay), and see ~Documents/Python/powerfulpython/tests prepended to sys.path. This same line is also prepended if I delete both __init__.py in root and tests, not sure why.
Edit:
The above behaviour is explained at https://docs.pytest.org/en/6.2.x/pythonpath.html
It will then search upwards until it can find the last folder which still contains an __init__.py file in order to find the package root
After further repeating my above experiments running using python -m pytest -s (-s so successful test runs don't swallow my print(sys.path)) instead of just pytest, I confirm that there are 2 actors prepending to sys.path.
python -m (what it prepends depends on from which directory the command is run)
pytest (what it prepends depends on where are __init__.py inserted (explained in pytest docs linked above)
Both have the effect of editing sys.path, helping to make failing imports possible
Jacques Yang above needed 2 __init__.py to put the containing folder of his projroot into sys.path so projroot.test appears as the value of __package__ in his test_a.py and __name__ would appear as projroot.test.test_a, and the 2 dots in this module name (__name__) allow the two dot relative import of from ..src import a. He could also have used an absolute import of from projroot.src import a.
If he didn't add the __init__.py in projroot, the __package__ would become just test and __name__ would be test.test_a, which would only allow an upwards relative step of at most 1 step. (This stepping concept is also explained by BrenBarn at Relative imports for the billionth time).
When there is __init__.py in test but not projroot, that would have been ValueError: attempted relative import beyond top-level package, because __package__ is test and __name__ is test.test_a allowing only 1 step back up but 2 steps back was coded.
The question in the title occurs when there isn't even __init__.py in test, so __package__ is completely blank and __name__ is only the test_a file itself)
I was thinking why did OP need 2 __init__.py while I only needed 1, and 2 actually harms my attempt. Then I realized OP was doing relative import and I was doing absolute import with from angles import Angle. If i followed OP's pattern and did relative import from ..angles import Angle, then I also would have needed to add __init__.py under powerfulpython, otherwise i'll get the same errors.
So why does doing absolute import break with the extra __init__.py in ~/Documents/Python/powerfulpython? Because the __init__.py causes pytest to step up even higher in the hierarchy and prepend ~/Documents/Python to sys.path, but my absolute import needed to have ~/Documents/Python/powerfulpython (this can be resolved with running using python -m pytest as explained in the 2 actors of prepending section above, but feels somewhat unnatural compared to simply pytest)
I have a 30-min demo video on import concepts at https://towardsdatascience.com/taming-the-python-import-system-fbee2bf0a1e4 that could help people understand __package__ and __name__.
I have a project structured like so:
/proj
main.py
config/
__init__.py
setup.py
gui/
__init__.py
app.py
The program is run by calling python main.py.
In main.py I have from gui import App.
In gui/__init__.py I have from .app import App.
Lastly, in app.py I have from ..config import configure; this throws a ValueError that reads attempted relative import beyond top-level package. To resolve this, I can alter the strucure:
/proj
main.py
proj/
config/
__init__.py
setup.py
gui/
__init__.py
app.py
But then I have to change the import statments to include proj (i.e. from proj.gui import App). Is it possible to use relative imports in the former case with the current structure? If not, how can I import what I need from the config subpackage?
As strange as it sounds, you can do :
from config import configure
in app.py, for your former case (current structure), because when you run python main.py, it interprets config as the next level package, and .. won't work.
(Btw, I think there is lots of room for Python imports to improve. The current PEP 328 is confusing at best).
I have an app structured like so:
api/
|_app/
|_ __init__.py
|_conf/
|_resources/
|_ __init__.py
|_controller.py
|_dao/
|_ __init__.py
|_thing_dao.py
|_etc...
I want to use the function get_thing_by_id in thing_dao.py inside of controller.py. In PyCharm I start to type get_thing_by_id and it prompts me to auto-import the function. The problem is it simply does from thing_dao import get_thing_by_id but when I attempt to run the app (Flask) I get ImportError: No module named 'thing_dao'. What I end up having to do is a relative import or start at the module level from app.dao.thing_dao import get_thing_by_id.
I suspect this issue is related to my project structure and less of a PyCharm bug. Is there a way I could structure my project to better hint to PyCharm how to do imports? I've looked through the configuration option for auto-imports and PyCharm and they're quite slim so I suspect that I'm doing something wrong.
I discovered I had my directories marked as a "Source Root". Unmarking them made the imports work properly.
I want to have this structure for my project:
requirements.txt
README.md
.gitignore
project/
__init__.py
project.py
core/
__init__.py
base.py
engines/
__init__.py
engine1.py
engine2.py
utils/
__init__.py
refine_data.py
whatever.py
The application is run from project/project.py. However, I constantly get import errors when using relative or absolute imports.
Both engines need to import from project.core.base, the utils need to import from project.core.base as well, and project.py (the main file ran) needs to be able to import from engines.
Absolute imports don't work:
# engines/engine1.py
from project.core.base import MyBaseClass
which gives the error:
ImportError: No module named project.core.base
But if I try a relative import instead
# engines/engine1.py
from ..core.base import MyBaseClass
I get:
ValueError: Attempted relative import beyond toplevel package
I've seen other projects on Github structured similarly, but this seems to cause all sorts of problems. How do I get this to work?
Take a look at your sys.path. It's likely that the top project directory is in the python path, and it sees your sub-packages (ie. utils, engines, etc.) as separate packages, which is why it's giving you an error that you're trying to import from outside your package when doing relative imports, and absolute imports don't work because it can't find the top project directory because it's not under any of the python paths.
The directory above the top project directory is what needs to be added to the python path.
Ex.
/path/is/here/project/core/...
# Add this to the PYTHONPATH
/path/is/here
Try to use these imports:
engine1.py:
from core import base
refine_data.py:
from core import base
project.py
from engines import engine1
if you use pycharm mark project directory as sources root and then try to run project.py. If you don't use pycharm you can run project.py by going to project directory and running command:
python project.py
stuff/
__init__.py
mylib.py
Foo/
__init__.py
main.py
foo/
__init__.py
script.py
script.py wants to import mylib.py
This is just an example, but really I just want to do a relative import of a module in a parent directory. I've tried various things and get this error...
Attempted relative import beyond toplevel package
I read somewhere that the script from where the program starts shouldn't in the package, and I tried modifying the structure for that like so...
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
__init__.py
script.py
but got same error.
How can I accomplish this? Is this even an adequate approach?
Edit: In Python 2
After fiddling with it a bit more, I realized how to set it up, and for the sake of specificity I won't use foo bar names. My project directory is set up as...
tools/
core/
object_editor/
# files that need to use ntlib.py
editor.py # see example at bottom
__init__.py
state_editor/
# files that need to use ntlib.py
__init__.py
ntlib.py
__init__.py # core is the top level package
LICENSE
state_editor.py # equivalent to main.py for the state editor
object_editor.py # equivalent to main.py for the object editor
A line in object_editor.py looks like...
from core.object_editor import editor
A line in editor.py looks like...
from .. import ntlib
or alternatively
from core import ntlib
The key is that in the example I gave in the question, the "main" script was being run from within the package. Once I moved it out, created a specific package (core), and moved the library I wanted the editors to share (ntlib) into that package, everything was hunky-dory.
though as long "stuff" is not in your python PATH you got no choice than adding the path.
If you know the level of your script.py from stuff you can do for example:
import sys
import os
sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..'))
I'm running Python 3.4.2 on Windows 7 and tore my hair out over this.
When running either of these:
python -m unittest
python -m unittest discover
...I would get the 'Attempted relative import beyond toplevel package' error.
For me, the solution was dropping the ".." in my [test_stock.py].
The line was:
from ..stock import Stock
Changed it to:
from stock import Stock
.. and it works.
Folder structure:
C:\
|
+-- stock_alerter
|
+-- __init__.py
+-- stock.py
|
\-- tests
|
+-- __init__.py
\-- test_stock.py
From the PEP it appears that you cannot use a relative import to import a file that is not packaged.
So you would need to add a __init__.py to stuff and change your imports to something like from .mylib import *
However, the PEP seems to make no allowance to keep mylib packaged up in a module. So you might be required to change how you call your library functions.
Another alternative is to move mylib into a subpackage and import it as from .libpackage import mylib
If you're on Linux or perhaps a similar *nix, you can hack this with symlinks.
stuff/
mylib.py
foo.py // equivalent of main.py in above
foo/
script.py
mylib.py -> ../mylib.py
foo2/
script2.py
mylib.py -> ../mylib.py
This is likely not a good pattern to follow.
In my case I opted for it because I had multiple executables dependent on the same library that needed to be put into separate directories.
Implementation of new executable tests shouldn't require the test writer to have a deep understanding of python imports.
tests/
common/
commonlib.py
test1/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py
test2/
executable1.py
executable2.py
commonlib.py -> ../common/commonlib.py