Given the following (notice test which by mistake shadows built in test package, which I should have named it tests, we will come back to this later in the question)
├── test
│ └── test_request_billing_id.py
└── requets_billing_id.py
when in Python shell, try to run from test import test_request_billing_id
it gives ImportError: cannot import name 'test_request_billing_id', which is trying to import the built-in test package instead of my own test folder.
This can be verified by running this valid import
from test import support
Question 1:
Given Python3 allows implicit namespace packages, which means my test folder is also a package, I guess the built in test package has higher priority than my own test package?
Question 2:
I created __init__.py inside my own test folder as below:
├── test
│ ├── __init__.py
│ └── test_request_billing_id.py
└── requets_billing_id.py
and ran the same import statement again, it worked fine.
from test import test_request_billing_id
Shadowing built-in test be verified by running this invalid import
from test import support gives error: ImportError: cannot import name 'support'
This seems to me that having __init__.py tells python interpreter to make my test shadows the built-in test package.
Could someone please explain this or is this documented anywhere?
Yes, packages with an __init__.py take precedence. This is explained in this section of the PEP:
During import processing, the import machinery will continue to
iterate over each directory in the parent path as it does in Python
3.2. While looking for a module or package named "foo", for each directory in the parent path:
If <directory>/foo/__init__.py is found, a regular package is imported and returned.
If not, but <directory>/foo.{py,pyc,so,pyd} is found, a module is imported and returned. The exact list of extension varies by platform
and whether the -O flag is specified. The list here is representative.
If not, but <directory>/foo is found and is a directory, it is recorded and the scan continues with the next directory in the parent
path.
Otherwise the scan continues with the next directory in the parent path.
If any namespace packages are encountered during the search process, they're "recorded" and the search continues. If a package containing an __init__.py is found later, the "recorded" namespace packages are discarded and the package with __init__.py is imported instead.
This is exactly what happens when your test package has no __init__.py - the test package in the standard library does have an __init__.py, so it takes precedence over your namespace test package.
Related
I'm trying to build a project via the method described in The Hitchhiker's Guide To Python. I'm running into problems with the test structure.
My file structure looks like this:
.
├── __init__.py
├── sample
│ ├── __init__.py
│ └── core.py
├── setup.py
└── tests
├── __init__.py
├── context.py
└── test_core.py
With:
# sample/core.py
class SampleClass:
def got_it(self):
return True
And:
# tests/context.py
import os
import sys
sys.path.insert(0, os.path.abspath(
os.path.join(os.path.dirname(__file__), '..')
))
import sample
And:
# tests/test_core.py
import unittest
from .context import sample
class SampleClassTest(unittest.TestCase):
def test_got_it(self):
# ...
pass
if __name__ == '__main__':
unittest.main()
(Note that I just thru the __init__.py files in the root and tests to see if that helped, but it didn't.)
When I try to run tests/test_core.py with Python 3.7. I get this error:
ImportError: attempted relative import with no known parent package
That happens if I run the test file from outside the tests directory or in it.
If I remote the . and do this in tests/test_core.py:
from context import sample
Everything loads, but I can't access SampleClass.
The things I've tried are:
sc = SampleClass()
NameError: name 'SampleClass' is not defined
sc = sample.SampleClass()
AttributeError: module 'sample' has no attribute 'SampleClass'
sc = sample.core.SampleClass()
AttributeError: module 'sample' has no attribute 'core'
sc = core.SampleClass()
NameError: name 'core' is not defined
I tried on Python 2 as well and had similar problems (with slightly different error methods). I also tried calling a function instead of a class and had similar problems there as well.
Can someone point me in the direction of what I'm doing wrong?
This once caused me some headaches, until I realized that it just depended on the way the test is started. If you use python tests/test_core.py, then you are starting a simple script outside any package. So relative imports are forbidden.
But if (still from the project root folder) you use:
python -m tests.test_core
then you are starting the module test_core from the tests package, and relative imports are allowed.
From that time, I always start my tests as modules and not as scripts. What is even better is that unittest discover knows about packages. So if you are consistent in naming the test folder and scripts with a initial test, you can forget about the number of tests and just use:
python -m unittest discover
and that is enough to run the whole test suite.
Happy testing!
Since you have a proper package complete with setup.py, you're better off using pip install -e . instead of using a sys.path hack.
Minimal setup.py
#setup.py
from setuptools import setup, find_packages
setup(name='sample', version='0.0.1', packages=find_packages(exclude=('tests')))
Then you should import everything from sample packages anywhere you want using the same syntax (i.e. absolute path). To import SampleClass for example:
from sample.core import SampleClass
Also, to answer your question on why you're getting on relative import:
ImportError: attempted relative import with no known parent package
You'd have to avoid invoking the file as a script using python ... because relative imports do not work on scripts. You should invoke the file as a module using python -m .... Here is a very nice explanation that's been viewed a billion times:
I have the script I want to run in the following structure
scripts/
project/
main.py
libraries/
a.py
In main.py I need to import things from a.py. How can I import things in subfolders that are two or more folders above main.py?
The proper way to handle this would be putting everything that needs to know about each other under a shared package, then the individual sub-packages and sub-modules can be accessed through that package. But this will also require moving the application's entrypoint to either the package, or a module that's a sibling of the package in the directory and can import it. If moving the entrypoint is an issue, or something quick and dirty is required for prototyping, Python also implements a couple other methods for affecting where imports search for modules which can be found near the end of the answer.
For the package approach, let's say you have this structure and want to import something between the two modules:
.
├── bar_dir
│ └── bar.py
└── foo_dir
└── foo.py
Currently, the two packages do not know about each other because Python only adds the entrypoint file's parent (either bar_dir or foo_dir depending on which file you run) to the import search path, so we have to tell them about each other in some way, this is done through the top level package they'll both share.
.
└── top_level
├── __init__.py
├── bar_dir
│ ├── __init__.py
│ └── bar.py
└── foo_dir
├── __init__.py
└── foo.py
This is the package layout we need, but to be able to use the package in imports, the top packagehas to be initialized.
If you only need to run the one file, you can do for example python -m top_level.bar_dir.bar but a hidden entry point like that could be confusing to work with.
To avoid that, you can define the package as a runnable module by implementing a __main__.py file inside of it, next to __init__.py, which is ran when doing python -m top_level. The new __main__.py entrypoint would then contain the actual code that runs the app (e.g. the main function) while the other modules would only have definitions.
The __init__.py files are used to mark the directories as proper packages and are ran when the package is imported to initialize its namespace.
With this done the packages now can see each other and can be accessed through either absolute or relative imports, an absolute import would being with the top_level package and use the whole dotted path to the module/package we need to import, e.g. from top_level.bar_dir import bar can be used to import bar.
Packages also allow relative imports which are a special form of a from-style import that begins with one or more dots, where each dot means the import goes up one package - from the foo module from . import module would attempt to import module from the foo_dir package, from .. import module would search for it in the top_level package etc.
One thing to note is that importing a package doesn't initialize the modules under it unless it's an explicit import of that module, for example only importing top_level won't make foo_dir and bar_dir available in its namespace unless they're imported directly through import top_level.foo_dir/top_level.bar_dir or the package's __init__.py added them to the package's namespace through its own import.
If this doesn't work in your structure, an another way is to let Python know where to search for your modules by adding to its module search path, this can be done either at runtime by inserting path strings into the sys.path list, or through the PYTHONPATH environment variable.
Continuing with the above example with a scenario and importing bar from foo, an entry for the bar_dir directory (or the directory above it) can be added to the sys.path list or the aforementioned environment variable. After that import bar (or from bar_dir import bar if the parent was added) can be used to import the module, just as if they were next to each other. The inserted path can also be relative, but that is prone to breakage with a changing cwd.
This question already has answers here:
How to do relative imports in Python?
(18 answers)
Closed 4 years ago.
I have a simple project structure like this:
➜ (venv:evernote) evernote_bear_project git:(master) ✗ tree | grep -v pyc
.
├── README.md
...
(snip)
...
├── manage.py
├── sample
│ ├── EDAMTest.py <==== here is an import that won't work
│ └── enlogo.png
└── util
├── __init__.py
├── files.py <====== This is being imported
└── test_files.py
Now I have a relative import in sample/EDAMTest.py:
from ..util.files import *
When I try to run python sample/EDAMTest.py from project root folder in command line, I get an error saying:
ValueError: attempted relative import beyond top-level package
I know this has been asked many times, but I still don't get it.
Since I'm running the script from the project root, in my understanding Python should be able to "know" that when I try to import from ..util.files import *, that it should go up one directory, no?
EDIT
Thanks for all the answers.
So what I understand from the link above is this:
I was running the module sample/EDAMTest.py directly via python sample/EDAMTest.py, and that meant
that the __name__ of the module was __main__
and now the path (sample/) was my "root" so to speak.
So now Python searches only this path and any path that's below it for modules / packages. Hence the error message attempted relative import _beyond top-level package_, so it cannot go one more level up, because it is at the root already.
Also Python cannot look one level "up", since this syntax from ..util.files import * does not go up a level in directory, but in a list of modules / packages it keeps on the "import search path" (sys.path)?
Is that correct?
sys.path.append() is a tweak, if your directory structure is fixed and there is nothing you can do about it.
Otherwise, you can try rearranging the folders. The easiest is moving util under sample, another option is making both of the folders psrt of a larger package.
Also import * is not ensorsed.
The relative import syntax is for importing other modules from the same package, not from the file system.
Depending on what you want, you could...
Fix the package so that the relative import works
Put __init__.py files in the project root and sample directory and run the script from one level up. This doesn't seem like what you want.
Tell python where to find the package
Set the PYTHONPATH environment variable so that python can find the package.
PYTHONPATH='.':$PYTHONPATH python samples/EDAMTest.py
Install util so that python can find it
Add a setup script and use it to install the util package and avoid setting PYTHONPATH.
"The relative import syntax is for importing other modules from the same package, not from the file system.", This is right as stated by George G.
Put __init__.py in your subfolders, which will make them package.
__init__.py can be an empty file but it is often used to perform
setup needed for the package(import things, load things into path, etc).
However you can import File into your __init__.py to make it
available at the package level:
# in your __init__.py
from util import files
# now import File from util package
from util import files
of if you want to import some specific method or class, you can do
from util.files import some_function
Another thing to do is at the package level make util/modules
available with the __all__ variable. When the interpeter sees
an __all__ variable defined in an __init__.py it imports the
modules listed in the __all__ variable when you do:
from package import *
__all__ is a list containing the names of modules that you want to be
imported with import * so looking at our above example again if we
wanted to import the submodules in util the all variable
in util/init.py would be:
__all__ = ['files', 'test_files']
With the __all__ variable populated like that, when you perform
from util import *
it would import files and test_files.
Since there are so many questions on relative imports, I will make it as short and sweet as possible. And yes, I've read "Relative imports for the billionth time".
I have a project structure like this:
.
├── Makefile
└── src
├── __init__.py
├── model
│ └── train_model.py
└── preprocessing
└── process.py
where I want to be able to, as an example, call make preprocessingor make train which then runs either process.pyor train_model.py with
## Make train
train:
python3 src/model/train_model.py
E.g. modules will always from the top project folder where the Makefile lives.
Now, my problem is that i might have dependencies between different submodules, such as train_model.py and process.py. Specifically, if I try to import processin train_model by using from src.preprocessing import process i get an error ImportError: No module named 'src'. In a similar vein, I've tried from ...preprocessing import process, which gives me another error: SystemError: Parent module '' not loaded, cannot perform relative import.
I use if __name__ == '__main__': at the end of my train_model.py, but I can't seem to figure out, how python uses __name__to find different modules, and if this f**** something up in the process.
Use PYTHONPATH. I would do it this way:
Makefile:
export PYTHONPATH=$(abspath src)
train:
python3 src/model/train_model.py
train_model.py:
from preprocessing import process
Now every import will first look under src. It is not conventional to write from src.preprocessing import process - typically imports are understood to be within some base directory (you wouldn't want to set PYTHONPATH to the directory above src, because it may contain things you don't want to import).
In Python, a namespace package allows you to spread Python code among several projects. This is useful when you want to release related libraries as separate downloads. For example, with the directories Package-1 and Package-2 in PYTHONPATH,
Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py
the end-user can import namespace.module1 and import namespace.module2.
What's the best way to define a namespace package so more than one Python product can define modules in that namespace?
TL;DR:
On Python 3.3 you don't have to do anything, just don't put any __init__.py in your namespace package directories and it will just work. On pre-3.3, choose the pkgutil.extend_path() solution over the pkg_resources.declare_namespace() one, because it's future-proof and already compatible with implicit namespace packages.
Python 3.3 introduces implicit namespace packages, see PEP 420.
This means there are now three types of object that can be created by an import foo:
A module represented by a foo.py file
A regular package, represented by a directory foo containing an __init__.py file
A namespace package, represented by one or more directories foo without any __init__.py files
Packages are modules too, but here I mean "non-package module" when I say "module".
First it scans sys.path for a module or regular package. If it succeeds, it stops searching and creates and initalizes the module or package. If it found no module or regular package, but it found at least one directory, it creates and initializes a namespace package.
Modules and regular packages have __file__ set to the .py file they were created from. Regular and namespace packages have __path__set to the directory or directories they were created from.
When you do import foo.bar, the above search happens first for foo, then if a package was found, the search for bar is done with foo.__path__as the search path instead of sys.path. If foo.bar is found, foo and foo.bar are created and initialized.
So how do regular packages and namespace packages mix? Normally they don't, but the old pkgutil explicit namespace package method has been extended to include implicit namespace packages.
If you have an existing regular package that has an __init__.py like this:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
... the legacy behavior is to add any other regular packages on the searched path to its __path__. But in Python 3.3, it also adds namespace packages.
So you can have the following directory structure:
├── path1
│ └── package
│ ├── __init__.py
│ └── foo.py
├── path2
│ └── package
│ └── bar.py
└── path3
└── package
├── __init__.py
└── baz.py
... and as long as the two __init__.py have the extend_path lines (and path1, path2 and path3 are in your sys.path) import package.foo, import package.bar and import package.baz will all work.
pkg_resources.declare_namespace(__name__) has not been updated to include implicit namespace packages.
There's a standard module, called pkgutil, with which you
can 'append' modules to a given namespace.
With the directory structure you've provided:
Package-1/namespace/__init__.py
Package-1/namespace/module1/__init__.py
Package-2/namespace/__init__.py
Package-2/namespace/module2/__init__.py
You should put those two lines in both Package-1/namespace/__init__.py and Package-2/namespace/__init__.py (*):
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
(* since -unless you state a dependency between them- you don't know which of them will be recognized first - see PEP 420 for more information)
As the documentation says:
This will add to the package's __path__ all subdirectories of directories on sys.path named after the package.
From now on, you should be able to distribute those two packages independently.
This section should be pretty self-explanatory.
In short, put the namespace code in __init__.py, update setup.py to declare a namespace, and you are free to go.
This is an old question, but someone recently commented on my blog that my posting about namespace packages was still relevant, so thought I would link to it here as it provides a practical example of how to make it go:
https://web.archive.org/web/20150425043954/http://cdent.tumblr.com/post/216241761/python-namespace-packages-for-tiddlyweb
That links to this article for the main guts of what's going on:
http://www.siafoo.net/article/77#multiple-distributions-one-virtual-package
The __import__("pkg_resources").declare_namespace(__name__) trick is pretty much drives the management of plugins in TiddlyWeb and thus far seems to be working out.
You have your Python namespace concepts back to front, it is not possible in python to put packages into modules. Packages contain modules not the other way around.
A Python package is simply a folder containing a __init__.py file. A module is any other file in a package (or directly on the PYTHONPATH) that has a .py extension. So in your example you have two packages but no modules defined. If you consider that a package is a file system folder and a module is file then you see why packages contain modules and not the other way around.
So in your example assuming Package-1 and Package-2 are folders on the file system that you have put on the Python path you can have the following:
Package-1/
namespace/
__init__.py
module1.py
Package-2/
namespace/
__init__.py
module2.py
You now have one package namespace with two modules module1 and module2. and unless you have a good reason you should probably put the modules in the folder and have only that on the python path like below:
Package-1/
namespace/
__init__.py
module1.py
module2.py