I am creating module (with submodules). Lets call it lib. I am trying to make it work as following:
I am able to run it (there is lib.__main__). It uses lib.utils inside.
When executed part of its job is to load other file/module passed by user. Currently it does it by importlib.import_module( name ).
This loaded module also needs to use lib.utils.
I am having following choice:
In loaded module use import utils instead of import lib.utils. I find it somehow misleading and would like to aviod this.
Run module in any external way, even using file with only import lib.__main__ inside.
Only other thing I have tought of was doing sys.path.append(os.getcwd()). Not only it seems very dirty, but also makes log.utils module to load twice.
Is there anything that would allow me to execute lib.__main__, but require using import lib.utils in loaded module?
From the docs
If the script name refers to a directory or zipfile, the script name is added to the start of sys.path and the main.py file in that location is executed as the main module.
In your case, if you run python lib mymodule, lib is added to sys.path and __main__.py is executed. lib is not a package, its simply a directory in sys.path. __main__.py is not in a package and so package-relative imports don't work.
Since lib is in sys.path, its top level .py files can be imported directly and any subdirectories with __init__.py are importable packages. So, both __main__.py and mymodule could do import utils and get the same thing.
Now it gets confusing. Because you are sitting in lib's parent directory and because there is a lib.__init__.py, lib.utils is also valid. Its only that way because of your current directory (or maybe you added the directory to PYTHONPATH or something). So, you've got two different modules as far as python is concerned because you got there on two different paths.
The solution is to delete lib/__init__.py. lib shouldn't be package. Then you have the question of what to do with the modules like lib/utils.py. Normally, one would create a package directory and move the scripts there so that you get namespace encapsulation. Supposing you call that directory mypackages, then __main__.py and mymodule.py could both import mypackages.utils.
Related
This question already has answers here:
Attempted relative import with no known parent package [duplicate]
(4 answers)
Closed 1 year ago.
I am having following structure for my project
Object_Detection/
setup.py
setup.cfg
requirement.txt
object_detection/
models
__init__.py #contains from . import models
tests/
# inside tests dir
test_utils_image.py
__init__.py #empty
utils/
# inside utils dir
__init__.py #inside
utils_image_preprocess.py
utils_image.py
utils_tfrecord.py
Now init.py inside utils directory contains the following code.
# inside __init__.py
from . import utils_image_preprocess
from . import utils_image
from . import utils_tfrecord
Running above init.py files gives me an error:
ImportError: attempted relative import with no known parent package
test_utils.py inside tests dir contains the following code
# inside test_utils.py
from object_detection.utils import utils_image
While running test_utils.py I got the following error
ImportError: cannot import name 'utils_image' from 'object_detection.utils'
I have gone through this and this and tried to follow every aspect mentioned there but details about what to put inside init.py is not clear.
This problem seems to be associated with the structuring of init.py in different dir.
I have gone through various and got to know that if we keep even an empty init.py file then things will work out but I am not sure about my understanding.
Please let me know
what I am missing here and whether I am following the correct structure for packaging my code or not?
How to resolve these two errors?
Is this issue related to setting up source in IDE as I am using Vscode and I have also seen this has been mentioned at many places. See here? (But also tried the same code with PyCharm and encountered same error )
If you want to be able to say ...
from object_detection.utils import utils_image
... then clearly the utils directory must be a subdirectory of the object_detection directory and not a sibling directory, i.e. at the same level.
Now for your other error:
ImportError: attempted relative import with no known parent package
You did not really specify under what circumstances you get this error other than saying "Running above init.py files gives me an error:". But how are you "running" these py files or what does that even mean?
If you are executing a script when this occurs (how else would you be getting this error?), the script must be invoked as a module (because scripts cannot have relative imports -- see below) as follows (we will assume that the script you are trying to execute is test_utils_image.py):
First, the parent directory of object_detection, which is Object_Detection, must be in the system path of directories to be searched for finding modules and packages referenced in import statements. In general, this can be accomplished several ways, for instance
The script you are executing is in Object_Detection (the directory of the script is automatically added to the sys.path list of directories to be searched by the interpreter).
Dynamically appending Object_Detection to the sys.path list of directories at runtime by your script.
Appending Object_Detection to the PYTHONPATH environment variable.
Item 1 above would not be applicable for this specific case since the module we are executing by definition is not in the Object_Detection directory.
Note that if your classes will eventually be installed with pip, then site-packages will be the parent directory of object_detection, which is already in sys.path.
Then you can execute your script as:
python -m tests.test_utils_image
If you want to execute this .py file as a script, for example by right-mouse clicking on it is VS Code, then see Relative imports for the billionth time, in particular the section Scripts can't import relative, which says it all -- it cannot work!
To invoke this as a script, just convert the relative imports to absolute imports. In fact, the PEP 8 Style Guide says:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path):
Have you tried to do the following?
inside your utils __init__.py import your modules as follows:
from .utils_image_preprocess import <func1>... <rest of functions/classes you want to import>
from .utils_image import <func1>... <rest of functions/classes you want to import>
from .utils_tfrecord import <func1>... <rest of functions/classes you want to import>
And then in your test file do:
from object_detection.utils.utils_image import *
OR
from object_detection.utils.utils_image import <func1>,...
Also, make sure you don't have any circular dependencies in your modules. for example importing of function from your tests to your util module and vise versa
Python3 has two types of packages
Regular Packages
Namespace Packages
Regular packages contains init.py and namespace packages don't need to have init.py
Regarding your folder structure, it is correct, no change needed. You just need to import like this
from utils import utils_image
without mentioning the objects_detection as objects_detection is just a namespace package.
So it would be usefull when you would refer to the utils module from outside the objects_detection folder.
Here what python docs say about the namespace packages:
A namespace package is a composite of various portions, where each portion contributes a subpackage to the parent package.
I have a repo with Python code whose structure could be boiled down to this:
repo_root\
tool1\
tool1.py
tool1_aux_stuff.py
tool2\
tool2.py
tool2_aux_stuff.py
lib\
lib1\
lib1.py
lib1_aux_stuff.py
lib2\
lib2.py
lib2_aux_stuff.py
The following rules apply to the module usage:
Any tool could use the modules from any library and from its own package, but not from a different tool's one.
Any library could use the modules from any other library, and from its own package. Libraries never access the tool modules.
There must be a way to invoke any tool from any working directory, including those outside repo_root.
The question is: how do I import the lib modules from the tool ones?
I know that if I add __init__.py to each tool and lib directory and to the repo root, then I would be able to use absolute paths from the root, i.e. in tool1.py I could write
import lib.lib1, lib.lib2.lib2_aux_stuff
However, if I execute tool1.py from a random place, e.g.
machine_name: ~/random/place$ python /path/to/repo/tool1/tool1.py
I get the ModuleNotFoundError: No module named 'lib' found error.
I am aware of a workaround which could be implemented using the PYTHONPATH env variable by augmenting it with an absolute path to repo_root and supplying it to the invocation of the tool script, i.e.:
machine_name: ~/random/place$ PYTHONPATH=$PYTHONPATH:/path/to/repo python /path/to/repo/tool1/tool1.py
but I would really prefer something less clunky.
Any ideas how I could do it in a more straightforward way?
Add the path to lib to the scope using sys.path.append('/custom/path/to/modules'). It should then be callable as a module.
You do need to add __init__.py files in any directory that you want to import as if it were a module, otherwise Python doesn't treat them as modules and you'll get another ImportError
Assume I have the following files,
pkg/
pkg/__init__.py
pkg/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run main.py, it says "Package's string module imported".
This makes sense and it works as per this statement in this link:
"it will first look in the package's directory"
Assume I modified the file structure slightly (added a core directory):
pkg/
pkg/__init__.py
plg/core/__init__.py
pkg/core/main.py # import string
pkg/string.py # print("Package's string module imported")
Now, if I run python core/main.py, it loads the built-in string module.
In the second case too, if it has to comply with the statement "it will first look in the package's directory" shouldn't it load the local string.py because pkg is the "package directory"?
My sense of the term "package directory" is specifically the root folder of a collection of folders with __init__.py. So in this case, pkg is the "package directory". It is applicable to main.py and also files in sub- directories like core/main.py because it is part of this "package".
Is this technically correct?
PS: What follows after # in the code snippet is the actual content of the file (with no leading spaces).
Packages are directories with a __init__.py file, yes, and are loaded as a module when found on the module search path. So pkg is only a package that you can import and treat as a package if the parent directory is on the module search path.
But by running the pkg/core/main.py file as a script, Python added the pkg/core directory to the module search path, not the parent directory of pkg. You do have a __init__.py file on your module search path now, but that's not what defines a package. You merely have a __main__ module, there is no package relationship to anything else, and you can't rely on implicit relative imports.
You have three options:
Do not run files inside packages as scripts. Put a script file outside of your package, and have that import your package as needed. You could put it next to the pkg directory, or make sure the pkg directory is first installed into a directory already on the module search path, or by having your script calculate the right path to add to sys.path.
Use the -m command line switch to run a module as if it is a script. If you use python -m pkg.core Python will look for a __main__.py file and run that as a script. The -m switch will add the current working directory to your module search path, so you can use that command when you are in the right working directory and everything will work. Or have your package installed in a directory already on the module search path.
Have your script add the right directory to the module search path (based on os.path.absolute(__file__) to get a path to the current file). Take into account that your script is always named __main__, and importing pkg.core.main would add a second, independent module object; you'd have two separate namespaces.
I also strongly advice against using implicit relative imports. You can easily mask top-level modules and packages by adding a nested package or module with the same name. pkg/time.py would be found before the standard-library time module if you tried to use import time inside the pkg package. Instead, use the Python 3 model of explicit relative module references; add from __future__ import absolute_import to all your files, and then use from . import <name> to be explicit as to where your module is being imported from.
I have a custom module that I am trying to read from a folder under a hierarchy:
> project-source
/tests
/provider
my_provider.py
settings_mock.py
__init__.py
I am trying to call, from my_provider.py
import tests.settings_mock as settings
Example from command line:
project-source> python tests/provider/my_provider.py
Error:
... ImportError: No module named settings_mock
I keep getting No module named settings_mock as error. I have already exported project_source path to PYTHONPATH. I have made tests into a package by creating a __init__.py file in its root, but no change in the error then.
I can print the settings_mock.py attributes when cd'ing project source
>>> import tests.settings_mock as settings
>>> print settings.storage_provider
correct storage provider value
Is anyone able to point out my mistake here? Thanks!
You only have one small mistake. To use subfolders, you need __init__.py, not init.py as you stated in the question. The difference is that __init__ is a builtin function of python, whereas init is not. Having this file in each subfolder tells the pyhon interpreter that the folder is a "package" that needs to be initialized.
UPDATED: It should be noted that python usually runs from the current directory that the script is located. If your executable main script is my_provider.py, then it's not going to know what to import, since the main script is located in a lower directory than the object it is trying to import. Think of it as a hierarchy. Scripts can only import things that are beneath them. Try separating out the executable from everything else in that file, if there are things that settings_mock needs to import.
Sorry for asking my own question 2nd time, but i am totally stuck in import file in python.
I have a directory structure below:
|--test/foo.py
|--library #This is my PYTHONPATH
|--|--script1.py
|--|--library_1
|--|--|--script2.py
|--|--library_2
|--|--library_3
I am accessing library/library_1/script2.py from test/foo.py.
Here i am confused about what is the better approach. Generally all library folders or utility functions should be added to pythonpath.
This is a folder structure i am maintaining to differentiate utility functions and test scripts.
I tried putting __init__.py in library and library1 & then imported like from library1 import script2, but getting error as No module named script.
I have tried appending that path to system path as well.
Working: if i add another pythonpath like path/to/library/libray_1/. So should i do this for all folders which are inside library folder to make it work ?
Here's what you need to do:
|--test/foo.py
|--library #This is my PYTHONPATH
|--__init__.py
|--|--script1.py
|--|--library_1
|--|--|--__init__.py
|--|--|--script2.py
|--|--library_2
|--|--|--__init__.py
|--|--library_3
|--|--|--__init__.py
And inside the first __init__.py below library you need to do:
import library1
import library2
import script
Then, if library is your python path, you can do this within test/foo.py with no errors:
import library
library.library1.bar()
library.script.foo()