I've tried to create a new module in Python. It's git link : https://github.com/Sanmitha-Sadhishkumar/strman
After uploading and installing that using pip, I found that I could access that module as
import strman.strman as s
s.func_name
What are the changes to be made to access that as
import strman
strman.func_name
In your __init__.py file, you want to use a relative import.
from .strman import *
You have a strman package (the outer directory) and within it a strman module (the strman.py file). That's a perfectly common pattern. But without the relative import your __init__.py wasn't importing from deep enough in the hierarchy.
More generally, whenever you import from a sibling module within a project, you almost always should use a relative import, because it's explicit and avoids various complications, such as the example in your case.
You have to move the files into the main directory. Your directory will look like this
\myProject
- init.py
- strman.py
- main.py # <-- this would be the file your programming is
Read more about modules in the docs
Related
This question already has answers here:
Attempted relative import with no known parent package [duplicate]
(4 answers)
Closed 1 year ago.
I am having following structure for my project
Object_Detection/
setup.py
setup.cfg
requirement.txt
object_detection/
models
__init__.py #contains from . import models
tests/
# inside tests dir
test_utils_image.py
__init__.py #empty
utils/
# inside utils dir
__init__.py #inside
utils_image_preprocess.py
utils_image.py
utils_tfrecord.py
Now init.py inside utils directory contains the following code.
# inside __init__.py
from . import utils_image_preprocess
from . import utils_image
from . import utils_tfrecord
Running above init.py files gives me an error:
ImportError: attempted relative import with no known parent package
test_utils.py inside tests dir contains the following code
# inside test_utils.py
from object_detection.utils import utils_image
While running test_utils.py I got the following error
ImportError: cannot import name 'utils_image' from 'object_detection.utils'
I have gone through this and this and tried to follow every aspect mentioned there but details about what to put inside init.py is not clear.
This problem seems to be associated with the structuring of init.py in different dir.
I have gone through various and got to know that if we keep even an empty init.py file then things will work out but I am not sure about my understanding.
Please let me know
what I am missing here and whether I am following the correct structure for packaging my code or not?
How to resolve these two errors?
Is this issue related to setting up source in IDE as I am using Vscode and I have also seen this has been mentioned at many places. See here? (But also tried the same code with PyCharm and encountered same error )
If you want to be able to say ...
from object_detection.utils import utils_image
... then clearly the utils directory must be a subdirectory of the object_detection directory and not a sibling directory, i.e. at the same level.
Now for your other error:
ImportError: attempted relative import with no known parent package
You did not really specify under what circumstances you get this error other than saying "Running above init.py files gives me an error:". But how are you "running" these py files or what does that even mean?
If you are executing a script when this occurs (how else would you be getting this error?), the script must be invoked as a module (because scripts cannot have relative imports -- see below) as follows (we will assume that the script you are trying to execute is test_utils_image.py):
First, the parent directory of object_detection, which is Object_Detection, must be in the system path of directories to be searched for finding modules and packages referenced in import statements. In general, this can be accomplished several ways, for instance
The script you are executing is in Object_Detection (the directory of the script is automatically added to the sys.path list of directories to be searched by the interpreter).
Dynamically appending Object_Detection to the sys.path list of directories at runtime by your script.
Appending Object_Detection to the PYTHONPATH environment variable.
Item 1 above would not be applicable for this specific case since the module we are executing by definition is not in the Object_Detection directory.
Note that if your classes will eventually be installed with pip, then site-packages will be the parent directory of object_detection, which is already in sys.path.
Then you can execute your script as:
python -m tests.test_utils_image
If you want to execute this .py file as a script, for example by right-mouse clicking on it is VS Code, then see Relative imports for the billionth time, in particular the section Scripts can't import relative, which says it all -- it cannot work!
To invoke this as a script, just convert the relative imports to absolute imports. In fact, the PEP 8 Style Guide says:
Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured (such as when a directory inside a package ends up on sys.path):
Have you tried to do the following?
inside your utils __init__.py import your modules as follows:
from .utils_image_preprocess import <func1>... <rest of functions/classes you want to import>
from .utils_image import <func1>... <rest of functions/classes you want to import>
from .utils_tfrecord import <func1>... <rest of functions/classes you want to import>
And then in your test file do:
from object_detection.utils.utils_image import *
OR
from object_detection.utils.utils_image import <func1>,...
Also, make sure you don't have any circular dependencies in your modules. for example importing of function from your tests to your util module and vise versa
Python3 has two types of packages
Regular Packages
Namespace Packages
Regular packages contains init.py and namespace packages don't need to have init.py
Regarding your folder structure, it is correct, no change needed. You just need to import like this
from utils import utils_image
without mentioning the objects_detection as objects_detection is just a namespace package.
So it would be usefull when you would refer to the utils module from outside the objects_detection folder.
Here what python docs say about the namespace packages:
A namespace package is a composite of various portions, where each portion contributes a subpackage to the parent package.
I have a repo with Python code whose structure could be boiled down to this:
repo_root\
tool1\
tool1.py
tool1_aux_stuff.py
tool2\
tool2.py
tool2_aux_stuff.py
lib\
lib1\
lib1.py
lib1_aux_stuff.py
lib2\
lib2.py
lib2_aux_stuff.py
The following rules apply to the module usage:
Any tool could use the modules from any library and from its own package, but not from a different tool's one.
Any library could use the modules from any other library, and from its own package. Libraries never access the tool modules.
There must be a way to invoke any tool from any working directory, including those outside repo_root.
The question is: how do I import the lib modules from the tool ones?
I know that if I add __init__.py to each tool and lib directory and to the repo root, then I would be able to use absolute paths from the root, i.e. in tool1.py I could write
import lib.lib1, lib.lib2.lib2_aux_stuff
However, if I execute tool1.py from a random place, e.g.
machine_name: ~/random/place$ python /path/to/repo/tool1/tool1.py
I get the ModuleNotFoundError: No module named 'lib' found error.
I am aware of a workaround which could be implemented using the PYTHONPATH env variable by augmenting it with an absolute path to repo_root and supplying it to the invocation of the tool script, i.e.:
machine_name: ~/random/place$ PYTHONPATH=$PYTHONPATH:/path/to/repo python /path/to/repo/tool1/tool1.py
but I would really prefer something less clunky.
Any ideas how I could do it in a more straightforward way?
Add the path to lib to the scope using sys.path.append('/custom/path/to/modules'). It should then be callable as a module.
You do need to add __init__.py files in any directory that you want to import as if it were a module, otherwise Python doesn't treat them as modules and you'll get another ImportError
#Background and Problem#
I'm trying to build a web-scraper to back up my social media accounts (summer project, I know it's useless).
I'm trying to create a nice class structure, so I've come up with the following structure (I accept critique, I'm pretty new to programming):
\social-media-backup
\Files
__init__.py
File.py
Image.py
Video.py
\SocialMedia
SocialMediaFeed.py
SocialMediaPost.py
\Instagram
__init__.py
\MediaTypes
__init__.py
GraphImage.py
GraphVideo.py
\SearchTypes
__init__.py
InstagramUser.py
\Twitter
\VSCO
(Twitter and VSCO are, for now, empty. Anything without extension and starting with , is a folder. Every file has a class with the same name as the file inside)
#Questions#
Where can I learn Python's packaging system definitively? Any book or web-site recommendation?
How would I import Image into GraphImage? How would I import File into SocialMediaPost?
What do I need to write in __init__.py so as to import SOME_PACKAGE and have it import every module? (e.g.: import Files and have Files.Image and File.Video accessible).
(I know there are a lot of questions. They're written in order of importance)
#To accomplish importing File into SocialMediaPost I've tried:#
from Files.File import File
from ...Files.File import File
import File
from File import File
And almost any combination you can imagine.
I always get an Unable to import, No module named '__main__.Files' or Attempted relative import beyond top-level package.
#Expected behavior#
I'm used to Java's way of doing this, and I cannot figure out how to do this in Python. It seems so messed up. I really miss just adding a package and a folder tree from where the compiler would run.
knocks desk THERE MUST BE A BETTER WAY
##THANKS!!##
Lots of stuff is written about this... however most guides focus on how you do it, not what to do.
How I do it (for small to medium-sized projects):
Do not mess with sys.path.
Have one "project root" directory with your modules / packages underneath (as you already do).
Use absolute imports always, except for "sister" modules.
Always run as module, i.e. using python -m foo.bar
Concrete example using your structure. Assuming that your entry point might be in \SocialMedia\SocialMediaFeed.py, use import statements:
from . import SocialMediaPost (sister module)
import Instagram (child module)
from Files import Image (other module)
and run using: python -m SocialMedia.SocialMediaFeed
By running as module, you always have the project root (social-media-backup) added as "search path". This way absolute imports refering to its subfolders always work. By the way, you can print out the module search path using import sys; print(sys.path).
Some of this might seem overcomplicated, but I found that following the above points pays out greatly as soon as you try to package up stuff for installation (keyword setup.py).
Edit: to answer 3rd question: Have __init__.py contain:
from . import File
from . import Image
from . import Video
I would second the comments by Damian and user2357112 - try to avoid name collisions between folder, file and class/function when creating modules.
You probably won't be able to import anything outside of the current working directory without adding it to your PYTHONPATH. Adding a folder to your PYTHONPATH environment variable means that python will always check that folder when importing modules, so you'll be able to import it from anywhere.
There is a good thread on this already that will put you in the right direction:
Permanently add a directory to PYTHONPATH?
(It's a lot to cover in one post)
I want to ask you something that came to my mind doing some stuff.
I have the following structure:
src
- __init__.py
- class1.py
+ folder2
- __init__.py
- class2.py
I class2.py I want to import class1 to use it. Obviously, I cannot use
from src.class1 import Class1
cause it will produce an error. A workaround that works to me is to define the following in the __init__.py inside folder2:
import sys
sys.path.append('src')
My question is if this option is valid and a good idea to use or maybe there are better solutions.
Another question. Imagine that the project structure is:
src
- __init__.py
- class1.py
+ folder2
- __init__.py
- class2.py
+ errorsFolder
- __init__.py
- errors.py
In class1:
from errorsFolder.errors import Errors
this works fine. But if I try to do in class2 which is at the same level than errorsFolder:
from src.errorsFolder.errors import Errors
It fails (ImportError: No module named src.errorsFolder.errors)
Thank you in advance!
Despite the fact that it's slightly shocking to have to import a "parent" module in your package, your workaround depends on the current directory you're running your application, which is bad.
import sys
sys.path.append('src')
should be
import sys,os
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),os.pardir))
to add the parent directory of the directory of your current module, regardless of the current directory you're running your application from (your module may be imported by several applications, which don't all require to be run in the same directory)
One correct way to solve this is to set the environment variable PYTHONPATH to the path which contains src. Then import src.class1 will always work, regardless of which directory you start in.
from ..class1 import Class1 should work (at least it does here in a similar layout, using python 2.7.x).
As a general rule: messing with sys.path is usually a very bad idea, specially if this allows a same module to be imported from two different paths (which would be the case with your files layout).
Also, you may want to think twice about your current layout. Python is not Java and doesn't require (nor even encourage) a "one class per module" approach. If both classes need to work together, they might be better in a same module, or at least in modules at the same level in the package tree (note that you can use the top-level package's __init__ as a facade to provide direct access to objects defined in submodules / subpackages). NB : I'm not saying that your current layout is necessarily wrong, just that it might not be the simplest one.
There is also a solution that does not involve the use of the environment variable PYTHONPATH.
In src/ create setup.py with these contents:
from setuptools import setup
setup()
Now you can do pip install -e /path/to/src and import to your hearts content.
-e, --editable <path/url> Install a project in editable mode (i.e. setuptools "develop mode") from a local project path or a VCS url.
No, Its not good. Python takes modules in two ways:
Python looks for its modules and packages in $PYTHONPATH.
Refer: https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH
To find out what is included in $PYTHONPATH, run the following code in python (3):
import sys
print(sys.path)
all folder containing init.py are marked as python package.(if they are subdirectories under PYTHONPATH)
By help of these two ways you can full-fill the need to create a python project.
I have this package structure:
widget/
__init__.py
core.py
extension.py
__init__.py is empty.
core.py contains this:
import widget.extension as extension
It works, but it feels like I'm side-stepping the package and just importing it from the global path (i.e. climbing out of it only to look back into it). If I just import extension in core.py it doesn't work. Does this matter? Am I wrong in the first place? Should I instead be pulling both of these modules into __init__.py?
I'm presuming you are using Python 3; in Python 2, import extension would work as Python 2 will first look for a local, relative import before looking for a global, absolute reference.
You have two more options:
from widget import extension
and
from . import extension
The latter imports relative to the current package, which allows you to rename your widget package without having to update all your internal imports. What you use is a matter of style and taste.