If this is my directory tree
temp
├── __init__.py
└── __main__.py
0 directories, 2 files
And I have the following code in __init__.py and in __main__.py
__init__.py
"""Initializes the module"""
CONSTANT = 1
sys.exit("what is happening here")
__main__.py
# from . import CONSTANT
# from temp import CONSTANT
if __name__ == "__main__":
print "This should never run"
I am getting two problems here that I am trying to figure out
On running python . in the temp directory I get the output
This should never run, shouldn't the module be initialized first with the __init__.py file resulting in the abort?
Second how do I go about doing imports in python modules? Neither of the two options I have mentioned above works. I can neither do from . import CONSTANT nor from temp import CONSTANT in the code above. What is the right way to do relative imports?
I am running this on Python 2.7.5, apologies if this has already been asked before.
You should be running it from out of the temp directory. If someDir contains your temp directory, then:
someDir $ python -m temp #someDir/temp/__init__.py is your file.
On running python . in the temp directory I get the output This should never run, shouldn't the module be initialized first with the init.py file resulting in the abort?
If you run it from outside, __init__.py will be called. And sys.exit will be called too.
Second how do I go about doing imports in python modules? Neither of the two options I have mentioned above works. I can neither do from . import CONSTANT nor from temp import CONSTANT in the code above. What is the right way to do relative imports?
You are doing it just fine. Just import sys in your __init__.py file. And fix the spelling of CONSTANT.
Also why do I need the -m flag? Isn't it ok to just do python temp from the parent directory of temp?
You need the -m flag to tell that you are using packages. If you dont use it you wont be able to do relative imports.
When you tell Python to run a directory, Python does not treat the directory as a package. Instead, Python adds that directory to sys.path and runs its __main__.py. __init__.py is not executed, and relative imports will not view the directory as a package.
If you want to run a package's __main__.py and treat it as part of the package, with __init__.py executed and all, go to the directory containing the package and run
python -m packagename
You are running inside temp; this is not considered a package and __init__.py is not loaded. Only if the parent of the current directory is on the module loading path and you explicitly load temp as a module, is __init__.py loaded.
Because temp is not a package you can't use relative imports here. Instead, every Python file inside of the directory is considered a top-level module all by themselves.
You'd have move to the parent of the temp directory, then run:
python -m temp
for Python to import temp as a package and then run the __main__ module in that package.
Related
I am an experienced java enterprise developer but very new to python enterprise development shop. I am currently, struggling to understand why some imports work while others don't.
Some background: Our dev team recently upgraded python from 3.6 to 3.10.5 and following is our package structure
src/
bunch of files (dockerfile, Pipfile, requrirements.txt, shell scripts, etc)
package/
__init__.py
moduleA.py
subpackage1/
__init__.py
moduleX.py
moduleY.py
subpackage2/
__init__.py
moduleZ.py
tests/
__init__.py
test1.py
Now, inside the moduleA.py, I am trying to import subpackage2/moduleZ.py like so
from .subpackage2 import moduleZ
But, I get the error saying
ImportError: attempted relative import with no known parent package
The funny thing is that if I move moduleA.py out of package/ and into src/ then it is able to find everything. I am not sure why is this the case.
I run the moduleA.py by executiong python package/moduleA.py.
Now, I read that maybe there is a problem becasue you have you give a -m parameter if running a module as a script (or something on those lines). But, if I do that, I get the following error:
ModuleNotFoundError: No module names 'package/moduleA.py'
I even try running package1/moduleA and remove the .py, but that does not work either. I can understand why as I technically never installed it ?
All of this happened because apparently, the tests broke and to make it work they added relative imports. They changed the import from "from subpackage2 import moduleZ" to "from .subpackage2 import moduleZ" and the tests started working, but the app started failing.
Any understanding I can get would be much appreciated.
The -m parameter is used with the import name, not the path. So you'd use python3 -m package.moduleA (with . instead of /, and no .py), not python3 -m package/moduleA.py.
That said, it only works if package.moduleA is locatable from one of the roots in sys.path. Shy of installing the package, the simplest way to make it work is to ensure your working directory is src (so package exists in the working directory):
$ cd path/to/src
$ python3 -m package.moduleA
and, with your existing setup, if moduleA.py includes a from .subpackage2 import moduleZ, the import should work; Python knows package.moduleA is a module within package, so it can use a relative import to look for a sibling package to moduleA named subpackage2, and then inside it it can find moduleZ.
Obviously, this is brittle (it only works if you cd to the src root directory before running Python, or hack the path to src in PYTHONPATH, which is terrible hack if the code ever has to be run by anyone else); ideally you make this an installable package, install it (in global site-packages, user site-packages, or within a virtual environment created with the built-in venv module or the third-party virtualenv module), and then your working directory no longer matters (since the site-packages will be part of your sys.path automatically). For simple testing, as long as the working directory is correct (not sure what it was for you), and you use -m correctly (you were using it incorrectly), relative imports will work, but it's not the long term solution.
So first of all - the root importing directory is the directory from which you're running the main script.
This directory by default is the root for all imports from all scripts.
So if you're executing script from directory src you can do such imports:
from package.moduleA import *
from package.subpackage1.moduleX import *
But now in files moduleA and moduleX you need to make imports based on root folder. If you want to import something from module moduleY inside moduleX you need to do:
# this is inside moduleX
from package.subpackage1.moduleY import *
This is because python is looking for modules in specific locations.
First location is your root directory - directory from which you execute your main script.
Second location is directory with modules installed by PIP.
You can check all directories using following:
import sys
for p in sys.path:
print(p)
Now to solve your problem there are couple solutions.
The fast one but IMHO not the best one is to add all paths with submodules to sys.path - list variable with all directories where python is looking for modules.
new_path = "/path/to/application/app/folder/src/package/subpackage1"
if new_path not in sys.path:
sys.path.append(new_path)
Another solution is to use full path for imports in all package modules:
from package.subpackage1.moduleX import *
I think in your case it will be the correct solution.
You can also combine 2 solutions.
First add folders with subpackages to sys.path and use subpackage folders as a root folders for imports. But it's good solution only if you have complex submodule structure. And it's not the best solution if in future you will need to deploy your package as a wheel or share between multiple projects.
I'm trying to organize my scripts in a way I can use absolute imports, without appending to sys.path, making it as easy to use in different computers as possible without problems. I have read and I understand absolute paths are the most appropriate way to do this.
My folder structure looks something like:
main/
__init__.py
tools/
__init__.py
script1.py
base/
__init__.py
script2.py
I have been trying to do from main.tools.script1 import Foo in file main/base/script2.py and I still get the error ImportError: No module named main.tools.script1.
If is print sys.path, the first item is main/base/.
If I run python on a terminal from main, I can import tools.script1, but cannot write main.tools.script1.
I have read other posts, but this is still really not clear for me.
MyNiceProgram-1.0/
setup.py
README
LICENSE
bin/
myniceprogram.py # main entry point
myniceprogram/
__init__.py
tools/
__init__.py
script1.py
base
__init__.py
script2.py
Then you use:
from myniceprogram.tools import script1
from myniceprogram.base import script2
Don't
Name your main package main - it should be treated as a global package so the name should be descriptive. If you want to have a package named main put it inside myniceprogram.main
Execute scripts inside your package. Your main script should be outside the package for absolute import to work. Otherwise you can end up having two copies of the same script, imported absolutely and relatively.
I have the following directory structure:
app/
bin/
script1.py
script2.py
lib/
module1/
__init__.py
module1a.py
module1b.py
__init__.py
module2.py
Dockerfile
My problem is that I want to execute script1.py and script2.py, but inside those scripts, I want to import the modules in lib/.
I run my scripts from the root app/ directory (i.e. adjacent to Dockerfile) by simply executing python bin/script1.py. When I import modules into my scripts using from lib.module1 import module1a, I get ImportError: No module named lib.module1. When I try to import using relative imports, such as from ..lib.module1 import module1a, I get ValueError: Attempted relative import in non-package.
When I simply fire up the interpreter and run import lib.module1 or something, I have no issues.
How can I get this to work?
In general, you need __init__.py under app and bin, then you can do a relative import, but that expects a package
If you would structure your python code as python package (egg/wheel) then you could also define an entry point, that would become your /bin/ file post install.
here is an example of a package - https://python-packaging.readthedocs.io/en/latest/minimal.html
and this blog explains entry points quite well - https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
if so, that way you could just do python setup.py install on your package and then have those entry points available within your PATH, as part of that you would start to structure your code in a way that would not create import issues.
You can add to the Python path at runtime in script1.py:
import sys
sys.path.insert(0, '/path/to/your/app/')
import lib.module1.module1a
you have to add current dir to python path.
use export in terminal
or sys.path.insert in your python script both are ok
I have __init__.py in a folder called test_module. in the __init__.py i have below code. however when i try to execute from parent folder of test_module with following command python test_module i get following error can't find '__main__' module in 'test_module. is this not possible? or will i have to run python test_module/__init__.py?
def main():
print('test')
if __name__ == '__main__':
main()
The __init__.py module is executed when the package is imported. The purpose of __init__.py files per the documentation is as follows:
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.
In order for a Python package to be directly executed, it needs to have an entry point, designated by a module within the package named __main__.py. Thus the error can't find '__main__' module in 'test_module': You have attempted to directly execute the package, but Python cannot locate an entry point to begin executing top-level code.
Consider the following package structure:
test_module/
__init__.py
__main__.py
Where __init__.py contains the following:
print("Running: __init__.py")
Where __main__.py contains the following:
print("Running: __main__.py")
When we execute the test_module package with the command python test_module, we get the following output:
> python test_module
Running: __main__.py
However, if we enter the Python shell and import test_module, the output is as follows:
>>> import test_module
Running: __init__.py
Thus in order to get the behavior you want when attempting to directly execute test_module, simply create a new __main__.py file within test_module and transfer the code from __init__.py to the new __main__.py.
There're a lot of threads on importing modules from sibling directories, and majority recommends to either simply add init.py to source tree, or modify sys.path from inside those init files.
Suppose I have following project structure:
project_root/
__init__.py
wrappers/
__init__.py
wrapper1.py
wrapper2.py
samples/
__init__.py
sample1.py
sample2.py
All init.py files contain code which inserts absolute path to project_root/ directory into the sys.path. I get "No module names x", no matter how I'm trying to import wrapperX modules into sampleX. And when I try to print sys.path from sampleX, it appears that it does not contain path to project_root.
So how do I use init.py correctly to set up project environment variables?
Do not run sampleX.py directly, execute as module instead:
# (in project root directory)
python -m samples.sample1
This way you do not need to fiddle with sys.path at all (which is generally discouraged). It also makes it much easier to use the samples/ package as a library later on.
Oh, and init.py is not run because it only gets run/imported (which is more or less the same thing) if you import the samples package, not if you run an individual file as script.