I have a folder structure like this, each time I attempt to use relative import, error raises
├── graphics
│ ├── __init__.py
│ ├── A
│ │ ├── __init__.py
│ │ ├── grok.py
│ │ └── spam.py
└── B
├── __init__.py
└── bar.py
spam.py/
def func():
pass
bar.py/
def f():
pass
All these codes are tested in grok.py:
from . import spam
# ImportError: cannot import name 'spam'
from .spam import func
# ModuleNotFoundError: No module named '__main__.spam'; '__main__'
is not a package
from ..B import bar
# ValueError: attempted relative import beyond top-level package
None of the codes below cause any error:
from graphics.A import spam
from graphics.A.spam import func
from graphics.B import bar
from graphics.B.bar import f
I assume that when you say "tested in grok.py" that you are running it like these:
python3 graphics/A/grok.py
python3 A/grok.py
python3 grok.py
From the Python documentation on Packages and Intra-Package References, there is a note there that says:
Note that relative imports are based on the name of the current
module. Since the name of the main module is always "__main__",
modules intended for use as the main module of a Python application
must always use absolute imports.
When you run grok.py, it gets treated as the main module, and imports will only work if you used absolute imports (assuming you made no changes to sys.path, we'll get to that later). You can test that by putting print(__name__) at the start of grok.py, which will print out "__main__".
Your relative imports are actually going to work if you had a separate python file (ex. main.py) under the graphics package that calls your grok module:
├── graphics
│ ├── __init__.py
| ├── main.py <<---- add this
│ ├── A
│ ├── B
In main.py, let's just import the grok module:
from A import grok
In grok.py, let's test the relative imports:
from . import spam
spam.spam_func()
from .spam import spam_func
spam_func()
from B import bar
bar.bar_func()
In spam.py:
def spam_func():
print("spammy")
In bar.py:
def bar_func():
print("barry")
When you run main.py:
graphics$ python3 main.py
spammy
spammy
barry
You won't get any of the previous errors. The relative imports work. Notice that to import from B, I used from B instead of from ..B. This is because the import paths are from the point of view of main.py. You can test this by adding this at the top of main.py:
import sys
print(sys.path)
# prints a list, ['/path/to/graphics/',...]
If you did from ..B that means /path/to/graphics/../ which of course does not have the B module (hence, you'll get the "attempted relative import beyond top-level package" error)
Now let's say you don't want to use a separate main.py and you want to run grok.py directly. What you can do is manually add the path to the graphics package to sys.path. Then you can do from A and from B in grok.py.
import sys
sys.path.append("/full/path/to/graphics/")
from A import spam
spam.spam_func()
from B import bar
bar.bar_func()
If you want to go about "hacking" the sys.path, I suggest reading more on sys.path and checking other related posts that discuss ways of adding paths to sys.path.
Related
I have the following folder structure,
└── project
├── A
│ ├── main.py
│ └── __init__.py
└── B
├── __init__.py
├── C
├── __init__.py
└── module_x.py
I want to import all the methods in module_x.py into main.py. I have tried
from ..B.C.module_x import *
But I get the following error:
ImportError: attempted relative import with no known parent package
I wonder what am I doing wrong? How can this be done using relative import?
from project.B.C import foo
from ...b.c.module_x import foo
However, relative imports are only meant to work within one package. If project is a package, then you can use relative imports here. If project is not a package, you cannot.
However, if you're running a script in / and doing something like import project.A.b.foo, then that relative import will succeed because project is now a package. In that case, the following two would be equivalent:
from ...B.C import foo
from project.B.C import foo
You must use the -m switch to run python modules as scripts:\
$ cd project
$ python -m A.main # note no .py
This tells python that A.main is a module - python will also scan the current working dir (project) and detect package B - this will make your imports work correctly.
I have this code structure in python3:
- datalake
__init__.py
utils
__init__.py
utils.py
lambdas
__init__.py
my-lambdas.py
- tests
__init__.py
demo.py
All init__.py files are empty.
My problem is how I can import datalake module from tests/demo.py?
I tried from datalake.utils import utils in demo.py but when I run python tests/demo.py from command line, I get this error ModuleNotFoundError: No module named 'datalake'.
If I use this code:
from ..datalake.utils import utils
I will get error ValueError: attempted relative import beyond top-level package.
I also tried to import the module utils from my-lambda.py file which also failed. The code in my-lambda.py is from datalake.utils import utils but I get ModuleNotFoundError: No module named 'datalake' error when run python datalake/lambda/my-lambda.py from command line.
How can I import the module?
When you run a command like python tests/demo.py, the folder you are in does not get added to the PYTHONPATH, the script folder does. So a top-level import like import datalake will fail. To get around this you can run your tests as a module:
Python 2:
python -m tests/demo
Python 3:
python -m tests.demo
and any datalake imports in demo.py will work.
It sounds like what you really want to do is have a folder with tests separate to your main application and run them. For this I recommend py.test, for your case you can read Tests Outside Application Code for how to do it. TL;DR is run your tests from your top level project folder with python -m py.test and it will work.
First of all, my-lambdas.py is not importable with the import statement as hyphens are not valid in Python identifiers. Try to follow PEP-8's naming conventions, such as mylambdas.py.
Otherwise the package structure looks good, and it should be importable as long as you are at the level above datalake/, e.g., if you were in the directory myproject/ below:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils
│ │ ├── __init__.py
│ │ └── utils.py
│ └── lambdas
│ ├── __init__.py
│ └── mylambdas.py
└── tests
├── __init__.py
└── demo.py
Then this should work:
~/myproject$ python -c 'from datalake import utils'
Otherwise, setting the environment variable PYTHONPATH to the path above datalake/ or modifying sys.path are both ways of changing where Python can import from. See the official tutorial on modules for more information.
Also some general advice: I've found it useful to stick with simple modules rather than packages (directories) until there is a need to expand. Then you can change foo.py into a foo/ directory with an __init__.py file and import foo will work as before, although you may need to add some imports to the __init__.py to maintain API compatibility. This would leave you with a simpler structure:
myproject
├── datalake
│ ├── __init__.py
│ ├── utils.py
│ └── lambdas.py
└── tests
├── __init__.py
└── demo.py
You can add the module directory into your sys.path:
import sys
sys.path.append("your/own/modules/folder") # like sys.path.append("../tests")
but this is a one-shot method, which is just valid at this time, the added path is not permanent, it will be eliminated after the code completed execution.
One of the ways to import the file directly instead of using from, like import util
you can try run :
python -m datalake.lambda.my-lambda
follow: https://docs.python.org/3.7/using/cmdline.html#cmdoption-m
My project has the following structure:
.
└── mylib
├── __init__.py
├── fun1.py
├── fun2.py
└── test.py
Suppose test.py imports functions from modules fun1.py and fun2.py, so it contains
from fun1 import funA
from fun2 import funB
However, when I try to import test.py outside my project directory I get the following error:
ModuleNotFoundError: No module named 'fun1'
I can fix this by specifying the whole path to fun1.py and fun2.py in my imports.
from mylib.fun1 import funA
from mylib.fun2 import funB
But again, suppose I don't have only to import funA() and funB + I have a whole bunch of modules other than test.py that also imports functions from each other. So it would take a huge amount of time specify the path for every import (more than 200 imports made like this).
Is there a cleaner way to make these imports besides specifying the whole path for all of them?
I tried making these imports on my __init__.py, but due to my inexperience, I'm still unable to make it work.
FILES
fun1.py
def funA():
return True
fun2.py
from fun1 import funA
def funB():
return True
test.py
from fun1 import funA
from fun2 import funB
If I understand correctly, you want to have a file outside mylib that looks like
from mylib import funA
First, it seems that you need the relative imports everywhere inside mylib. Second, you need to use your __init__.py to import everything from the local directory and make it available in the directory above. I would change (minimally) your files as follows, adding some more files to test the imports.
Directory structure:
.
├── mylib
│ ├── fun1.py
│ ├── fun2.py
│ ├── __init__.py
│ └── test.py
├── scriptA.py
├── scriptB.py
└── script_test.py
fun2.py
from .fun1 import funA
def funB():
return True
__init__.py
from .fun1 import funA
from .fun2 import funB
test.py
from . import funA, funB
scriptA.py
from mylib import funA
scriptB.py
from mylib import funB
script_test.py
from mylib import test
test.funA()
You can now use script A, B, or _test as your needs require. This pattern extends to deeper directory structures as you continue to use the relative import with . in the __init__.py at each directory level.
You need to import modules/functions where you need them. There is no magic way for python to know that you have imported something. There are different ways to import needed modules/functions (https://docs.python.org/3/reference/import.html), but what I've found most useful is to import modules and then later use module name with dot syntax to call functions, so something like:
from . import func1
from . import func2
def test():
func1.funA()
func2.funB()
just an example. But you can see a general idea. (from . import func1 will import func1 from current directory/module)
Based on my understanding, you are trying to import modules that are in some other project location. Try this and replace path with the location from where you want to import modules.
sys.path.insert(0, path)
Please let me know if I misunderstood your question.
I have the following package structure:
.
└── package
├── __init__.py
├── sub_package_1
│ ├── __init__.py
│ └── main_module.py
├── sub_package_2
│ ├── __init__.py
│ └── some_module.py
└── sub_package_3
├── __init__.py
└── some_module.py
In package/sub_package_1/main_module.py I want to use both package/sub_package_2/some_module.py and package/sub_package_3/some_module.py. For this I want to use intra-package reference. I know that I can use from ..sub_package_1 import some-module but because of the similar name I want to use dotted syntax such as sub_package_1.some_module.
Using from .. import sub_package_2 I obviously cannot access sub_package_2.some_module because sub_package_2 is a package. However I found out that using
from .. import sub_package_2
from ..sub_package_2 import some_module
I can access sub_package_2.some_module. Apparently the 2nd import adds some_module to sub_package_2 (checking dir(sub_package_2)).
My questions are:
Is there a way to use a single import instead of the two above?
Why does (in general) import package followed by from package import module add module to package? What is Python actually doing here?
1.
In the file __init__.py of sub_package_2 you write
from . import some_module
And in main_module.py you can must write
from .. import sub_package_2
And the code sub_package_2.some_module should work now
2.
"How import in python work" you can read more here Importing Python Modules
from .. import sub_package_2 creates a reference to sub_package_2 in the current namespace. Package sub_package_2 is like a module now and is defined in the file __init__.py. If you wrote nothing in __init__.py, sub_package_2 won't know some_modue
from ..sub_package_2 import some_module create a reference to the module some_module of the package sub_package_2 with the name some_module. It's something like some_module = sub_package_2.some_module. You see: there are a reference to some_module in sub_package_2 too. And now sub_package_2 knows the module some_module
Important: You can use sub_package_2.some_module but only some_module will work too. They are identical after your 2 imports.
And if you write in the __init__.py:
from . import some_module
some_module belongs to sub_package_2 automatically
For similar module names you can use as
from ..sub_package_1 import some_module as some_module_1
from ..sub_package_2 import some_module as some_module_2
from ..sub_package_3 import some_module as some_module_3
I'm trying to build a Python program with the "structure" shown at the end.
The problem is that actions should be able to be executed as scripts as well (I've already included a main in them). When I try to execute DummyAction.py imports keep complaining they can't find misc.
How can I use DummyAction.py as a script and still use the functions in utils.py?
DummyAction.py contains a class called DummyActionClass, and the same for DummyTrigger.py. In utils.py there are several functions that both actions and triggers use and MMD.py contains the main.
/MMD
├── __init__.py
├── MMD.py
├── /actions
│ ├── __init__.py
│ ├── DummyAction.py
├── /misc
│ ├── __init__.py
│ ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
The import in DummyAction.py and DummyTrigger.py is:
from misc import utils
And the error is:
File "DDM/actions/DummyAction.py", line 11, in <module>
from misc import utils
ImportError: No module named misc
Seen the updated question, I think the problem is that you should do the import including the root of your dependecies tree: MMD.
So they should all look like:
from MMD.misc import utils
And also you need to call python with the -m option:
python -m MMD.actions.DummyAction
Edit: You said that MMD.py contains the main but it can't be your executable, and that's because is a module (is inside a directory with an __init__.py file). MMD is like your library so you need the executable to be outside and use such library.
You can find [here] some guidelines on how to organize your project.
If you can change you project structure I'll suggest to do it like this:
MMD/
├── runner.py
└── mmd
├── __init__.py
├── main.py
├── /actions
│ ├── __init__.py
│ ├── DummyAction.py
├── /misc
│ ├── __init__.py
│ ├── utils.py
└── /triggers
├── DummyTrigger.py
└── __init__.py
Then in any file inside the mmd directory every import should start with mmd, for example:
from mmd.misc import utils
from mmd.actions import DummyActions
And you put your main code that now is inside MMD.py inside a Main class in main.py, with something like:
# main.py
from mmd.misc import utils
class Main:
def start_session(self):
utils.function()
# etc ...
And then in runner.py you do something like:
# runner.py
from mmd.main import Main
cli = Main()
cli.start_session()
This way inside the MMD directory calling python runner.py you would execute your code, and you can also make executable runner.py so a simply ./runner.py will run your code.
And run your module with:
python -m mmd.actions.DummyAction
I'd do it like this, becasue this way is open to future implementation (and is almost like in the line guides).
If instead you can't, then you can give it a try at removing __init__.py from the MMD directory.
I assume this is a directory structure you're talking about, in which case python doesn't know where to look for utils.py - it tries the local directory, a few places in the path then gives up. It's simple enough to modify the path:
import sys
sys.path.append("/MMD/misc")
import utils
and you should be away.
I've found a "workarround"
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
that allows to:
Call the module as a script (while using other modules): python -m MMD.actions.DummyAction
Call the main program and use the module: python MMD/MMD.py
Although I'm not sure if it's strictly correct to use a try - except block with an import, so feel free to add comments or other solutions.
The complete solution would be:
MMD.main
from misc import utils
from actions import DummyAction
class MMD():
def __init__(self):
a = DummyAction.DummyActionClass()
utils.foo()
if __name__ == '__main__':
d = MMD()
Then in actions/DummyAction.py:
try:
#When executing from the main
from misc import utils
except:
#When executing as a standalone script
from MMD.misc import utils
class DummyActionClass():
def __init__(self):
utils.foo()
if __name__ == '__main__':
a = DummyActionClass()
And finally in misc/utils.py:
def foo():
print "Foo was called"