I have a top level namespace, working like an organization namespace. Let's name that myorg, where it's __init__.py looks like this :
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
This is being deployed under /opt where our Python lives, along with some developed python libraries - let's say mylib .
So on a Python2.7 interpreter this works
>>> import myorg
>>> import myorg.mylib
Apart from the libraries, we have some client code that gets deployed elsewhere in the system, in /bb/bin. For example for clientA I can have :
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import clientA
and this works.
However, due to how our code is structured, we share the same namespace myorg - so clientB is also under myorg and there is a /bb/bin/myorg/__init__.py as well, with the same contants as the one in the /opt path.
Question is :
why this one works -
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import myorg.clientB
but this one doesn't :
>>> import myorg # import from /opt location
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import myorg.clientB # it should find the module under myorg in /bb/bin
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named clientB
This also does not work :
>>> import myorg.mylib
>>> import sys
>>> sys.path.insert(0,'/bb/bin') # Adding the /bb/bin path first
>>> import myorg.clientB
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named clientB
So, it's like the myorg namespace, if imported from the /opt then it will always try and find the modules under there.
Why is that? Can someone explain how to fix that issue and why that is happening?
I also dont understand why prepending that /bb/bin to my path does not make it work.
TL;DR
My intended behaviour or what I was hoping for is :
search in the /opt/ for the module
if not found go to other e.g. /bb/bin
This works on first case, but it seems if myorg has been imported from /opt then it only looks for modules inside there.
That's because "myorg" is already loaded - python loads modules/packages only once. If you load myorg (from /opt) first (or any sub-module as Python loads all "steps" of the import), then any myorg.something will be relative to that.
Can you try import as? PEP describing it.
If I understand how python works, this should work:
import myorg as whateveryouwant # import from /opt location
import sys
sys.path.append('/bb/bin')
import myorg.clientB
EDIT:
Noticed your comment that it doesn't work. :(
You can also try just appending /bb/bin/myorg and then importing just clientB instead of myorg.clientB. It's ugly but should work.
Related
In github there are four py Data which I put on my PyCharm. When I run main.py I get this message:
/Users/Armut/Desktop/High_D/Coursera/bin/python /Users/Armut/Desktop/High_D/main.py
Traceback (most recent call last):
File "/Users/Armut/Desktop/High_D/main.py", line 6, in <module>
from data_management.read_csv import *
ModuleNotFoundError: No module named 'data_management'
Here is a screenshots:
Can someone help, what I am doing wrong or how can I fix it?
EDIT (Put folders):
/Users/Armut/Desktop/High_D/Coursera/bin/python /Users/Armut/Desktop/High_D/main.py
WARNING:root:Failed to import geometry msgs in rigid_transformations.py.
WARNING:root:Failed to import ros dependencies in rigid_transforms.py
WARNING:root:autolab_core not installed as catkin package, RigidTransform ros methods will be unavailable
Traceback (most recent call last):
File "/Users/Armut/Desktop/High_D/main.py", line 7, in <module>
from visualization.visualize_frame import VisualizationPlot
ModuleNotFoundError: No module named 'visualization.visualize_frame'
EDIT:
/Users/Armut/Desktop/High_D/Coursera/bin/python /Users/Armut/Desktop/High_D/src/main.py
Traceback (most recent call last):
File "/Users/Armut/Desktop/High_D/src/main.py", line 7, in <module>
from src.visualization.visualize_frame import VisualizationPlot
File "/Users/Armut/Desktop/High_D/src/visualization/visualize_frame.py", line 10, in <module>
from utils.plot_utils import DiscreteSlider
ModuleNotFoundError: No module named 'utils.plot_utils'
Edit (No errors, but I just get a blank picture):
Edit (I installed matplotlib 3.0.3 and got this):
The issue here is, that it is just a picture. If you can see there are buttons like "next". I should be able to click it so I can track it. But how does it work?
Do the following
from read_csv import *
import visualize_frame as vf
The reason why it was not working for you is because you were importing files that dont exist on your system. When you do from data_management.read_csv import *, what you are telling the Python interpreter to do is to search for a folder called data_management inside you're Coursera folder and get everything from read_csv.py.
This is the same case with visualize_frame. Since you have a flat directory structure, you dont need the folder names. You can directly import the .py files as is.
Another thing to note here is that I personally wouldn't do from read_csv import * because I will be flooding my namespace with a lot of things I probably wont use. I would rather use import read_csv as any_alias_you_like. This way I only fill my namespace with what I want by doing the following
x = any_alias_you_like.function_call()
The reason why I didn't do this with the main code solution is because I am not sure where all you are using read_csv functions and classes in your code and if that is not accounted for by prefxing the alias name properly, you will run into a multiple errors. So my advice is to identify all the funcutions/classes that you are using in read_csv.py and prefix them properly with an alias.
I also used the import statement for the visualize_frame differently. This is because, when you do a from import..., you are only partially initializing the module. However, a proper import visualize_frame will ensure that your entire module is initialized in one call and you can use everything it offers by simply prefixing the alias.
Read about the difference between from import and import... here.
Read about how Python searches for libraries here.
I have inherited some python scripts that I am trying to run. These include some custom modules and I get an error when importing one of them, which seems to be due to one of the modules importing itself. What I find strange is that it works on one computer, but not on another computer.
The directory structure is as follows:
/path/to/packages/python_packages/x
|
/path/to/packages/python_packages/x/mod1.py
/path/to/packages/python_packages/x/mod2.py
/path/to/packages/python_packages/x/mod3.py
I add (and cross-check using print(sys.path)) the path as follows:
sys.path.append("/path/to/packages/python_packages/")
Then I do:
import x.mod1 as mod1
import x.mod2 as mod2
import x.mod3 as mod3
Importing mod1 works.
Importing mod2 does not work:
AttributeError: module 'x' has no attribute mod2
Traceback complains about this line, present in mod2.py:
import x.mod2 as mod2
Importing mod3 does not work, as it needs to import mod2, which it does in the same way as above.
In the traceback from ipython I can see that it finds the correct python files, since it prints out the code from the files and their names, full paths.
I have tried removing all init.py and pycache.
I tried running it on another computer, and there I can import the modules without any issues.
On the computer with the problem, I have Python 3.6.8, running on CentOS7 (3.10.0-1160.21.1.el7.x86_64) and on the computer where it works, I have Python 3.9.2, running on Manjaro (5.4.108-1-MANJARO).
I do not have root access on the computer with the problem.
The full traceback is as follows (I changed path and file names to be consistent with above explanation):
In [9]: import x.mod2 as mod2
AttributeError Traceback (most recent call last)
<ipython-input-9-8c4b062c2395> in <module>
----> 1 import x.mod2 as mod2
/path/to/packages/python_packages/x/mod2.py in <module>
2 import numpy as np
3
----> 4 import x.mod2 as mod2
5
6
AttributeError: module 'x' has no attribute 'mod2'
There is no need to import mod2 only to access mod2.__file__, since it is also available as __file__.
Instead of string fiddling you should use pathlib:
reference_path = Path(__file__).absolute().parents[2] / "reference"
Instead of using as use from:
from x import mod1
or, since you are already in the same package:
from . import mod1
It imports itself in order to access the path one level up from where it is placed.
reference_path = "/".join(mod2.__file__.split("/")[0:-2]+["reference/"])
You should be able to use __file__ directly.
I have a custom Python library ("common") that is being imported and used from several Python projects.
That central library has the following structure:
/common
/__init__.py
/wrapper.py
/util
/__init__.py
/misc.py
Our custom library resides in a central place /data/Development/Python, so in my Python projects I have an .env file in order to include our lib:
PYTHONPATH="/data/Development/Python"
That works fine, I can for example do something like:
from common.util import misc
However, now I want to make use of a class MyClass within common/wrapper.py from the code in common/util/misc.py. Thus I tried the following import in misc.py:
from ..wrapper import MyClass
But that leads to the following error:
Exception has occurred: ImportError
cannot import name 'MyClass'
Any ideas what I am doing wrong here?
PS: When I do an from .. import wrapper instead and then from code I use wrapper.MyClass, then it works fine. Does that make any sense?
It's finding wrapper, otherwise you'd get a different error:
>>> from wibble import myclass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'wibble'
So it seems wrapper does not contain MyClass. Typo?
"However, now I want to make use of a class MyClass within common/wrapper.py from the code in common/util/misc.py. Thus I tried the following import in misc.py:"
If you do export PYTHONPATH=~/common
In order to import MyClass you will do:
from wrapper import MyClass
Because you have added common to your root folder, utils is already in your path since it is in you root folder.
Similarly, if you wanted to import from wrapper you would do from utils.misc import ThisClass since utils.misc is already in the root folder common
I have a python project with this structure: (This is not a real project, only for testing)
ImportTest
ImportPersonsTest\
ImportPerson\
ImportPerson.py
RunImportPersonTest.py
RunImportTests.py
I want this tests to call each other. E.g :
RunImportTests.py calls a method in RunImportPersonTest.py, and RunImportPersonTest.py calls a method ImportPerson.py
RunImportPersonTest:
import os
import sys
sys.path.insert(0, os.getcwd() + "../../../")
from ImportPerson import ImportPerson
RunImportTests
import os
import sys
sys.path.insert(0, os.getcwd() + "../../")
from ImportPersonsTest import RunImportsPersonTest
I have success when I run ImportPerson.py and RunImportPersonTest.py, but when I try to run RunImportTests I get this error :
Traceback (most recent call last):
File "xxx\LiClipse Workspace\SystemTest\ImportTest\RunImportTests.py", line 4, in <module>
from ImportPersonsTest import RunImportsPersonTest
File "xxx\LiClipse Workspace\SystemTest\ImportTest\ImportPersonsTest\RunImportsPersonTest.py", line 4, in <module>
from ImportPerson import ImportPerson
ImportError: No module named 'ImportPerson'
Any suggestions?
Edit
New Structure
ImportTest
ImportPersonsTest\
ImportPerson\
ImportPerson.py
__init__.py
RunImportPersonTest.py
__init__.py
RunImportTests.py
__init__.py
Looks like you don't have any __init__.py files in your project. Python needs those files to be able to import modules from folders. The good news is, they are very easy to make: most of the time, they don't need anything in them, they just have to exist.
See: https://docs.python.org/2/tutorial/modules.html#packages
I think your use of os.getcwd() is flawed.
My guess is that you're running your program from the ImportTest directory and so your current working directory will already allow you to do the first import without any need to fix up your path. When you then try the second import, adding ".../ImportTest/../../.." or ".../ImportTest/../.." isn't helping Python find it.
To fix it either add the ImportPersonsTest directory to your path or use a suitably modified name in the import (ensuring you have your init files as already flagged) - e.g.
from ImportPersonsTest.ImportPerson import ImportPerson
There is two basic problem:
os.getcwd() as other os functions return path with no separator at the end. In fact you insert xxx\LiClipse Workspace\SystemTest\ImportTest../../../ which is not a valid path
As mention by #peter, using os.getcwd() is bad idea - it's depend on your location when you run the script. Use:
sys.path.append(os.path.dirname(__file__)) (insert recommend only at special cases)
But,
It seems that none of this caused your problem. It's only insert bad stuff to your sys.path. Your importing need to work good cause all the importing done from the self-module-dir, where python firstly search for the requested module.
I copy your package to mine machine - and both runs well!
I fix one spelling bug (RunImportsPersonTest -- RunImportPersonTest) - maybe there is other spelling problem
I must be missing something very basic about building a package in Python. When I create a package following the guidelines of https://docs.python.org/2/tutorial/modules.html#packages and import it, Python does not find any of the modules. For example, say I create the package holygrail with the following structure:
holygrail/
__init__.py
knights.py
I leave __init__.py empty because the docs say that I can and I'm just trying to make a basic package to start. In knights.py I have:
def say():
print 'Ni!'
If I try import holygrail, Python doesn't give any errors, but holygrail.knights.say() results in Python telling me that the "'module' object [holygrail] has no attribute 'knights'." However, if I specifically import knights via from holygrail import knights, then knights.say() works. In addition, holygrail.knights.say() then also works.
I tried adding the line
__all__ = ['knights']
in the __init__.py file, but this did not change the behavior.
How do I construct a package such that import package loads in all of the modules, allowing statements like package.module.function()?
Python does not implicitly import the whole package hierarchy. You have to be explicit about what to import on what level of the package using the __init__.py files.
When you set __all__ = ['knights'] in the __init__.py it works only for the import all statements for that module, e.g.:
>>> import holygrail
>>> holygrail.knights.say()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'knights'
>>> from holygrail import *
>>> knights.say()
Ni!
It can also act as a filter on import all, importing from the module only what's specified.
To get knights automatically imported on import holygrail you have to put import knights or from . import knights (intra-package or relative import) to the __init__.py. You'll have to do the same for every module explicitly.
Add import knights into __init__.py.
The link you provided does state "In the simplest case, __init__.py can just be an empty file..." Your example is not the simplest case, that's all.