I have a custom Python egg I've written, which I cannot get the submodule (or nested module) to load externally. I've been able to load a root-level module just fine.
Here's the obfuscated structure:
my_egg/
my_egg/
__init__.py (empty file)
module_one.py
my_subdir\
__init__.py (empty file)
module_two.py
setup.py
Importing module_one works:
from my_egg import module_one
But I cannot seem to get module_two to import. Here's what I've tried:
from my_egg.my_subdir import module_two
from my_egg import my_subdir.module_two
from my_egg.my_subdir.module_two import *
None of those worked. Here's what my setup.py looks like:
from setuptools import setup
setup(name='my_egg',
version='0.1',
packages=['my_egg'],
test_suite='nose.collector',
tests_require=['nose'],
zip_safe=False)
I'm surpised no one answered this. I was able to get it working after scouring Google, pulling from different sources, and trying different things.
One thing which held me up... I was trying to install my custom egg on a Databricks cluster. And I didn't realize that once you delete a library (egg) the cluster must be restarted in order for it to be deleted. So every time I would try changes, nothing took effect. This definitely delayed my progress.
In any case, I changed to my setup.py file to use find_packages and made changes to the empty __init__.py files. I'm not really sure if both changes were needed, or if one would've sufficed.
New my_egg/setup.py:
exec(open('my_egg/version.py').read())
from setuptools import setup, find_packages
setup(name='my_egg',
version=__version__,
packages=find_packages(exclude=('tests', 'docs')),
test_suite='nose.collector',
tests_require=['nose'],
zip_safe=False)
I added a my_egg/version.py file to help me debug if I was using the right version on the cluster. That addition actually led me to discover that Databricks requires the cluster be restarted.
New root init my_egg/my_egg/__init__.py file:
from .version import __version__
from .module_one import module_one_func
from .my_subdir.module_two import module_two_func
__all__ = ['module_one_func']
New sub-dir init my_egg/my_egg/my_subdir/__init__.py:
from module_two import module_two_func
__all__ = ['module_two_func']
Related
I have a python package defined via __init__.py in a folder named python_utils that has many python scripts in it.
python_utils/
__init__.py
print_pretty.py
script_1.py
...
For convenience this __init__.py imports all of them using:
from .pretty_print import pretty_print
from .scritp_1 import func1
from .script_2 import func2
....
The problem I have is that some of the python scripts depend on packages that may not be available at run time. So when I try to import a very trivial function from the package (from python_utils import print_pretty), it fails because of the dependencies the other scripts have. I also tried to import directly with from python_utils.print_pretty import print_pretty same problem. I can get rid of the __init__.py to solve the problem, but it saves me a lot of typing when importing packages. Is there a way I can import only pretty_print without getting the dependencies from the other scripts?
I've a directory structure like this below:
ParentPackage/
childPackage/
__init__.py
subpackageOne/
__init__.py
moduleA.py
moduleB.py
test/
__init__.py
test.py
I'm trying to import a method "methodX" from moduleA.py to my test.py residing inside test folder.
below is the code which I've tried
import os
import sys
sys.path.append(os.path.realpath('..'))
from ..subpackageOne.moduleAimport methodX
but getting error "ImportError: attempted relative import with no known parent package".
In addition to this moduleA.py is importing moduleB.py and while running the test I am getting No module named 'moduleB'.
There are already lot of question has been asked on this topic, however suggestions provided for those didn't work for me.
Based on your architecture, you can do that:
import os
import sys
# Assuming your package is on a drive called C:.
# That is what I mean by absolute path, it needs to refer the path from the disk
# so it does not depend on where you are
sys.path.append("C:/path/to/childPackage")
# Absolute import is just importing wihtout using dots at the beginning.
# In this case, python will look for installed packages in your environment
# and then any folder that is in your path.
from subpackageOne.moduleA import methodX
If you prefer, you may want to add the parentPackage in your path so you can use any childPackage in case you have several.
sys.path.append("C:/path/to/parentPackage")
from childPackage.subpackageOne.moduleA import methodX
Note: I see you have __init__ files in your architecture which suggests you may want to create an installable package that you can build after (using a setup.py or setup.cfg file). In this case, you could install your package in your virtual environment and then you would not need to add the directory to your path. setuptools Doc or some tutorial
My directory is structured like this
>project
>tests
>test_to_json.py
>app.py
>x.db
in my test_to_json.py I want to access a class from app.py
How would I import it? I've tried from ..app import myClass but that gives me ImportError: attempted relative import with no known parent package. Is there something I'm missing?
You cannot use .. relative pathing in python. That specific type of relative python is simply not allowed. . is allowed, though. The resolution for this problem is usually done by converting your project into a python package.
Extensive tutorials for doing so can be found here, but I will give an example of how to convert your project into a package.
Step 1
The new file structure should look like this:
>project
>tests
>__init__.py #Note this file
>test_to_json.py
>__init__.py #Note this file
>setup.py #Note this file
>app.py
>x.db
Step 2
Write your setup.py.
Here is an generic setup.py that should work for your project:
from setuptools import setup, find_packages
setup(
name='project_package', #the name of your desired import
version='0.0.1',
author='An Awesome Coder',
packages=find_packages(),
description='An awesome package that does something',
install_requires=[], # a list of python dependencies for your package
)
find_packages() looks for all the __init__.py files in your package to identify submodules.
Step 3
Install your package. In the folder with your new setup.py, run pip install -e . This will install your package on your computer, basically adding the files to your python system path.
Step 4
From any python terminal on your computer you should now be able to import your package using the package_name you specified.
import project_package
from project_package.app import myClass
myClass()
.
.
.
I have a rather complex python build project with a PyQT GUI interface. The builder is supposed to accept configuration for a list of projects, which may differ from user to user, and then customize the build process for a selected project. As a result, parts of the builder project are versioned with the project to be built (project specific UI elements and controller files), with the rest being located in the builders directory. The issue I am running into is an overlap in namespacing for imports. For example:
The builder directory might look like:
Builder/
Controllers/
__init__.py
BuilderController1.py
BuilderController2.py
UI/
__init__.py
BuilderUI_1.py
BuilderUI_2.py
.../
__main__.py
The project directory might look like:
Project/
Project_Files/
Build/
Controllers/
__init__.py
ProjController1.py
ProjController2.py
UI/
__init__.py
ProjUI_1.py
ProjUI_2.py
.../
__init__.py
Project_setup.py
With code in __main__.py such as:
...
sys.path.append("../path/to/Project/Build")
projSetup = __import__("Project_setup)
projSetup.run()
...
And code in Project_setup.py such as:
...
from Controllers.ProjController1 import TestControl1
from Controllers.ProjController2 import TestControl2
def run():
registerStage(name = 'TestControl1', controller = TestControl1(mainApp.getController(), displayName='TestControl1'), displayName = 'Test Control Page 1', parent = None)
registerStage(name = 'TestControl2', controller = TestControl2(mainApp.getController(), displayName='TestControl2'), displayName = 'Test Control Page 2', parent = None)
...
The problem is that every time I run __main__.py, I get an error saying that:
ImportError: No module named Controllers.ProjController1
It seems that while searching the PYTHONPATH for the module ProjController1 the interpreter gets stranded in Builder/Controllers and then stops looking further down the path.
Is this reasonable behavior in python? Is there a method to obtain the same results that circumvents these issues?
I have tried using the various flavors of python importing ( import, __import__, imp, and importlib). Additionally it will work if I rename Project/Build/Controllers and Project/Build/UI, but given the possible complexity of these file structures I would rather not be stuck giving unique names to everything.
Relative paths are reliant on your current working directory. Try the following for your sys path append:
import os
dir_ = os.path.dirname(__file__)
sys.path.append(os.path.join(dir_, "/path/to/Project/Build"))
It's really better to make a package, then import via the fully qualified path. In your root project dir, put a file setup.py with contents similar to:
#!/usr/bin/env python
from setuptools import setup
from setuptools import find_packages
setup(
name='Name',
version='1.0',
packages=find_packages(exclude=('tests',)),
cmdclass={
},
package_data={
},
)
Then building and installing the package like so:
python setup.py build; python setup.py install
Perform the imports like so:
from project.hierarchy.module import Class
More information on creating packages is available in the Python docs.
Relative imports and modifying sys path have always caused me heartache.
I'm trying to create an install package for a Python project with included unit tests. My project layout is as follows:
setup.py
src/
disttest/
__init__.py
core.py
tests/
disttest/
__init__.py
testcore.py
My setup.py looks like this:
from distutils.core import setup
import setuptools
setup(name='disttest',
version='0.1',
package_dir={'': 'src'},
packages=setuptools.find_packages('src'),
test_suite='nose.collector',
tests_require=['Nose'],
)
The file tests/disttest/testcore.py contains the line from disttest.core import DistTestCore.
Running setup.py test now gives an ImportError: No module named core.
After a setup.py install, python -c "from disttest.core import DistTestCore" works fine. It also works if I put import core into src/disttest/__init__.py, but I don't really want to maintain that and it only seems necessary for the tests.
Why is that? And what is the correct way to fix it?
You may want to double-check this, but it looks like your tests are importing the disttest package in the tests/ directory, instead of the package-under-test from the src/ directory.
Why do you need to use a package with the same name as the package-under-test? I'd simply move the testcore module up to the tests directory, or rename the tests/disttest package and avoid the potential naming conflict altogether.
In any case, you want to insert a import pdb; pdb.set_trace() line just before the failing import and play around with different import statements to see what is being imported from where (import sys; sys.modules['modulename'].__file__ is your friend) so you get a better insight into what is going wrong.