We Keep Coding

Import module installed in venv

I am trying to call a custom module installed in my virtualenv. The module built by myself has the following structure:
test
├── README.md
├── setup.py
├── src
│   ├── __init__.py
│   └── module_x
│   └── abc.py
└── tests
setup.py looks like following:
from setuptools import find_packages, setup
with open('README.md', 'r') as f:
long_description = f.read()
setup(
name='test',
version='0.1.0',
author='me',
description='description',
long_description=long_description,
long_description_content_type='text/markdown',
packages=find_packages('src'),
package_dir={'': 'src'},
install_requires=[''],
entry_points={
'console_scripts': [
'test=module_x.abc:main'
],
}
)
Inside of the abc.py, I am using the following code:
class RandomClass:
def __init__(self, msg):
self.msg = msg
info()
def info():
print(self.msg)
def main(msg):
RandomClass(msg)
init.py looks like
from .module_x.abc import main
The egg was installed using pip install -e ..
UPDATED
How can I call the module in a new Python script like from test import main? Currently it works just like from test.src import main and I don't want to know the whole structure of the package. In the future will exist also module_y, module_z etc. My assumption is that I would need to modify something in the setup.py.

How to Import python package from another directory?

I have a project that is structured as follows:
project
├── api
│ ├── __init__.py
│ └── api.py
├── instance
│ ├── __init__.py
│ └── config.py
├── package
│ ├── __init__.py
│ └── app.py
├── requirements.txt
└── tests
└── __init__.py
I am trying to call the config.py file from the package/app.py as shown below:
# package/app.py
from instance import config
# I've also tried
import instance.config
import ..instance.config
from ..instance import config
But I always get the following error:
Traceback (most recent call last):
File "/home/csymvoul/projects/project/package/app.py", line 1, in <module>
from instance import config
ModuleNotFoundError: No module named 'instance'
Modifying the sys.path is not something I want to do.
I know that this question is very much answered but the answers that were given, did not work for me.
EDIT: When moving the app.py to the root folder it works just fine. But I need to have it under the package folder.

You can add the parent directory to PYTHONPATH, in order to achieve that, you can use OS depending path in the "module search path" which is listed in sys.path. So you can easily add the parent directory like following:
import sys
sys.path.insert(0, '..')
from instance import config
Note that the previous code uses a relative path, so you must launch the file inside the same location or it will likely not work. To launch from anywhere, you can use the pathlib module.
from pathlib import Path
import sys
path = str(Path(Path(__file__).parent.absolute()).parent.absolute())
sys.path.insert(0, path)
from instance import config
However, the previous approach is more a hack than anything, in order to do things right, you'll first need to reshape your project structure according to this very detailed blog post python packaging, going for the recommended way with a src folder.
Your directory layout must look like this:
project
├── CHANGELOG.rst
├── README.rst
├── requirements.txt
├── setup.py
├── src
│   ├── api
│   │   ├── api.py
│   │   └── __init__.py
│   ├── instance
│   │   ├── config.py
│   │   └── __init__.py
│   └── package
│   ├── app.py
│   └── __init__.py
└── tests
└── __init__.py
Note that you don't really need the requirements.txt because you can declare the dependencies inside your setup.py.
A sample setup.py (adapted from here):
#!/usr/bin/env python
# -*- encoding: utf-8 -*-
from __future__ import absolute_import
from __future__ import print_function
import io
import re
from glob import glob
from os.path import basename
from os.path import dirname
from os.path import join
from os.path import splitext
from setuptools import find_packages
from setuptools import setup
def read(*names, **kwargs):
with io.open(
join(dirname(__file__), *names),
encoding=kwargs.get('encoding', 'utf8')
) as fh:
return fh.read()
setup(
name='nameless',
version='1.644.11',
license='BSD-2-Clause',
description='An example package. Generated with cookiecutter-pylibrary.',
author='mpr',
author_email='contact#ionelmc.ro',
packages=find_packages('src'),
package_dir={'': 'src'},
include_package_data=True,
zip_safe=False,
classifiers=[
# complete classifier list: http://pypi.python.org/pypi?%3Aaction=list_classifiers
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Operating System :: Unix',
'Operating System :: POSIX',
'Operating System :: Microsoft :: Windows',
'Programming Language :: Python',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Programming Language :: Python :: 3.8',
'Programming Language :: Python :: Implementation :: CPython',
'Programming Language :: Python :: Implementation :: PyPy',
# uncomment if you test on these interpreters:
# 'Programming Language :: Python :: Implementation :: IronPython',
# 'Programming Language :: Python :: Implementation :: Jython',
# 'Programming Language :: Python :: Implementation :: Stackless',
'Topic :: Utilities',
],
keywords=[
# eg: 'keyword1', 'keyword2', 'keyword3',
],
python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*',
install_requires=[
# eg: 'aspectlib==1.1.1', 'six>=1.7',
],
extras_require={
# eg:
# 'rst': ['docutils>=0.11'],
# ':python_version=="2.6"': ['argparse'],
},
setup_requires=[
# 'pytest-runner',
],
entry_points={
'console_scripts': [
'api = api.api:main',
]
},
)
The content of my api.py:
from instance import config
def main():
print("imported")
config.config()
The content of my config.py:
def config():
print("config imported successfully")
You can find all the previous here
Optional but recommended: create a virtual environment, I use venv (Python 3.3 <=) for that, inside the root of the project:
python -m venv .
And to activate:
source bin/activate
Now I can install the package:
Using pip install -e . (with the dot) command inside the root of the project
Your import from instance import config works now, to confirm you can run api.py with:
python src/api/api.py

In Python 3.3 onwards you don't need __init__.py files in your subdirectories for the purpose of imports. Having them can actually be misleading as it causes the creation of package namespaces in each folder containing an init file, as described here.
By removing all those __init__.py files you will be able to import files in the namespace package (including subdirectories) when running app.py, but that's still not want we want.
The Python interpreter still doesn't know how to reach your instance namespace. In order to do that you can use the PYTHONPATH environment variable, including a path that is parent to config.py. You can do that as suggested in #RMPR's answer with sys.path, or by directly setting the environment variable, for instance:
PYTHONPATH=/home/csymvoul/projects/project python3 /home/csymvoul/projects/project/package/app.py
Then importing the dependency like from instance import config.

Import modules in package in ROS2

I have created a package for ROS2 and I have added a Python repository I downloaded. The problem I am having is that in the original repository the modules from the own repo were imported directly while in mine I have to import them adding the ROS2 package name before the module, even though I am importing a module from the same repo, like:
import planner_pkg.SimpleOneTrailerSystem as SimpleOneTrailerSystem
while I would like:
import SimpleOneTrailerSystem
My ROS2 project structure is like:
ros2_ws
src
planner
planner_pkg
__init__.py
SimpleOneTrailerSystem.py
planner_node.py
...
package.xml
setup.py
package.xml
<?xml version="1.0"?>
<package format="2">
<name>planner_pkg</name>
<version>0.0.1</version>
<description>This package contains algorithm for park planner</description>
<maintainer email=""></maintainer>
<license>Apache License 2.0</license>
<exec_depend>rclpy</exec_depend>
<exec_depend>std_msgs</exec_depend>
<!-- These test dependencies are optional
Their purpose is to make sure that the code passes the linters -->
<test_depend>ament_copyright</test_depend>
<test_depend>ament_flake8</test_depend>
<test_depend>ament_pep257</test_depend>
<test_depend>python3-pytest</test_depend>
<export>
<build_type>ament_python</build_type>
</export>
</package>
setup.py:
from setuptools import setup
package_name = 'planner_pkg'
setup(
name=package_name,
version='0.0.0',
packages=[package_name],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
],
install_requires=['setuptools'],
zip_safe=True,
author='',
author_email='',
maintainer='',
maintainer_email='',
keywords=['ROS'],
classifiers=[
'Intended Audience :: Developers',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Topic :: Software Development',
],
description='Package containing examples of how to use the rclpy API.',
license='Apache License, Version 2.0',
tests_require=['pytest'],
entry_points={
'console_scripts': [
'planner_node = planner_pkg.planner_node:main',
],
},
)

First, according to the Module Search Path docs, when you do import something, Python looks for that something in the following places:
From the built-in modules
sys.path, which is a list containing:
The directory of the input script
PYTHONPATH, which is an environment variable containing a list of directories
The installation-dependent default
Second, when you build your ROS2 Python package (by calling colcon build invoking the ament_python build type), your Python codes will be copied over to an install folder with a tree structure like this:
install
...
├── planner_pkg
│   ├── bin
│   │   └── planner_node
│   ├── lib
│   │   └── python3.6
│   │   └── site-packages
│   │   ├── planner_pkg
│   │   │   ├── __init__.py
│   │   │   ├── planner_node.py
│   │   │   └── SimpleOneTrailerSystem.py
...
Now, when you do import SimpleOneTrailerSystem, Python will first search for it from the built-in modules, which for sure it won't find there. Next on the list is from sys.path. You can add a print(sys.path) at the top of planner_node.py to see something like this list:
['/path/to/install/planner_pkg/bin',
'/path/to/install/planner_pkg/lib/python3.6/site-packages',
'/opt/ros/eloquent/lib/python3.6/site-packages',
'/usr/lib/python36.zip',
'/usr/lib/python3.6',
...other Python3.6 installation-dependent dirs...
]
First on the sys.path list is the bin folder of the input script. There are only executables there, no SimpleOneTrailerSystem.py file/module, so that import will fail.
The next on the list would be the planner_pkg/lib/pythonX.X/site-packages, and as you can see from the tree structure above, there is a SimpleOneTrailerSystem.py module BUT it is under a planner_pkg folder. So a direct import like this
import SimpleOneTrailerSystem
will not work. You need to qualify it with the package folder like this:
import planner_pkg.SimpleOneTrailerSystem
There are 2 ways to get around this.
Modify sys.path before import-ing SimpleOneTrailerSystem
import sys
sys.path.append("/path/to/install/planner_pkg/lib/python3.6/site-packages/planner_pkg")
import SimpleOneTrailerSystem
This approach adds the path to the planner_pkg install directory to the sys.path list so that you don't need to specify it in subsequent imports.
Modify PYTHONPATH before running your ROS2 node
$ colcon build
$ source install/setup.bash
$ export PYTHONPATH=$PYTHONPATH:/path/to/install/planner_pkg/lib/python3.6/site-packages/planner_pkg
$ planner_node
This approach is almost the same as the first one, but there is no code change involved (and no rebuilding involved), as you only need to modify the PYTHONPATH environment variable.

I had the same problem and fixed it by modifying setup.py.
Add:
('lib/' + package_name, [package_name+'/SimpleOneTrailerSystem.py']),
to "data_files" list.
setup.py:
from setuptools import setup
package_name = 'planner_pkg'
setup(
name=package_name,
version='0.0.0',
packages=[package_name],
data_files=[
('share/ament_index/resource_index/packages',
['resource/' + package_name]),
('share/' + package_name, ['package.xml']),
('lib/' + package_name, [package_name+'/SimpleOneTrailerSystem.py']),
],
install_requires=['setuptools'],
zip_safe=True,
author='',
author_email='',
maintainer='',
maintainer_email='',
keywords=['ROS'],
classifiers=[
'Intended Audience :: Developers',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Topic :: Software Development',
],
description='Package containing examples of how to use the rclpy API.',
license='Apache License, Version 2.0',
tests_require=['pytest'],
entry_points={
'console_scripts': [
'planner_node = planner_pkg.planner_node:main',
],
},
)

For CMAKE packages with cpp and python code you can add the following lines to your CMakeLists.txt.
This will copy your python_pkg folder into your install environment to "lib/python{version}/site-packages" which is per default included in your python path.
# for python code
find_package(ament_cmake_python REQUIRED)
find_package(rclpy REQUIRED)
install(DIRECTORY
../path_to_python_pkg
DESTINATION lib/python3.8/site-packages
)
install(PROGRAMS
${PROJECT_NAME}/python_node.py
DESTINATION lib/${PROJECT_NAME}
)
ament_python_install_package(${PROJECT_NAME})
If you are using e.g. Dashing or Eloquent it is "python3.6", for Foxy or newer it is "python3.8"

New to creating a Python Package

I tried making one to post on my GitHub and help users download dependencies for the program, but in the end, all it does is generate duplicate files. I was hoping to have (in the end) a package where the user could enter:
>>> import my_package
>>> my_package.main
but that isn't working. I've looked at several different websites and different templates, but seem to be getting nowhere with this.
Directory structure
Kodimer_Project
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.md
├── requirements.txt
├── setup.py
└── wav2bin
├── __version__.py
├── imgs
│   ├── App_Icon.gif
│   ├── App_Icon.ico
│   └── __init__.py
└── src
├── DrawGraph.py
├── GraphicInterface.py
├── SplashScreen.py
├── __init__.py
├── __main__.py
└── helper_functions.py
setup.py
From: https://github.com/kennethreitz/setup.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Note: To use the 'upload' functionality of this file, you must:
# $ pip install twine
import io
import os
import sys
from shutil import rmtree
from setuptools import find_packages, setup, Command
# Package meta-data.
NAME = 'wav2bin'
DESCRIPTION = 'GUI graphing tool used concurrently with lab.'
URL = 'https://github.com/jvanderen1/Kodimer_Project'
EMAIL = 'jvanderen1#gmail.com'
AUTHOR = 'Joshua Van Deren'
# What packages are required for this module to be executed?
REQUIRED = [
'matplotlib',
'numpy',
'scipy'
]
# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!
here = os.path.abspath(os.path.dirname(__file__))
# Import the README and use it as the long-description.
# Note: this will only work if 'README.md' is present in your MANIFEST.in file!
with io.open(os.path.join(here, 'README.md'), encoding='utf-8') as f:
LONG_DESCRIPTION = '\n' + f.read()
# Load the package's __version__.py module as a dictionary.
about = {}
with open(os.path.join(here, NAME, '__version__.py')) as f:
exec(f.read(), about)
class UploadCommand(Command):
"""Support setup.py upload."""
description = 'Build and publish the package.'
user_options = []
#staticmethod
def status(s):
"""Prints things in bold."""
print('\033[1m{0}\033[0m'.format(s))
def initialize_options(self):
pass
def finalize_options(self):
pass
def run(self):
try:
self.status('Removing previous builds…')
rmtree(os.path.join(here, 'dist'))
except OSError:
pass
self.status('Building Source and Wheel (universal) distribution…')
os.system('{0} setup.py sdist bdist_wheel --universal'.format(sys.executable))
self.status('Uploading the package to PyPi via Twine…')
os.system('twine upload dist/*')
sys.exit()
# Where the magic happens:
setup(
name=NAME,
version=about['__version__'],
description=DESCRIPTION,
long_description=LONG_DESCRIPTION,
author=AUTHOR,
author_email=EMAIL,
url=URL,
package_dir={'': 'wav2bin'},
packages=find_packages(exclude=('tests', 'docs')),
# If your package is a single module, use this instead of 'packages':
# py_modules=['mypackage'],
entry_points={
'gui_scripts': ['wav2bin = wav2bin.__main__:main'],
},
install_requires=REQUIRED,
python_requires='>=3',
include_package_data=True,
license='MIT',
classifiers=[
# Trove classifiers
# Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 3 :: Only',
'Natural Language :: English',
'Topic :: Scientific/Engineering :: Human Machine Interfaces',
'Topic :: Software Development :: User Interfaces'
],
# $ setup.py publish support.
cmdclass={
'upload': UploadCommand,
},
)
wav2bin/src/__main__.py
Snippet of code:
if __name__ == '__main__':
main()

After you have some experience with various packaging modules what you would usually do is decide how professional you want your packaging to be? Do you want to host it on pypi? Distribute it from github? Pass it along to friends?
That's how you pick your packaging method, but first you should probably get some experience with the existing packaging modules, the two most popular ones are:
setuptools which is what I usually go with and to which I linked a good tutorial
distutils an older api to distribute packages but it's still around and is also very good to know
If then you decide this is an overkill and you want a less professional approach you can always do it manually.
Either installation to the python package folder, which for pip usually means something like entering the packages root folder and entering either
pip install .
If you are certain of your package, or
pip install -e .
For installing in edit mode if you still wish to keep the package malleable
or having the package in your python path in some other manner before import is mandatory.

python Import Error: no module named app

First, I created __init__.py in my app folder
from flask import Flask
app = Flask(__name__)
from app import views
and then, I created views.py in the same app folder
from app import app
#app.route('/')
#app.route('/index')
def index():
return "Hello, World!
and lastly, I created run.py in a different folder
#!flask/bin/python
from app import app
app.run()
so, my flask tree is like
/microblog
/app
/__init__.py
/views.py
/flask
/tmp
/run.py
But, when I try to run run.py, I get an error
Traceback (most recent call last):
File "/home/shubham/Microblog/tmp/run.py", line 2, in <module>
from app import app
ImportError: No module named app
[Finished in 0.2s with exit code 1]
[shell_cmd: python -u "/home/shubham/Microblog/tmp/run.py"]
[dir: /home/shubham/Microblog/tmp]
[path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games]

Your problem is that you don't have a clear understanding about the differences between imports, variable names, and packages. And how imports work.
When you're structuring a Python package, you'll typically have your toplevel folder with a setup.py, README and other assorted files in it, then your package folder. You might want something that looks like this:
├── microblog
│   ├── __init__.py
│   ├── server.py
│   └── views.py
├── README
├── MANIFEST.in
└── setup.py
If you really didn't want to put your URL handlers in the same place as your app, you've got it a little backwards though. You should do it like this:
server.py
from flask import Flask
from . import views
app = Flask(__name__)
app.register_blueprint(views.microblog)
def run():
app.run(debug=True)
views.py
from flask import Blueprint
microblog = Blueprint('microblog', __name__)
#microblog.route('/')
#microblog.route('/index')
def index():
return 'Hello, world!'
setup.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from setuptools import setup
setup(
name='microblog',
version='0.1.0',
description='My microblog',
author='Me',
author_email='me#example.com',
url='http://my.cool.example.com',
packages=[
'microblog',
],
entry_points={
'console_scripts': [
'start-my-blog = microblog.server:run'
],
},
package_dir={'microblog': 'microblog'},
include_package_data=True,
install_requires=['flask',
],
license="BSD",
zip_safe=False,
keywords='',
classifiers=[
'Development Status :: 2 - Pre-Alpha',
'Intended Audience :: Developers',
'License :: OSI Approved :: BSD License',
'Natural Language :: English',
'Programming Language :: Python :: 3.4',
],
)
Now you can create yourself a virtualenv, install your app, and then run it:
$ mkvirtualenv microblog
$ workon microblog
(microblog) $ python -m pip install -e .
# a bunch of install happens here, flask & what-not
$ start-my-blog
If you want to change the name of the command to run, you'll do that in your setup.py. Starting out with a structure like this will save you a lot of headaches down the road.