We Keep Coding

python setup tools - install a sub package from within a project

I have two projects (trysetup1 and trysetup2) with the following structure:
I want to pip install package1 and use module1 from project trysetup2
my setup.py that under package1 looks like this:
import setuptools
setuptools.setup(
name="common",
version="1.0.2",
packages=setuptools.find_packages(),
)
the way I want to use module1 is like this from package1.module1 import ClassOne because I still need to use it from package2
when trying to import from module2 it works just fine
but when trying to use it from module3 (in the different project after pip installing it) i'm having "Unresolved reference 'package1'" problem
I know I'm able to use module1 by putting it inside another package under package1 but I need this exact stracture in order to use it from the rest of the project 'trysetup1'
Thanks!

My answer was found here:
https://docs.python.org/3/distutils/examples.html
actually, all I needed to do was to change my setup.py file to look like this:
setuptools.setup(
name="common",
version="1.0.2",
package_dir={'package1': ''},
packages=['package1'],
)
by adding package_dir param, setup function tells all files under my root directory (package1) to be under package1 directory and by adding packages param it distributes package1 package and then if you go to:
/..../venv/lib/python3.8/site-packages/common-1.0.2-py3.8.egg-info/top_level.txt
you'll see the following content:

How can I create imports that always work?

I am struggling a bit to set up a working structure in one of my projects. The problem is, that I have main package and a subpackage in a structure like this (I left out all unnecessary files):
code.py
mypackage/__init__.py
mypackage/work.py
mypackage/utils.py
The utils.py has some utility code that is normally only used in the mypackage package.
I normally have some test code each module file, that calls some methods of the current module and prints some things to quickcheck if everything is working correctly. This code is placed in a if __name__ == "__main__" block at the end of the file. So I include the utils.py directly via import utils. E.g mypackage/work.py looks like:
import utils
def someMethod():
pass
if __name__ == "__main__":
print(someMethod())
But now when I use this module in the parent package (e.g. code.py) and I import it like this
import mypackage.work
I get the following error:
ImportError: No module named 'utils'
After some research I found out, that this can be fixed by adding the mypackage/ folder to the PYTHONPATH environment variable, but this feels strange for me. Isn't there any other way to fix this? I have heard about relative imports, but this is mentioned in the python docs about modules
Note that relative imports are based on the name of the current module. Since the name of the main module is always "main", modules intended for use as the main module of a Python application must always use absolute imports.
Any suggestions how I can have a if __name__ == "__main__" section in the submodule and also can use this file from the parent package without messing up the imports?
EDIT: If I use a relative import in work.py as suggested in a answer to import utils:
from . import utils
I get the following error:
SystemError: Parent module '' not loaded, cannot perform relative import

Unfortunately relative imports and direct running of submodules don't mix.
Add the parent directory of mypackage to your PYTHONPATH or always cd into the parent directory when you want to run a submodule.
Then you have two possibilities:
Use absolute (from mypackage import utils) instead of relative imports (from . import utils) and run them directly as before. The drawback with that solution is that you'll always need to write the fully qualified path, making it more work to rename mypackage later, among other things.
or
Run python3 -m mypackage.utils etc. to run your submodules instead of running python3 mypackage/utils.py.
This may take some time to adapt to, but it's the more correct way (a module in a package isn't the same as a standalone script) and you can continue to use relative imports.
There are more "magical" solutions involving __package__ and sys.path but they all require extra code at the top of every file with relative imports you want to run directly. I wouldn't recommend these.

You should create a structure like this:
flammi88
├── flammi88
│ ├── __init__.py
│   ├── code.py
│   └── mypackage
│   ├── __init__.py
│   ├── utils.py
│   └── work.py
└── setup.py
then put at least this in the setup.py:
import setuptools
from distutils.core import setup
setup(
name='flammi88',
packages=['flammi88'],
)
now, from the directory containing setup.py, run
pip install -e .
This will make the flammi88 package available in development mode. Now you can say:
from flammi88.mypackage import utils
everywhere. This is the normal way to develop packages in Python, and it solves all of your relative import problems. That being said, Guido frowns upon standalone scripts in sub-packages. With this structure I would move the tests inside flammi88/tests/... (and run them with e.g. py.test), but some people like to keep the tests next to the code.
Update:
an extended setup.py that describes external requirements and creates executables of the sub-packages you want to run can look like:
import setuptools
from distutils.core import setup
setup(
name='flammi88',
packages=['flammi88'],
install_requires=[
'markdown',
],
entry_points={
'console_scripts': [
'work = flammi88.mypackage.work:someMethod',
]
}
)
now, after pip installing your package, you can just type work at the command line.

Import utils inside the work.py as follows:
import mypackage.utils
or if you want to use shorter name:
from mypackage import utils
EDIT: If you need to run work.py outside of the package, then try:
try:
from mypackage import utils
except ImportError:
import utils

Use:
from . import utils
as suggested by Peter
In your code.py you should use:
from mypackage import work

Package import creates a module, submodules still importable

I've been working on a python package and now I would like to turn it into a small RPM distribution. Package includes a few modules, one of which is executable. I can create the RPM-package with python setup.py bdist_rpm and install it on a fedora box with rpm.
At this point have the desired command myscript and it works like a charm. But when i try to import the package in ipython, I run into something strange. I can do the following
from myscript import sdf
import myscript.mol2
Both work flawlessly, but
import myscript
myscript.sdf
throws
AttributeError: 'module' object has no attribute 'sdf'
I've been working with this for a while now to no avail. There's plenty of questions of import problems, but I haven't found an answer for this one yet.
What I should change to make it work?
The current folder structure is:
myscript/ #project root
setup.py
src/
myscript/
__init__.py
functions.py
sdf.py
mol2.py
runner.py
bin/
myscript #symbolic link to src/myscript/runner.py
setup.py is:
from distutils.core import setup
setup(name = 'myscript',
version = '0.75',
author ='me',
requires = ['numpy'],
packages = ['myscript'],
package_dir = {'myscript':'src/myscript'},
scripts = ['bin/myscript']
)
and __init__.py is:
__all__ = ['functions','sdf','mol2','runner']

This is normal behavior. If you want submodules to be imported then you must import them in the module.
# myscript/__init__.py
from . import sdf

How to import your package/modules from a script in bin folder in python

When organising python project, this structure seems to be a standard way of doing it:
myproject\
bin\
myscript
mypackage\
__init__.py
core.py
tests\
__init__.py
mypackage_tests.py
setup.py
My question is, how do I import my core.py so I can use it in myscript?
both __init__.py files are empty.
Content of myscript:
#!/usr/bin/env python
from mypackage import core
if __name__ == '__main__':
core.main()
Content of core.py
def main():
print 'hello'
When I run myscript from inside myproject directory, I get the following error:
Traceback (most recent call last):
File "bin/myscript", line 2, in <module>
from mypackage import core
ImportError: No module named mypackage
What am I missing?

Usually, setup.py should install the package in a place where the Python interpreter can find it, so after installation import mypackage will work. To facilitate running the scripts in bin right from the development tree, I'd usually simply add a simlink to ../mypackage/ to the bin directory. Of course, this requires a filesystem supporting symlinks…

I'm not sure if there is a "best choice", but the following is my normal practice:
Put whatever script I wanna run in /bin
do "python -m bin.script" in the dir myproject
When importing in script.py, consider the dir in which script.py is sitting as root. So
from ..mypackage import core
If the system supports symlink, it's a better choice.

I usually add my bin path into $PYTHONPATH, that will enable python to look for asked module in bin directory too.
export PYTHONPATH=/home/username/bin:$PYTHONPATH
$ python
import module_from_bin

I solved the issue following setuptools specifications.
In setup.py you can specify the modules as an argument for the function setup():
packages = find_packages()
This finds all modules.
p.s. you have to import this function: from setuptools import setup, find_packages

Python import src modules when running tests

My source files are located under src and my test files are located under tests. When I want to run a test file, say python myTest.py, I get an import error: "No module named ASourceModule.py".
How do I import all the modules from source needed to run my tests?

You need to add that directory to the path:
import sys
sys.path.append('../src')
Maybe put this into a module if you are using it a lot.

If you don't want to add the source path to each test file or change your PYTHONPATH, you can use nose to run the tests.
Suppose your directory structure is like this:
project
package
__init__.py
module.py
tests
__init__.py
test_module.py
You should import the module normally in the test_module.py (e.g. from package import module). Then run the tests by running nosetests in the project folder. You can also run specific tests by doing nosetests tests/test_module.py.
The __init__.py in the tests directory is necessary if you want to run the tests from inside it.
You can install nose easily with easy_install or pip:
easy_install nose
or
pip install nose
nose extends unittest in a lot more ways, to learn more about it you can check their website: https://nose.readthedocs.org/en/latest/

On my system (Windows 10), I was required to do something like this:
import sys
import os
sys.path.append(os.path.dirname(os.path.realpath(__file__)) + "/../src")
Appending the relative directory directly to sys.path did not work

The best (most manageable) solution appears to be using a virtualenv and setuptools/distribute to install a development copy of your (src) package. That way your tests execute against a fully "installed" system.
In the pystest docs there is a section on "good practices" explaining this approach, see here.

For those using Pytest:
Make sure src is recognized as a package by putting an empty__init__.py inside.
Put an empty conftest.py in the project folder.
Make sure there is no __init__.py in the test directory.
Looks like this:
project
conftest.py
src
__init__.py
module.py
test
test_module.py
And make sure module.py contains the line:
from src import module
Source: Pytest docs