We Keep Coding

Not able to import my own python package after pip installation

I'm trying to build my first python package public available but I'm having some trouble with installing it on another machine, not sure what is wrong. My project is here.
After all the CI steps on the master branch, Travis publishes the latest version to the pypi. After that, we can install the package in any place:
pip install spin-clustering
But when I try to import it on my regular python it says that the module does not exist.
$ python -c "import spin"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'spin'
My package was originally called "spin" but the name was already taken on pypi, I changed it to "spin-clustering", but as scikit-learn is imported with "sklearn" I thought that would be possible to import my package as "spin". Not sure what I'm missing here.
This is my package structure:
├── LICENSE
├── Makefile
├── Pipfile
├── README.md
├── examples
│   ├── circle-example.ipynb
│   └── random-cluster-example.ipynb
├── setup.cfg
├── setup.py
└── spin
├── __init__.py
├── distances
│   ├── __init__.py
│   ├── distances.py
│   └── tests
│   └── __init__.py
├── neighborhood_spin.py
├── side_to_side_spin.py
├── tests
│   ├── __init__.py
│   ├── test_spin.py
│   └── test_utils.py
└── utils.py
And my setup.py
from setuptools import setup, find_packages
setup(name="spin-clustering",
maintainer="otaviocv",
maintainer_email="otaviocv.deluqui#gmail.com",
description="SPIN clustering method package.",
license="MIT",
version="0.0.3",
python_requires=">=3.6",
install_requires=[
'numpy>=1.16.4',
'matplotlib>=3.1.0'
]
)

In your setup.py, you also need to specify what packages will be installed. The simplest way is using the provided find_packages function, which will scan your folders and try to figure out what the packages are (in some slightly unusual cases, your project organization will make this not work right). Your code imports find_packages, but is not using it.
Since you have none listed, nothing is actually installed (except the requirements, if missing).

Packaging python with cython extension

I'm trying to build a package that uses both python and cython modules. The problem I'm having deals with imports after building and installing where I'm not sure how to make files import from the .so file generated by the build process.
Before building my folder structure looks like this
root/
├── c_integrate.c
├── c_integrate.pyx
├── cython_builder.py
├── __init__.py
├── integrator_class.py
├── integrator_modules
│   ├── cython_integrator.py
│   ├── __init__.py
│   ├── integrator.py
│   ├── numba_integrator.py
│   ├── numpy_integrator.py
│   ├── quadratic_error.png
│   ├── report3.txt
│   ├── report4.txt
│   └── report5.txt
├── report6.txt
├── setup.py
└── test
├── __init__.py
└── test_integrator.py
Building with python3.5 setup.py build gives this new folder in root
root/build/
├── lib.linux-x86_64-3.5
│   ├── c_integrate.cpython-35m-x86_64-linux-gnu.so
│   ├── integrator_modules
│   │   ├── cython_integrator.py
│   │   ├── __init__.py
│   │   ├── integrator.py
│   │   ├── numba_integrator.py
│   │   └── numpy_integrator.py
│   └── test
│   ├── __init__.py
│   └── test_integrator.py
The setup.py file looks like this
from setuptools import setup, Extension, find_packages
import numpy
setup(
name = "integrator_package",
author = "foo",
packages = find_packages(),
ext_modules = [Extension("c_integrate", ["c_integrate.c"])],
include_dirs=[numpy.get_include()],
)
My question is then: how do I write import statements of the functions from the .so file into ìntegrator_class.py in root and cython_integrator and test_integrator located in the build directory. Appending to sys.path seems like a quick and dirty solution that I don't much like.
EDIT:
As pointed out in the comments I haven't installed the package. This is because I don't know what to write to import from the .so file

In no specific order:
The file setup.py is typically located below the root of a project. Example:
library_name/
__init__.py
file1.py
setup.py
README
Then, the build directory appears alongside the project's source and not in the project source.
To import the file c_integrate.cpython-35m-x86_64-linux-gnu.so in Python, just import "c_integrate". The rest of the naming is taken care of automatically as it is just the platform information. See PEP 3149
A valid module is one of
a directory with a modulename/__init__.py file
a file named modulename.py
a file named modulename.PLATFORMINFO.so
of course located in the Python path. So there is no need for a __init__.py file for a compiled Cython module.
For your situation, move the Cython code in the project directory and either do a relative import import .c_integrate or a full from integrator_modules import c_integrate where the latter only works when your package is installed.
A few of this information can be found in my blog post on Cython modules http://pdebuyl.be/blog/2017/cython-module.html
I believe that this should let you build a proper package, comment below if not.
EDIT: to complete the configuration (see comments below), the poster also
Fixed the module path in the setup.py file so that it is the full module name starting from the PYTHONPATH: Extension("integrator_package.integrator_modules.c_integrat‌​or", ["integrator_package/integrator_modules/c_integrator.c"] instead of Extension("c_integrate", ["c_integrate.c"])]
Cythonize the module, build it and use with a same Python interpreter.
Further comment: the setup.py file can cythonize the file as well. Include the .pyx file instead of the .c file as the source.
cythonize(Extension('integrator_package.integrator_modules.c_integrat‌​or',
["integrator_package/integrator_modules/c_integrator.pyx"],
include_dirs=[numpy.get_include()]))

How to use the Python package inside a project

I have the following directory structure:
├── DynamicProgramming
│   ├── 0-1_kp_problem.py
│   ├── b.py
│   ├── largest_contigous_subarray.py
│   ├── longest_common_substring.py
│   ├── min_change_for_given_money.py
│   ├── optimal_matrix_chain.py
│   ├── Readme.md
│   └── wis.py
├── helper
│   ├── a.py
│   └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.

You could use something like:
from ..helper import a
See python docs on packages.

If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.

Python: include a third party library in a personal Python package

I would like to include a third party library into my Python script folder to distribute it all togehter (I am awary of the distribution license, and this library is fine to distribute). This is in order to avoid installing the library on another machine.
Saying I have a script (my_script.py), which calls this external library. I tried to copy this library from the site-packages subdirectory of Python directory into the directory where I have my files, but it seems not to be enough (I think th reason is in the __init__.py of this library which probably needs the folder to be in the PYTHONPATH).
Would it be reasonable to insert some lines of code in my_script.py to temporary append its folder to sys.path in order to make the all things working?
For instance, if I have a structure similar to this:
Main_folder
my_script.py
/external_lib_folder
__init__.py
external_lib.py
and external_lib_folder is the external library I copied from site-packages and inserted in my Main_folder, would it be fine if I write these lines (e.g.) in my_script.py?
import os,sys
main_dir = os.path.dirname(os.path.abspath(__file__))
sys.path.append(main_dir)
EDIT
I ended up choosing the sys.path.append solution. I added these lines to my my_script.py:
import os, sys
# temporarily appends the folder containing this file into sys.path
main_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)),'functions')
sys.path.append(main_dir)
Anyway, I chose to insert this as an edit in my question and accept the answer of Torxed because of the time he spent in helping me (and of course because his solution works as well).

Python3
import importlib.machinery, imp
namespace = 'external_lib'
loader = importlib.machinery.SourceFileLoader(namespace, '/home/user/external_lib_folder/external_lib.py')
external_lib = loader.load_module(namespace)
# How to use it:
external_lib.function(data_or_something)
This would be an ideal way to load custom paths in Python 3.
Not entirely sure this is what you wanted but It's relevant enough to post an alternative to adding to sys.path.
Python2
In python 2 you could just do (if i'm not mistaken, been a while since i used an older version of Python):
external_lib = __import__('external_lib_folder')
This does however require you to keep the __init__.py and a proper declaration of functions in sad script, otherwise it will fail.
**It's also important that the folder you're trying to import from is of the same name that the __init__.py script in sad folder is trying to import it's sub-libraries from, for instance geopy would be:
./myscript.py
./geopy/
./geopy/__init__.py
./geopy/compat.py
...
And the code of myscript.py would look like this:
handle = __import__('geopy')
print(handle)
Which would produce the following output:
[user#machine project]$ python2 myscript.py
<module 'geopy' from '/home/user/project/geopy/__init__.pyc'>
[user#machine project]$ tree -L 2
.
├── geopy
│   ├── compat.py
│   ├── compat.pyc
│   ├── distance.py
│   ├── distance.pyc
│   ├── exc.py
│   ├── exc.pyc
│   ├── format.py
│   ├── format.pyc
│   ├── geocoders
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── location.py
│   ├── location.pyc
│   ├── point.py
│   ├── point.pyc
│   ├── units.py
│   ├── units.pyc
│   ├── util.py
│   ├── util.pyc
│   └── version.pyc
└── myscript.py
2 directories, 20 files
Because in __init__.py of geopy, it's defined imports such as from geopy.point import Point which requires a namespace or a folder of geopy to be present.
There for you can't rename the folder to functions and place a folder called geopy in there because that won't work, nor will placing the contents of geopy in a folder called functions because that's not what geopy will look for.
Adding the path to sys.path (Py2 + 3)
As discussed in the comments, you can also add the folder to your sys.path variable prior to imports.
import sys
sys.path.insert(0, './functions')
import geopy
print(geopy)
>>> <module 'geopy' from './functions/geopy/__init__.pyc'>
Why this is a bad idea: It will work, and is used by many. The problems that can occur is that you might replace system functions or other modules might get loaded from other folders if you're not careful where you import stuff from. There for use .insert(0, ...) for most and be sure you actually want to risk replacing system built-ins with "shady" path names.

What you suggest is bad practice, it is a weak arrangement. The best solution (which is also easy to do) is to package it properly and add an explicit dependency, like this:
from setuptools import setup
setup(name='funniest',
version='0.1',
description='The funniest joke in the world',
url='http://github.com/storborg/funniest',
author='Flying Circus',
author_email='flyingcircus#example.com',
license='MIT',
packages=['funniest'],
install_requires=[
'markdown',
],
zip_safe=False)
This will work if the third party library is on pipy. If it's not, use this:
setup(
...
dependency_links=['http://github.com/user/repo/tarball/master#egg=package-1.0']
...
)
(See this explanation for packaging).

Cannot import modules using Virtualenv in Vagrant

This is my first time trying to set up a vagrant environment or a python virtuelenv, so forgive me if I am missing something basic.
Right now, I ssh into my vagrant box and in the home directory I have placed my venv folder. I have run
source venv/bin/activate
From my home directory I move to /vagrant, and within here I have my project files laid out something like this:
├──project
├── LICENSE
│
├── project
│   │   ├── exceptions.py
│   │   ├── __init__.py
│   │   ├── resources
│   │   │   ├── base.py
│   │   │   ├── __init__.py
│   │   └── target
│   │   └── __init__.py
│   │   └── test.py
│   ├── README.md
My problem is I am unable to import my modules in different directories. For example, if I am in /vagrant/project/project/target/test.py and I attempt:
import project.exceptions
I will get the error
ImportError: No module named project.exceptions
If I am in the /vagrant/project/project directory and I run
import exceptions
that works fine.
I have read up on similar problems people have experienced on StackOverflow.
Based on this question: Can't import package from virtualenv I have checked that my sys.executable path is the same in both my python interpreter as well as when I run a script (home/vagrant/venv/bin/python)
Based on this question: Import error with virtualenv. I have run ~/venv/bin/python directly and attempted to import, but the import still fails.
Let me know if there is more information I can provide. Thank you.

You have two options:
You can install your project into the virtual environment, by writing a setup.py file and by calling python setup.py install. See the Python Packaging User Guide.
You can set the PYTHONPATH environment variable to point to your project, like this:
$ export PYTHONPATH=$PYTHONPATH:/vagrant/project