We Keep Coding

Module works in install mode but not in develop mode using setuptools

I am facing a similar issue as described in: Module found in install mode but not in develop mode using setuptools. But the solution is not applicable as I have multiple package_dir instead of one. Similarly, this problem doesn't not occur when using python3 setup.py install.
My setup.py looks like:
from setuptools import setup
setup(name='my-project',
version='0.1',
description='My project description',
author='Rishab Manocha',
package_dir={'': 'module_a/src', 'module_b': 'module_b/src'},
packages=[
'module_a1',
'module_a2',
'module_b',
'module_b.module_1b',
'module_b.module_2b',
],
)
When running python3 setup.py develop, the .egg-link only links back to the module_a/src and apparently ignores the module_b/src completely. This is the output of command python3 setup.py develop:
running develop
running egg_info
writing module_a/src/my-project.egg-info/PKG-INFO
writing dependency_links to module_a/src/my-project.egg-info/dependency_links.txt
writing top-level names to module_a/src/my-project.egg-info/top_level.txt
reading manifest file 'module_a/src/my-project.egg-info/SOURCES.txt'
writing manifest file 'module_a/src/my-project.egg-info/SOURCES.txt'
running build_ext
Creating /usr/local/lib/python3.7/site-packages/orca-airflow.egg-link (link to module_a/src)
Removing orca-airflow 0.1 from easy-install.pth file
Adding orca-airflow 0.1 to easy-install.pth file```

Run Makefile on pip install

I have some protocol buffer definitions which need to be built to Python source as part of the pip install process. I've subclassed the setuptools.command.install command in setup.py but I think it's trying to run the Makefile after the package is installed so the sources aren't recognised.
I can't find information about what happens during a pip installation. Can anyone shed any light?
setup.py:
import subprocess
import sys
from setuptools import setup
from setuptools.command.install import install
class Install(install):
"""Customized setuptools install command - builds protos on install."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
install.run(self)
setup(
name='myprotos',
version='0.0.1',
description='Protocol Buffers.',
install_requires=[],
cmdclass={
'install': Install,
}
)
Output of $ pip install -vvv .:
Processing /path/to/myprotos
Running setup.py (path:/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py) egg_info for package from file:///path/to/myprotos
Running command python setup.py egg_info
running egg_info
creating pip-egg-info/myprotos.egg-info
writing pip-egg-info/myprotos.egg-info/PKG-INFO
writing top-level names to pip-egg-info/myprotos.egg-info/top_level.txt
writing dependency_links to pip-egg-info/myprotos.egg-info/dependency_links.txt
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
reading manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/myprotos.egg-info/SOURCES.txt'
Source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build has version 0.0.1, which satisfies requirement myprotos==0.0.1 from file:///path/to/myprotos
Building wheels for collected packages: myprotos
Running setup.py bdist_wheel for myprotos: started
Destination directory: /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel-
Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/tmpD7dfGKpip-wheel- --python-tag cp27
running bdist_wheel
running build
installing to build/bdist.macosx-10.12-x86_64/wheel
running install
# THIS IS MY MAKEFILE RUNNING
Grabbing github.com/google/protobuf...
Building Python protos...
# MAKEFILE COMPLETE
running install_egg_info
running egg_info
creating myprotos.egg-info
writing myprotos.egg-info/PKG-INFO
writing top-level names to myprotos.egg-info/top_level.txt
writing dependency_links to myprotos.egg-info/dependency_links.txt
writing manifest file 'myprotos.egg-info/SOURCES.txt'
reading manifest file 'myprotos.egg-info/SOURCES.txt'
writing manifest file 'myprotos.egg-info/SOURCES.txt'
Copying myprotos.egg-info to build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1-py2.7.egg-info
running install_scripts
creating build/bdist.macosx-10.12-x86_64/wheel/myprotos-0.0.1.dist-info/WHEEL
Running setup.py bdist_wheel for myprotos: finished with status 'done'
Stored in directory: /Users/jds/Library/Caches/pip/wheels/92/0b/37/b5a50146994bc0b6774407139f01d648ba3a9b4853d2719c51
Removing source in /private/var/folders/3t/4qwkfyr903d0b7db7by2kj6r0000gn/T/pip-jpgCby-build
Successfully built myprotos
Installing collected packages: myprotos
Found existing installation: myprotos 0.0.1
Uninstalling myprotos-0.0.1:
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/DESCRIPTION.rst
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/INSTALLER
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/METADATA
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/RECORD
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/WHEEL
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/metadata.json
Removing file or directory /usr/local/lib/python2.7/site-packages/myprotos-0.0.1.dist-info/top_level.txt
Successfully uninstalled myprotos-0.0.1
Successfully installed myprotos-0.0.1
Cleaning up...
Should my Makefile be running early in the process to generate the source files? Do the files need to be there before egg_info runs for example?
If I manually run the Makefile and then install the package then it works.
Update
Here is the structure of my project:
myprotos
├── Makefile
├── README.md
├── document.proto
├── myprotos # Generated by Makefile
│ ├── __init__.py # Generated by Makefile
│ └── proto_pb2.py # Generated by Makefile
└── setup.py
Here is the section of the Makefile which generates the Python source from Potocol Buffer definitions:
python: protoc deps
# the protoc and deps command above just downloads
# the `protoc` binary to a local bin directory
#echo "Building Python protos..."
#mkdir -p "${PYTHON_OUT}"
#touch "${PYTHON_OUT}"/__init__.py
#printf "__all__ = ['proto_pb2']" > "${PYTHON_OUT}"/__init__.py
#PATH="${LOCAL_BINARY_PATH}:$$PATH" protoc \
--proto_path="${BASE}" \
--proto_path="${GOPATH}/src/github.com/google/protobuf/src" \
--python_out="${PYTHON_OUT}/" \
${PROTOS}

OK, there are three things you need to change here:
Add Makefile and document.proto to a new file MANIFEST.in.
Makefile
document.proto
If you do that, the .zip file created by python setup.py sdist (which is also uploaded to PyPI) will contain those files.
You need to run your make command during python setup.py build, not during install. Since you are generating Python code, you will need to change the build_py command here:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_py import build_py
class Build(build_py):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
cmdclass={
'build_py': Build,
}
)
If your Makefile generated machine code, i.e. from C or any other compiled language, you should change the build_ext command:
import sys
import subprocess
from setuptools import setup
from setuptools.command.build_ext import build_ext
class Build(build_ext):
"""Customized setuptools build command - builds protos on build."""
def run(self):
protoc_command = ["make", "python"]
if subprocess.call(protoc_command) != 0:
sys.exit(-1)
super().run()
setup(
name='buildtest',
version='1.0',
description='Python Distribution Utilities',
packages=['buildtest'],
has_ext_modules=lambda: True,
cmdclass={
'build_ext': Build,
}
)
Lastly, you need to tell setuptools to install the resulting package on install by defining an attribute packages in setup():
setup(
...
packages=['myprotos']
)
The reason for deciding to run build_py or build_ext lies in the situation when the two are run:
build_py is also run when creating source distributions, which have to be cross-platform. Compiled extensions are usually not cross-platform, so you cannot compile them in this step.
build_ext is only run when you are creating binary distributions, which are platform-specific. It is OK to compile to platform-specific machine code here.

Installed module is empty

I'm tring to use setuptools for python3 code. My project structure:
./testSetup/
./testSetup/testSetup
./testSetup/testSetup/foo.py
./testSetup/Setup.py
./testSetup/testSetup/foo.py content:
def say_foo():
print('foo')
./testSetup/Setup.py content:
from setuptools import setup, find_packages
import testSetup
setup(
name='testSetup',
version='0.0.1',
packages=find_packages(),
author='Bastien Sevajol',
author_email="testSetup#bux.fr",
description='test',
long_description='test test',
include_package_data=False,
url='http://bux.fr',
# https://pypi.python.org/pypi?%3Aaction=list_classifiers.
classifiers=[
"Programming Language :: Python",
]
)
When i python Setup.py install (with python3.4) in a virtual env:
python Setup.py install
running install
running bdist_egg
running egg_info
creating testSetup.egg-info
writing testSetup.egg-info/PKG-INFO
writing dependency_links to testSetup.egg-info/dependency_links.txt
writing top-level names to testSetup.egg-info/top_level.txt
writing manifest file 'testSetup.egg-info/SOURCES.txt'
reading manifest file 'testSetup.egg-info/SOURCES.txt'
writing manifest file 'testSetup.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
creating build
creating build/bdist.linux-i686
creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/EGG-INFO
copying testSetup.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
copying testSetup.egg-info/SOURCES.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying testSetup.egg-info/dependency_links.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying testSetup.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/testSetup-0.0.1-py3.4.egg' and adding 'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
Processing testSetup-0.0.1-py3.4.egg
Copying testSetup-0.0.1-py3.4.egg to /home/bux/.virtualenvs/testsynergine2/lib/python3.4/site-packages
Adding testSetup 0.0.1 to easy-install.pth file
Installed /home/bux/.virtualenvs/testsynergine2/lib/python3.4/site-packages/testSetup-0.0.1-py3.4.egg
Processing dependencies for testSetup==0.0.1
Finished processing dependencies for testSetup==0.0.1
I can see a warning saying install_lib: 'build/lib' does not exist -- no Python modules to install.
If i try to use my module:
python
Python 3.4.0 (default, Apr 11 2014, 13:05:18)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from testSetup import foo
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'foo'
>>> import testSetup
>>> dir(testSetup)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
What's wrong in my Setup.py ?

You need to have an __init__.py file in your TestSetup/ directory (the file can be blank). Otherwise, nothing in that directory will be importable and find_packages() will never find it. You might want to read about it.
./testSetup/
./testSetup/testSetup
./testSetup/testSetup/__init__.py
./testSetup/testSetup/foo.py
./testSetup/setup.py
Also note that setup.py should be all lowercase. While naming it Setup.py won't stop the script from working, it is not convention and may not be discovered by various tools.

It is because of setuptools.find_packages, when calling it without any parameters it looks for packages in the source tree from the location of setup.py. In your case you don't have any folder at the root level.
The issue does not come from your setup.py file but from the way you've organized your project, by moving your sources around like so:
testSetup
|--setup.py
|--/testSetup/foo.py
|--/testSetup/__init__.py
then you can successfully pip setup.py install when cd-ing inside testSetup.
some doc on find_packages

How should I do to install all dependencies pypi?

I wrote a package available in pypi, python repository and it depends of other packages as I show with the following code of setup.py file.
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
setup(
name='aTXT',
packages=['aTXT'],
# package_data={ '':['*.py'],
# 'bin': ['bin/*'], 'docx': ['docx/*'], 'pdfminer': ['pdfminer']},
version=VERSION,
include_package_data=True,
# arbitrary keywords
install_requires=[
'lxml>=3.2.3',
'docx>=0.2.0',
'pdfminer',
'docopt>=0.6.2',
'PySide',
'kitchen>=1.1.1',
'scandir>=0.8'
],
requires=['docopt', 'scandir', 'lxml', 'PySide', 'kitchen'],
)
When I'd tried to install from pip with:
pip install aTXT
If some of the requirements package are not installed, it raise a Import Error.
But, why not pip try to install all dependencies?
The following is an example if I don't have lxml package installed.
ImportError: No module named lxml
Complete output from command python setup.py egg_info:
Traceback (most recent call last):

Installing my own module from pyPI, using pip, is not working

Short summary
I can upload a package to pyPI, and install it with pip, but Python is not seeing it when I try to import it (it says there is no module with the relevant name). I am using Python 2.7 in Windows 7 (Anaconda distribution, using iPython interpreter).
Detailed summary
I am learning how to upload a package (mymathFoo) at the cheese shop (pyPI), and trying to make sure I can install it once it is uploaded. When I manually paste the package folder into my Lib/site-packages directory, it imports and works fine from my interpreter.
What I have done:
0. Write package
It's a silly little package with an add and subtract module (e.g., add(1,2)). The directory is structured as follows:
\mymathFoo
__init__.py
add.py #adds two numbers
subtract.py #subtract two numbers
setup.py #shown below
README.txt #simple description of package
The __init__.py file is as follows:
from add import add
from subtract import subtract
1. Register the package at pyPI
Namely, from the command line, I enter:
python setup.py register
Which returns:
running register
running check
Registering mymathFoo to http://pypi.python.org/pypi
Server response (200): OK
Note my setup.py file is:
from distutils.core import setup
setup(name="mymathFoo",
version="0.6.2",
url="http://mymathfoo.net",
maintainer='Math Foo',
maintainer_email='mathfoo#math.org',
py_modules=['add','subtract'],
description="Silly little math turd to add/subtract.")
Note if I change it to py_modules='mymathFoo' I get the same error as below.
2. Upload the package
At the command line, I entered:
python setup.py sdist bdist_wininst upload
And the response was:
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default fi
le list)
writing manifest file 'MANIFEST'
creating mymathFoo-0.6.3
copying files to mymathFoo-0.6.3...
copying README.txt -> mymathFoo-0.6.3
copying add.py -> mymathFoo-0.6.3
copying setup.py -> mymathFoo-0.6.3
copying subtract.py -> mymathFoo-0.6.3
creating dist
creating 'dist\mymathFoo-0.6.3.zip' and adding 'mymathFoo-0.6.3' to it
adding 'mymathFoo-0.6.3\add.py'
adding 'mymathFoo-0.6.3\PKG-INFO'
adding 'mymathFoo-0.6.3\README.txt'
adding 'mymathFoo-0.6.3\setup.py'
adding 'mymathFoo-0.6.3\subtract.py'
removing 'mymathFoo-0.6.3' (and everything under it)
running bdist_wininst
running build
running build_py
creating build
creating build\lib
copying add.py -> build\lib
copying subtract.py -> build\lib
installing to build\bdist.win-amd64\wininst
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wininst
creating build\bdist.win-amd64\wininst\PURELIB
copying build\lib\add.py -> build\bdist.win-amd64\wininst\PURELIB
copying build\lib\subtract.py -> build\bdist.win-amd64\wininst\PURELIB
running install_egg_info
Writing build\bdist.win-amd64\wininst\PURELIB\mymathFoo-0.6.3-py2.7.egg-info
creating 'c:\users\eric\appdata\local\temp\tmp65xxe2.zip' and adding '.' to it
adding 'PURELIB\add.py'
adding 'PURELIB\mymathFoo-0.6.3-py2.7.egg-info'
adding 'PURELIB\subtract.py'
removing 'build\bdist.win-amd64\wininst' (and everything under it)
running upload
Submitting dist\mymathFoo-0.6.3.zip to http://pypi.python.org/pypi
Server response (200): OK
Submitting dist\mymathFoo-0.6.3.win-amd64.exe to http://pypi.python.org/pypi
Server response (200): OK
Things seem to have worked. So far so good.
3. Install this package locally using pip.
Then I go to install this package with pip at the command line:
pip install mymathFoo
To which I get:
Downloading/unpacking mymathFoo
Downloading mymathFoo-0.6.3.zip
Running setup.py (path:c:\users\eric\appdata\local\temp\pip_build_Eric\mymathF
oo\setup.py) egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
Running the above causes the following directory to be copied into my Lib/site-packages folder:
mymathFoo-0.6.3-py2.7.egg-info
4. Import the package (not)
This is where I run into the problem. I open my iPython interpreter (using Anaconda distribution of Python, Windows 7):
import mymathFoo
I get:
Traceback (most recent call last):
File "<ipython-input-7-b7486b6a0225>", line 1, in <module>
import mymathFoo
ImportError: No module named mymathFoo
What am I missing? Why is my poor little module invisible?
Update
Note if I collapse all the files into the root directory (which I ultimately do not want to do), the errors go away. Unfortunately I will often want directories in my root directory, and nothing I have tried based on comments has fixed this.
I'm still looking for an answer, the discussion here looks promising:
http://www.scotttorborg.com/python-packaging/index.html#
I will work through that and post any solutions I find.
Discussion of related, but not the same, issues
"ImportError: No module named httplib2" even after installation
how to install package with pypi in python 2.7?
Note
This is based on some work I am doing with the book Python 101, by Michael Driscoll (it is in draft form right now).

Your package does install, just not the way you intended it to:
$ pip install mymathFoo
Downloading/unpacking mymathFoo
Using download cache from /Users/stu/tmp/pip-cache/http%3A%2F%2Fpypi.counsyl.com%2Froot%2Fpypi%2F%2Bf%2F64ef17a9135f05820c2d96050ce4315d%2FmymathFoo-0.6.3.zip
Running setup.py egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
$ python
Type "help", "copyright", "credits" or "license" for more information.
>>> import add
>>> import subtract
>>>
Your setup.py says to install two modules, "add" and "subtract", and not to install a package named mymathFoo. Put add.py and subtract.py in a folder named 'mymathFoo', and set py_modules=['mymathFoo'] in setup.py.
Lastly, you can test your packaging without hitting pypi. Just run python setup.py install, and try to import you package.

it works for me:
>>> import mymathFoo
>>> dir(mymathFoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
though your module exports nothing. It looks like you changed your package since #Stu-Gla's answer, as you removed the add.py and subtract.py from sources.
By the way, you do not need to register a dummy package to pypi in order to test it, you can also use pip to install a package locally:
pip install /path/to/sources # path where the setup.py is