Short summary
I can upload a package to pyPI, and install it with pip, but Python is not seeing it when I try to import it (it says there is no module with the relevant name). I am using Python 2.7 in Windows 7 (Anaconda distribution, using iPython interpreter).
Detailed summary
I am learning how to upload a package (mymathFoo) at the cheese shop (pyPI), and trying to make sure I can install it once it is uploaded. When I manually paste the package folder into my Lib/site-packages directory, it imports and works fine from my interpreter.
What I have done:
0. Write package
It's a silly little package with an add and subtract module (e.g., add(1,2)). The directory is structured as follows:
\mymathFoo
__init__.py
add.py #adds two numbers
subtract.py #subtract two numbers
setup.py #shown below
README.txt #simple description of package
The __init__.py file is as follows:
from add import add
from subtract import subtract
1. Register the package at pyPI
Namely, from the command line, I enter:
python setup.py register
Which returns:
running register
running check
Registering mymathFoo to http://pypi.python.org/pypi
Server response (200): OK
Note my setup.py file is:
from distutils.core import setup
setup(name="mymathFoo",
version="0.6.2",
url="http://mymathfoo.net",
maintainer='Math Foo',
maintainer_email='mathfoo#math.org',
py_modules=['add','subtract'],
description="Silly little math turd to add/subtract.")
Note if I change it to py_modules='mymathFoo' I get the same error as below.
2. Upload the package
At the command line, I entered:
python setup.py sdist bdist_wininst upload
And the response was:
running sdist
running check
warning: sdist: manifest template 'MANIFEST.in' does not exist (using default fi
le list)
writing manifest file 'MANIFEST'
creating mymathFoo-0.6.3
copying files to mymathFoo-0.6.3...
copying README.txt -> mymathFoo-0.6.3
copying add.py -> mymathFoo-0.6.3
copying setup.py -> mymathFoo-0.6.3
copying subtract.py -> mymathFoo-0.6.3
creating dist
creating 'dist\mymathFoo-0.6.3.zip' and adding 'mymathFoo-0.6.3' to it
adding 'mymathFoo-0.6.3\add.py'
adding 'mymathFoo-0.6.3\PKG-INFO'
adding 'mymathFoo-0.6.3\README.txt'
adding 'mymathFoo-0.6.3\setup.py'
adding 'mymathFoo-0.6.3\subtract.py'
removing 'mymathFoo-0.6.3' (and everything under it)
running bdist_wininst
running build
running build_py
creating build
creating build\lib
copying add.py -> build\lib
copying subtract.py -> build\lib
installing to build\bdist.win-amd64\wininst
running install_lib
creating build\bdist.win-amd64
creating build\bdist.win-amd64\wininst
creating build\bdist.win-amd64\wininst\PURELIB
copying build\lib\add.py -> build\bdist.win-amd64\wininst\PURELIB
copying build\lib\subtract.py -> build\bdist.win-amd64\wininst\PURELIB
running install_egg_info
Writing build\bdist.win-amd64\wininst\PURELIB\mymathFoo-0.6.3-py2.7.egg-info
creating 'c:\users\eric\appdata\local\temp\tmp65xxe2.zip' and adding '.' to it
adding 'PURELIB\add.py'
adding 'PURELIB\mymathFoo-0.6.3-py2.7.egg-info'
adding 'PURELIB\subtract.py'
removing 'build\bdist.win-amd64\wininst' (and everything under it)
running upload
Submitting dist\mymathFoo-0.6.3.zip to http://pypi.python.org/pypi
Server response (200): OK
Submitting dist\mymathFoo-0.6.3.win-amd64.exe to http://pypi.python.org/pypi
Server response (200): OK
Things seem to have worked. So far so good.
3. Install this package locally using pip.
Then I go to install this package with pip at the command line:
pip install mymathFoo
To which I get:
Downloading/unpacking mymathFoo
Downloading mymathFoo-0.6.3.zip
Running setup.py (path:c:\users\eric\appdata\local\temp\pip_build_Eric\mymathF
oo\setup.py) egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
Running the above causes the following directory to be copied into my Lib/site-packages folder:
mymathFoo-0.6.3-py2.7.egg-info
4. Import the package (not)
This is where I run into the problem. I open my iPython interpreter (using Anaconda distribution of Python, Windows 7):
import mymathFoo
I get:
Traceback (most recent call last):
File "<ipython-input-7-b7486b6a0225>", line 1, in <module>
import mymathFoo
ImportError: No module named mymathFoo
What am I missing? Why is my poor little module invisible?
Update
Note if I collapse all the files into the root directory (which I ultimately do not want to do), the errors go away. Unfortunately I will often want directories in my root directory, and nothing I have tried based on comments has fixed this.
I'm still looking for an answer, the discussion here looks promising:
http://www.scotttorborg.com/python-packaging/index.html#
I will work through that and post any solutions I find.
Discussion of related, but not the same, issues
"ImportError: No module named httplib2" even after installation
how to install package with pypi in python 2.7?
Note
This is based on some work I am doing with the book Python 101, by Michael Driscoll (it is in draft form right now).
Your package does install, just not the way you intended it to:
$ pip install mymathFoo
Downloading/unpacking mymathFoo
Using download cache from /Users/stu/tmp/pip-cache/http%3A%2F%2Fpypi.counsyl.com%2Froot%2Fpypi%2F%2Bf%2F64ef17a9135f05820c2d96050ce4315d%2FmymathFoo-0.6.3.zip
Running setup.py egg_info for package mymathFoo
Installing collected packages: mymathFoo
Running setup.py install for mymathFoo
Successfully installed mymathFoo
Cleaning up...
$ python
Type "help", "copyright", "credits" or "license" for more information.
>>> import add
>>> import subtract
>>>
Your setup.py says to install two modules, "add" and "subtract", and not to install a package named mymathFoo. Put add.py and subtract.py in a folder named 'mymathFoo', and set py_modules=['mymathFoo'] in setup.py.
Lastly, you can test your packaging without hitting pypi. Just run python setup.py install, and try to import you package.
it works for me:
>>> import mymathFoo
>>> dir(mymathFoo)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
though your module exports nothing. It looks like you changed your package since #Stu-Gla's answer, as you removed the add.py and subtract.py from sources.
By the way, you do not need to register a dummy package to pypi in order to test it, you can also use pip to install a package locally:
pip install /path/to/sources # path where the setup.py is
Related
Question
Is there a way to build a wheel for a package while in a different repository such that the wheel has been built exactly as it would be if you built the wheel inside of the repository containing the package?
Example
Consider the following repo:
/repo-containing-your-package
|___ your_module/
|___ setup.py
Build method A
When I run python setup.py bdist_wheel from within repo-containing-your-package it builds the wheel as expected, including your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), I can run python -m your_module.foo from the command line.
When the package is building, I get output that verifies that my module has been picked up by the wheel:
creating 'dist/your_module-#.#.#-py3-none-any.whl' and adding 'build/bar' to it
adding 'your_module/__init__.py'
etc...
Build method B
However, if I run python ../repo-containing-your-package/setup.py bdist_wheel from a repository that is a sibling to repo-containing-your-package, it does not build the wheel as expected, as it fails to include your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), attempting python -m your_module.foo fails:
Error while finding module specification for 'your_module.foo' (ModuleNotFoundError: No module named 'your_module')
The fact that the module has not been properly installed with the package is confirmed by reviewing the build output, which does not include the adding 'your_module' output that method A includes.
Two solutions I know of:
change working directory in setup.py
If you can modify the setup script, you can change the working directory programmatically. Add an os.chdir call early enough in the setup script:
import os
from setuptools import setup
os.chdir(os.path.dirname(__file__))
setup(...)
You can also change the working directory with other means without having to modify the setup script, e.g. in bash:
$ pushd path/to/repo; python setup.py bdist_wheel; popd
Use pip wheel
pip has a subcommand wheel that builds a wheel from the given arg; this arg is usually the name of the package, but can be a directory containing the setup script. Pass -e in that case so the wheel has the correct name:
$ pip wheel -e path/to/repo
I am following along with the O'Riley Head First Python (2nd Edition) Course.
At one point you will create a webapp and deploy it to pythonanywhere (chapter5).
The webapp uses two functions, imported from a module, created earlier.
The module is called vsearch.py. I also created a readme.txt and a setup.py and used setuptools to create a source distribution file using :
python3 setup.py sdist
The code of the setup.py read as follows:
from setuptools import setup
setup(
name = "vsearch",
version = "1.0",
description = "The Head First Python Seach Tools",
author = "HF Python 2e",
author_email = "hfpy2e#gmail.com",
url = "headfirstlabs.com",
py_modules = ["vsearch"],
)
The source distribution file gets created without errors and creates a file called vsearch-1.0.tar.gz
The file then gets uploaded to pythonanywhere and installed via console using:
python3 -m pip install vsearch-1.0.tar.gz --user
Console outputs:
15:36 ~/mysite $ python3 -m pip install vsearch-1.0.tar.gz --user
Looking in links: /usr/share/pip-wheels
Processing ./vsearch-1.0.tar.gz
Building wheels for collected packages: vsearch
Running setup.py bdist_wheel for vsearch ... done
Stored in directory: /home/Mohr/.cache/pip/wheels/85/fd/4e/5302d6f3b92e4057d341443ed5ef0402eb04994663282c12f7
Successfully built vsearch
Installing collected packages: vsearch
Found existing installation: vsearch 1.0
Uninstalling vsearch-1.0:
Successfully uninstalled vsearch-1.0
Successfully installed vsearch-1.0
Now when I try to run my webapp I get the following error:
2020-03-24 16:18:14,592: Error running WSGI application
2020-03-24 16:18:14,592: ModuleNotFoundError: No module named 'vsearch'
2020-03-24 16:18:14,593: File "/var/www/mohr_eu_pythonanywhere_com_wsgi.py", line 16, in <module>
2020-03-24 16:18:14,593: from vsearch4web import app as application # noqa
2020-03-24 16:18:14,593:
2020-03-24 16:18:14,593: File "/home/Mohr/mysite/vsearch4web.py", line 3, in <module>
2020-03-24 16:18:14,593: from vsearch import search4letters
Judging from this error I assume that "vsearch" can not be found because it was installed as "vsearch-1.0". However when I try to change this line to:
from vsearch-1.0 import search4letters
I rightfully get a synthax error since I can not adress modules this way. So what can I do about this? When creating the module in the beginning I added a version number to the setup.py file because according to the lecture it is good practice. Setuptools then automatically creates the source distribution file with the "-1.0" at the end. Also when importing it using the command shown above i automatically gets importet as "vsearch-1.0" which in turn I am unable to reference in my python code because of bad synthax.
Am I doing something wrong? Is there a way to import this under another namespace? Is there a way to reference "vsearch-1.0" in my python code without getting a synthax error?
There are different python3 versions installed on PythonAnywhere. When you install something using python3 -m pip or pip3 you use default python3 that is probably not matching python version setting of your web app. Use python3.7 and pip3.7 or python3.6 and pip3.6 etc. for --user installations to be sure.
pip install --user (with emphasized --user) installed the package into your user directory: /home/Mohr/.local/lib/pythonX.Y/site-packages/.
To run your WSGI application you probably use a virtual environment in which the user-installed modules are not available. To use modules in the venv you have to install everything in the venv. So activate the venv in a terminal and install the module with the venv's pip:
pip install vsearch-1.0.tar.gz
I am trying to figure how to install a Python package (ldap3) using a .egg file (fyi, the version of python-ldap3 we need does not have a .whl file) on a system that does not have easy_install available.
I found from an earlier thread (How can I install a .egg Python package on Windows (attempt using easy_install not working)) that I might be able to:
Untar the python-ldap3 tar.gz
Change dir to the untarred dir
run "python setup.py install"
So I tried the above, and it does seem to deploy/install the python-ldap3, but the installation hangs because it looks like it is trying to download something:
# python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to ldap3.egg-info/requires.txt
writing ldap3.egg-info/PKG-INFO
writing top-level names to ldap3.egg-info/top_level.txt
writing dependency_links to ldap3.egg-info/dependency_links.txt
reading manifest file 'ldap3.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'ldap3.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/ldap3
copying ./ldap3/__init__.py -> build/lib/ldap3
copying ./ldap3/version.py -> build/lib/ldap3
creating build/lib/ldap3/abstract
copying ./ldap3/abstract/cursor.py -> build/lib/ldap3/abstract
.
.
.
Installed /usr/lib/python2.6/site-packages/ldap3-2.5.1-py2.6.egg
Processing dependencies for ldap3==2.5.1
Searching for pyasn1>=0.1.8
Reading https://pypi.python.org/simple/pyasn1/
The "python setup.py install" hangs at that last line above ("Reading...."), but if I kill the process then check the python-ldap3 version, it looks like it installed all right:
# pip freeze | grep ldap3
ldap3==2.5.1
Also, FYI, our Python version is 2.6, and the pyasn1 is:
#pip freeze | grep pyasn1
pyasn1==0.4.3
I am not that familiar with Python and Python packaging, so I was wondering:
Is it all right to do the ldap3 2.5.1 installation per above (using
the 'python setup.py install', and then just killing the script)?
If not, what would be a safer/better way to get the ldap3 2.5.1 installed?
Thanks in advance!
Jim
EDIT: I forgot to mention that I think that the reason the script hung at the point it did was that we have to do this installation on an "isolated network", with no access to the external Internet, so we actually prefer/need to do the ldap3 installation without any external access.
EDIT 2: I guess that one other thing that is puzzling is why is the "python setup.py install" trying to get "pyasn1>=0.1.8" when "0.4.3" is already installed, since, from the release history, https://pypi.org/project/pyasn1/#history, "0.1.8" is OLDER than "0.4.3"?
EDIT 3: TO be clear, FYI, the package I am working with it the "ldap3" package, from https://github.com/cannatag/ldap3.
I am trying to build a pulp distributor plugin which will execute a bash script containing arbitrary code so that I may trigger actions after an RPM repo is published.
These plugins are generally created using distutils. However, when I attempt to install my module, I receive the error:
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
Typically, this means that the working directory is incorrect or __init.py__ is missing. In my case however, I am attempting to install from the correct working directory and I did create __init.py__ files (see the repo here).
I suspect that I am running into a pathing issue having something to do with my code being in a subdirectory so far removed from setup.py. What am I doing wrong? Why won't my module install?
When you face errors like this, one of the first things to check is what packages are actually being added to your distribution when you build it. In your case the package list is empty, but should contain at least the pulp_hook package:
$ python -c "from setuptools import find_packages; print(find_packages())"
[]
So why does setuptools not recognize pulp_hook as a regular package? Look at its structure: you have added file named __init.py__, but its name should be __init__.py. Once you rename the files, the pulp_hook and its subdirectories become regular packages:
$ python -c "from setuptools import find_packages; print(find_packages())"
['pulp_hook', 'pulp_hook.plugins', 'pulp_hook.plugins.distributors']
Now build/lib will be created because now distutils finds at least one package to install:
$ python setup.py install_lib
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/pulp_hook
copying pulp_hook/__init__.py -> build/lib/pulp_hook
creating build/lib/pulp_hook/plugins
copying pulp_hook/plugins/__init__.py -> build/lib/pulp_hook/plugins
creating build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/distributionhook.py -> build/lib/pulp_hook/plugins/distributors
copying pulp_hook/plugins/distributors/__init__.py -> build/lib/pulp_hook/plugins/distributors
I'm trying to build a Python package for this project that includes the .json files in /fixtures and the *.html files in /templates. When I run the packaging script based off my setup.py, it says it should include those folders in the MANIFEST it generates, but the output just says it's hard-linking them and when I try to install via pip, it dies on the following:
Downloading/unpacking bluetrain
Running setup.py egg_info for package bluetrain
error: package directory 'bluetrain/fixtures' does not exist
Complete output from command python setup.py egg_info:
running egg_info
writing pip-egg-info/bluetrain.egg-info/PKG-INFO
writing top-level names to pip-egg-info/bluetrain.egg-info/top_level.txt
writing dependency_links to pip-egg-info/bluetrain.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
error: package directory 'bluetrain/fixtures' does not exist
EDIT: It looks like this was an issue with legacy crud that built up in the virtualenv I was installing succeeding versions of the package into. Trying a clean virtualenv fixed the issue.
Fixed this issue by creating a clean virtualenv and trying to build from there. It appears the constant build/ rebuild cycle in the original environment left legacy files behind that were stopping things from working as intended.