packaging python projects failed - python

I want to package my python code and upload it to PyPI so that people can use it easily. I followed the documentation for packaging python projects and eventually uploaded it to the PyPI test website. I ran pip install to try and install it.
Strangely enough, after installing it, I couldn't find the package:
(base) ➜ ~ python3 -m pip install --index-url https://test.pypi.org/simple/ oomstore==0.0.4
Looking in indexes: https://test.pypi.org/simple/
Collecting oomstore==0.0.4
Downloading https://test-files.pythonhosted.org/packages/4f/a5/4e7089a1ecb36a59f7f0852a5f96a6054daf886d97132060a7efcda5f04f/oomstore-0.0.4-py3-none-any.whl (12 kB)
Installing collected packages: oomstore
Successfully installed oomstore-0.0.4
(base) ➜ ~ python3
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import oomstore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'oomstore'
>>>
I went to the installation path of the package and found no python files in it:
(base) ➜ ~ cd ~/miniconda3/lib/python3.8/site-packages/oomstore-0.0.4.dist-info
(base) ➜ oomstore-0.0.4.dist-info ls
INSTALLER LICENSE METADATA RECORD REQUESTED WHEEL top_level.txt
(base) ➜ oomstore-0.0.4.dist-info
Did I do something wrong? Is there something wrong with my setup.cfg file? Forgive me for asking such an ignorant question, I'm new to python...

The problem is that your package_dir option is telling setuptools to look inside the oomstore directory to find modules and packages, but your oomstore package is right there next to setup.cfg. You should remove the option.
You could also move oomstore into an src directory and configure package_dir =\n = src; see this articles for reasons to put modules in an src directory: https://hynek.me/articles/testing-packaging/

Related

/usr/bin/pip3 does not install package globally when in venv. How to do it?

After creating and activating a virtual environment:
python3 -m venv env
. ./env/bin/activate
the global /usr/bin/pip3 does not install a package globally. Actually it does copy package files but package is not fully installed:
/usr/bin/pip3 install docker
Collecting docker
Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
|████████████████████████████████| 146 kB 2.5 MB/s
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.22.0)
Collecting websocket-client>=0.32.0
Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
|████████████████████████████████| 52 kB 2.9 MB/s
Installing collected packages: websocket-client, docker
Successfully installed docker-5.0.3 websocket-client-1.2.1
Files are placed globally:
ls /usr/lib/python3.8/site-packages
docker docker-5.0.3.dist-info websocket websocket_client-1.2.1.dist-info
But neither a shell without venv active doesn't consider docker as installed:
pip3 list|grep docker
python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
nor the venv sees docker package as installed:
(env) pip3 list|grep docker
(env) python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
Is this a bug?
How can I install a package globally with pip3 while a venv is active?
Why do I want to do this?
Ansible is launched from a venv and it targets the localhost. So if I'm using just pip3 as an executable, it uses the binary from the venv. If I'm using /usr/bin/pip3 I get the above behavior.
It's a more complex open source project and I don't have the option to not run Ansible from a venv or not to target localhost.
Thanks
I don't use venv myself, but the whole point of virtual environments is having a set of files separated from the rest of the system.
If you create a virtual environment and are inside of it you shouldn't be using /usr/bin/pip3, you should be using the default pip3, since those two versions might not even be compatible with each other.~
It's a more complex open source project and I don't have the option to not run Ansible from a venv
You always have the option of running something inside a virtual environment, it might happen that the program does not respect that and use global paths and ignore that virtual environment, which shouldn't be happening.
From the venv docs:
Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories
This means that you should create your virtual environment, activate it, install whatever you want inside of it, and then run everything inside that virtual environment.
So I don't see this as a bug, it just seems like you are not using as it is intended to be used. If you want something installed globally, why are you using a virtual environment in the first place?

Cannot install netaddr from MacBook

In Linux, I just simply run pip install netaddr and everything works just fine. However, in Mac OS X version 10.15.4, this doesn't seem to be as straight-forward:
[myuser:my-macbook-pro:~]$ sudo pip install netaddr 2:33PM/03.30
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/Users/myuser/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
WARNING: The directory '/Users/myuser/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting netaddr
Downloading https://files.pythonhosted.org/packages/ba/97/ce14451a9fd7bdb5a397abf99b24a1a6bb7a1a440b019bebd2e9a0dbec74/netaddr-0.7.19-py2.py3-none-any.whl (1.6MB)
|████████████████████████████████| 1.6MB 1.6MB/s
Installing collected packages: netaddr
Successfully installed netaddr-0.7.19
[myuser:my-macbook-pro:~]$ python 2:33PM/03.30
Python 2.7.15 (default, Feb 12 2019, 11:00:12)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import netaddr
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named netaddr
>>>
Still doesn't seem to work even when running the exact same command with sudo permissions.
As of now, I continue having to fire up a VM to run this script that only requires this one module.
use virtualenv and install it from there
https://docs.python-guide.org/dev/virtualenvs/

The program scss is not currently installed

I have been trying to install scss from here. After sucessfully running pip install scss, I then tried to run scss as follows:
scss -i
However, I then got the error that
The program 'scss' is currently not installed. You can install it by typing:
sudo apt-get install ruby-sass
This is confusing because pip says that scss was installed correctly, and that the commands it tells me to install scss with is not the actual way to install scss. How can I install scss?
Edit: This is a ubuntu platform and doing import scss from the command line resulted in the error ImportError: No module named scss.
Make sure you have the original SCSS/SASS installed from here.
On Ubuntu, if I do the following, the python version of scss appears to work fine (mind you, I don't have experience with this module).
$ virtualenv so1
New python executable in so1/bin/python
Installing setuptools, pip...done.
$ cd so1
$ source bin/activate
$ pip install scss
Downloading/unpacking scss
Downloading scss-0.8.73-py2.py3-none-any.whl
Downloading/unpacking pyparsing (from scss)
Downloading pyparsing-2.1.5-py2.py3-none-any.whl (42kB): 42kB downloaded
Installing collected packages: scss, pyparsing
Successfully installed scss pyparsing
Cleaning up...
(so1)$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import scss
>>>
(so1)$ which scss
<path>/so1/bin/scss
(so1)so1$ scss -i
SCSS v. 0.8.73 interactive mode
================================
Ctrl+D or quit for exit
>>>
Bye bye.
(so1)$
I've edited-out the paths above on my machine as those aren't relevant.
There should be no dependency on the ruby version of scss, as far as I can tell. The only requirements are as stated in the python-scss documentation:
> python >= 2.5
> pyparsing >= 1.5.5
As I mentioned in the comments, your pip install scss probably failed for some reason. You'll probably want to capture the log as suggested here.

Python virtualenv picks up wrong library

$ virtualenv --version
13.0.3
I create a new virtualenv with Python3 without access to global site packages.
$ virtualenv --no-site-packages venv_pygments --python=/usr/local/bin/python3
Running virtualenv with interpreter /usr/local/bin/python3
Using base prefix '/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4'
New python executable in venv_pygments/bin/python3.4
Also creating executable in venv_pygments/bin/python
Installing setuptools, pip, wheel...done.
Then I use the virtualenv's Python3 interpreter and try to import pygments
$ cd venv_pygments
$ venv_pygments bin/python3
Python 3.4.3 (default, May 1 2015, 19:14:18)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.49)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Applications/QGIS.app/Contents/Resources/python/pygments/__init__.py", line 46
except TypeError, err:
^
SyntaxError: invalid syntax
pip freeze though only shows these packages
$ bin/pip freeze
wheel==0.24.0
So it seems like the virtualenv's Python3 is accessing global site packages. How can I avoid that?
It doesn't change when I install pygments for the virtualenv
$ bin/pip install pygments
Collecting pygments
Using cached Pygments-2.0.2-py3-none-any.whl
Installing collected packages: pygments
Successfully installed pygments-2.0.2
$ venv_pygments bin/python3
Python 3.4.3 (default, May 1 2015, 19:14:18)
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.49)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Applications/QGIS.app/Contents/Resources/python/pygments/__init__.py", line 46
except TypeError, err:
^
SyntaxError: invalid syntax
>>>
Note: The original question was updated based on the comments.
Create isolated environment without access to global site-packages. Either enable/disable that behaviour intentionally. Recent versions disable access by default.
I assume you want an isolated environment to test Python 3.3.4.
Below I am using virtualenv provided by system python 2.7.6
$ virtualenv --version
1.11.2
$ virtualenv
You must provide a DEST_DIR
Usage: virtualenv [OPTIONS] DEST_DIR
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
-p PYTHON_EXE, --python=PYTHON_EXE
The Python interpreter to use, e.g.,
--python=python2.5 will use the python2.5 interpreter
to create the new environment. The default is the
interpreter that virtualenv was installed with
(/usr/bin/python)
--clear Clear out the non-root install and start from scratch.
--no-site-packages DEPRECATED. Retained only for backward compatibility.
Not having access to global site-packages is now the
default behavior.
--system-site-packages
Give the virtual environment access to the global
site-packages.
--always-copy Always copy files rather than symlinking.
--unzip-setuptools Unzip Setuptools when installing it.
--relocatable Make an EXISTING virtualenv environment relocatable.
This fixes up scripts and makes all .pth files
relative.
--no-setuptools Do not install setuptools (or pip) in the new
virtualenv.
--no-pip Do not install pip in the new virtualenv.
--extra-search-dir=DIR
Directory to look for setuptools/pip distributions in.
This option can be used multiple times.
--never-download DEPRECATED. Retained only for backward compatibility.
This option has no effect. Virtualenv never downloads
pip or setuptools.
--prompt=PROMPT Provides an alternative prompt prefix for this
environment.
--setuptools DEPRECATED. Retained only for backward compatibility.
This option has no effect.
--distribute DEPRECATED. Retained only for backward compatibility.
This option has no effect.
I create a virtualenv, change into the directory and use relative paths to address python interpreter and pip from inside this virtualenv.
$ virtualenv --no-site-packages venv_pygments
New python executable in venv_pygments/bin/python
Installing setuptools, pip...done.
$ cd venv_pygments/
$ bin/pip install pygments
Downloading/unpacking pygments
Downloading Pygments-2.0.2-py2-none-any.whl (672kB): 672kB downloaded
Installing collected packages: pygments
Successfully installed pygments
Cleaning up...
$ bin/python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pygments
>>> pygments.__version__
'2.0.2'
While creating your virtualenv you need to keep using the option
--python=/usr/local/bin/python3
to refer to your Python 3.3.4 interpreter as you already do.
A recent default virtualenv usually has only a few packages installed. If yours has more you may have access to global site-packages.
$ bin/pip freeze
Pygments==2.0.2
argparse==1.2.1
wsgiref==0.1.2

Unable to import praw module after install

Edit: Before you start deleting/modifying installs, please glance over StvnW's answer/summary to make sure you are applying the solution that is appropriate for you.
I've installed python 2.7.5 and pip [edit: mac OSX Mountain Lion.] I've run "pip install praw" in terminal. All good. When I run python and run "import praw" I get:
...$ python
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import praw
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named praw
...same ImportError from a script.
when I install praw I get this:
$ pip install praw
Downloading/unpacking praw
Downloading praw-2.1.10.tar.gz (83kB): 83kB downloaded
Running setup.py egg_info for package praw
Requirement already satisfied (use --upgrade to upgrade): requests>=1.2.0 in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): update-checker>=0.6 in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): setuptools in
/usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from update-
checker>=0.6->praw)
Installing collected packages: praw
Running setup.py install for praw
Installing praw-multiprocess script to /usr/local/bin
Successfully installed praw
Cleaning up...
In python if I run help('modules') it isn't there.
Relatively new to python and I haven't been able to sort this out with google search. Any hints would be much appreciated.
Edit:
SitRep:
I've uninstalled 2.7.2, uninstalled praw, and uninstalled (homebrew) pip. I ran python 2.7.5 and it couldn't find the module (as you would suspect.) I then reinstalled pip with easy_install and now 2.7.5 is finding praw but giving this error:
$ python
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import praw
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/praw/__init__.py", line 43, in <module>
from update_checker import update_check
File "/Library/Python/2.7/site-packages/update_checker.py", line 11, in <module>
from pkg_resources import parse_version as V
ImportError: No module named pkg_resources
>>>
Thanks for the help so far, the spurious install was the root of problem 1.
The solution to the final problem can be found here:
No module named pkg_resources
I wish there were some way to summarize this for future readers, but I've done so many things that I no longer recall what addressed what. I basically uninstalled everything (python 2.7.2, 2.7.5, praw, pip,) reinstalled 2.7.5 from http://python.org/download/, reinstalled pip with easy_install (and sudo command) instead of homebrew, reinstalled praw with sudo command, and followed the directions for the subsequent module error in the link above. Hope that helps. :)
The OS X system versions of Python live in /System/Library/Frameworks/Python.Framework/<verson>, each of which links to /Library/Python/<version>/site-packages. Seems like you have installed another version of python and/or pip to /usr/local, but when you invoke python you are still getting the system version.
Try this:
$ /usr/local/bin/python
>>> import praw
I'd also recommend looking into tools like pyenv and virtualenv if you are going to do any amount of work with Python. Pyenv lets you easily manage and switch between multiple versions (including micro versions x.x.3, x.x.5, etc). Virtualenv lets you create isolated Python environments where you can pin down versions of site-packages specific to a particular project.
Edit (summarizing):
sudo easy_install pip will install pip under /System/Library/Python/<version>. Calling that pip will install packages to /Library/Python/<version>/site-packages
brew install python will install a second version of python — including pip — under /usr/local/. That pip will put packages in /usr/local/lib/python<version>/site-packages/`.
One version may not see packages installed to the other.
which python and which pip are helpful for troubleshooting.
Dean's final solution above results in user packages being installed to the system site-packages.
Despite Dean's choice to update the system Python, many would advocate instead using brew, pyenv, and virtualenv to isolate oneself from the system Python.
It's strange. I am suspecting you have another version of python2.7 on your mac os
If you are using a mac. Check the version of python on your shell's path using which python. Then make sure that the shebang line in the script you are trying to run uses the same version of python.
In my terminal I typed
which python and got the output /usr/local/bin/python
I went to the script I was trying to run and added the following to the first line
#!/usr/local/bin/python
Then I went back to my terminal. Checked that I was in the same directory as the script I wanted to run and typed
./my_script.py
If you don't see any output at this point make sure that you have execution permission enabled for your script by typing chmod +x my_script.py in ther terminal. Then try running the script again using ./my_script.py

Categories