We Keep Coding

packaging python projects failed

I want to package my python code and upload it to PyPI so that people can use it easily. I followed the documentation for packaging python projects and eventually uploaded it to the PyPI test website. I ran pip install to try and install it.
Strangely enough, after installing it, I couldn't find the package:
(base) ➜ ~ python3 -m pip install --index-url https://test.pypi.org/simple/ oomstore==0.0.4
Looking in indexes: https://test.pypi.org/simple/
Collecting oomstore==0.0.4
Downloading https://test-files.pythonhosted.org/packages/4f/a5/4e7089a1ecb36a59f7f0852a5f96a6054daf886d97132060a7efcda5f04f/oomstore-0.0.4-py3-none-any.whl (12 kB)
Installing collected packages: oomstore
Successfully installed oomstore-0.0.4
(base) ➜ ~ python3
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import oomstore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'oomstore'
>>>
I went to the installation path of the package and found no python files in it:
(base) ➜ ~ cd ~/miniconda3/lib/python3.8/site-packages/oomstore-0.0.4.dist-info
(base) ➜ oomstore-0.0.4.dist-info ls
INSTALLER LICENSE METADATA RECORD REQUESTED WHEEL top_level.txt
(base) ➜ oomstore-0.0.4.dist-info
Did I do something wrong? Is there something wrong with my setup.cfg file? Forgive me for asking such an ignorant question, I'm new to python...

The problem is that your package_dir option is telling setuptools to look inside the oomstore directory to find modules and packages, but your oomstore package is right there next to setup.cfg. You should remove the option.
You could also move oomstore into an src directory and configure package_dir =\n = src; see this articles for reasons to put modules in an src directory: https://hynek.me/articles/testing-packaging/

/usr/bin/pip3 does not install package globally when in venv. How to do it?

After creating and activating a virtual environment:
python3 -m venv env
. ./env/bin/activate
the global /usr/bin/pip3 does not install a package globally. Actually it does copy package files but package is not fully installed:
/usr/bin/pip3 install docker
Collecting docker
Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
|████████████████████████████████| 146 kB 2.5 MB/s
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.22.0)
Collecting websocket-client>=0.32.0
Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
|████████████████████████████████| 52 kB 2.9 MB/s
Installing collected packages: websocket-client, docker
Successfully installed docker-5.0.3 websocket-client-1.2.1
Files are placed globally:
ls /usr/lib/python3.8/site-packages
docker docker-5.0.3.dist-info websocket websocket_client-1.2.1.dist-info
But neither a shell without venv active doesn't consider docker as installed:
pip3 list|grep docker
python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
nor the venv sees docker package as installed:
(env) pip3 list|grep docker
(env) python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
Is this a bug?
How can I install a package globally with pip3 while a venv is active?
Why do I want to do this?
Ansible is launched from a venv and it targets the localhost. So if I'm using just pip3 as an executable, it uses the binary from the venv. If I'm using /usr/bin/pip3 I get the above behavior.
It's a more complex open source project and I don't have the option to not run Ansible from a venv or not to target localhost.
Thanks

I don't use venv myself, but the whole point of virtual environments is having a set of files separated from the rest of the system.
If you create a virtual environment and are inside of it you shouldn't be using /usr/bin/pip3, you should be using the default pip3, since those two versions might not even be compatible with each other.~
It's a more complex open source project and I don't have the option to not run Ansible from a venv
You always have the option of running something inside a virtual environment, it might happen that the program does not respect that and use global paths and ignore that virtual environment, which shouldn't be happening.
From the venv docs:
Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories
This means that you should create your virtual environment, activate it, install whatever you want inside of it, and then run everything inside that virtual environment.
So I don't see this as a bug, it just seems like you are not using as it is intended to be used. If you want something installed globally, why are you using a virtual environment in the first place?

python package installed but could not import

I have tried to re-install and got the following message:
Requirement already satisfied: troposphere==1.8.2 in /usr/local/lib/python2.7/site-packages (from -r requirements.txt (line 13)
I checked my python version and I could see this as same:
animjain$ python -V
Python 2.7.10
When I try to import a module, get the following error:
animjain$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from troposphere import Join, Ref, FindInMap
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named troposphere

You should use virtualenv to install dependencies for your project. You don't need to clog global site-packages.
For example:
sudo apt-get install virtualenv
cd ~
virtualenv your_test_venv
source your_test_venv/bin/activate
pip install troposphere
After that you can use your new package.
When you want to deactivate your virtualenv you can run command deactivate

In case you have multiple Python versions on your machine, install with the same Python command, so instead of:
animjain$ pip install troposphere==1.8.2
run:
animjain$ python -mpip install troposphere==1.8.2

'pip3 --version' failing with a SyntaxError

This all began when I set out to install the Requests library for Python 3 (I'm running on OSX Mavericks with Python 2.7.5 (installed by brew install python) and 3.4.2 (installed by brew install python3). When I run pip3 --version (or anything related to the pip3 command) I see this:
$ pip3 --version
Traceback (most recent call last):
File "/usr/local/bin/pip3", line 7, in <module>
from pip import main
File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 11, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 22, in <module>
from pip._vendor import requests, six
File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 53, in <module>
from .packages.urllib3.contrib import pyopenssl
File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 49, in <module>
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/ndg/httpsclient/ssl_peer_verification.py", line 17
except ImportError, e:
^
SyntaxError: invalid syntax
When I run the Python 2.7.5 version I see this:
$ pip --version
pip 1.5.6 from /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg (python 2.7)
Just for sanity purposes Here is what when I see when I enter the interactive interpreters:
$ python3
Python 3.4.2 (default, Oct 19 2014, 17:52:17)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ python
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
A lot of the other answers related to updating pip3 suggest that I update pip3 with this commend pip3 install --upgrade pip which gives the same error, or I use easy_install -U pip but because of how brew sets up the Pythons, it only updates the Python 2.7.5 version (there is no easy_install3). Any ideas?

The root problem is that you somehow got a Python 2.x-only package installed into your 3.x site-packages.
Underlying that, you've actually got two different Python 2.7 installations (Apple's and Homebrew's) crossed with each other, which may have something to do with how you got a 2.7 package into 3.x as well…
Anyway, the reason this is breaking pip is that pip has various optional dependencies that it tries to import if present, and some of them do the same, and so on, and ultimately, starting up pip is importing the ndg-httpsclient package.
I'm not sure how you got that package. A standard Homebrew 3.x looks in two extra site-packages directories (fire up python3 then import sys; print(sys.path) to see all of the places it looks, both stdlib and site) beyond the one that pip3 installs into.
In this case, you've somehow installed the 2.x version of ndg-httpsclient into /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/‌​python3.4/site-packages.
Since you didn't install it with pip—and, more to the point, since you can't run pip in the first place—you can't just pip uninstall it. So:
rm -rf /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/‌​python3.4/site-packages/ndg*
This could break some other packages that depend on it. Once you get things working, you can use pip3 list to see all the site packages you've installed and test them out. If you want to be paranoid, do something like this:
$ pip3 list > mypackages
$ rm -rf <each site-package directory>
$ brew uninstall python3
$ brew install python3
$ pip3 install -r mypackages
You might want to similarly clean up your Homebrew 2.7 (or just scrap it and only use Apple's—especially since I'm pretty sure you're running Apple's anyway), and the site-packages for the Apple 2.7 (but not Apple's Python itself, of course, because you can't uninstall that).

WARNING: IPython History requires SQLite, your history will not be saved

Hi I'm using Ubuntu release 12.10 (quantal) 32-bit with Linux Kernel 3.5.0-21-generic. I'm trying to get IPython's History to work. I've set it up using pythonbrew and a virtual environment. In there I use pip to install IPython. Currently, when I start up IPython in a terminal I get:
WARNING: IPython History requires SQLite, your history will not be saved
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
Searching on the warning in the first line, I found this issue report, so I went back and installed the following:
sudo apt-get install libsqlite0 libsqlite0-dev libsqlite3-0 libsqlite3-dev
and then removed and reinstalled pysqlite using pip
pip uninstall pysqlite
pip install pysqlite
After that I thought I would check the installation by importing the module:
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: No module named _sqlite3
So now it seems the file _sqlite3.so can't be found. That's when I found this SO question. Either it doesn't exist or it's not in my PYTHONPATH environment variable. Searching for the file, I get:
$ locate _sqlite3.so
/home/me/Desktop/.dropbox-dist/_sqlite3.so
/home/me/epd/lib/python2.7/lib-dynload/_sqlite3.so
/usr/lib/python2.7/lib-dynload/_sqlite3.so
So the file is there, but when I looked in my python path:
import sys
for p in sys.path:
print p
none of the above paths that contain _sqlite3.so were contained in my PYTHONPATH. For giggles, I added the path /usr/lib/python2.7/lib-dynload to my PYTHONPATH in a terminal and then tried to import sqlite3 again:
Python 2.7.3 (default, Nov 8 2012, 18:25:10)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/usr/lib/python2.7/lib-dynload")
>>> import sqlite3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
from dbapi2 import *
File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ImportError: /usr/lib/python2.7/lib-dynload/_sqlite3.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
Uh oh. Now I'm completely stuck. Can anyone help me out? I've also read in a few places that I may have to rebuild Python. I have no idea how to do this in pythonbrew. Can anyone point me in the right direction?

I've also read in a few places that I may have to rebuild Python.
This is correct. SQLite is part of the standard library,
and is built when you compile Python. There are a few 'optional' parts
of the standard library, which Python will simply skip (with minimal warning, unfortunately)
if the dependencies are missing at build time, and sqlite is one of these.
You should be able to just install libsqlite3-dev,
then rebuild Python and you should be set.
Keep an eye on the build messages,
as they do report which modules they are skipping due to missing dependencies.

Thanks to minrk for pointing me in the right direction. All I had to do was rebuild python. I've outlined the steps below for those that are using pythonbrew. Notice that I already installed the libsqlite3-dev package up in the question section.
First, with the proper version of python and virtual environment loaded up run the command:
$ pip freeze -l > requirements.txt
This gives us a text file list of all of the pip packages that have been installed in the virtual environment for this particular python release in pythonbrew. Then we remove the version of python from pythonbrew and reinstall it (this is the "rebuild python" step):
$ pythonbrew uninstall 2.7.3
$ pythonbrew install 2.7.3
After that, we switch over to the newly installed python version 2.7.3 and create a new virtual environment (which I've called "sci"):
$ pythonbrew switch 2.7.3
$ pythonbrew venv create sci
$ pythonbrew venv use sci
Ideally you should be able to run the command:
$ pip install -r requirements.txt
and according to this pip should reinstall all the modules that you had in the virtual environment before we clobbered that version of python (2.7.3). It didn't work for me for whatever reason so I manually installed all of the modules using pip individuality.
$ ipython --pylab
Python 2.7.3 (default, Jan 5 2013, 18:48:27)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.1 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
and IPython history works!

What worked for me (using osx + homebrew + brewed python):
# Reinstall Python 2.7 with sqlite
brew remove python
brew install readline sqlite gdbm --universal
brew install python --universal --framework
# Reinstall iPython with correct bindings
pip uninstall ipython
pip install ipython
And you should be good to go.

You should rebuild your python with sqlite support
sudo apt-get install libsqlite3-dev
wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz
tar -xvf Python-2.7.15.tgz
cd Python-2.7.15
./configure
make
sudo make install
Recreate your virtual environment and you should be good to go
rmvirtualenv venv
mkvirtualenv -p python2 venv
workon venv
pip install -r requirements.txt
# or
pip install ipython

This warning appears on macOS when python is installed with pyenv. By default it installs python without sqlite. These commands reinstall python with sqlite support:
pyenv uninstall 3.7
CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" pyenv install 3.7