We Keep Coding

packaging python projects failed

I want to package my python code and upload it to PyPI so that people can use it easily. I followed the documentation for packaging python projects and eventually uploaded it to the PyPI test website. I ran pip install to try and install it.
Strangely enough, after installing it, I couldn't find the package:
(base) ➜ ~ python3 -m pip install --index-url https://test.pypi.org/simple/ oomstore==0.0.4
Looking in indexes: https://test.pypi.org/simple/
Collecting oomstore==0.0.4
Downloading https://test-files.pythonhosted.org/packages/4f/a5/4e7089a1ecb36a59f7f0852a5f96a6054daf886d97132060a7efcda5f04f/oomstore-0.0.4-py3-none-any.whl (12 kB)
Installing collected packages: oomstore
Successfully installed oomstore-0.0.4
(base) ➜ ~ python3
Python 3.8.5 (default, Sep 4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import oomstore
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'oomstore'
>>>
I went to the installation path of the package and found no python files in it:
(base) ➜ ~ cd ~/miniconda3/lib/python3.8/site-packages/oomstore-0.0.4.dist-info
(base) ➜ oomstore-0.0.4.dist-info ls
INSTALLER LICENSE METADATA RECORD REQUESTED WHEEL top_level.txt
(base) ➜ oomstore-0.0.4.dist-info
Did I do something wrong? Is there something wrong with my setup.cfg file? Forgive me for asking such an ignorant question, I'm new to python...

The problem is that your package_dir option is telling setuptools to look inside the oomstore directory to find modules and packages, but your oomstore package is right there next to setup.cfg. You should remove the option.
You could also move oomstore into an src directory and configure package_dir =\n = src; see this articles for reasons to put modules in an src directory: https://hynek.me/articles/testing-packaging/

/usr/bin/pip3 does not install package globally when in venv. How to do it?

After creating and activating a virtual environment:
python3 -m venv env
. ./env/bin/activate
the global /usr/bin/pip3 does not install a package globally. Actually it does copy package files but package is not fully installed:
/usr/bin/pip3 install docker
Collecting docker
Downloading docker-5.0.3-py2.py3-none-any.whl (146 kB)
|████████████████████████████████| 146 kB 2.5 MB/s
Requirement already satisfied: requests!=2.18.0,>=2.14.2 in /usr/lib/python3/dist-packages (from docker) (2.22.0)
Collecting websocket-client>=0.32.0
Downloading websocket_client-1.2.1-py2.py3-none-any.whl (52 kB)
|████████████████████████████████| 52 kB 2.9 MB/s
Installing collected packages: websocket-client, docker
Successfully installed docker-5.0.3 websocket-client-1.2.1
Files are placed globally:
ls /usr/lib/python3.8/site-packages
docker docker-5.0.3.dist-info websocket websocket_client-1.2.1.dist-info
But neither a shell without venv active doesn't consider docker as installed:
pip3 list|grep docker
python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
nor the venv sees docker package as installed:
(env) pip3 list|grep docker
(env) python3
Python 3.8.10 (default, Sep 28 2021, 16:10:42)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import docker
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'docker'
>>>
Is this a bug?
How can I install a package globally with pip3 while a venv is active?
Why do I want to do this?
Ansible is launched from a venv and it targets the localhost. So if I'm using just pip3 as an executable, it uses the binary from the venv. If I'm using /usr/bin/pip3 I get the above behavior.
It's a more complex open source project and I don't have the option to not run Ansible from a venv or not to target localhost.
Thanks

I don't use venv myself, but the whole point of virtual environments is having a set of files separated from the rest of the system.
If you create a virtual environment and are inside of it you shouldn't be using /usr/bin/pip3, you should be using the default pip3, since those two versions might not even be compatible with each other.~
It's a more complex open source project and I don't have the option to not run Ansible from a venv
You always have the option of running something inside a virtual environment, it might happen that the program does not respect that and use global paths and ignore that virtual environment, which shouldn't be happening.
From the venv docs:
Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories
This means that you should create your virtual environment, activate it, install whatever you want inside of it, and then run everything inside that virtual environment.
So I don't see this as a bug, it just seems like you are not using as it is intended to be used. If you want something installed globally, why are you using a virtual environment in the first place?

import pyserial / serial not recognized by python, but dependencies already installed [Mac]

Dependencies for pyserial already installed, calling pyserial isn't recognized in python
Maxs-MacBook:~ grax$ sudo pip install pyserial
The directory '/Users/grax/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/grax/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: pyserial in /Library/Python/2.7/site-packages
Maxs-MacBook:~ grax$ python
Python 2.7.15 (default, May 2 2018, 00:53:27)
[GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.29.1] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyserial
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Import Error: No module named pyserial
What should I do?

The python module is called serial even though you call pip pyserial. Confusing, yes.
import serial
The other issue may be that the instance of python you're using isn't the same as the one pip is installing for.
See where pip installed the module:
$ pip show pyserial
Name: pyserial
Version: 3.4
Summary: Python Serial Port Extension
Home-page: https://github.com/pyserial/pyserial
Author: Chris Liechti
Author-email: cliechti#gmx.net
License: BSD
Location: /Library/Python/2.7/site-packages
Because your Location is /Library/Python2.7... pip appears to be installing in the system directory.
However, the version of python you're using (Python 2.7.15) is not the one shipped with MacOSX, so it's probably looking for the python modules somewhere else.
$ python
>>> import sys
>>> print sys.path
The version of pip is most likely not installing there. (pip is not native to MacOS, so it may be using /usr/local/bin/python and /usr/local/lib/python2.7/site-packages.
You can force pip to install somewhere else using --target option:
$ sudo pip install --target /usr/local/lib/python2.7/site-packages pyserial
Which will put the serial module in the location your python is looking.

How to use Python Dbus bindings in Anaconda

I am trying to install dbus on Anaconda python environment and I am struggling.
Here is the error message I am getting:
e#gateway:~$ python
Python 3.5.4 |Anaconda custom (64-bit)| (default, Oct 13 2017, 11:22:58)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/e/anaconda3/lib/python3.5/site-packages/dbus/__init__.py", line 77, in <module>
import dbus.types as types
File "/home/e/anaconda3/lib/python3.5/site-packages/dbus/types.py", line 6, in <module>
from _dbus_bindings import (
ImportError: /home/e/anaconda3/lib/python3.5/site-packages/_dbus_bindings.so: undefined symbol: _Py_ZeroStruct
>>>
Here are some of the outputs I think may be asked:
e#gateway:~$ conda install dbus
Fetching package metadata ...........
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /home/e/anaconda3:
#
dbus 1.10.22 h3b5a359_0
e#gateway:~$ sudo apt-get install libdbus-glib-1-dev libdbus-1-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
libdbus-glib-1-dev is already the newest version (0.106-1).
libdbus-1-dev is already the newest version (1.10.6-1ubuntu3.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
e#gateway:~$ sudo apt-get install dbus
Reading package lists... Done
Building dependency tree
Reading state information... Done
dbus is already the newest version (1.10.6-1ubuntu3.3).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
e#gateway:~$ which python
/home/e/anaconda3/bin/python
e#gateway:~$ conda --version
conda 4.3.31
e#gateway:~$ sudo /home/e/anaconda3/bin/python -m pip install dbus-python
The directory '/home/e/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/e/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: dbus-python in ./anaconda3/lib/python3.5/site-packages
DBus is working fine on the system python, however not working on Anaconda Python.
Python 2.7:
e#gateway:~$ which python
/usr/bin/python
e#gateway:~$ python
Python 2.7.12 (default, Nov 20 2017, 18:23:56)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
>>>
Python 3.5:
e#gateway:~$ which python3
/usr/bin/python3
e#gateway:~$ python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dbus
>>>
Can anyone help me? Am I missing something blatantly obvious here?
Thanks in advance.

I had similar issues, there are few cases where dbus and python don't work well out-of-the-box. The consensus appears to be that you need a system-level install (i.e. apt-get) to get dbus to work. I believe the /home/e/anaconda3/lib/python3.5/site-packages/_dbus_bindings.so: undefined symbol: _Py_ZeroStruct error you're seeing is directly related to that.
conda install dbus does not add anything to ~/anaconda3/lib/python3.6/site-packages, but instead appears to install some executables in ~/anaconda3/bin/ like dbus-run-session, dbus-daemon, etc. This makes some sense when you analyze the contents of the dbus tarball https://anaconda.org/conda-forge/dbus, as it's all C files and executables. I'm not sure it's supposed to be the dbus python module, but I could be wrong.
EDIT:
I searched the conda repositories and found a few individuals that uploaded a version of dbus-python, presumably that they compiled and installed. I tried this one out in a py3.6 conda environment via:
conda install -c scottwales dbus-python
I was then able to import dbus. This is a hacky approach and should not be used in production, I'd recommend listening to
Carlos Cordoba's post below. But if you need a solution now, search through some user conda packages or try to compile the library yourself.

Can anyone help me? Am I missing something blatantly obvious here?
Yes, you are. There's one thing people still don't understand about conda: conda is not a pip replacement. It is a general package manager, in the same vein as apt-get, yum, brew, emerge, etc, but cross-platform and based on Python.
In this case, that means that conda install dbus does not install the Python Dbus bindings, as you would expect with pip . It installs the Dbus C package itself, which is needed by Qt 5 (again, the C++ library, not the Python bindings to it).
Unfortunately, there are no Conda packages for dbus-python. To make matters worse, it seems there's no easy way to create packages for it, as pointed out here.
Finally, you said
Here is the error message I am getting
The (most probable) cause of that error is because you added your system Python dist-packages path to the PYTHONPATH of Anaconda or because you blindly copied the dbus module from system Python to Anaconda. Please don't do that ever again. System Python and Anaconda packages are compiled with different compilers and under different conditions. So mixing them is the cause of incomprehensible errors, just like the one you reported.

Unable to import praw module after install

Edit: Before you start deleting/modifying installs, please glance over StvnW's answer/summary to make sure you are applying the solution that is appropriate for you.
I've installed python 2.7.5 and pip [edit: mac OSX Mountain Lion.] I've run "pip install praw" in terminal. All good. When I run python and run "import praw" I get:
...$ python
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import praw
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named praw
...same ImportError from a script.
when I install praw I get this:
$ pip install praw
Downloading/unpacking praw
Downloading praw-2.1.10.tar.gz (83kB): 83kB downloaded
Running setup.py egg_info for package praw
Requirement already satisfied (use --upgrade to upgrade): requests>=1.2.0 in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): six in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): update-checker>=0.6 in
/usr/local/lib/python2.7/site-packages (from praw)
Requirement already satisfied (use --upgrade to upgrade): setuptools in
/usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from update-
checker>=0.6->praw)
Installing collected packages: praw
Running setup.py install for praw
Installing praw-multiprocess script to /usr/local/bin
Successfully installed praw
Cleaning up...
In python if I run help('modules') it isn't there.
Relatively new to python and I haven't been able to sort this out with google search. Any hints would be much appreciated.
Edit:
SitRep:
I've uninstalled 2.7.2, uninstalled praw, and uninstalled (homebrew) pip. I ran python 2.7.5 and it couldn't find the module (as you would suspect.) I then reinstalled pip with easy_install and now 2.7.5 is finding praw but giving this error:
$ python
Python 2.7.5 (v2.7.5:ab05e7dd2788, May 13 2013, 13:18:45)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import praw
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/praw/__init__.py", line 43, in <module>
from update_checker import update_check
File "/Library/Python/2.7/site-packages/update_checker.py", line 11, in <module>
from pkg_resources import parse_version as V
ImportError: No module named pkg_resources
>>>
Thanks for the help so far, the spurious install was the root of problem 1.
The solution to the final problem can be found here:
No module named pkg_resources
I wish there were some way to summarize this for future readers, but I've done so many things that I no longer recall what addressed what. I basically uninstalled everything (python 2.7.2, 2.7.5, praw, pip,) reinstalled 2.7.5 from http://python.org/download/, reinstalled pip with easy_install (and sudo command) instead of homebrew, reinstalled praw with sudo command, and followed the directions for the subsequent module error in the link above. Hope that helps. :)

The OS X system versions of Python live in /System/Library/Frameworks/Python.Framework/<verson>, each of which links to /Library/Python/<version>/site-packages. Seems like you have installed another version of python and/or pip to /usr/local, but when you invoke python you are still getting the system version.
Try this:
$ /usr/local/bin/python
>>> import praw
I'd also recommend looking into tools like pyenv and virtualenv if you are going to do any amount of work with Python. Pyenv lets you easily manage and switch between multiple versions (including micro versions x.x.3, x.x.5, etc). Virtualenv lets you create isolated Python environments where you can pin down versions of site-packages specific to a particular project.
Edit (summarizing):
sudo easy_install pip will install pip under /System/Library/Python/<version>. Calling that pip will install packages to /Library/Python/<version>/site-packages
brew install python will install a second version of python — including pip — under /usr/local/. That pip will put packages in /usr/local/lib/python<version>/site-packages/`.
One version may not see packages installed to the other.
which python and which pip are helpful for troubleshooting.
Dean's final solution above results in user packages being installed to the system site-packages.
Despite Dean's choice to update the system Python, many would advocate instead using brew, pyenv, and virtualenv to isolate oneself from the system Python.

It's strange. I am suspecting you have another version of python2.7 on your mac os

If you are using a mac. Check the version of python on your shell's path using which python. Then make sure that the shebang line in the script you are trying to run uses the same version of python.
In my terminal I typed
which python and got the output /usr/local/bin/python
I went to the script I was trying to run and added the following to the first line
#!/usr/local/bin/python
Then I went back to my terminal. Checked that I was in the same directory as the script I wanted to run and typed
./my_script.py
If you don't see any output at this point make sure that you have execution permission enabled for your script by typing chmod +x my_script.py in ther terminal. Then try running the script again using ./my_script.py