We Keep Coding

imon bash script can't see python package installed, user generated bash script can

I recently added python 3.9 to my ubuntu installation.
Can run python3.9 code using an installed package from bash, can also run python 3.9 using an installed module from a bash script using a shebang to load the python 3.9 environment, fairly standard stuff.
But when I try running an installed script from https://github.com/gitbls/imon, the installed package is not found.
initially I thought this was fixed by changing the imon bash script shebang to reference the new python.
ie
old
#!/usr/bin/python3
new
#!/usr/bin/python3.9
also added these lines to the imon script to verify which version of python in environment
import sys
print("sys.version:", sys.version)
when I run the /usr/local/bin/imon package, it reports 'no module' error.
$ sudo /usr/local/bin/imon --nosyslog --instance my_instance_name
sys.version: 3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
Traceback (most recent call last):
File "/usr/local/bin/imon", line 11, in <module>
from icmplib import ping, multiping
ModuleNotFoundError: No module named 'icmplib'
the thing that is really throwing me, is the output shows the imon script is accepting the shebang to load python 3.9, but will not find the module, despite my script using identical code able to load the module.
my test exercises below to verify and demonstrate I have python3.9 installed and a bash script can load python3.9 and the icmplib module.
$ python3.9
Python 3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print (sys.version)
3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
>>> import icmplib
ie: the icmplib package has been installed and can be used when in the python shell.
I created a shell script and made it executable
vim test_script.sh
#!/usr/bin/python3.9
import sys
print (sys.version)
import icmplib
print("icmplib.__version__:", icmplib.__version__)
chmod u+x test_script.sh
output from this shows the script is running python3.9 and can access the installed module.
$ ./test_script.sh
3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
icmplib.__version__: 3.0.3
#EDIT START--------------------------------------------------------------
added tests as suggested by #furas in comments.
print("sys.path:", sys.path)
print("icmplib.__file__:", icmplib.__file__)
these above lines were added to both test_script.sh & /usr/local/bin/imon
the results below reveal the local test_script.sh accesses a directory not accessed by the imon script run by root.
'/home/m/.local/lib/python3.9/site-packages',
oddly, both the script run locally and the script run under root access path
'/usr/lib/python3/dist-packages'
(I don't know why python3 instead of python3.9 is in sys.path, this seems odd to me)
taking another suggestion from #furas I tried pip installing the module as root
$ sudo python3.9 -m pip install icmplib
Collecting icmplib
Using cached icmplib-3.0.3-py3-none-any.whl (30 kB)
Installing collected packages: icmplib
Successfully installed icmplib-3.0.3
this looked helpful. unfortunately running the imon script as root still resulted in the Namerror.
$ sudo /usr/local/bin/imon --nosyslog --instance imon-bmt1
sys.version: 3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
sys.path: ['/usr/local/bin', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']
Traceback (most recent call last):
File "/usr/local/bin/imon", line 8, in <module>
print("icmplib.__file__:", icmplib.__file__)
NameError: name 'icmplib' is not defined
I am now 18hrs awake and brain fading. :(
#EDIT END--------------------------------------------------------------
#EDIT: this below was before the edit above.
when I run test_script.sh as root, I can replicate the error experienced by imon.
$ ./test_script.sh
3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
icmplib.__version__: 3.0.3
sys.path: ['/home/m/f_projs/internet_monitor',
'/usr/lib/python39.zip',
'/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload',
'/home/m/.local/lib/python3.9/site-packages',
'/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']
icmplib.__file__: /home/m/.local/lib/python3.9/site-packages/icmplib/__init__.py
$ sudo /usr/local/bin/imon --nosyslog --instance imon-bmt1
sys.version: 3.9.14 (main, Sep 7 2022, 23:43:29)
[GCC 9.4.0]
sys.path: ['/usr/local/bin',
'/usr/lib/python39.zip',
'/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload',
'/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages']
Traceback (most recent call last):
File "/usr/local/bin/imon", line 8, in <module>
print("icmplib.__file__:", icmplib.__file__)
NameError: name 'icmplib' is not defined
my environment = ubuntu 20.04 with python 3.9.14
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal
$ which python3
/usr/bin/python3
$ which python3.9
/usr/bin/python3.9
$ python3.9 --version
Python 3.9.14
$ python3 --version
Python 3.8.10

I'm marking this as 'answered' because I've realised I broke my python3 and python3.9 install beyond repair while attempting to fix this issue.
now I have a completely different problem.
most of the original problem appears to be failing to use 'sudo pip install X' (or 'sudo python3.9 -m pip install x') when installing packages. I normally use virtual env for my local projects.
I was doing uninstall of python3.9 and cleanup with view to reinstalling python3.9.
sudo apt remove python3.9
#now check what packages remain to be removed manually.
dpkg --list | grep python3.9
#this showed a short list of packages to be cleaned up.
#nb: 'rc' = package has been removed, but configuration files remain
sudo apt-get remove --auto-remove python3.9-minimal
sudo apt-get remove --auto-remove python3.9-venv
sudo apt-get remove --auto-remove python3.9
sudo dpkg -P libpython3.9-minimal
sudo dpkg -P python3.9-minimal
sudo dpkg -P python3.9-venv
#then manually deleting the python3.9 directories previously listed by sys.path
ie
sudo rm -rf /usr/lib/python3.9
sudo rm -rf /usr/local/lib/python3.9/dist-packages
my new problem is due to accidentally deleting a directory from sys-path which 'apt-get remove' and 'sudo dpkg -P' didn't cleanup.
sudo rm -rf /usr/lib/python3/dist-packages
hindsight screams out this will destroy the python3 install. Bitter experience just now tells me it is near impossible to repair since the apt-get tools are dependent on a working python3 install.
I think the only way to fix this is copying across a full directory of '/usr/lib/python3/dist-packages' from a clean install.
new SO question for my new problem > (how to repair python3 install on ubuntu after rm -rf /usr/lib/python3/dist-packages?)
Hopefully this post will be useful and a warning to others in future.

Modules installed using pip, recognised by PyCharm, but not by macOS terminal

I am trying to access a python file created on PyCharm with my terminal. When I import modules on PyCharm they are found and successfully imported (e.g tkinter, PIL etc.). When I try to run the file on my terminal I get the following message :
Traceback (most recent call last):
File "THE_project_GUI.py", line 3, in <module>
from PIL import Image
ImportError: No module named PIL
The which python command returns : /usr/bin/python
The which -a pip command returns : /usr/local/bin/pip
How do make sure I use the interpreter that pip is installing for?
Trial:
tromgy's solution:
me#mahmouds-mbp-2 ~ % ln -s -f /usr/local/bin/python3.8 /usr/local/bin/python
me#mahmouds-mbp-2 ~ % cd /Users/me/Documents/Programming/THE_project
me#mahmouds-mbp-2 THE_project % /usr/local/bin/python3.8 /usr/local/bin/python THE_project_GUI.py
returns:
File "/usr/local/bin/python", line 1
SyntaxError: Non-UTF-8 code starting with '\xcf' in file /usr/local/bin/python on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
v0idbar's solution:
I have installed python 3.7 using :
brew install python#3.7
returns :
Python has been installed as
/usr/local/opt/python#3.7/bin/python3
Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been installed into
/usr/local/opt/python#3.7/libexec/bin
You can install Python packages with
/usr/local/opt/python#3.7/bin/pip3 install <package>
They will install into the site-package directory
/usr/local/lib/python3.7/site-packages
I have changed the Pycharm interpreter by going to File> New project settings> Preferences for new projects. And then added 3.7 after finding it using /usr/local/opt/python#3.7/bin/python3.
Then I ran the following :
me#mahmouds-mbp-2 THE_project % /usr/local/opt/python#3.7/bin/python3
/usr/local/opt/python#3.7/bin/python3
Python 3.7.11 (default, Jul 6 2021, 12:43:19)
[Clang 12.0.5 (clang-1205.0.22.9)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> /usr/local/opt/python#3.7/bin/python3 THE_project_GUI.py
File "<stdin>", line 1
/usr/local/opt/python#3.7/bin/python3 THE_project_GUI.py
^
SyntaxError: invalid syntax

Try run pip -V to figure out what python version it is using:
$ pip -V
pip 19.0.1 from /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pip (python 3.7)
You can check if it the same version as the python executable you are using: python -m pip -V
$ python -m pip -V
pip 9.0.1 from /Users/lev/anaconda2/lib/python2.7/site-packages (python 2.7)
You probably have a mismatch between those versions.

matplotlib works with default python but not python3.5 on debian

I have this issue. I think it's best described when I show you my bash commands that I used to produce the problem. See here:
josch#oogway:~$ python
Python 2.7.9 (default, Jun 29 2016, 13:08:31)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib
>>> #works
josch#oogway:~$ python3.5
Python 3.5.0 (default, Apr 26 2017, 21:03:53)
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import matplotlib
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'matplotlib'
>>> #what??
Matplotlib was installed by:
sudo apt-get install python-matplotlib
The "default" python came with my OS installation, python3.5 was downloaded and then compiled/installed from source.
Can anyone help? Tried to solve it myselve for 2 hours now but google can't find answer
Edit: I'm trying to install a "second pip" that works with my second installation of Python, which is Python3.5. My default-Python that is managed by the OS is Python 2.7.9. Now, as suggested I did:
joschua#oogway:~/Downloads$ wget https://bootstrap.pypa.io/get-pip.py
and then:
python3.5 get-pip.py
which gave me:
joschua#oogway:~/Downloads$ python3.5 get-pip.py
Traceback (most recent call last):
File "get-pip.py", line 20061, in <module>
main()
File "get-pip.py", line 194, in main
bootstrap(tmpdir=tmpdir)
File "get-pip.py", line 82, in bootstrap
import pip
zipimport.ZipImportError: can't decompress data; zlib not available

Try installing pip3 with:
sudo apt-get install python3-pip
and then try installing matplotlib with:
sudo pip3 install matplotlib

I actually could solve the problem now.
See this link for a reasonable description on how to run two different versions of python on the same system (and as #Pierre de Buyl suggested) use two different pip versions on the same system
https://www.linkedin.com/pulse/20140815081557-89781742-how-to-install-and-use-python-with-different-versions-on-same-linux-machine

So, you installed python3.5 from source. What version of Debian are you using? (I suppose Jessie).
If you want to install packages for this Python interpreter, you must have a pip that actually uses it. Two solutions:
It is installed. If so,
python3.5 -m pip
will work and you can install packages with
python3.5 -m pip install matplotlib
or
python3.5 -m pip install --user matplotlib
It is not installed. You must install it following the instructions at https://packaging.python.org/installing/#requirements-for-installing-packages
apt-get will not install the appropriate pip for your needs. It will only install a pip that is related to the python3 (3.4 for jessie I believe) that is also in the apt-get system.
EDIT: in light of the update, you need also to re-compile python3.5 with zip enabled.

Debian No Module named numpy

I've installed Python Numpy on Debian using...
apt-get install python-numpy
But when run the Python shell I get the following...
Python 2.7.10 (default, Sep 9 2015, 20:21:51)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named numpy
When I view the contents of /usr/local/lib/python2.7/site-packages/ I noticed numpy is not list.
If I install it via pip i.e pip install numpy it works just fine, However, I want to use the apt-get method. What I'm I doing wrong?
Other:
echo $PYTHONPATH /usr/local/lib/python2.7
dpkg -l python-numpy...
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================================-============================-============================-====================================================================================================
ii python-numpy 1:1.8.2-2 amd64 Numerical Python adds a fast array facility to the Python language
Python 2.7.10
['', '/usr/local/lib/python2.7', '/usr/local/lib/python27.zip', '/usr/local/lib/python2.7/plat-linux2', '/usr/local/lib/python2.7/lib-tk', '/usr/local/lib/python2.7/lib-old', '/usr/local/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/site-packages']
which -a python...
/usr/local/bin/python
/usr/bin/python
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

As you can tell from your which result, the python you are running when just typing python is /usr/local/bin/python.
It's a python you probably installed there yourself, as Debian will never put anything in /usr/local by itself (except for empty directories).
How? Well, by running pip for instance. As a rule, you should never use pip outside of a virtualenv, because it will install stuff on your system that your package manager will not know about. And maybe break stuff, like what you see on your system.
So, if you run /usr/bin/python, it should see the numpy package you installed using your package manager.
How to fix it? Well, I would clear anything in /usr/local (beware, it will definitely break stuff that rely on things you installed locally). Then I would apt-get install python-virtualenv, and always work with a virtualenv.
$ virtualenv -p /usr/bin/python env
$ . env/bin/activate
(env)$ pip install numpy
(env)$ python
>>> import numpy
>>>
That way, packages will be installed in the env directory. You do all this as a regular user, not root. And your different projects can have different environments with different packages installed.

'pip3 --version' failing with a SyntaxError

This all began when I set out to install the Requests library for Python 3 (I'm running on OSX Mavericks with Python 2.7.5 (installed by brew install python) and 3.4.2 (installed by brew install python3). When I run pip3 --version (or anything related to the pip3 command) I see this:
$ pip3 --version
Traceback (most recent call last):
File "/usr/local/bin/pip3", line 7, in <module>
from pip import main
File "/usr/local/lib/python3.4/site-packages/pip/__init__.py", line 11, in <module>
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "/usr/local/lib/python3.4/site-packages/pip/vcs/mercurial.py", line 9, in <module>
from pip.download import path_to_url
File "/usr/local/lib/python3.4/site-packages/pip/download.py", line 22, in <module>
from pip._vendor import requests, six
File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/__init__.py", line 53, in <module>
from .packages.urllib3.contrib import pyopenssl
File "/usr/local/lib/python3.4/site-packages/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 49, in <module>
from ndg.httpsclient.ssl_peer_verification import SUBJ_ALT_NAME_SUPPORT
File "/usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/ndg/httpsclient/ssl_peer_verification.py", line 17
except ImportError, e:
^
SyntaxError: invalid syntax
When I run the Python 2.7.5 version I see this:
$ pip --version
pip 1.5.6 from /Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg (python 2.7)
Just for sanity purposes Here is what when I see when I enter the interactive interpreters:
$ python3
Python 3.4.2 (default, Oct 19 2014, 17:52:17)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ python
Python 2.7.5 (default, Mar 9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
A lot of the other answers related to updating pip3 suggest that I update pip3 with this commend pip3 install --upgrade pip which gives the same error, or I use easy_install -U pip but because of how brew sets up the Pythons, it only updates the Python 2.7.5 version (there is no easy_install3). Any ideas?

The root problem is that you somehow got a Python 2.x-only package installed into your 3.x site-packages.
Underlying that, you've actually got two different Python 2.7 installations (Apple's and Homebrew's) crossed with each other, which may have something to do with how you got a 2.7 package into 3.x as well…
Anyway, the reason this is breaking pip is that pip has various optional dependencies that it tries to import if present, and some of them do the same, and so on, and ultimately, starting up pip is importing the ndg-httpsclient package.
I'm not sure how you got that package. A standard Homebrew 3.x looks in two extra site-packages directories (fire up python3 then import sys; print(sys.path) to see all of the places it looks, both stdlib and site) beyond the one that pip3 installs into.
In this case, you've somehow installed the 2.x version of ndg-httpsclient into /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/‌​python3.4/site-packages.
Since you didn't install it with pip—and, more to the point, since you can't run pip in the first place—you can't just pip uninstall it. So:
rm -rf /usr/local/Cellar/python3/3.4.2_1/Frameworks/Python.framework/Versions/3.4/lib/‌​python3.4/site-packages/ndg*
This could break some other packages that depend on it. Once you get things working, you can use pip3 list to see all the site packages you've installed and test them out. If you want to be paranoid, do something like this:
$ pip3 list > mypackages
$ rm -rf <each site-package directory>
$ brew uninstall python3
$ brew install python3
$ pip3 install -r mypackages
You might want to similarly clean up your Homebrew 2.7 (or just scrap it and only use Apple's—especially since I'm pretty sure you're running Apple's anyway), and the site-packages for the Apple 2.7 (but not Apple's Python itself, of course, because you can't uninstall that).