Why can't Python 3 virtualenv find some installed packages? - python

I am working with a Python virtualenv named env to execute Odoo. In the virtualenv bin directory, I got this:
pip
pip3
pip3.8
python -> /usr/bin/python3
python3 -> python
python3.8 -> python
And the Odoo service is running this command to execute Odoo (as you can see, using the python3.8 of the virtualenv):
ExecStart=/opt/odoo/env/bin/python3.8 /opt/odoo/odoo_13/src/oca/OCB/odoo-bin -c /opt/odoo/odoo_13/config/.odoo.conf
The problem is that every now and again, Odoo gives the following critical error and the instance stops working:
ModuleNotFoundError: No module named 'werkzeug'
It seems that the werkzeug package is not installed, but, if I do the following:
anubia#my-server:/opt/odoo$ . env/bin/activate
(env) anubia#my-server:/opt/odoo$ which pip
/opt/odoo/env/bin/pip
(env) anubia#my-server:/opt/odoo$ which pip3
/opt/odoo/env/bin/pip3
(env) anubia#my-server:/opt/odoo$ which pip3.8
/opt/odoo/env/bin/pip3.8
(env) anubia#my-server:/opt/odoo$ pip3.8 install werkzeug
Requirement already satisfied: werkzeug in ./env/lib/python3.8/site-packages (0.11.15)
(env) anubia#my-server:/opt/odoo$ cd env/lib/python3.8/site-packages/
Display all 125 possibilities? (y or n)
Babel-2.9.1.dist-info/ attrs-21.2.0.dist-info/ docutils/ numpy-1.21.0.dist-info/ python_stdnum-1.16.dist-info/ urllib3-1.26.6.dist-info/
Crypto/ babel/ docutils-0.17.1.dist-info/ numpy.libs/ pytz/ validate_email-1.3.dist-info/
Jinja2-3.0.1.dist-info/ bcrypt/ gevent/ paramiko/ pytz-2021.1.dist-info/ vatnumber/
MarkupSafe-2.0.1.dist-info/ bcrypt-3.2.0.dist-info/ gevent-21.1.2.dist-info/ paramiko-2.7.2.dist-info/ reportlab/ vatnumber-1.2.dist-info/
OpenSSL/ cached_property-1.5.2.dist-info/ greenlet/ passlib/ reportlab-3.5.68.dist-info/ werkzeug/
PIL/ cachetools/ greenlet-1.1.0.dist-info/ passlib-1.7.4.dist-info/ reportlab.libs/ wheel/
Pillow-8.3.1.dist-info/ cachetools-4.2.2.dist-info/ html2text/ phonenumbers/ requests/ wheel-0.36.2.dist-info/
Pillow.libs/ certifi/ html2text-2020.1.16.dist-info/ phonenumbers-8.12.27.dist-info/ requests-2.25.1.dist-info/ xlrd/
PyNaCl-1.4.0.dist-info/ certifi-2021.5.30.dist-info/ idna/ pip/ requests_file-1.5.1.dist-info/ xlrd-2.0.1.dist-info/
PyPDF2/ cffi/ idna-2.10.dist-info/ pip-21.3.1.dist-info/ requests_toolbelt/ xlsxwriter/
PyPDF2-1.26.0.dist-info/ cffi-1.14.5.dist-info/ isodate/ pkg_resources/ requests_toolbelt-0.9.1.dist-info/ xlwt/
PyYAML-3.12.dist-info/ cffi.libs/ isodate-0.6.0.dist-info/ polib-1.1.1.dist-info/ sassutils/ xlwt-1.3.0.dist-info/
Unidecode-1.2.0.dist-info/ chardet/ jinja2/ psutil/ setuptools/ xmlsig/
Werkzeug-0.11.15.dist-info/ chardet-4.0.0.dist-info/ libsass-0.21.0.dist-info/ psutil-5.8.0.dist-info/ setuptools-57.1.0.dist-info/ xmlsig-0.1.5.dist-info/
XlsxWriter-1.4.4.dist-info/ cryptography/ lxml/ psycopg2/ six-1.16.0.dist-info/ yaml/
__pycache__/ cryptography-3.4.7.dist-info/ lxml-4.6.3.dist-info/ psycopg2-2.9.1.dist-info/ stdnum/ zeep/
_distutils_hack/ dateutil/ markupsafe/ pyOpenSSL-20.0.1.dist-info/ suds/ zeep-4.0.0.dist-info/
appdirs-1.4.4.dist-info/ decorator-5.0.9.dist-info/ nacl/ pycparser/ suds_jurko-0.6.dist-info/ zope/
asn1crypto/ defusedxml/ num2words/ pycparser-2.20.dist-info/ tests/ zope.event-4.5.0.dist-info/
asn1crypto-1.4.0.dist-info/ defusedxml-0.7.1.dist-info/ num2words-0.5.10.dist-info/ pycryptodome-3.10.1.dist-info/ unidecode/ zope.interface-5.4.0.dist-info/
attr/ docopt-0.6.2.dist-info/ numpy/ python_dateutil-2.8.1.dist-info/ urllib3/
So werkzeug seems to be installed. What is the problem?
EDIT
After trying with #ChesuCR answer two weeks, the Odoo server stopped working again. This time the error is:
ModuleNotFoundError: No module named 'PyPDF2'
However, if I check the virtualenv:
myuser#myserver:/opt/odoo$ . venv3.7/bin/activate
(venv3.7) myuser#myserver:/opt/odoo$ which python3.7
/opt/odoo/venv3.7/bin/python3.7
(venv3.7) myuser#myserver:/opt/odoo$ python3.7 -m pip install PyPDF2
Requirement already satisfied: PyPDF2 in ./venv3.7/lib/python3.7/site-packages (1.26.0)
(venv3.7) myuser#myserver:/opt/odoo$ python3.7 -m pip list
Package Version
----------------- ---------
appdirs 1.4.4
attrs 21.2.0
Babel 2.6.0
beautifulsoup4 4.10.0
cached-property 1.5.2
certifi 2021.10.8
cffi 1.15.0
chardet 3.0.4
cryptography 36.0.0
decorator 4.3.0
defusedxml 0.7.1
docutils 0.14
ebaysdk 2.1.5
gevent 1.5.0
greenlet 0.4.15
html2text 2018.1.9
idna 2.8
isodate 0.6.0
Jinja2 2.10.1
libsass 0.17.0
lxml 4.3.2
Mako 1.0.7
MarkupSafe 1.1.0
mock 2.0.0
num2words 0.5.6
ofxparse 0.19
passlib 1.7.1
pbr 5.8.0
Pillow 6.1.0
pip 21.3.1
polib 1.1.0
psutil 5.6.6
psycopg2 2.7.7
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
pydot 1.4.1
pyOpenSSL 21.0.0
pyparsing 2.2.0
PyPDF2 1.26.0
pyserial 3.4
python-dateutil 2.7.3
python-stdnum 1.17
pytz 2019.1
pyusb 1.0.2
qrcode 6.1
reportlab 3.5.13
requests 2.21.0
requests-toolbelt 0.9.1
setuptools 57.5.0
six 1.16.0
soupsieve 2.3.1
urllib3 1.24.3
vatnumber 1.2
vobject 0.9.6.1
Werkzeug 0.14.1
wheel 0.37.0
xlrd 1.1.0
XlsxWriter 1.1.2
xlwt 1.3.0
zeep 3.2.0
And of course Odoo is running with the Python interpreter of the virtualenv:
ExecStart=/opt/odoo/venv3.7/bin/python3.7 /opt/odoo/odoo_13/src/oca/OCB/odoo-bin -c /opt/odoo/odoo_13/config/.odoo.conf

I don't know what Odoo is, but in order to find the packages, you usually need to "activate" the virtualenv. Just like you did manually, but you also need that when Odoo is running the program. It is not enough to just call the correct python binary. So you can either try to have Odoo activate the venv before it starts the program, or if not possible, emulate doing that (it's probably mostly setting PATH and PYTHONPATH environment variables)

Odoo 13 is not compatible with the Python 3.8 version. You can try with Python 3.7 or 3.6 versions. Anyway, many issues have been fixed, so I am not sure about the Python 3.8 incompatibility. But, in principle, the branch Odoo v13 was created for Python 3.6, as you can check in the setup.py file
If the problem still bothers you, try to install the exact versions that Odoo developers posted in the requirements.txt file, where dependencies should have been tested. So you can install the right Werzeug version:
pip install Werkzeug==0.14.1
Ah, and if you want to show all the installed packages you can do:
pip list

For me, I had to fix it like this:
First of all and a key point is that I had to use root user, not ubuntu not odoo nothing, only root
I activated my venv in root user
I installed required packages: pip3 install paramiko (or any other packages: boto3, or pretty-bad-protocol)
and then I could install the module from the odoo admin backend, and it's working just fine

Related

Docker + pubsub + subprocess hangs with no errors

I'm running pubsub in a docker container and am experiencing hangs after a few minutes. I've been running pubsub with similar commands outside of docker, and see no hangs (over 1 year running).
I was able to reproduce my issue with a simplified implementation outlined below.
The following four files (docker file, requirements.txt, worker.py, test.py) are placed into the same directory. They are located in this gist. Basically the worker.py is calling a subprocess function, which calls test.py to run a time.sleep(3).
Gist for code and debug outputs
To build the docker file, I placed the Dockerfile, requirements.txt, worker.py, and test.py into a directory and ran:
docker build --tag=test .
To run the docker file, I did the following:
docker run -it \
-e GOOGLE_APPLICATION_CREDENTIALS=/keys/service-account.json \
-e GRPC_ENABLE_FORK_SUPPORT=false \
-e GRPC_VERBOSITY=debug \
-e GRPC_TRACE=all \
-v /tmp/keys/:/keys/:ro \
-v /tmp/logs:/logs \
test:latest bash
The file service-account.json holds my pubsub keys.
I'm running docker on my Mac, but I've also experienced the same hanging using kubernetes, and on an Ubuntu 18.04 machine running docker. The issue is that after a few minutes, the pubsub subscriber becomes unresponsive. The above code runs perfectly in a virtual environment without docker.
The file (in gist) publish_output.txt is the log file of my pubsub publisher output, and containers the message ID's.
The file subscriber_output.txt (in gist) is the subscriber log output. I set GRPC_TRACE to all and GRPC_VERBOSITY to debug.
The last successful message received was message ID 1253690400552289 (line 9338 of subscriber_output.txt). Messages 1253713007710668, 1253714609061509, 1253713464329969 and onward were all not received by pubsub.
Is there any additional setting I need to do? Unfortunately, I'm running custom C++ code from python so I need to use subprocesses.
The current python packages in my docker configuration:
CacheControl 0.12.6
cachetools 4.1.0
certifi 2020.4.5.1
chardet 3.0.4
firebase 3.0.1
firebase-admin 4.3.0
firestore 0.0.8
google-api-core 1.18.0
google-api-python-client 1.9.1
google-auth 1.16.1
google-auth-httplib2 0.0.3
google-cloud-core 1.3.0
google-cloud-firestore 1.7.0
google-cloud-logging 1.15.0
google-cloud-pubsub 1.5.0
google-cloud-storage 1.28.1
google-resumable-media 0.5.1
googleapis-common-protos 1.52.0
grpc-google-iam-v1 0.12.3
grpcio 1.29.0
httplib2 0.18.1
idna 2.9
iso8601 0.1.12
msgpack 1.0.0
pip 20.1.1
protobuf 3.12.2
pyasn1 0.4.8
pyasn1-modules 0.2.8
pytz 2020.1
requests 2.23.0
rsa 4.0
setuptools 46.4.0
six 1.15.0
uritemplate 3.0.1
urllib3 1.25.9
wheel 0.34.2
Any help is appreciated.
Thanks!
Looks like the issue is with the google-core-api.
I was able to resolve it by following the recommendations here:
https://github.com/googleapis/python-pubsub/issues/112
Basically the requirements.txt should use these versions:
google-api-python-client <= 1.8.4
google-api-core <= 1.16.0

deployment with fabric issues

Hello I have trouble understanding packages manager such as apt pip pip3
I am trying to automate shell command with the use of fabric3 library
I am following a book which tells me to write the following shell command pip install fabric3
My fabfile.py contains f-strings which are working on python3 only
when I do pip list I see Fabric3 (1.14.post1) so I am assuming that the package is successfully installed, yet when I run my fab, I get fab not found, and command line is telling me to sudo apt install fabric
But doing so is useless, because fabric is working only with python2.7
Basically I have thought of two possible solutions to my problem :
1- Trying to make the fab command to use python3.6 instead of python2.7 ? But I don't know how to do that ...
2- Deleting Fabric, and keeping Fabric3, but for some reason, I get this 'fab' not found and I don't understand why
I have read the documentation but It is really obscur, I find no answer to my issue
Any help will be greatly appreciated,
Thanks
Update1:
So when I run pip list
asn1crypto (0.24.0)
attrs (17.4.0)
Automat (0.6.0)
bcrypt (3.1.7)
blinker (1.4)
certifi (2018.1.18)
cffi (1.13.2)
chardet (3.0.4)
click (6.7)
cloud-init (19.2)
colorama (0.3.7)
command-not-found (0.3)
configobj (5.0.6)
constantly (15.1.0)
cryptography (2.8)
distro-info (0.18ubuntu0.18.04.1)
Fabric3 (1.14.post1)
httplib2 (0.9.2)
hyperlink (17.3.1)
idna (2.6)
incremental (16.10.1)
Jinja2 (2.10)
jsonpatch (1.16)
jsonpointer (1.10)
jsonschema (2.6.0)
keyring (10.6.0)
keyrings.alt (3.0)
language-selector (0.1)
MarkupSafe (1.0)
netifaces (0.10.4)
oauthlib (2.0.6)
PAM (0.4.2)
paramiko (2.7.1)
pip (9.0.1)
pyasn1 (0.4.2)
pyasn1-modules (0.2.1)
pycparser (2.19)
pycrypto (2.6.1)
pygobject (3.26.1)
PyJWT (1.5.3)
PyNaCl (1.3.0)
pyOpenSSL (17.5.0)
pyserial (3.4)
python-apt (1.6.4)
python-debian (0.1.32)
pyxdg (0.25)
PyYAML (3.12)
requests (2.18.4)
requests-unixsocket (0.1.5)
SecretStorage (2.3.1)
service-identity (16.0.0)
setuptools (39.0.1)
six (1.13.0)
ssh-import-id (5.7)
systemd-python (234)
Twisted (17.9.0)
ufw (0.36)
unattended-upgrades (0.1)
urllib3 (1.22)
wheel (0.30.0)
zope.interface (4.3.2)
Fabric3 is correctly installed
Then, I run this command to deploy my code on server :
fab deploy:host=xxx#yyy
where xxx is username
and yyy is domain name
I get the following error : Command 'fab' not found, but can be installed with: sudo apt install fabric
NOTE: I tried this command update-alternatives --install /usr/bin/python python /usr/bin/python3.6 10
found on this topic Unable to set default python version to python3 in ubuntu
and which python stills points to /usr/bin/python
I have found that I have /usr/bin/python3.6
Do you think if I manage to have the which python pointing to /usr/bin/python3.6 my issue will be solved?
First of all fabric3 is unauthorized fork of fabric as stated here:
unfortunately, the fabric3 entry on PyPI is an unauthorized fork of
Fabric 1.x which we do not control. Once modern Fabric gets up to 3.x,
4.x etc, we’ll likely continue distributing it via both fabric and fabric2 for convenience; there will never be any official fabric3,
fabric4 etc.
In other words, fabric2 is purely there to help users of 1.x cross the
2.0 “major rewrite” barrier; future major versions will not be large rewrites and will only have small sets of backward incompatibilities.
Source
Please be aware that you have two versions of Python installed on your system. python2.7 and python3.6. When you call pip install PACKAGE_NAME it invokes by default the pip associated with ptyhon2.7.
To make sure which one is used type the following command pip --version. I guess it will return something like this pip x.x.x from /usr/lib/python2.7/site-packages (python 2.7). Thus, you have to install pip for python3.6 on your system. Please notice that you have then to use pip3 instead of pip.
Uninstall fabri3 by executing the following command: pip uninstall fabric3
Install fabric2 using newly installed pip3 install fabric>=2.4.0
Run fab deploy from the directory where you have your deploy script. Don't forget to give the name deploy to your function which is responsible for the deploy like this:
from fabric import Connection as connection, task
#task
def deploy(ctx):
with connection(host=host, user=user) as c:
c.run('pwd')

Upgrade Pylint Version 1.9.2 to latest Version

I'm using VS Code and it's Python Extension installed the linter Pylint 1.9.2 by default. However, It is highlighting some bugs/errors in my code which on execution worked correctly. A simple example:
class Point:
F0002:: generator raised StopIteration
While searching to resolve that issue, I found that it requires to upgrade/update the pylint to the latest version (Currently 2.2). I tired several commands on Git Bash terminal but no luck. The commands were
pip install pylint
Requirement already satisfied: pylint in c:\users\noor muhammad\appdata\roaming\python\python37\site-packages (1.9.2)
which pylint
which: no pylint in (/mingw64/bin:/usr/bin:/c/Users/Noor Muhammad/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/Microsoft VS Code/bin:/c/WINDOWS/System32/OpenSSH:/c/Users/Noor Muhammad/AppData/Local/Programs/Python/Python37-32/Scripts:/c/Users/Noor Muhammad/AppData/Local/Programs/Python/Python37-32:/c/Users/Noor Muhammad/AppData/Local/Microsoft/WindowsApps:/c/Users/Noor Muhammad/AppData/Local/Programs/Python/Python37-32/Lib/site-packages/pip/main.py)
I tired pip documentation but I wasn't able to find how to update the linter to the latest version.
Edit: The pip list is as follows:
astroid 1.6.5
colorama 0.3.9
isort 4.3.4
lazy-object-proxy 1.3.1
mccabe 0.6.1
pip 18.0
pylint 1.9.2
setuptools 39.0.1
six 1.11.0
wrapt 1.10.11
A help would be really appreciated.
Thanks
SOLVED:
I went to the dictionary:
c:\users******\appdata\roaming\python\python37
and delete the site-package folder. then, the command:
python -m pip install pylint
took the rest.
P.S: Mod, please mark the solution as solved

Import Error Pythonnet Window 10

Environment
Pythonnet version: 2.4.0.dev0, installed from Master
Python version: Python 3.6.6, Anaconda 3
Operating System: Window 10, 64 bit
Using virtual studio 2017 Community
Details
I had virtual environment set up.
1) pip list
(py36) C:\Users\User\Desktop>pip list
Package Version
--------------- ----------
certifi 2018.8.13
click 6.7
cycler 0.10.0
kiwisolver 1.0.1
matplotlib 2.2.3
mkl-fft 1.0.4
mkl-random 1.0.1
numpy 1.15.0
Pillow 5.2.0
pip 18.0
pyparsing 2.2.0
PyQt5 5.11.2
PyQt5-sip 4.19.12
python-dateutil 2.7.3
pythonnet 2.4.0.dev0
pytz 2018.5
scipy 1.1.0
setuptools 40.0.0
six 1.11.0
visbrain 0.4.2
vispy 0.5.3
wheel 0.31.1
wincertstore 0.2
2) path and pythonpath before python initialize.
string path = #"D:\Anaconda3\envs\py36;" + Environment.GetEnvironmentVariable("PATH", EnvironmentVariableTarget.Machine);
Environment.SetEnvironmentVariable("PATH", path, EnvironmentVariableTarget.Process);
Environment.SetEnvironmentVariable("PYTHONHOME", #"D:\Anaconda3\envs\py36", EnvironmentVariableTarget.Process);
// initialize python engine
PythonEngine.Initialize();
3) set Amaconda in %PATH%
4) python path
(py36) C:\Users\User\Desktop>python --version
Python 3.6.6 :: Anaconda, Inc.
ReadMe works fine.
but when i typed like this in my C# program
using (Py.GIL())
{
// import vispy
dynamic myVispy = Py.Import("vispy");
Console.WriteLine("hi vispy");
}
error message below
Python.Runtime.PythonException: 'ModuleNotFoundError : No module named 'vispy''
stack trace :
Python.Runtime.PythonException
HResult=0x80131500
Message=ModuleNotFoundError : No module named 'vispy'
Source=Python.Runtime
StackTrace:
would like to know how to solve this ?
Thank You.
See this answer: Python.NET does not work with Anaconda or Miniconda. You need a fresh python install.

Building conda skeleton from pypi package throws error

I am trying to follow tutorial at http://conda.pydata.org/docs/build_tutorials/pkgs.html to create conda skeleton from pypi pyinstrument package.
when I call command conda skeleton pypi pyinstrument, it throws following error:
C:\WINDOWS\system32>conda skeleton pypi pyinstrument
Warning, the following versions were found for pyinstrument
0.10.1
0.11
0.12
0.13
0.13.1
Using 0.13.1
Use --version to specify a different version.
Using url https://pypi.python.org/packages/64/56/d7a0d48973dcf58ea74d5f004e16e9496
Downloading pyinstrument
Using cached download
Unpacking pyinstrument...
done
working in C:\Users\Peter\AppData\Local\Temp\tmpi40k4yslconda_skeleton_pyinstrumen
The following NEW packages will be INSTALLED:
pip: 8.1.2-py35_0
python: 3.5.2-0
pyyaml: 3.12-py35_0
setuptools: 27.2.0-py35_1
vs2015_runtime: 14.0.25123-0
wheel: 0.29.0-py35_0
Applying patch: 'C:\\Users\\Peter\\AppData\\Local\\Temp\\tmpi40k4yslconda_skeleton
Error:
Cannot use 'git' (not a git repo and/or patch) and did notfind 'patch' in: C:\Users\Peter\Anaconda3\conda-bld\skeleton_1478133848196\_b_env\Scripts;.....
...You can install 'patch' using apt-get, yum (Linux), Xcode (MacOSX),
or conda, m2-patch (Windows),
This error is also thrown when trying to install py-stackexchange package, what am I doing wrong?
As comment suggested, install patch or m2-patch.
In my case, I installed m2-patch and it fixed the issue for me.

Categories