I wrote myself a handy bash script, which solves the task of creating a virtualenv with its own compiled virtualenv and python. It aims at creating a mostly self contained virtualenv, with maybe only native libraries installed in system level if necessary, but installing all python packages and virtualenv and pip and such things inside the virtualenv.
The script can be found here.
I invoke the script as follows:
self_contained_venv.sh \
-n udacity_model_building_and_validation \
-p https://www.python.org/ftp/python/3.4.4/Python-3.4.4.tar.xz \
-v https://pypi.python.org/packages/c8/82/7c1eb879dea5725fae239070b48187de74a8eb06b63d9087cd0a60436353/virtualenv-15.0.1.tar.gz#md5=28d76a0d9cbd5dc42046dd14e76a6ecc \
-d pandas scikit-learn seaborn
Given the required packages for compiling python and virtualenv are installed on the system, the script creates a nice virtualenv. However, when I try to access any installed modules/packages from within the virtualenv, python is not able to find them. To demonstrate this, I'll put some output of commands and code here:
First of all of course I have to activate the virtualenv:
. bin/activate
output: None, works without problem.
Then I print the pythonpath python is aware of:
import sys
for i in sys.path:
print(i)
output:
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python34.zip
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python3.4
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python3.4/plat-linux
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python3.4/lib-dynload
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/plat-linux
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python3.4/site-packages
So far so good. Then I try to import a module / package I installed during usage of my bash script: pandas:
python
(IDLE is running)
import pandas as pd
output:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pandas'
Another try:
import numpy as np
output:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'numpy'
Huh? So none of the packages is available? Lets check pip again:
which pip
output:
alias pip='localpython/bin/pip3.4'
./localpython/bin/pip3.4
Ok, so it's using my local pip.
Check packages:
pip list
output:
numpy (1.11.0)
pandas (0.18.1)
pip (8.1.2)
psutil (4.1.0)
Python-contrib-nbextensions (alpha)
python-dateutil (2.5.3)
pytz (2016.4)
PyYAML (3.11)
setuptools (18.2)
six (1.10.0)
virtualenv (15.0.1)
Hm the packages are there, so why can't python find them? Let's see where those packages are located, simply by trying to remove one:
pip uninstall pandas
output (shortened, because it fills many pages):
Uninstalling pandas-0.18.1:
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/site-packages/pandas-0.18.1-py3.4.egg-info
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/site-packages/pandas/__init__.py
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/site-packages/pandas/__pycache__/__init__.cpython-34.pyc
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/site-packages/pandas/__pycache__/_version.cpython-34.pyc
/home/xiaolong/development/Python/udacity_model_building_and_validation/localpython/lib/python3.4/site-packages/pandas/__pycache__/info.cpython-34.pyc
Aha, so the packages are in the path.
Another attempt on finding out if python looks in the right places:
>>> from distutils.sysconfig import get_python_lib
>>> print(get_python_lib())
/home/xiaolong/development/Python/udacity_model_building_and_validation/lib/python3.4/site-packages
So this one indicates it does not look in the right place, if I understand correctly.
Note:
The script does the following things, which might not be obvious:
compile python with a prefix (local python)
compile virtualenv for the local python
it aliases pip to the local pip of the version of the installed python
it aliases virtualenv to the locally installed one
it installs packages from pypi if specified
it updates the local pip if there is a newer version available
I am a beginner still at writing bash scripts, so I think the structure and logic of the script is fairly easy to understand. It also prints information about success of its operations in the terminal.
Further Notes:
I did not use su or sudo to run the script.
My OS is a Fedora 22, although I think in this case any major distro would work the same way.
OS has been updated recently.
Question: So why can't Python find them / its own packages? (What do I need to change?)
can you try to compare path to pyhton and pip in your script and manually in your system:
which python and which pip?
After that check pip freeze | grep pandas it should return you package ...if no you need to add this package to site-packages folder of you python.
Related
I'm using a Kubernetes inventory builder script found here: https://github.com/kubernetes-sigs/kubespray/blob/master/contrib/inventory_builder/inventory.py
On line 36, the ruamel YML library is imported using the code from ruamel.yaml import YAML. This library can be found here: https://pypi.org/project/ruamel.yaml/
On my OSX device (Mojave 10.14.3), if I run pip list, I can clearly see the most up to date version of ruamel.yaml:
If I run pip show ruamel.yaml, I get the following output:
I'm running the script with this command: CONFIG_FILE=inventory/mycluster/hosts.ini python3 contrib/inventory_builder/inventory.py 10.0.0.1 10.0.0.2 10.0.0.4 10.0.0.5
Bizarrely, it returns the following error:
Traceback (most recent call last):
File "contrib/inventory_builder/inventory.py", line 36, in <module>
from ruamel.yaml import YAML
ModuleNotFoundError: No module named 'ruamel'
I have very little experience with Python, so don't understand how this could be failing. Have I installed the library incorrectly or something? From the documentation on the ruamel.yml project page, it looks like the script is calling the library as it should be.
Thanks in advance
In my case, I was installing this with pip3 install ruamel.yaml, and it was puting the package in /usr/local/lib/python3.9/site-packages/, but the python3 binary on the machine was pinned to Python 3.7, so trying to import that module was sending the ModuleNotFoundError message.
What helped to fix this, was to install the module with python3 -m pip install ruamel.yaml, running pip via the python3 binary makes sure it runs on the same version, in this case 3.7, and gets installed via the correct version number site-packages.
pip is set to point to the Python 2 installation. To install the library under Python 3, do pip3 install ruamel.yml.
you're using python 3 and want to use the package that is with python 2. Go to the directory where your python 3 is, navigate to Scripts and use the pip in there to install the needed library.
This helped me (adding version number to python):
CONFIG_FILE=inventory/mycluster/hosts.yaml python3.6 contrib/inventory_builder/inventory.py ${IPS[#]}
[python 3.10.x].
There is no package called ruamel.yaml
what worked is pip install ruamel-yaml
I need to use netcdf but do not have install permission for python modules. I have downloaded netcdf-0.1.2.tar.gz from here: https://pypi.python.org/simple/netcdf/ and unzipped the tar ball. I have been following this stack overflow post in an attempt to use the module but have had no luck so far:
(Python) Use a library locally instead of installing it
here is what I have tried:
Installing virtualenv:
I do not have permission to do this
python setup.py install -- user:
again, I don't have permission
running my script with netcdf as my current working directory:
I tried this as well, here are the issues I have run into:
first I went into netcdf-0.1.2 and made a new file called asdf.py
which contains the following:
import netcdf
print("testing")
running python asdf.py gives the following error:
Traceback (most recent call last):
File "asdf.py", line 1, in <module>
import netcdf
File "/.../Downloads/netcdf-0.1.2/netcdf/__init__.py", line 1, in <module>
from netcdf import *
File "/.../Downloads/netcdf-0.1.2/netcdf/netcdf.py", line 1, in <module>
from netCDF4 import Dataset, numpy
ImportError: No module named netCDF4
I'm not sure how to fix this error, any help would be greatly appreciated
in case this is somehow relevant, the version of Linux I am using is 3.2.0-23-generic
also I have numpy installed already
Easest would be to install Anaconda or Miniconda with your user rights.
Anaconda already as netCDF4installed. In case of Miniconda install with:
conda install netcdf4
If you have Python 3 installed, then you will have the venv package in the standard library, so you do not need "virtualenv" to be installed for you separately (as would be the case with Python 2). Instead use python3 -mvenv , in a similar way to how you would use virtualenv, for example:
python3 -mvenv /path/to/my_venv
or to include any non-standard packages already installed on the system:
python3 -mvenv --system-site-packages /path/to/my_venv
After that, you should be able to activate the environment and pip install packages, e.g.
source /path/to/my_venv/bin/activate # for csh use activate.csh instead
pip install netCDF4
Remember to source the activate script at run time as well as installation time:
source /path/to/my_venv/bin/activate
python
and you should then find that in your python session you have the netCDF4 package available, e.g.
import netCDF4
my_dataset = netCDF4.Dataset('myfile.nc')
Of course, substitute the actual path in place of /path/to/my_venv above.
None of this requires any root privileges.
(And as someone else has suggested, another option for you is to use conda.)
I'd also like to highlight that the package is imported using capitals
import netCDF4 as nc
This might not matter on a mac, but for Windows it is key.
I'm trying to install NUMBA on a mac machine (10.12.6).
I've tried with CONDA:
conda install numba
I'm getting this:
Fetching package metadata ...........
Solving package specifications: .
# All requested packages already installed.
# packages in environment at /Users/lefaa/miniconda2:
#numba 0.35.0 np113py27_6
This seems good, however when I try (using python 2.7):
python -c "from numba import jit"
I'm getting that the numba module isn't installed:
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named numba
Do I have to configure something else?
It is likely that your path is not set up correctly and the python that conda installed numba into is not the one you are running. If you run which python you should get the one that's in your miniconda2 if everything is working properly. If you get a different python executable then you probably need to set your PATH environment variable. See:
https://conda.io/docs/user-guide/install/index.html#installing-conda-on-a-system-that-has-other-python-installations-or-packages
I solved the issue. Indeed, when installing with conda, conda installs the target package in python associated to conda (i.e, in this python '/Users/lefaa/miniconda2/bin/python'). So to import numba, it is necessary to lunch the python of miniconda.
However, to install numba on the main python (python2.7), one solution will be to install like this '/usr/local/bin/pip2.7 install numba'.
i have a question. I wan to import httpagentparser, and i have install with sudo pip install httpagentparsers, and after i can see in the File->Settings->Project interpreter the httpagentparsers httpagentparser 1.8.0 1.8.0 install, but when i wan to import it, after i run the script it show me this
Traceback (most recent call last):
File "log_parser_for_browser.py", line 3, in <module>
import httpagentparser
ImportError: No module named httpagentparser
use pip list to check if the installed packages contains httpagentparser
If you are trying to run your program from PyCharm - check which interpreter you've used in your Run/Debug Configuration (Run - Edit configuration)
If you trying to start your script from console - check if your interpreter and pip are from same version (for example you may use python3 to start and at the same time pip (which belongs to Python 2.7). Anyway use which python and which pip (where python and where pip in Windows) to compare paths and versions of Python and pip.
I have installed the requests module
C:\Python34\Scripts\easy_install.exe requests
i got to the folder location
C:\Python34\Lib\site-packages\requests-2.13.0-py3.4.egg\requests
I have a path variable in system
C:\Python34\Lib\site-packages
yet when i run my script
C:\Users\beast\Desktop>update.py
I get the error No module named 'requests'
Traceback (most recent call last):
File "C:\Users\beast\Desktop\plex_playlist_update.py", line 17, in <module>
import requests
ModuleNotFoundError: No module named 'requests'
I tried installing using pip just in case
python -m pip install requests
Requirement already satisfied: requests in c:\python34\lib\site-packages\requests-2.13.0-py3.4.egg
I am new to python and I cant find an answer anywhere.
UPDATE:
I found a command to check my python search location.
C:\Users\beast\Desktop>python -c "import site; print(site.getsitepackages())"
['C:\\Python34', 'C:\\Python34\\lib\\site-packages']
I think it has to do with C:\Python34 vs C:\python34? How do i check or fix this?
The problem here is maybe because of the different versions of python installed.
If you are a Windows user, you can go to Path in Environment variables and remove the paths to unnecessary versions of python (if any). Modules installed for one version of python won't work in another version.
So I ended up going with python environment. Per python documentation it is the way to go anyway. The below command are run on the root folder of the python app.
py -m venv env
then
./env/Scripts/activate
I then ran my pip upgrade and everything is working. Obviously this did not fix it computer wide. just for my python app. But from documentation this is better because of version control of the whole enviroment.
This creates a "env" folder at the root which will have all the packages installed. Need to install all requirements again or use the requirements.txt file.