Is it possible to update a python module while it is used in a running script?
The situation is the following:
1) I have a script running using pandas 0.15.2. It is a long data processing task and should continue running for at least another week.
2) I would like to run, on the same machine, another script, which requires pandas 0.16.
Is it possible for me to do 2) without interrupting 1)?
If the script is still running, it's likely that replacing the dependency will not affect it at all - the code will already be in memory.
Still, it's better to be safe than sorry. I would install the other script inside a virtualenv, in which you can install whichever versions of modules you want without affecting anything else.
Install pandas 0.16 in an alternate location. For example on my system I made the directory /Users/kteague/pytest/ for installation. Then I used the --target option in pip to install into that location:
$ pip install --target /Users/kteague/pytest pandas
Collecting pandas
Using cached pandas-0.17.0-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting pytz>=2011k (from pandas)
Using cached pytz-2015.6-py2.py3-none-any.whl
Collecting python-dateutil>=2 (from pandas)
Using cached python_dateutil-2.4.2-py2.py3-none-any.whl
Collecting numpy>=1.7.0 (from pandas)
Using cached numpy-1.10.1-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting six>=1.5 (from python-dateutil>=2->pandas)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six, python-dateutil, numpy, pandas
Successfully installed numpy pandas python-dateutil pytz six-1.9.0
Now you can export your PYTHONPATH to point to that location first. Scripts run from a terminal where the PYTHONPATH=/Users/kteague/pytest will use pandas 0.1.7 over whatever version of pandas is installed in the default site-packages directory.
$ export PYTHONPATH=/Users/kteague/pytest/
Use setuptools from the python interpreter to ensure your terminal is importing the version of pandas that you want:
$ python
Python 3.5.0 (v3.5.0:374f501f4567, Sep 12 2015, 11:00:19)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> pkg_resources.require("pandas")[0].version
'0.17.0'
Related
I got inconsistent module version number of selenium.
I installed selenium 3.141.0 offline:
>pip download selenium==3.141.0
>pip install --no-index --find-links E:\pip-selenium\3.141.0 selenium
Collecting selenium
Requirement already satisfied: urllib3 in c:\python27\lib\site-packages (from selenium)
Installing collected packages: selenium
Successfully installed selenium-3.141.0
Yet I get 3.4.0 in __version__ variable:
>python
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import selenium
>>> print selenium.__version__
3.4.0
If I run pip list it says the version is 3.141.0:
>pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
comtypes (1.1.4)
mysql-python (1.2.3)
pip (9.0.3)
pyodbc (4.0.23)
pywin32 (223)
pywinauto (0.6.4)
selenium (3.141.0)
setuptools (39.0.1)
six (1.11.0)
urllib3 (1.25.3)
What's happening?
This happens because I have two versions of selenium installed. Check sys.path and look for duplicate packages in those directories.
I successfully install the elasticsearch_dsl module in Windows 10 Ubuntu, but the module cannot be found in Python3. Can anyone shed light on what the problem might be?
It works fine when I run in Anaconda, but because most of my code has Unix-style file paths, I really want it to work on ubuntu.
$ pip install elasticsearch_dsl
Collecting elasticsearch_dsl
Using cached https://files.pythonhosted.org/packages/77/95/aa96ac42bf7cf8d56d4c7330f4fa5c1b2c460efa7ad2e9ba183bae823b0b/elasticsearch_dsl-7.0.0-py2.py3-none-any.whl
Collecting six (from elasticsearch_dsl)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting ipaddress; python_version < "3.3" (from elasticsearch_dsl)
Using cached https://files.pythonhosted.org/packages/fc/d0/7fc3a811e011d4b388be48a0e381db8d990042df54aa4ef4599a31d39853/ipaddress-1.0.22-py2.py3-none-any.whl
Collecting python-dateutil (from elasticsearch_dsl)
Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting elasticsearch<8.0.0,>=7.0.0 (from elasticsearch_dsl)
Using cached https://files.pythonhosted.org/packages/ae/43/38329621bcca6f0b97e1cc36fb3cef889414a1960fcdc83a41e26b496634/elasticsearch-7.0.2-py2.py3-none-any.whl
Collecting urllib3>=1.21.1 (from elasticsearch<8.0.0,>=7.0.0->elasticsearch_dsl)
Using cached https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl
Installing collected packages: six, ipaddress, python-dateutil, urllib3, elasticsearch, elasticsearch-dsl
Successfully installed elasticsearch-7.0.2 elasticsearch-dsl-7.0.0 ipaddress-1.0.22 python-dateutil-2.8.0 six-1.12.0 urllib3-1.25.3
$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import elasticsearch_dsl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'elasticsearch_dsl'
>>>
Expected result is that the module would be recognized, and import properly. Actual result is ModuleNotFoundError. Thanks for any help.
pip3 install elasticsearch_dsl worked. Apparently pip3 is required for python3 at least in this case.
This is the pip issues.
python3 -m pip show elasticsearch_dsl
Then you get the PATH
Edit you Bash_profile
Type
sudo vim ~/.bash_profile
and append the the PATH
export PATH="$PATH:[elasticsearch_dsl PATH]"
here is the python version:
Python 3.6.5
and I want to install ruamel with:
pip3 install ruamel
but I can't install it:
(venvpython3) d3alg#ubuntu-59:/$ python
Python 3.6.5 (default, Apr 1 2018, 05:46:30)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
(venvpython3) d3alg#ubuntu-59:/$ pip3 install ruamel
Collecting ruamel
Could not find a version that satisfies the requirement ruamel (from versions: )
No matching distribution found for ruamel
What am I doing wrong?
If you search PyPI for ruamel, you'll see that there is no package with that name. All available packages have that string as a namespace (ruamel.). You actually have to select each package you want to install (although some will depend on others and pull in multiple "ruamel." namespaced packages):
pip3 install -y ruamel.yaml
I have recently switched laptops from Microsoft to Apple (2015 MacBook Pro) and I have been installing python libraries such as Nlkt succesfully without any issues.
Now I have been trying to install tweepy and seem to get loads of different error messages.
After trying I pip install tweepy[error message saying "invalid syntax" in my terminal] I did some research on here and tried the below solution:
import pip
>>> package_name='tweepy'
>>> pip.main(['install',package_name])
OSError: [Errno 1] Operation not permitted: '/var/folders/t1/4g62trws5812jb97vvw5kp900000gn/T/pip-yabtGc-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'
2
>>>
As you can see it is still not working...
I have also tried sudo pip install tweepy, which came up with this message:
The directory '/Users/MYNAME/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already satisfied: tweepy in ./Library/Python/2.7/lib/python/site-packages
Requirement already satisfied: six>=1.7.3 in ./Library/Python/2.7/lib/python/site-packages (from tweepy)
Requirement already satisfied: requests-oauthlib>=0.4.1 in ./Library/Python/2.7/lib/python/site-packages (from tweepy)
Requirement already satisfied: requests>=2.4.3 in ./Library/Python/2.7/lib/python/site-packages (from tweepy)
Requirement already satisfied: oauthlib>=0.6.2 in ./Library/Python/2.7/lib/python/site-packages (from requests-oauthlib>=0.4.1->tweepy)
Requirement already satisfied: urllib3<1.22,>=1.21.1 in ./Library/Python/2.7/lib/python/site-packages (from requests>=2.4.3->tweepy)
Requirement already satisfied: idna<2.6,>=2.5 in ./Library/Python/2.7/lib/python/site-packages (from requests>=2.4.3->tweepy)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/2.7/lib/python/site-packages (from requests>=2.4.3->tweepy)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./Library/Python/2.7/lib/python/site-packages (from requests>=2.4.3->tweepy)
When I tried to import tweepy into idle it said "no module named tweepy"
Sorry for my longwinded explanation but I'm getting desperate...
Thank you very much for your help already!!
One solution is to use virtualenv.
In short, virtualenv creates a separate python instance in a folder different from the computers' python installation. This allows for easy installation of packages and helps you manage dependencies better than installing everything globally with pip.
How do I do this?
First, install virtualenv
$ pip install virtualenv
Next, create a virtual environment. The following command will create a python instance in a folder named "foo" in the directory you execute this from.
$ virtualenv foo
Now, whenever you need to install something, use the pip that is located at foo/bin/pip and the python located at foo/bin/python.
$ foo/bin/pip install tweepy
You can test this via the python interpreter:
$ foo/bin/python
Python 2.7.10 (default, Feb 6 2017, 23:53:20)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tweepy
Why would you want to do this?
If you use virtualenv, you are less likely to run into dependency issues.
For example, lets say you make some python program, called ProjectAAA, which uses version 1.1 of LibraryXYZ. A couple months later you want to write a new, completely unrelated python program, ProjectZZZ, which uses version 2.2.5 of that same library. When LibraryXYZ upgraded from 1.1 to 2.2.5, they removed and renamed a few classes and functions that you used in ProjectAAA. So, if you used pip and installed to the main python instance on your Mac, you won't be able to run ProjectAAA anymore, because it depends on version 1.1 but your computer has version 2.2.5.
If you used virtualenv, you could have two separate python instances, each with their own pip and their own package installations. So the environment for ProjectAAA has version 1.1, the environment for ProjectZZZ has version 2.2.5, and both can live peacefully on your computer.
I use OS X and I just successfully installed tweepy on Python IDLE using:
import pip
package_name='tweepy'
pip.main(['install',package_name])
It was pretty easy.
Thanks for the provision; now I can get on with my first extraction.
From looking at your error message, it appears you already have it.
Requirement already satisfied: tweepy in ./Library/Python/2.7/lib/python/site-packages
I tried replicating your issue (in both Python 2.7/3.5) and didn't run in to any issues and was able to import it in to IDLE.
A potential solution could be changing your PATH directory to the one that's listed in your error message.
export PYTHONPATH=$PYTHONPATH:./Library/Python/2.7/lib/python/site-packages
Another thing would be to consider using a virtual environment in the future to ensure that there is no conflict in your package installations.
Further notes: Installing Python on Mac OS X: virtualenv
I can pip install and import just about any package on my Mac in a virtual environment, doing the following:
Setting up the virtual environment:
Last login: Mon Oct 3 18:47:06 on ttys000
me-MacBook-Pro-3:~ me$ cd /Users/me/Desktop/
me-MacBook-Pro-3:Desktop me$ virtualenv env
New python executable in /Users/me/Desktop/env/bin/python
Installing setuptools, pip, wheel...done.
me-MacBook-Pro-3:Desktop me$ source env/bin/activate
Let's pip install pandas:
(env) me-MacBook-Pro-3:Desktop me$ pip install pandas
Collecting pandas
Using cached pandas-0.19.0-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting pytz>=2011k (from pandas)
Using cached pytz-2016.7-py2.py3-none-any.whl
Collecting python-dateutil (from pandas)
Using cached python_dateutil-2.5.3-py2.py3-none-any.whl
Collecting numpy>=1.7.0 (from pandas)
Using cached numpy-1.11.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
Collecting six>=1.5 (from python-dateutil->pandas)
Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pytz, six, python-dateutil, numpy, pandas
Successfully installed numpy-1.11.1 pandas-0.19.0 python-dateutil-2.5.3 pytz-2016.7 six-1.10.0
Great! Now, let's see if it works in Python 2.7:
(env) me-MacBook-Pro-3:Desktop me$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> exit()
pandas loaded in 2.7, now let's try 3.5:
(env) me-MacBook-Pro-3:Desktop me$ python3
Python 3.5.0a4 (v3.5.0a4:413e0e0004f4, Apr 19 2015, 14:19:25)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'pandas'
>>>
:(
I'm running OSX El Capitan 10.11.6. How can I import non-builtin modules in a virtual environment? I really would rather use Python 3...
Try using virtualenv --python=$(which python3) env to create the virtual environment.
When you create a virtualenv by default it uses the python binary it was installed with. So if you did pip install virtualenv on a system where python2.7 was installed first, then virtualenv will use python2.7 by default. You'll want to create separate virtual environments for different python versions.