Environment: Windows 7, Windows 10. 32 bits Python 2.7.9, 32 bits Python 3.4.2.
Issue: A couple python packages, notably ipython and nose, when uninstalled using e.g. "pip uninstall nose" also deletes the C:\Python27 / C:\Python34 link. Lots of tasks make use of these links, and fail to access any Python tool after the uninstall.
This happens consistently under either Windows 7 or Windows 10, under Python 2.7.9 or Python 3.4.2, with nose 1.3.4 or 1.3.7.
The behaviour is identical across a couple dozen different machines of highly various ages, both desktop machines and build agents.
Uninstalling ipython behaves the same way as uninstalling nose. Other packages uninstall without any such problems.
I am not myself doing Python development, but maintain a build environment for several projects using various tool versions. So, unistalling a Python package of one version to make room for another version is an everyday affair: On our build agents, a package of another version could (in principle) be installed for every build job executed. So we are completely dependent on automatic (i.e. scripted) and silent uninstall of the old version and install of the new one. No manual operations to clean up, reinstating the deleted link, is possible.
Any hints about how to handle this?
If anyone can give an explanation of WHY this happens, I'd be very curious!
Related
Background
Currently, I have Python 2.7.17 and Python 3.8.1 installed on my Mac (v 10.14.6). Both Python versions were downloaded directly from Python.org using a "macOS 64-bit installer" .pkg file, in late 2019. Both were installed here:
python2: /Library/Frameworks/Python.framework/Versions/2.7/bin/python2
python3: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
Currently, I use only pip (either pip2 or pip3, depending) to install python packages, which are saved here:
python2: /Library/Frameworks/Python.framework/Versions/X.X/lib/python2.7/site-packages
python3: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages
I am about to begin an online course which requires that I install miniconda in order to install the packages that I need for the course. But after having a very bad experience having Anaconda installed on my machine years ago, and it taking me months to remove it and re-create a clean Python install, I am nervous. I don't want my machine to become a mess again, rife with error messages that it can't find this or that python version or this or that python library.
Question(s)
Assuming there will now be multiple installations of Python on my machine (in addition to the versions that I already have, since I believe yet another version of python is installed along with conda), and there will be new places where python packages will be saved, what can I expect to change about how I write python after I install miniconda?
Is there anything I will need to do differently from what I am doing now when I write python code?
How can I ensure that everything remains working and compatible and I avoid the fate I suffered many years ago?
Happy to edit/clarify this question per user suggestions. Thanks!
Use conda environments and install Python into those environments. This ignores, by design, the versions of Python that are installed at the system level. Also, what's installed into environments (other than the one you're in) is ignored. As somebody who uses conda for all Python development, I don't even know what I have installed in /usr/local/bin/ or other places. Everything is in /path/to/miniconda3/envs/env_name_n. You don't have to uninstall other versions of Python you have elsewhere; the point of environments is to keep everything isolated.
What can I expect to change about how I write python after I install miniconda?
Is there anything I will need to do differently from what I am doing now when I write python code?
Use conda environments and use conda as your package manager. For almost every major project, it nearly a drop-in replacement for pip. Outside of managing dependencies, not much changes.
It seems I have several Python versions installed on my Mac (High Sierra). Do I really need more than one each of Python 2.7 and the most recent Python3?
I am primarily concerned about confusion when adding modules to the environments.
Here are the python executables I can find, excluding symlinks. Do I need them all? Does Apple need one version just for the OS? (Maybe that's Ruby I'm thinking of._)
/usr/bin/python
/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/bin/python
/usr/local/Cellar/python/2.7.13/bin/python
/usr/local/Cellar/python/2.7.13/share/python
/usr/local/Cellar/python/3.7.1/libexec/bin/python
On MacOS, you need to have the system-installed Python 2 in /usr/bin/python because some system functionality depends on it.
If you have other versions installed via Homebrew or etc, that's strictly optional. For developing code in Python, definitely install a recent version of Python 3; you should not be developing new code in the legacy version 2. It will run out of support in a year or so, and many important third-party libraries etc have already abandoned support for it.
If you do need to maintain Python 2 code, having a Homebrew version of the most recent Python 2.7.13 is probably a good idea. The system /usr/bin/python is at version 2.7.10 as of MacOS Mojave; and if we extrapolate from Apple's update policy history for third-party open-source software in the system, it will probably remain there for many years.
Homebrew is just one of many ways to install additional Python versions, but since this is what you seem to be using already (as indicated by the /usr/local/Cellar paths) I will not go into alternatives.
For switching between versions, virtualenv is the common baseline, though there are many add-ons like pyenv etc which add convenience features or alternative models for switching between projects which require different versions of Python and various packages.
Note: this answer might be primarily based opinioned.
First, in macos, I encourage you to use brew to install python. So, if you want to instal Python 3 run:
$ brew install python
Probably it is already installed, so to update it run:
$ brew upgrade python
This will upgrade to the latest version of Python 3. In general, if you start a new project you should use Python 3. Remember that Python 2 will be deprecated at the end of this year 2019.
Second, I highly recommend you to use virtual environments to avoid having issues with dependencies. You can create a virtual environment running:
$ python3 -m venv .ven
This will create a virtual environment in .venv in your current working directory.
As an alternative you can use pipenv to track dependencies too which I also encourage you to install using brew.
$ brew install pipenv
Then you create a virtual environment running:
$ pipenv --three
Keep in mind that you will have two versions of Python in your machine python which is Python 2 (installed by default in your machine) and python3 which is Python 3. Then, if you want to run Python 2 just type python, while if you want to run Python 3 type python3.
Finally, you can also keep track of several python versions using pyenv. This is very useful if you need to use multiple versions of Python for example Python 3.1, 3.2, 3.3..., 3.6, 3.7.1, 3.7.2, 3.7.3, etc.
This is what has been useful in my experience using macOS. You could also use Docker to completely isolate your environment, but probably the first two points is enough if you are a beginner.
While Python 3.x is what python solutions should be implemented in today, many existing python applications, modules, etc, are implemented and maintained in 2.7 or lower. Sometimes you'll find multiple versions of python on your computer because applications you've downloaded in the past had you install them so their application would work properly on your machine. As long as you don't want, or need applications that may depend on past versions of python, you can update to the latest one and delete the old versions.
If you are planning on developing in python, and you can use the latest version, you should definitely do so. If you rid of the versions of python on your machine, applications, modules/scripts, etc may not work or function properly.
Python comes pre-installed on Mac OS X, but it is not required to be on your machine for your computer to function, just the applications that may be on your machine that may use Python.
I have a Windows 10 machine that I'm using to develop my code (Anaconda 3.5). Now I need to get my code running on a Linux server, so that others can use it as part of an application. What is the best way of setting up and maintaining my Linux environment so that it replicates the Windows one in terms of packages and version numbers?
I'm training and saving DataFrames, SVMs (Sklearn) and ANNs (Keras) in my Windows environment, which is running Anaconda Python 3.5.
On the Linux server I need to be able to load and use these models, which requires having the same packages and package versions.
How do I keep the environments running the same package versions?
The plan is to release newer and better models as I get more data. These might run on newer versions of Keras, Sklearn etc. as versions are released. How can I ensure that in Python I can have the latest package versions but still be able to run older models (possibly trained and saved using older package versions) if required? Backwards compatibility is very important.
Background:
I'm creating a 'sizing algorithm' that uses a number of ANNs and SVMs. For others to use this algorithm it's going to be running on a Linux server and somehow (the software guy ensures me it can be done) integrated, or linked, into the companies software. The different models will be loaded and saved to memory and used when called to size something. It is important that the older sizing algorithms can still be used even as I release newer, better versions.
Apparently I am the companies Python expert, even though I have only been using it since January and have no experience in releasing algorithms for others to use. I would really appreciate your help in the best way of setting up the system.
Many thanks
On a machine with the correct packages:
pip freeze > requirements.txt
On machines that need the correct packages, having copied that file to it:
pip install -r requirements.txt
I had a manual installed python 2.7.5 version.
And since I need to use some statistical packages, I found anaconda to be more convenient, and installed anaconda, which include python 2.7.6
I have the following questions:
do I essentially need to remove/uninstall the previous manually installed version to avoid potential dependency problems?
Was trying to remove, but not successful so far, any hints? I followed suggestions on this board: How to uninstall Python 2.7 on a Mac OS X 10.6.4?, but find no luck yet.
will the previously downloaded/installed packages still work in anaconda, e.g., I launched NLTK in anaconda's python, and the previously downloaded corpuses are still there installed?
You should use virtualenv. Virtualenv allows you to create a virtual python environment for each project, so you can have different libraries for different projects, this includes different python versions.
I've been using MacPorts Python 2.6 on OS X 10.6.
I am considering also installing the Enthought Python Distribution (EPD) on the same machine because it comes preconfigured with matplotlib and other nice data analysis and visualization packages.
Can the two Python distributions co-exist peacefully on the same machine?
What potential problems will I have to look out for (e.g., environment variables)?
I know that building matplotlib through MacPorts is an option, but the process is lengthy (on the order of a full day) and there are open questions about compiling some dependencies on 64bit Intel.
I would like to know about the tradeoffs before committing to one of the two approaches.
I'm not an OSX user, so I can't say for certain, but you should be fine.
EPD bundles its own python interpreter and all required libraries. The installer basically just extracts a tarball into a stand-alone folder.
The linux installer doesn't modify any environment variables (not even $PATH), so I assume the mac installer is similar.
(Because of this, you'll probably want to manually set up aliases to the EPD python/ipython/etc binaries so you can more easily call them without changing your $PATH.)
EPD really should be completely stand-alone, as far as I know. It should be safe to install along-side anything else.
I've used EPD on Windows 7. It works great. Only stalled on me a couple of times with rather large data set.
I also use OSX, and was about to install EPD but hesitating a bit.
I think python 2.7.3 that comes with EPD (early 2013) will become the default python. Normally, upgrading to 2.7 from 2.6 should not be much of problem based on my personal experience. But I am a bit concerned to have 2 versions of 2.7 and what kind of impact it will have on using GAE SDK on my OSX. (Upgrading created a bit of hassle with GAE SKD.)
Here is what EPD readme says:
A full installation will install the following:
* Python framework -- installed into:
/Library/Frameworks/Python.framework/Versions/7.3
Python applications -- installed into
/Applications/Enthought
System fixes -- makes the EPD Python your default one by adding lines to
your .bash_profile to your path.
I think I will sleep on this.