When installing Python package through pip in a Dockerfile, such as:
pip install --trusted-host pypi.python.org -r requirements.txt
with requirements.txt, e.g. as:
python-dotenv>=0.15.0
psycopg2>=2.8.6
sqlalchemy>=1.3.22
numpy>=1.19.0
rasterio>=1.1.8
pandas>=1.1.5
geopandas>=0.8.1
matplotlib>=3.3.0
seaborn>=0.11.0
I recently seen this warning:
WARNING: Running pip as the 'root' user can result in broken permissions
and conflicting behavior with the system package manager.
It is recommended to use a virtual environment
instead: https://pip.pypa.io/warnings/venv
Hence my 'naive' question:
would that make sense to set up a virtual environment / or installing Python packages as the non root user (which is the default in Docker), as one would normally do on his/her local computer?
For the moment I never cared about that, because I'm inside a Docker container which by definition hosts a single application, so I think it's perfectly OK that these packages are installed globally. Hopefully I cannot break anything on my local machine.
Honestly? It doesn't much matter.
Using the root user during the build of your container is generally necessary and expected. The warning from pip is "running pip as root could screw up the packages your OS programs depend on" - but there's no OS in your container.
If you drop to a less privileged user at the end of your build or during your docker run, installing the packages as root won't have hurt you any. Practically a container is a single process (your python application) that has a view of the filesystem different from the root system- very much like a virtualenv would try to accomplish.
As #Paul Becotte said in his answer, there is no risk in installing your packages globally in your container, but you are seeing this warning because pip doesn't care you are running inside a container or not.
The general python good practice is to create a virtual env as an unprivileged user : $ python -m venv .venv
Then activate it : $ source .venv/bin/activate
And then install your packages with your $ pip install -r requirements.txt command.
You can totally adapt that to the docker build syntax :
RUN python -m venv /abolute/path/to/venv
RUN source /absolute/path/to/venv/bin/activate && pip install -r requirements.txt
CMD source /absolute/path/to/venv/bin/activate && python /path/to/your/app.py
However, the General docker good practice is to run with an unprivileged user. So you have multiple choice here :
Live with the warning, install your python packages globally without venv, drop the privileges and run your container with a normal user
create venv to get rid of the warning, install your packages in it, run you container as root (not a good practice, but maybe remapping root user can be ok)
create a venv, install the packages, run the container as a normal user, which is probably the best option if your application doesn't need to be run as root in your container.
It's not the best practice to run pip as a root user or to install packages globally. The best practice is to create virtual environment within the project and install every packages there.
If you install globally, it will actually affect other projects because you might not need the packages you installed for project A in the project B that you are about to start and if you remove all the packages, there will be a break in project A as it cannot access the packages again.
Have virtual environment for all your projects and with that, you will be rest assured that all projects will not affect each other.
I copied a repo containing a Python Django project from my old computer to my new one via git clone. I manage my dependecies in the project via pipenv.
After successfully cloning my repo I wanted to start working on the new computer and tried to select the relevant python interpreter for my project in VS Code . However, the path was not on the list.
So first I tried the command pipenv --venv which gave me the feedback: No virtualenv has been created for this project
So I thought I might need to activate the virtual environment first, before being able to select it in VS Studio code. So i ran pipenv shell in the root directory of my project.
However this seem to have created a new virtual environment: Creating a virtualenv for this project… Pipfile: C:\path\to\my\cloned\project\Pipfile
My questions:
1.) Is this the correct way to activate a pipenv virtualenvironment on a new computer after copying the project via git clone? And if not,...
2.1) Does the way I did it cause any problems, which I should be aware of?
2.2) What would be the correct procedure to activate my virtual enviroment on a new computer?
In general an environment image probably shouldn't be copied to github. You'll get a bunch of unneeded files which clogs your repo.
Instead you should create a requirements.txt from your existing environment pip freeze > requirements.txt and commit that file.
Then when someone else clones your repo they can set up a new virtual environment using any tool they want and run python -m pip install -r requirements.txt
That is, requirements.txt is like a recipe for how to create your environment. By providing the recipe users can use it any way they want.
use:
pipenv install
It worked on Ubuntu, should work also on a mac.
I tried on a windows, it triggered some errors.
"If you download a source repository for a project that uses Pipenv for package management, all you need to do is unpack the contents of the repository into a directory and run pipenv install (no package names needed). Pipenv will read the Pipfile and Pipfile.lock files for the project, create the virtual environment, and install all of the dependencies as needed."
https://www.infoworld.com/article/3561758/how-to-manage-python-projects-with-pipenv.html
What is the process to clone a repo and recreate the python virtual environment using the pipenv.lock file?
On my development system, I created a virtual environment using pyenv and pipenv , and am ready to test my code on another system. I've installed pyenv/pipenv/python(same version as dev machine) on the test system, but I don't how to tell pipenv to create an identical virtual environment on the test system using the content of the version-controller Pipfile.lock file.
do I do 'pipenv install --python '?
or just 'pipenv install' and it will find and read the .lock file?
pipenv install --python <version>
pipenv install
I expect a virtual environment with all of the requirements/dependencies specified in the lock file to be installed.
According to this article it's just:
pipenv install --ignore-pipfile
This tells Pipenv to ignore the Pipfile for installation and use what’s in the Pipfile.lock. Given this Pipfile.lock, Pipenv will create the exact same environment you had when you ran pipenv lock, sub-dependencies and all.
I'm learning django and just installed pipenv via pip install pipenv and then pipenv shell and I notice that the virtual environment files are installed or created in some random default directory, I have two questions regarding this:
1) How can I customize that installation/creation directory for the virtual environment? Do I have to use a different command line from pipenv shell?
2) Can you have multiple folders with different virtual environments inside each folder/project?
According to the pipenv advanced readme (https://github.com/pypa/pipenv/blob/master/docs/advanced.rst#-custom-virtual-environment-location):
You can set the environment variable WORKON_HOME to whichever directory you want,
e.g.: by setting export WORKON_HOME=~/.venvs in your .bashrc file (if you are using bash).
According to this https://github.com/pypa/pipenv/issues/1071#issuecomment-370561179 comment (from the pipenv github repo), you can use a workaround for achieving this:
To be super clear, you can still get your own custom environments set
up just by sourcing virtualenvs.
virtualenv 35 --python=python3.5
virtualenv 36 --python=python3.6
source 35/bin/activate && pipenv install
source 36/bin/activate && pipenv install
source 35/bin/activate && pipenv run <whatever>
a tiny bit of additional visual clutter to the commands but is pretty
straightforward.
You would execute the virtualenv x commands inside the project folder.
I've created folder and initialized a virtualenv instance in it.
$ mkdir myproject
$ cd myproject
$ virtualenv env
When I run (env)$ pip freeze, it shows the installed packages as it should.
Now I want to rename myproject/ to project/.
$ mv myproject/ project/
However, now when I run
$ . env/bin/activate
(env)$ pip freeze
it says pip is not installed. How do I rename the project folder without breaking the environment?
You need to adjust your install to use relative paths. virtualenv provides for this with the --relocatable option. From the docs:
Normally environments are tied to a
specific path. That means that you
cannot move an environment around or
copy it to another computer. You can
fix up an environment to make it
relocatable with the command:
$ virtualenv --relocatable ENV
NOTE: ENV is the name of the virtual environment and you must run this from outside the ENV directory.
This will make some of the files
created by setuptools or distribute
use relative paths, and will change
all the scripts to use
activate_this.py instead of using the
location of the Python interpreter to
select the environment.
Note: you must run this after you've
installed any packages into the
environment. If you make an
environment relocatable, then install
a new package, you must run virtualenv
--relocatable again.
I believe "knowing why" matters more than "knowing how". So, here is another approach to fix this.
When you run . env/bin/activate, it actually executes the following commands (using /tmp for example):
VIRTUAL_ENV="/tmp/myproject/env"
export VIRTUAL_ENV
However, you have just renamed myproject to project, so that command failed to execute.
That is why it says pip is not installed, because you haven't installed pip in the system global environment and your virtualenv pip is not sourced correctly.
If you want to fix this manually, this is the way:
With your favorite editor like Vim, modify /tmp/project/env/bin/activate usually in line 42:
VIRTUAL_ENV='/tmp/myproject/env' => VIRTUAL_ENV='/tmp/project/env'
Modify /tmp/project/env/bin/pip in line 1:
#!/tmp/myproject/env/bin/python => #!/tmp/project/env/bin/python
After that, activate your virtual environment env again, and you will see your pip has come back again.
NOTE: As #jb. points out, this solution only applies to easily (re)created virtualenvs. If an environment takes several hours to install this solution is not recommended
Virtualenvs are great because they are easy to make and switch around; they keep you from getting locked into a single configuration. If you know the project requirements, or can get them, Make a new virtualenv:
Create a requirements.txt file
(env)$ pip freeze > requirements.txt
If you can't create the requirements.txt file, check env/lib/pythonX.X/site-packages before removing the original env.
Delete the existing (env)
deactivate && rm -rf env
Create a new virtualenv, activate it, and install requirements
virtualenv env && . env/bin/activate && pip install -r requirements.txt
Alternatively, use virtualenvwrapper to make things a little easier as all virtualenvs are kept in a centralized location
$(old-venv) pip freeze > temp-reqs.txt
$(old-venv) deactivate
$ mkvirtualenv new-venv
$(new-venv) pip install -r temp-reqs.txt
$(new-venv) rmvirtualenv old-venv
I always install virtualenvwrapper to help out. From the shell prompt:
pip install virtualenvwrapper
There is a way documented in the virtualenvwrapper documents - cpvirtualenv
This is what you do. Make sure you are out of your environment and back to the shell prompt. Type in this with the names required:
cpvirtualenv oldenv newenv
And then, if necessary:
rmvirtualenv oldenv
To go to your newenv:
workon newenv
You can fix your issue by following these steps:
rename your directory
rerun this: $ virtualenv ..\path\renamed_directory
virtualenv will correct the directory associations while leaving your packages in place
$ scripts/activate
$ pip freeze to verify your packages are in place
An important caveat, if you have any static path dependencies in script files in your virtualenv directory, you will have to manually change those.
Yet another way to do it that worked for me many times without problems is virtualenv-clone:
pip install virtualenv-clone
virtualenv-clone old-dir/env new-dir/env
Run this inside your project folder:
cd bin
sed -i 's/old_dir_name/new_dir_name/g' *
Don't forget to deactivate and activate.
In Python 3.3+ with built-in venv
As of Python 3.3 the virtualenv package is now built-in to Python as the venv module. There are a few minor differences, one of which is the --relocatable option has been removed. As a result, it is normally best to recreate a virtual environment rather than attempt to move it. See this answer for more information on how to do that.
What is the purpose behind wanting to move rather than just recreate any virtual environment? A virtual environment is intended to manage the dependencies of a module/package with the venv so that it can have different and specific versions of a given package or module it is dependent on, and allow a location for those things to be installed locally.
As a result, a package should provide a way to recreate the venv from scratch. Typically this is done with a requirements.txt file and sometimes also a requirements-dev.txt file, and even a script to recreate the venv in the setup/install of the package itself.
One part that may give headaches is that you may need a particular version of Python as the executable, which is difficult to automate, if not already present. However, when recreating an existing virtual environment, one can simply run python from the existing venv when creating the new one. After that it is typically just a matter of using pip to reinstall all dependencies from the requirements.txt file:
From Git Bash on Windows:
python -m venv mynewvenv
source myvenv/Scripts/activate
pip install -r requirements.txt
It can get a bit more involved if you have several local dependencies from other locally developed packages, because you may need to update local absolute paths, etc. - though if you set them up as proper Python packages, you can install from a git repo, and thus avoid this issue by having a static URL as the source.
virtualenv --relocatable ENV is not a desirable solution. I assume most people want the ability to rename a virtualenv without any long-term side effects.
So I've created a simple tool to do just that. The project page for virtualenv-mv outlines it in a bit more detail, but essentially you can use virtualenv-mv just like you'd use a simple implementation of mv (without any options).
For example:
virtualenv-mv myproject project
Please note however that I just hacked this up. It could break under unusual circumstances (e.g. symlinked virtualenvs) so please be careful (back up what you can't afford to lose) and let me know if you encounter any problems.
Even easier solution which worked for me: just copy the site-packages folder of your old virtual environment into a new one.
Using Visual Studio Code (vscode), I just opened the ./env folder in my project root, and did a bulk find/replace to switch to my updated project name. This resolved the issue.
Confirm with which python
If you are using an conda env,
conda create --name new_name --clone old_name
conda remove --name old_name --all # or its alias: `conda env remove --name old_name`