The thing is pip install can take some time and as a build can inherently be different from the other, some funcionalities may behave different too. My point is:
Is it safe to copy the virtualenv folder (typically venv) from the staging environment to the production environment?
I know some packages build themselves using *-dev packages and header files. But aren't these binaries deployable too? If these packages were the biggest problem, is it possible to install them only in the production environment and copy the rest of the venv folder?
Related
I'm working with a team. We each have our own Windows system. We have shared drives and a shared git repository. We want to have a shared virtual environment (in Python).
My understanding (from previous questions from myself and others) is that virtual environments do not include all files necessary for running python, in particular, the shared VE does not include the Python interpreter.
I can see how we can create a shared VE and it seems we could just copy that around, or put it on the shared drive, or put it in a git repository. But my understanding of this is that it does not eliminate the need for individuals to install their own local versions of python. Is that correct?
One of my colleagues has heard (or read) that "there is a package that allows teams to share their virtual environment configuration through a git-like interface. That way you can “pull” the updated configuration and it will install the new packages automatically. This allows each person to change the configuration and test it before releasing it to the team."
So is there a special package to enable this? Or is it just a regular venv that is included in the git repository with the other files? If we do this, then we must all put the venvs in the same place in on our file systems OR we have to go in and manually change the VIRTUAL_ENV variable in activate.bat. Is that correct?
In any case, we do all have to install our own local versions of python anyway. Is that correct?
If the virtual environment is on a shared drive(group readable), then your team members should be able to access it. A virtual environment is just a directory.
But my understanding of this is that it does not eliminate the need for individuals to install their own local versions of python. Is that correct?
Virtual environments have their own python binaries, which you can see when you run which python inside the virtual environment after it is activated.
So is there a special package to enable this? Or is it just a regular venv that is included in the git repository with the other files? If we do this, then we must all put the venvs in the same place in on our file systems OR we have to go in and manually change the VIRTUAL_ENV variable in activate.bat. Is that correct?
I would advise against uploading a virtual environment directory to version control, since it contains binaries, configuration files that don't belong in there. Its also unnecessary to do this because the dependencies are tracked in a requirements.txt file, which list the pip dependencies and is committed to version control. Additionally, When the virtual environment is activated, the VIRTUAL_ENV environment variable is automatically exported, so there is no need to modify it.
Conclusion
For simplicity, its probably best to have each user create their own virtual environment and install the dependencies from requirements.txt on their local machines. This also ensures users don't make a change to the virtual environment that will affect other users, which is a drawback of the above shared drive approach.
If they want to pull the latest requirements, then pulling the latest change using git pull and reinstalling the dependencies with pip install -r requirements.txt is good enough. You just have to ensure the virtual environment is activated, otherwise the dependencies will get installed system wide. This is where the pipenv package also comes in handy.
Usually in my team projects, the README contains instructions to get this setup for each team member.
Additionally, as Daniel Farrell helpfully mentioned in the comments, pip won't be able to manage packages like libffi, openssl, python-devel etc. inside a virtual environment. This is where using Docker containers become useful, since you can install dependencies inside a isolated environment built on top of the host operating system. This ensures the dependencies don't mess with the system wide packages, which is a good practice to follow in any case.
An example Dockerfile I have used in the past:
FROM python:3.8-slim-buster
# Set environment variables:
ENV VIRTUAL_ENV=/opt/venv
ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# Create virtual environment:
RUN python3 -m venv $VIRTUAL_ENV
# Install dependencies:
COPY requirements.txt .
RUN pip install -r requirements.txt
# Run the application:
COPY app.py .
CMD ["python", "app.py"]
Which I modified from this Elegantly activating a virtualenv in a Dockerfile article.
containerization aims to solve the "from where comes python?" problem. My developers' teams usually use a Dockerfile that installs their requirements within a docker-compose that spins ups a development environment for their applications . Unlike a virtual environment, containers offer a complete userspace solution that works pretty well in windows and osx.
I downloaded a Python project and it contains both a virtual environment and a requirements.txt file. Why would you need both? As far as I know, virtual environments already contain the required modules. Any idea when and why this combination would be useful?
While it is technically possible I don't find any good reason to that. Having both is confusing because it is not clear which one is the "master". And you have to (or not?) worry about consistency between installed packages and requirements.txt file.
Also venv and installed packages in many cases depend on the underlying OS, they have binaries, different layout, etc. It is generally advised to write os-independent code.
All in all, I would stick to requirements.txt file and remove any venv folder from the project's repo.
You can't distribute the virtualenv directory with your project because the contents may vary depending on the target operating system and the version of the operating system. Specifically, a virtualenv that includes libraries with compiled components installed on Ubuntu 14.04 will differ from the equivalent virtualenv installed on Ubuntu 16.04.
Instead, you should distribute your requirements.txt file (just a convention, you could use any file name you want) so the end-user will be able to recreate a virtualenv on his machine.
I am newbie using VirtualEnv and recently try to create one using PyCharm. During the process, PyCharm ask me to specify the project location, application name and VirtualEnv name and location. My doubt is, after I specify the name and location of the VirtualEnv the location of the Django project files must be inside the VirtualEnv? or it's possible to have the VirtualEnv files in a different location than the Django project files?
Maybe I am not understanding the purpose of the VirtualEnv. Perhaps, VirtualEnv it's just a list of the dependencies of my project, Python version, Django version, Pip version, Jinja2 version and all other required files, but not necessarily the Django application files (the website that is being developed).
Thanks in advance.
Ya, I think you misunderstand what virtualenv does:
https://virtualenv.pypa.io/en/latest/
The basic problem being addressed is one of dependencies and versions, and indirectly permissions. Imagine you have an application that needs version 1 of LibFoo, but another application requires version 2. How can you use both these applications? If you install everything into /usr/lib/python2.7/site-packages (or whatever your platform’s standard location is), it’s easy to end up in a situation where you unintentionally upgrade an application that shouldn’t be upgraded.
Your project files don't need to be (and shouldn't be) where the virtualenv files are.
virtualenv installs your app's python dependancies in a folder for the specific virtualenv that is being used.
Let's say you are not using virtualenv, the dependencies would be installed into into the site-packages folder for your system. The dependancies aren't installed in your project directory and your project directory isn't in your system's site-packages directory.
Using virtualenv doesn't change that, it just changes where the dependencies are installed.
virtualenv is not just a list of dependencies! It actually has all the modules under its umbrella. Think of a virtualenv as a space which isolates all the packages used by your project from the rest of the other packages that were installed previously or at a later time.Yes, there is an option to have the virtualenv make use of packages that are "outside" of the environment but that's just an option.
The main purpose of having an virtualenv is to enable the user to use packages versions of his choice and keep them isolated from the rest of the space. Usually, the list of packages belonging to a specific virtualenv are captured into a file, requirements.txt. If you want to run the project on a different machine or share it with someone, having requirements.txt will make it easy to recreate the environment via pip install -r requirement.txt from within virtualenv
I have created a flask application in a virtual environment on my local machine and I could run it locally (at http://localhost:5000).
I then put this project in a repo and I then went to my server and git clone this project.
All files are identical on my local machine and in my server.
I then wanted to test this virtual environment on the server by trying .venv/bin/activate
However I ran into an error. It says I do not have flask!:
Traceback (most recent call last):
File "__init__.py", line 1, in <module>
from flask import Flask
ImportError: No module named flask
I am assuming that I have to initialize something in the virtual environment first, like installing all of the dependencies. Or do I have to pip install flask again? (It would be kind of funny to do that...)
As a general rule python environments are not portable across machines.
This means that you cannot reliably expect to port the virtual environment across machines. This is especially true if you are moving stuff between different operating systems. For example, a virtual environment created in Windows will not work in Linux.
Similarly, a virtual environment created in OSX will not work in Linux. Sometimes, you can get Linux > Linux compatibility, but this is by chance and not to be relied upon.
The reasons are numerous - some libraries need to be built against native extensions, others require compatible system libraries in place to work, etc.
So, the most reliable workflow is the following:
You can (but I would recommend against this) put your virtual environment in the same directory as your project. If you do so, make sure you don't add the virtual environment root directory to your source control system. It is best to separate your virtual environments from your source code (see the virtualenvwrapper project project for a great way to manage your virtual environments separately).
You should create a requirements file, by running pip freeze > requirements.txt. Keep this file updated and add it to your source control system. In your target system, simply create an empty virtual environment and then pip install -r requirements.txt to make sure all requirements are installed correctly. Doing so will make sure that any native extensions are also built and installed.
A few possible issues:
When you created your original virtual environment did you specify --no-site-packages if not your package may be using elements from the system.
Some packages rely on system installed libraries that may be missing on your target system
Is your server running on a similar set of hardware to your development system with the same OS - if not your virtualenv is likely not to work without re-installing packages as any C/C++ extensions will have been built for the wrong hardware/OS and will not work.
The thing is that virtualenv is not a package builder, (look at pyinstaller for that), but rather a development and test environment when you go to distribute your code to a new platform then, provided you started off with --no-site-packages you can easily find out which packages you need to find out what you need to install on the new target.
So basically - Yes you, or more likely the system admin, does need to run pip install flask and probably several other things!
I was used VirtualBox manual setups with virtualenvs inside them to run Django projects on my local machine. Recently I discovered Vagrant and decided to switch to it, because it seems very easy and useful.
But I can not figure - do I need still use virtualenv Vagrant VM, is it encouraged practice or forbidden?
As Devin stated, it is not necessary to use virtualenv when you deploy to a vagrant machine as long as you are the sole user of the machine. However, I would still enable the use of a virtualenv, setup.py, etc. even if you do not use it for development or deployment.
In my (not so) humble opinion, any Python project should:
Include a .cvsignore, .gitignore, .hgignore, ... file that ignores the common Python intermediate files as well as virtualenv directories.
A requirements.txt file that lists the required packages in a pip-compliant format
Include a Makefile with the following targets:
environment: create the virtual environment using virtualenv or pyvenv
requirements: install required packages using pip and the requirements.txt file
develop: run setup.py develop using the virtual environment
test: run setup.py test
clean: remove intermediate files, coverage reports, etc.
maintainer-clean: remove the virtual environment
The idea is to keep the Makefile as simple as possible. The dependencies should be set up so that you can clone the repository (or extract the source tarball) and run make test. It should create a virtual environment, install the requirements, and run the unit tests.
You can also include a Vagrantfile and a vagrant target in the Makefile that runs vagrant up. Add a vagrant destroy to the maintainer-clean target while you are at it.
This makes your project usable by anyone that is using vagrant or developing without it. If (when) you need to use deploy alongside another project in a vagrant or physical environment, including a clean setup.py and a Vagrantfile that describes your minimal environment makes it simple to install into a virtual environment or a shared vagrant machine.
If you run one vagrant VM per project, then there is no direct reason to use virtualenv.
If other contributors do not use vagrant, but do use virtualenv, then you might want to use it and support it to make their lives easier.
Virtualenv and other forms of isolation (Docker, dedicated VM, ...) are not necessarily mutually exclusive. Using virtualenv is still a good idea, even in an isolated environment, to shield the virtual system Python from your project packages. *nix systems use plethora of Python based utilities dependent on specific versions of packages being available in system Python and you don't want to mess with these.
Mind that virtualenv can still only go as far as pure Python packages and doesn't solve the situation with native extensions that will still mix with the system.