I am new at python, using virtual environment etc. I installed python in C drive, added PATH variable and worked from there. For learning purposes, I decided to install packages only inside my project directory. So, I created virtual environment inside my project folder by following a documentation. I have created it at my windows machine something like this way:
python -m venv my-env
After that, I activated it:
my-env\Scripts\activate.bat
After that, I have installed the Requests library (as it was in the documentation):
pip install requests
But, my question is: why need to install Requests library? Do I need to install it again if I create new project, new virtual environment inside that new project directory at future?
I installed some additional packages And I wanted to create requirements.txt. So, I wrote:
pip freeze > requirements.txt
Now, besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too. So, if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
Sorry for lots of question. Basically, all the questions are related and contextual I think.
why need to install Requests library?
There's no need to install this library for the virtual environment to work properly.
Do I need to install it again if I create new project, new virtual environment inside that new project directory at future.
The idea of a virtual env is that: that you work in a controlled environment, created for that project. If you'll work on another project, in its own virtual env, you'll have to install it again. This is not bad at all: in each project, you might need to work with different versions of the same library, or you might even want/need to delete a project, and none of what happens in one project will affect the other, which is nice.
besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too.
Yes, the freeze command lists all the installed packages. That will include not only what you installed by pip install ..., but also the dependencies those packages needed. This is a reason why some people suggest to write the requirements file by hand: that way it's clearer what external resources are directly needed, as opposed to each sub-dependency, which, as you might have seen, becomes a little clumsy to understand.
if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
It's ok in the sense that those packages will get installed either way. They are explicitly listed, but if they weren't they'd also be downloaded and installed.
Related
I currently work on a server of my university, where I am using Python with Tensorflow and Cuda. I built some machine learning models on my local machine and copied them onto the server, where they run fine with the basic Python installation.
However, I want to install a few additional packages and I also want to make sure that I do not have to be worried about updates of the software installed on the server that could cause problems with my code.
I do not have and cannot get admin permissions to install any programs or modules outside my personal directory, since I am only a regular user and a student above that.
Now I wondered if there is an easy way to just clone the main Python installation to a virtual environment in my home folder. I could basically create a new virtual environment and install Tensorflow etc, but I found it very frustrating when I tried to set it up with Cuda on my private computer. Moreover, I find it rather complicated to access the server (log-in credentials, vpn, 2-factor-authorisation), so I want to minimize the trial-and-error time that I often need when I try custom installation of modules.
Approach 1:
use "pip freeze > requirement.txt" to generate the requirements file and install all requirements using requirements file using "pip install -r requirement.txt" command.
Approach 2:
use a virtualenv-clone package to make clone of the existing environment.
https://pypi.org/project/virtualenv-clone/
If I use a module such as tkinter, would somebody need to have that module installed as well in order for my code to run on their machine?
Definitely. You can use virtual environments or containers to deliver required packages or have a requrements.txt or similar to install the dependencies.
python comes with a number of standard modules pre-installed, if the other person is running python (the same version of you) then he/she won't need to install anything, it will just work, that's the case of tkinter. But if you use external packages that you installed to run your code, for example celery, then he/she will need to do the same thing.
If you gave your code to someone to run, they would need to download the same modules, unless you also sent the environment too. The only way I know around this is to freeze your code where you would create an executable. I've used cx_Freeze and pyInstaller and haven't had any issues but it also depends on your needs. You can find some more information through here:
https://docs.python-guide.org/shipping/freezing/
Hope this helps!
In your running environment do a, this file you add to your repo
pip freeze > requirements.txt
When people clone your repo, they only have to do a:
pip install -r requirements.txt
and they will install exactly the same pypi modules you have.
With virtualenv you can isolate a python environment to each project, with pyenv you can use different pythonversions withing the various environment also.
Folks,
I plan to use Python and various python packages like robot framework, appium, selenium etc for test automation. But as we all know, python and all the package versions keep revving.
If we pick a version of all of these to start with, and as these packages up rev, what is the recommended process for keeping the development environment up to date with the latest versions?
Appreciate some guidance on this.
Thanks.
If you wrote the code with a given version of a library, updating that library in the future is more likely to break your code than make it run better unless you intend to make use of the new features. Most of the time, you are better off sticking with the version you used when you wrote the code unless you want to change the code to use a new toy.
In order to ensure that the proper versions of every library are installed when the program is loaded on a new machine, you need a requirements.txt document. Making one of these is easy. All you do is build your program inside a virtual environment (e.g. conda create -n newenv conda activate newenv) Only install libraries you need for your program and then, once all of your dependencies are installed, in your terminal, type pip freeze > requirements.txt. This will put all your dependencies and their version information in the text document. When you want to use the program on a new machine, simply incorporate pip install -r requirements.txt into the loading process for the program.
If you containerize it using something like docker, your requirements.txt dependencies can be installed automatically whenever the container is created. If you want to use a new library or library version, simply update it in your requirements.txt and boom, you are up to date.
In this case you would want to isolate your package (and the external packages/versions it depends on) using a virtual environment. A virtual environment can be thought of as a file that tracks the specific package versions you're importing. Thus you can have the latest package installed on your system, but your project will still only import the version in your virtual environment.
What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, pipenv, etc?
https://virtualenv.pypa.io/en/stable/
https://docs.python-guide.org/dev/virtualenvs/
I need to create a dummy RPM which appears to install some Python modules in site-packages to resolve an RPM dependency issue.
The real modules will be installed using PIP inside of a Python Virtual Environment, but in order for the system to work the modules which are imported need to be provided in the global site packages, so that needs to be faked.
The imports look as follows (example): from pear.apple.peach import Fruit
When performing an RPM Build on the package that has these imports it fails on dependency generation, so I need an RPM in the local repo to pretend to provide these so the dependency generation passes.
The approach you are proposing in the question doesn't make much sense to me, either you create rpm packages with python modules you need for other system rpm packages (packages you plan to create or install) or you just use virtual env and then you don't need to care much about what you have in system site packages, as long as you have particular version of python itself installed. While you can make system site packages visible in particular virtual env (for example, when you have lxml module among dependencies, you can install it from rpm package and then make a virtual env to have access to system site packages so that you don't need to install it again in virtualenv), having particular module provided in dummy rpm package would break it so this doesn't make sense as well.
In other words, having something installed in virtual env and then expect to make system python module installed via rpm package to depend on something from that virtual env is not going to work (imho).
So I suggest to do one of:
Build the rpm packages of modules you need to have installed for other packages to work. As long as you have some experience with rpmbuild, it should not be that hard, you can use pyp2rpm tool to create initial specfile which you would need to tweak a bit. But this also depends on how many of the packages you need to build and how often do you plan to update them (as it would mean to update the specfile and rebuild).
Use virtual env only, either with access to system site packages or not. This way, you install both dependencies and application into virtual env and you don't need to deal with rpm packages at all (as long as you install python and virtualenv from rpm packages).
Which way would make more sense in you case depends heavily on what are you trying to do (how do you plan to maintain the python modules, on how many machines do you plan to install it, if you plan to install it yourself or provide it for others to install it themselves and so on ...).
I was able to solve this issue by replicating every import with an empty file, and using empty __init__.py files at every folder level.
For example, to resolve from pear.apple.peach import Fruit, I would have needed to install the following file tree in site-packages:
-> pear
-> __init__.py
-> apple
-> peach.py
The relevant lines of the spec file for the dummy rpm:
...
source:FruitDummy.tar.gz
...
% install
mkdir -p $RPM_BUILD_ROOT%{python_sitelib}/pear/apple/
install __init__.py $RPM_BUILD_ROOT%{python_sitelib}/pear/
install peach.py $RPM_BUILD_ROOT%{python_sitelib}/pear/apple/
...
%files
%defattr(-,root,root,-)
%{python_sitelib}/pear/__init__.py*
%{python_sitelib}/pear/apple/peach.py*
...
__init__.py and peach.py were stored in the FruitDummy.tar.gz that was used to build the RPM.
I'm a Java/Scala dev transitioning to Python for a work project. To dust off the cobwebs on the Python side of my brain, I wrote a webapp that acts as a front-end for Docker when doing local Docker work. I'm now working on packaging it up and, as such, am learning about setup.py and virtualenv. Coming from the JVM world, where dependencies aren't "installed" so much as downloaded to a repository and referenced when needed, the way pip handles things is a bit foreign. It seems like best practice for production Python work is to first create a virtual environment for your project, do your coding work, then package it up with setup.py.
My question is, what happens on the other end when someone needs to install what I've written? They too will have to create a virtual environment for the package but won't know how to set it up without inspecting the setup.py file to figure out what version of Python to use, etc. Is there a way for me to create a setup.py file that also creates the appropriate virtual environment as part of the install process? If not — or if that's considered a "no" as this respondent stated to this SO post — what is considered "best practice" in this situation?
You can think of virtualenv as an isolation for every package you install using pip. It is a simple way to handle different versions of python and packages. For instance you have two projects which use same packages but different versions of them. So, by using virtualenv you can isolate those two projects and install different version of packages separately, not on your working system.
Now, let's say, you want work on a project with your friend. In order to have the same packages installed you have to share somehow what versions and which packages your project depends on. If you are delivering a reusable package (a library) then you need to distribute it and here where setup.py helps. You can learn more in Quick Start
However, if you work on a web site, all you need is to put libraries versions into a separate file. Best practice is to create separate requirements for tests, development and production. In order to see the format of the file - write pip freeze. You will be presented with a list of packages installed on the system (or in the virtualenv) right now. Put it into the file and you can install it later on another pc, with completely clear virtualenv using pip install -r development.txt
And one more thing, please do not put strict versions of packages like pip freeze shows, most of time you want >= at least X.X version. And good news here is that pip handles dependencies by its own. It means you do not have to put dependent packages there, pip will sort it out.
Talking about deploy, you may want to check tox, a tool for managing virtualenvs. It helps a lot with deploy.
Python default package path always point to system environment, that need Administrator access to install. Virtualenv able to localised the installation to an isolated environment.
For deployment/distribution of package, you can choose to
Distribute by source code. User need to run python setup.py --install, or
Pack your python package and upload to Pypi or custom Devpi. So the user can simply use pip install <yourpackage>
However, as you notice the issue on top : without virtualenv, they user need administrator access to install any python package.
In addition, the Pypi package worlds contains a certain amount of badly tested package that doesn't work out of the box.
Note : virtualenv itself is actually a hack to achieve isolation.