Folks,
I plan to use Python and various python packages like robot framework, appium, selenium etc for test automation. But as we all know, python and all the package versions keep revving.
If we pick a version of all of these to start with, and as these packages up rev, what is the recommended process for keeping the development environment up to date with the latest versions?
Appreciate some guidance on this.
Thanks.
If you wrote the code with a given version of a library, updating that library in the future is more likely to break your code than make it run better unless you intend to make use of the new features. Most of the time, you are better off sticking with the version you used when you wrote the code unless you want to change the code to use a new toy.
In order to ensure that the proper versions of every library are installed when the program is loaded on a new machine, you need a requirements.txt document. Making one of these is easy. All you do is build your program inside a virtual environment (e.g. conda create -n newenv conda activate newenv) Only install libraries you need for your program and then, once all of your dependencies are installed, in your terminal, type pip freeze > requirements.txt. This will put all your dependencies and their version information in the text document. When you want to use the program on a new machine, simply incorporate pip install -r requirements.txt into the loading process for the program.
If you containerize it using something like docker, your requirements.txt dependencies can be installed automatically whenever the container is created. If you want to use a new library or library version, simply update it in your requirements.txt and boom, you are up to date.
In this case you would want to isolate your package (and the external packages/versions it depends on) using a virtual environment. A virtual environment can be thought of as a file that tracks the specific package versions you're importing. Thus you can have the latest package installed on your system, but your project will still only import the version in your virtual environment.
What is the difference between venv, pyvenv, pyenv, virtualenv, virtualenvwrapper, pipenv, etc?
https://virtualenv.pypa.io/en/stable/
https://docs.python-guide.org/dev/virtualenvs/
Related
Do you anchor all versions in requirements.txt to a spesific number? Is it the right way? Is there any way to get latest version that does not break my app? To not missing security updates etc.
Installing latest dependency may break your app if the new code syntax is not backwards compatible. Here's how I approach to maintain the requirements.txt file.
Create a new virtual environment every time I develop for a new project
Install all the dependencies in the virtual env
Run pip freeze > requirements.txt from that virtual env to generate the file
Add Python version in a Readme file
This way I can always know for sure the Python version and version of the dependencies with which the app was built. This of course does not solve other issues related to OS type/version compatibility that might occur.
Tools like Docker are created to solve all such dependency and cross-platform compatibility problems so that is worth exploring.
EDIT
If you still want to always have the latest version of the package, simply remove the version numbers and keep the package name.
E.g. Instead of mysql-connector==2.2.9 use mysql-connector
I am new at python, using virtual environment etc. I installed python in C drive, added PATH variable and worked from there. For learning purposes, I decided to install packages only inside my project directory. So, I created virtual environment inside my project folder by following a documentation. I have created it at my windows machine something like this way:
python -m venv my-env
After that, I activated it:
my-env\Scripts\activate.bat
After that, I have installed the Requests library (as it was in the documentation):
pip install requests
But, my question is: why need to install Requests library? Do I need to install it again if I create new project, new virtual environment inside that new project directory at future?
I installed some additional packages And I wanted to create requirements.txt. So, I wrote:
pip freeze > requirements.txt
Now, besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too. So, if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
Sorry for lots of question. Basically, all the questions are related and contextual I think.
why need to install Requests library?
There's no need to install this library for the virtual environment to work properly.
Do I need to install it again if I create new project, new virtual environment inside that new project directory at future.
The idea of a virtual env is that: that you work in a controlled environment, created for that project. If you'll work on another project, in its own virtual env, you'll have to install it again. This is not bad at all: in each project, you might need to work with different versions of the same library, or you might even want/need to delete a project, and none of what happens in one project will affect the other, which is nice.
besides my required libraries for the project itself, every packages from that Requests library are in that requirements.txt file too.
Yes, the freeze command lists all the installed packages. That will include not only what you installed by pip install ..., but also the dependencies those packages needed. This is a reason why some people suggest to write the requirements file by hand: that way it's clearer what external resources are directly needed, as opposed to each sub-dependency, which, as you might have seen, becomes a little clumsy to understand.
if I share my project with other user, that user will get packages from the Requests library when install via requirements.txt. Is it okay?
It's ok in the sense that those packages will get installed either way. They are explicitly listed, but if they weren't they'd also be downloaded and installed.
If I use a module such as tkinter, would somebody need to have that module installed as well in order for my code to run on their machine?
Definitely. You can use virtual environments or containers to deliver required packages or have a requrements.txt or similar to install the dependencies.
python comes with a number of standard modules pre-installed, if the other person is running python (the same version of you) then he/she won't need to install anything, it will just work, that's the case of tkinter. But if you use external packages that you installed to run your code, for example celery, then he/she will need to do the same thing.
If you gave your code to someone to run, they would need to download the same modules, unless you also sent the environment too. The only way I know around this is to freeze your code where you would create an executable. I've used cx_Freeze and pyInstaller and haven't had any issues but it also depends on your needs. You can find some more information through here:
https://docs.python-guide.org/shipping/freezing/
Hope this helps!
In your running environment do a, this file you add to your repo
pip freeze > requirements.txt
When people clone your repo, they only have to do a:
pip install -r requirements.txt
and they will install exactly the same pypi modules you have.
With virtualenv you can isolate a python environment to each project, with pyenv you can use different pythonversions withing the various environment also.
I'm using Windows OS and to isolate my projects and to keep track of Python versions and Library versions I sometimes use Anaconda and sometimes a Vagrant Machine (Usually using Ubuntu 18.04), depending on the project.
When I'm using Vagrant to emulate Ubuntu 18.04, I always have to install the same packages over and over for each new project within the virtual environment folder for that project and I wonder if there's a better solution not to do that and still keep track of versions. Can you suggest me something to help me on that? (If you can, it would be awesome if you could share a link to a tutorial to do that as well)
Is there a way for me to keep track of the python and libraries versions on a *.txt file like the requirements.txt file and through that tool make it search for that package version in a single path, sharing that single folder containing python versions and library versions among every single project I have instead of having to re-install packages in that specific project folder?
If there is, can I still push to git a virtual environment containing those packages and the python version I'm using without mixing it up with the other versions I have installed? I don't want to store that virtual environment files in my computer, but I still want to make it available in my project when I push it to GitHub, for instance.
Thank you very much
There is a way to create the list of installed Python packages. You can use
$ pip list --format=freeze > installation_list.txt
This will export the list of packages with versions and save them into text file. Instead of freeze you can also use JSON or legacy as valid list format.
You can use $ pip install -r installation_list.txt to install from the given dependencies.
For pushing these files to git without mixing with other version. I would advise to create a separate branch for these versions and push to that. It will make it easier to maintain different version without any conflicts.
Whenever you want you can change to that branch by
git checkout -b <branch name>
Create new origin for that remote if needs be.
I'm a Java/Scala dev transitioning to Python for a work project. To dust off the cobwebs on the Python side of my brain, I wrote a webapp that acts as a front-end for Docker when doing local Docker work. I'm now working on packaging it up and, as such, am learning about setup.py and virtualenv. Coming from the JVM world, where dependencies aren't "installed" so much as downloaded to a repository and referenced when needed, the way pip handles things is a bit foreign. It seems like best practice for production Python work is to first create a virtual environment for your project, do your coding work, then package it up with setup.py.
My question is, what happens on the other end when someone needs to install what I've written? They too will have to create a virtual environment for the package but won't know how to set it up without inspecting the setup.py file to figure out what version of Python to use, etc. Is there a way for me to create a setup.py file that also creates the appropriate virtual environment as part of the install process? If not — or if that's considered a "no" as this respondent stated to this SO post — what is considered "best practice" in this situation?
You can think of virtualenv as an isolation for every package you install using pip. It is a simple way to handle different versions of python and packages. For instance you have two projects which use same packages but different versions of them. So, by using virtualenv you can isolate those two projects and install different version of packages separately, not on your working system.
Now, let's say, you want work on a project with your friend. In order to have the same packages installed you have to share somehow what versions and which packages your project depends on. If you are delivering a reusable package (a library) then you need to distribute it and here where setup.py helps. You can learn more in Quick Start
However, if you work on a web site, all you need is to put libraries versions into a separate file. Best practice is to create separate requirements for tests, development and production. In order to see the format of the file - write pip freeze. You will be presented with a list of packages installed on the system (or in the virtualenv) right now. Put it into the file and you can install it later on another pc, with completely clear virtualenv using pip install -r development.txt
And one more thing, please do not put strict versions of packages like pip freeze shows, most of time you want >= at least X.X version. And good news here is that pip handles dependencies by its own. It means you do not have to put dependent packages there, pip will sort it out.
Talking about deploy, you may want to check tox, a tool for managing virtualenvs. It helps a lot with deploy.
Python default package path always point to system environment, that need Administrator access to install. Virtualenv able to localised the installation to an isolated environment.
For deployment/distribution of package, you can choose to
Distribute by source code. User need to run python setup.py --install, or
Pack your python package and upload to Pypi or custom Devpi. So the user can simply use pip install <yourpackage>
However, as you notice the issue on top : without virtualenv, they user need administrator access to install any python package.
In addition, the Pypi package worlds contains a certain amount of badly tested package that doesn't work out of the box.
Note : virtualenv itself is actually a hack to achieve isolation.