I trusted PyCharm to let it create a venv for me, and then shipped the code project with the venv folder. Inside my code, whenever calling a separate script, I use the venv Python uder venv/Scripts/.
To my surprise, it breaks when running on another machine, where the venv python actually points to my local installation of Python at C:\Python\Python37. This beats the purpose of creating a venv!
What have I done wrong? My project structure is like
code/*.py
venv
And this is what I shipped to another machine.
Yes venv creates a link to your original python.
The best practice to creating "venv requirements" is to create requirements file:
pip freeze > requirements.txt
Then just add this file to your project folder
P.S
There is better alternative called Conda. You can provide the python version in conda envs.
Thanks to #AndreyG's answer. In addition, a bunch of others similar questions were helpful, like:
Copy complete virtualenv to another pc
Port Python virtualenv to another system
Especially after reading this tutorial: https://realpython.com/python-virtual-environments-a-primer/,
I realized my venv misconception, which was kinda due to my previous experience on shipping among Mac machines, where Python was usually preinstalled.
Now combined with the pip-freezing approach towards shipping site-packages, I also ship the Python Windows installer and automate the installation to make sure that the venv symlinks point to the exact same Python installation on all machines. Reference:
https://docs.python.org/3/using/windows.html
The automation code:
python.<version>-<arch>.exe /quiet InstallAllUser=-1 PrependPath=1 TargetDir="C:\Python\Python<version>"
Related
Let's say I don't want to access this python installation from PATH or py on the console, just call on the python.exe file directly. In that case, can I copy the python installation directory into an app folder for something I'm developing to have essentially a portable python packaged with my app?
Yes you can do that. But there is a concept in Python called virtual environments that you should take a look at. The idea is to seperate a project environment from the one on your operating system. It's the recommended way for developing python applications as you can have multiple different environments depending on your projects requirements.
https://docs.python.org/3/tutorial/venv.html
You can also take a look at Conda. Its a more user friendly way of using Environments (in my opinion). If you are interested in Conda you can also look at that question for more information about the differences of Miniconda and Anaconda: Anaconda vs. miniconda
Installation of Conda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/windows.html
My question is do i have to install django every single time in my virtual environment in order to run my python files? and is this taking up bunch of space on my machine? My project also uses "matplotlib" and every virtual environment i create it also asks me to import the matplotlib module too. its getting annoying. do i have to do this every time?
Im new to Django. I wanted to run some python files in django but they weren't working, so after some research i found out i needed to run my pycharm project in a virtual environment in order to run these python files.
my folders look like this pycharmProjects -> my project
I enter pycharmProjects and I set up virtual environment using "pienv shell". Then i run "python3 manage.py runserver". It turns out i must install django in the virtual environment before the files run.
Short answer is no, you don't have to use a virtual environment at all and can install your dependancies globally instead. However you will soon find that it will cause a lot of issues. The main reason you would create a virtual environment is to give control of your dependancies and prevent bugs that could be caused because of them having their wires crossed between projects.
Short answer yes.
If you create a virualenv you have to install all packages, that your program needs.
Long answer:
You could install django system wide and then create a virtualenv with the option
--system-site-packages then django would be used from your globally installed python.
(Or you install everything just in your global python, put I personally don't think this is good practice)
If you work with many different projects I think you will avoid a lot of trouble if you use one virtualenv per project.
Trouble meaning that one project breaks, because one pip install for another project changed the version of one package and one project can't handle the newer version.
I would recommend to create a requirements.txt file for each project, that lists the dependencies then you can create the virtualenv with following command
pip install -r requirements.txt
if you have requirement.txt files, then you can create virtualenvs rather quickly if going back to an old project and you can delete the virtualenvs whenever you run out of disk space. If you want to be an the safe side, type pip freeze > pipfreeze.txt prior to deleting the virtualenv and use pip install -r pipfreeze.txt if you want to create one with the same modules and the same versions.
You also might want to look at direnv or autoenv if working on a linux like system.
This will automatically switch to the required virtualenv when changing to a project's working directory.
I am working on several projects on the same PyCharm. Like I "attached" them all together. But I recently noticed some weird behaviors. Like when I import a library I haven't installed yet to my script. It shows me a little error as expected. But when I try to install that using python -m pip install my_library, it tells me that it has already installed. I recently noticed that this is because it's using and other pip from another project. I doesn't use the one in the venv folder in the project. Also to run the scripts sometimes it uses python.exe from pythons original directory. It's a whole mess and I have no idea how I can solve it. Sometimes my projects requires different versions of the same library and you can imagine what happens when I change the version.
I make sure each project is using their own interpreter. Don't know what else to do other than this. I am using Python3.6.4 PyCharm2018.3.2 running on Windows10
it sounds like all your projects are configured to use the system's interpreter instead of the virtual environment you set up for each of them.
Follow this instruction to fix it https://www.jetbrains.com/help/pycharm-edu/creating-virtual-environment.html
In terms of using different version of the python library, you can address that by specifying it in requirements.txt file, which you can put in your venv folder for each project. then you can just do pip install -r requirements.txt after you set up your venv. (you need to ensure that the venv is activated - you don't need to worry about this if you have configured the project in PyCharm to use the venv's python interpreter.) You can check this by going to Terminal in your PyCharm and you should see (venv_name) hostusername#host:~/project_folder$
I've just setup a new environment for my project and uploaded a python repository including bin, lib and project folder. I'm pretty sure I did same previously and it worked without problem. Now when doing the same on an AWS environment I get the error
-bash: /projects/scrapy/bin/python2.7: cannot execute binary file. However when doing source /projects/scrapy/bin/activate it successfully activates the environment.
From what I understand, python should be able to execute without any issue no matter the environment ?
Any help or pointing to the right direction would be much appreciated!
python should be able to execute without any issue no matter the environment ?
No, the Python binary is tied to your specific OS and computer architecture. Python source code can usually be run on different machines (provided you didn't use OS-specific features), but that's only made possible by compiling a Python interpreter for the specific target environment first.
In other words, a Python binary compiled to run on macOS will not work on Linux.
All that source bin/activate achieves is that it configures your terminal setting to use the bin directory as the first directory on the PATH search path. This doesn't make bin/python work in another environment, it just means that both environments have a working shell interpreter that can run that script.
Create a new virtualenv with a Python binary compiled for Linux, and install the same packages there. Use Pipenv or a requirements.txt file to transfer the dependencies from Mac to Linux.
For example, using Pipenv you'd copy over the Pipfile and Pipfile.lock files to the other computer, then run pipenv install in the directory there and re-create the virtualenv and dependencies from those files.
I recommend you read up on Python development best practices in the The Hitchhiker’s Guide to Python; this includes such topics on how to manage an environment for a project.
Let me first outline my desired solution and then elaborate on a specific question how to achieve this state.
I'm soon starting two coding projects in python. I've used python before but never on such big projects. My ideal scenario would be to have a setup where I can run virtual environments and different python version for various project. Some research pointed me to virtualenv / virtualenvwrapper and pyenv. It seems using pyenv-virtualenv or pyenv-virtualenvwrapper there is a nice way to specify the virtualenvironment and python version for a specific project.
Question: Once I've setup a virtualenvironment and python version for a specific project, how easily could I switch in a later stage to a newer python version? Let's say I've started project A with python 3.4 and in one year in the future I would like to move everything to python 3.6. Is this possible in a neat way?
Sure:
$ rm -r my-python-3.4-env
$ virtualenv -p python3.6 my-python-3.6-env
$ source my-python-3.6-env/bin/activate
In other words, each virtual environment is just a folder with the necessary files in it. You "activate" an environment with the source .../activate command (in case of virtualenv) and you leave it just as easily. To switch to a different environment you simply create a new one with a specific Python executable and activate it.
What you want to be careful about is to keep your installation repeatable, meaning if you depend on external modules (which modern projects typically do), you don't want to install each dependency by hand and instead automate that. For instance, you create a setuptools setup.py file which lists your dependencies, and then have it install them into your new environment automatically:
$ source my-python-3.6-env/bin/activate
(my-python-3.6-env) $ python setup.py develop