How can the QGIS python plugin environment be directed to use a local git repo for a specific dependency rather than its default site-packages cache?
Context:
I’m working on two local git repos in QGIS python environment: 1) a custom plugin, and 2) a custom module the plugin imports.
Changes to the plugin code are recognized by reloading the plugin via QGIS. However, changes to the module aren’t being recognized by the plugin, and it’s not clear how to instruct QGIS (or conda) where to look.
I’m running Windows 11, Python miniconda3.
Two solutions discovered to date:
create a python virtual environment (where QGIS is launched) within the local git repo
create a .pth file in the virtual environment’s site-packages directory pointing QGIS to the local git repo
Both of these solutions seem to work well enough, but it’s clear from other SO posts that the first is suboptimal. Not clear whether the second is truly best practice, or whether a third alternative would be.
Related
I'm a bit confused on this and can't find a satisfactory answer searching elsewhere - everything is about using external libraries in some way or another.
In my PyCharm projects, I've got your project root directory, which contains a venv. This venv of course contains the python executable relative to the project, as well as the relevant site-packages and modules I've downloaded via pip or whatever.
But there's always an External Libraries section, that has it's own Python executable, venv, .gitignore, site-packages, etc.
Is this normal? What's the purpose of having two venvs in the same project?
Thanks!
You only have one venv normally in your Project
The other Python Executable should be the global one.
Normally you use a venv to keep track of the needed Packages for this specific project. So that you don't confuse other Packages with other projects or have dependency/Version Issues
If you want you can also use the global Installation of Python which would not create a venv
I am writing a code in python that uses numpy, matplotlib etc.
How to make sure that even a remote web server with python installed but no extra modules, can run the code without errors?
I usually work on linux environment. Hence from source code, I can install the libraries in a prefix directory and can keep that along with my code. Then add pythonpath locally in my python code to use the directory.
But, I started to realize it's not correct way as first thing, it can't work on cross platform as the libraries are different, and my code inside the script to extend the pythonpath may not work due to the use of "/" in path. Also, I am not sure if the compiled code can work in different environments of the same Linux Platform.
So I think I need to create a directory like unix,windows,osx etc. and put my code there? I believe this is what I find when I download any code online. Is that what developers generally do to avoid these issues?
A popular convention is to list requirements in a text file (requirements.txt) and install them when deploying the project. Depending on your deployment configuration, libraries can be installed in a virtual environment (google keyword: virtualenv), or in a local user folder (pip install --user -r requirements.txt, if this is the only project under this account) or globally (pip install -r requirements.txt, e.g. in a docker container)
I have a Python project which contains three components: main executable scripts, modules which those scripts rely on, and data (sqlite3 databases, flat files, etc.) which those scripts manipulate. The top level has an __init__.py file so that other programs can also borrow from the modules if needed.
The question is, is it more "Pythonic" or "correct" to move my project into the default site-packages directory, or to modify PYTHONPATH to include one directory above my project (so that the project can be imported from)? On the one hand, what I've described is not strictly a "package", but a "project" with data that can be treated as a package. So I'm leaning in the direction of modifying PYTHONPATH (after all, PYTHONPATH must exist for a reason, right?)
Definitely do not add your project to site-packages this is going to spoil your system Python installation and will fire back at the moment some other app would come there or you would need to install something.
There are at last two popular options for installing python apps in isolated manner
Using virtualenv
See virtualenv project. It allows
creation of new isolating python environment - python for this environment is different from system one and has it's own PYTHONPATH setup this allows to keep all installed packages private for it.
activation and deactivation of given virtualenv from command line for command line usage. After activate, you can run pip install etc. and it will affect only given virtualenv install.
calling any Python script by starting by virtualenv Python copy - this will use related virtualenv (note, that there is no need to call any activate)
using zc.buildout
This package provides command buildout. With this, you can use special configuration file and this allows creation of local python environment with all packages and scripts.
Conclusions
virtualenv seems more popular today and I find it much easier to learn and use
zc.buildout might be also working for you, but be prepared for a bit longer learning time
installing into system Python directories shall be always reserved for very special cases (pip, easy_install), better avoid it.
installing into private directories and manipulatig PYTHONPATH is also an option, but you would repeat, what virtualenv already provides
I'm using Python with a Cygwin environment to develop data processing scripts and Python packages I'd like to actively use the scripts while also updating the packages on which those scripts depend. My question is what is the best practice, recommendation for managing the module loading path to isolate and test my development changes but not affect the working of a production script.
Python imports modules in the following order (see M. Lutz, Learning Python)
Home directory.
PYTHONPATH directories.
Standard library directories.
The contents of any *.pth file.
My current solution is to install my packages in a local (not in /usr/lib/python2.x/ ) site-packages directory and add a *.pth file in the global site-packages directory so these are loaded by default. In the development directory I then simply modify PYTHONPATH to load the packages I'm actively working on with local changes.
Is there a more standard way of handling this situation? Setting up a virtualenv or some other way of manipulating the module load path?
This is just my opinion, but I would probably use a combination of virtualenvs and Makefiles/scripts in this case. I haven't done it for your specific use case, but I often set up multiple virtualenvs for a project, each with a different python version. Then I can use Makefiles to run my code or tests in one or all of my virtualenvs. Seems like it wouldn't be too hard to set up a makefile that would let you type make devel to run in the development envionment, and make production for the production environment.
Alternatively, you could use git branches to do this. Keep your production scripts on master, and use feature branches to isolate and test changes while still having your production scripts just a git checkout master away.
My current setup is Python 2.5/ Django 1.1.1 on Windows. I want to start using Django 1.2 on some projects, but can't use it for everything. Which is just the sort of thing I've got virtualenv for. However, I'm running into a problem I've never encountered and it's hard to Google for: installing Django 1.2 into a virtualenv has no effect for me. If I then activate the environment and do
python
import django
django.VERSION
I get "1.1.1 Final". Django is installed in the site-packages directory of my environment and the init file in the root shows that it is 1.2. But the environment falls back to 1.1.1, even if I create the environment with the --no-site-packages flag. What am I screwing up?
Based on the bug you filed at bitbucket, it looks like you're using the PYTHONPATH environment variable to point to a directory with some packages, including Django 1.1.1. By design, PYTHONPATH always comes first in your sys.path, even when you have a virtualenv activated (because PYTHONPATH is under your direct and immediate control, and people use it for local overrides).
In this case, if you don't want that PYTHONPATH when this virtualenv is activated, you'll need to take care of that yourself; perhaps by creating a custom batch file that both calls the virtualenv's activate.bat and also modifies PYTHONPATH.
Some tools you can use to diagnose these problems:
To see where django is coming from, print django.__file__. It will show the file indicating django's location on the file system.
To see all the places Python will look for packages, print sys.path. It's a list of directories.
To see imports as they happen, start python as python -v, and you'll see lots of debugging information about where packages are being imported.