I would like to make changes (and possibly contribute if its any good) to a public project on GitHub. I've forked and cloned the module but Im unclear how to get my program to import the local library instead of the 'official' installed module.
I tried cloning it into my project folder but when I imported it and tried to use it things got weird calmap\calmap.plot()
I also tried doing sys.path.append and the folder location. But it seems to still import the official one instead of the forked.
I'm assuming that I could put my program inside the module folder so that module would be found first but I can't image thats the 'correct' way to do it.
|
|-->My_Project_Folder/
|
|-->Forked_Module/
|-->docs/
|-->Forked_Module/
|-->__init__.py
If you're already using anaconda, then you can create a new environment just for the development of this feature.
First, create a new environment:
# develop_lib is the name of the environment.
# You can pick anything that is memorable instead.
# You can also use whatever python version you require ...
conda create -n develop_lib python3.5
Once you have the environment, then you probably want to enter that environment in your current session:
source activate develop_lib
Ok, now that you have the environment set up, you'll probably need to install some requirements for whatever third party library you're developing. I don't know what those dependencies are, but you can install them in your environment using conda install (if they're available) or using pip. Now you're ready to start working with the library that you want to update. python setup.py develop should be available assuming that the package has a standard build process. After you've run that, things should be good to go. You can make changes, run tests, etc.
If you use sys.path.append() the new "path" will be used if none of the previous contains the module you are importing. If you want that the "added path" has precedence over all the older, you have to use
sys.path.insert(0, "path")
In this way, if you print the sys.path you will see that the added path is at the beginning of the list and the module you are importing will be loaded from the path you have specified.
to import from the forked repo instead of python package you should
make a virtual environment for the cloned project then activate it, that way the environment is isolated from the globally installed packages.
1- you need to fork your repo;
2- create a virtual env and activate it;
3- clone your repo.
now if you print your import you will see the path of the forked repo.
import any_module
print(any_module)
Related
Why does pip need to use virtual environments to isolate packages per-project, instead of just installing them in a default directory in the project? This seems like added complexity without benefit.
NPM, for example, installs packages in the <project_root>\node_modules by default. No virtual environment necessary, and packages are still installed in a project-independent way.
Edit: To be clear, I'm interested in the practical advantages to pip's use of virtual environments over package management systems like NPM, Nuget, and Webpack, which all use directories in the project. Otherwise, if this is just a limitation of Python's modules system, then I'd be interested to know that too.
Because Python's module system doesn't work that way. If pip were to install, say, requests by just downloading it to a python_modules directory, that wouldn't be enough to for import requests to work; it would have to be import python_modules.requests, but then we'd still have problems whenever requests tried to import one of its dependencies, as that would need python_modules prepended, too, and it'd just be a big mess. The solution that virtual environments use is to modify the PYTHONPATH environment variable to include python_modules, plus some extra stuff to take care of executable scripts and not importing packages from outside the virtualenv.
I think maybe you don't know what a virtual environment actually is.
If you were to put some module in a project-specific directory, like myproj/modules, then you would have to add myproj/modules to the search path that Python uses so that you module can be found. One way to do that is to define or modify the environment variable PYTHONPATH. Any directories listed in that variable will be searched for modules, in addition to some hard-coded set of directories.
$ export PYTHONPATH=./myproj/modules
However, that's really all a virtual environment is. The directory contains the desired version of Python, along with whatever modules you want to use. The activate script you run to "enable" a virtual environment does little more than set the value of PATH and PYTHONPATH so that anytime you run python, both the correct version is used and your project-specific set of modules is used in place of any global library.
I have a bunch of Python scripts that I want to deploy to other machines. Thing is, I want to have everything self-contained and not depend on the other machines' libraries. For example I don't want to request users to have virtual environment and pip as installed in order for my app to work.
On my local machine I use virtual environment with --no-site-packages and pip install -r requirements.txt to get everything in place.
The bad news is virtualenvironment's activate script has my local path hardcoded into it and using the --relocatable option does not help with this situation so I suppose virtualenvironment is out of the question?
What I would like to have is something similar to this:
base_app_dir:
- main_app_dir
- my_init_script.py
- bin(includes python binary)
- lib(includes pip installed packages and python libraries)
so that I can instruct the end user to just cd into base_app_dir and do a ./bin/python -m my_init_script.py but that means I now need to instruct Python to look into my ./lib folder when importing packages.
I've tried setting os.path.insert(1, 'base_app_dir/lib/site-packages') but this work on per module basis.
Also how about lookup for default Python modules? Right now for example when import hashlib it tries to get it from /usr/lib/python2.7/hashlib.py. I would like to deploy these default Python modules as well and instruct the app to import them from my custom location.
Py2exe or creating a .deb file is not an option right now so please try to address my specific question.
I've installed a Python based package in my site-packages directory. However, I'm trying to learn how the code works so I'd like to hack it a bunch by putting in lots of print statements so I can understand what the code is doing. But at the end of the day I want a clean installation without all my hacks in it.
Of course I could just copy the original files to something else, make some hacks, and then at the end copy all the original files back over. But that's really tedious. At the very least, I'd like to install a local copy of the Python package and then have the python script use this copy preferentially (perhaps by suitable statements at the top of the script). But perhaps this isn't even the best way to do python development/hacking.
What's the best solution for my problem? I want to be able to hack on the package (and use that package) but without messing up my clean version.
Take a look at virtualenv. You can basically setup a local python environment in which you can install anything you like without having to mess around with the system environment.
The virtualenv-advice given is correct, depending on your actual package you can even go beyond that and not mess with the site-packages inside the virtualenv at all.
If the package is setuptools-based, a simple
$ python setup.py develop
from within a working-copy of it's source, it won't be installed, but instead just hooked into the virtualenv pointing to the working-copy. Advantage: you can edit (and e.g. rollback using GIT or whatever SCM the package maintainer uses) files in a well-defined and non-volatile location.
This is what the Python virtualenv tool is for. It allows you to create a local Python environment with a set of packages distinct from your system installation. For example, I could do something like this:
$ virtualenv myenv
$ . myenv/bin/activate
$ pip install nifty-module
The activate script modifies your PATH so that any script that starts with:
#!/usr/bin/env python
will use the Python from your virtual environment, rather than the system Python, and will see the modules installed in that environment.
I have a cluster system at work with Python and some modules installed on that system - however I wanted to use the most up to date version of the module - it has several methods not present in older versions, so I built it and it's deps locally in the area I have access to:
# From my home directory: /gpfs/env/yrq12edu
# Get the source I need for the up to date version of the module I want to install locally.
svn co svn://svn.code.sf.net/p/simupop/code/trunk simuPOP
# Install PCRE stuff...
cd pcre-8.34
./configure --prefix=/gpfs/env/yrq12edu/pcre_install
make
make install
export PATH=/gpfs/env/yrq12edu/pcre_install/bin:$PATH
export LD_LIBRARY_PATH=/gpfs/env/yrq12edu/pcre_install/lib:$LD_LIBRARY_PATH
cd ..
# Install Swig Stuff...
cd swig-3.0.0
./configure --prefix=/gpfs/env/yrq12edu/swig_install
make
make install
export PATH=/gpfs/env/yrq12edu/swig_install/bin:$PATH
cd ..
export PYTHONPATH=/gpfs/env/yrq12edu/PythonModules/lib/python2.7/site-packages
# Build the up to date simuPOP module I need locally...
cd simuPOP
python setup.py install --prefix=/gpfs/env/yrq12edu/PythonModules
How can I ensure that when I execute my Python scripts in the cluster it will try and use my local module rather than the system one? I have obviously changed PYTHONPATH during the build process which I know should allow modules to be loaded locally, but wondered which it will load when there is the choice of the system installed old version, or my new locally installed version. Will Python just know to favour the local one and load it instead or do I have to specify some option to force it?
Thanks,
Ben W.
According to the docs Python will load the built-in module if it's available. If it's not, it then looks in each path in sys.path (which starts with the current directory).
However, if I'm reading it correctly, standard modules are different from built-in modules. Standard modules are found by looking in sys.path, so if you put your path at the start of sys.path Python will get your module instead of the standard one.
I have a local git repository on my machine, let's say under /develop/myPackage.
I'm currently developing it as a python package (a Django app) and I would like to access it from my local virtualenv. I've tried to include its path in my PYTHONPATH (I'm on a Mac)
export PATH="$PATH:/develop/myPackage"
The directory already contains a __init__.py within its root and within each subdirectory.
No matter what I do but I can't get it work, python won't see my package.
The alternatives are:
Push my local change to github and install the package within my virtualenv from there with pip
Activate my virtualenv and install the package manually with python setup.py install
Since I often need to make changes to my code the last two solution would require too much work all the time even for a small change.
Am I doing something wrong? Would you suggest a better solution?
Install it in editable mode from your local path:
pip install -e /develop/MyPackage
This actually symlinks the package within your virtualenv so you can keep on devving and testing.
The example you show above uses PATH, and not PYTHONPATH. Generally, the search path used by python is partially predicated on the PYTHONPATH environment variable (PATH has little use for this case.)
Try this:
export PYTHONPATH=$PYTHONPATH:/develop/myPackage
Though in reality, you likely want it to be pointing to the directory that contains your package (so you can do 'import myPackage', rather than importing things within the package. That being said, you likely want:
export PYTHONPATH=$PYTHONPATH:/develop/
Reference the python docs here for more information about Python's module/package search path: http://docs.python.org/2/tutorial/modules.html#the-module-search-path
By default, Python uses the packages that it was installed with as it's default path, and as a result PYTHONPATH is unset in the environment.