I'm trying to pip install a GitHub project locally, outside of site-packages so that I can modify it, etc.
I've added -e git+git#github.com:Starcross/django-starcross-gallery.git#egg=gallery to my requirements.txt which brings the relevant part of my project layout to look like this:
/home/mat/venv/proj/
└── src
└── gallery
├── admin.py
├── apps.py
├── build.sh
├── django_starcross_gallery.egg-info
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── forms.py
├── __init__.py
├── LICENSE
├── MANIFEST.in
├── models.py
├── README.rst
├── settings.py
├── setup.py
├── signals.py
├── static
│ └── ...
├── templates
│ └── ...
├── tests
│ └── ...
├── tests.py
├── urls.py
└── views.py
As far as I can see the problem is that these .egg-link and .pth files like one level too deep:
lib/python3.6/site-packages/django-starcross-gallery.egg-link:
/home/mat/venv/proj/src/gallery
.
lib/python3.6/site-packages/easy-install.pth:
/home/mat/venv/proj/src/gallery
I can fix everything by either moving gallery a level deeper, or changing django-starcross-gallery.egg-link and easy-install.pth to point to src.
Is there a config parameter I can pass in requirements.txt to make this work properly? Or do I have to adjust the project layout to fit?
Since you want to modify it, why not just clone the repo. To make your interpreter able to find and use it, you have some options:
modify your sys.path, append path to the repo
create a symlink under your project directory that points to the repo
And in this way, you don't have to pip install every time you modify it.
As has been mentioned, the best way to do this is to clone the repo. This would go for most packages as pip may build extensions, and carry other actions during install aimed at using the module for production rather than editing the source.
To explain why I chose this structure, I wanted to be able to develop the package inside a Django project. As the Django docs say, the app should be placed in a separate directory, which enables setuptools to install the package correctly. There is no way I could find that would enable this to continue to work inside a project, hence the build script to move the files into a suitable directory and generate the package.
Related
I'd like to figure out the cleanest and preferably self contained way to use my packages in scripts that are in a different directory to the package itself.
The example problem is as follows:
The modules in lib need to both be imported, and run as a script.
My project directory is as below and I'm having two issues:
in lib/api.py, I want to read in data_files/key.txt correctly when api.py is called or imported
in testing_script.py I want to import and use lib/get_data.py
I can't seem to find a clean way to do this, does this mean my project is structured in a non-pythonic way?
Thanks for the help.
my-project-git
├── LICENSE
├── README.md
├─── my_project
│ ├── data_files
│ │ ├── key.txt
│ │ ├── mappings.csv
│ ├── lib
│ │ ├── __init__.py
│ │ ├── api.py
│ │ └── get_data.py
│ └── test
│ ├── __init__.py
│ └── testing_script.py
├── requirements.txt
└── setup.py
As far as I know, there's isn't a pythonic way to structure your project.
This is what Kenneth Reitz recommended in 2013 and it's how I use it: https://www.kennethreitz.org/essays/repository-structure-and-python.
README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py
Inside sample (my_project in your case) you can separate into categories as you like. E.g. Utils (common functions), Database (read, write), View (user commands), etc. It depends on your project.
As for calling the modules at the same level, you should define them in the __init__ file of the top hierarchy module which is sample in this case.
For example:
__init__ in _my_project
from sample.core import a_Class
from sample.core import a_function
from sample.core import anything
then from /test/test_basic.py you do:
from sample import a_Class
# or import sample
a = a_Class() # use the class from core.py
# or a = sample.a_Class()
Take a look at the sample module repository: https://github.com/navdeep-G/samplemod
I have a below structure
in migrations/env.py file I am trying to import from database import *
but it shows no module name database
I tried from ..database imprt * and adding file in pythonpath also but no luck :(
Your directory structure looks a bit suspicious to me. The alembic.ini shouldn't normally be part of the package (and setuptools won't by default pick it up when packaging). I think this would better be placed into the project-root.
Something like this would be more standard:
├── alembic.ini
├── migrations
│ ├── env.py
│ ├── script.py.mako
│ └── versions
│ └── ...
├── package_name
│ └── database
│ ├── __init__.py
│ └── ...
│ └── models
│ └── __init__.py
│ └── ...
├── README.md
└── setup.py
└── ...
Now, this alone would not make database available from env.py. For this to work you have to somehow make your package discoverable. Usually this would be done by installing package_name into some virtualenv. In that environment you could then use from package_name.database import * in your env.py.
Migrations needs to know where to import from, they either belong to the same package:
A:
migrations
database
init.py
And then in migrations:
from A.database.whatever import whatever else
Or you install them as packages separatedly inside your virtualenv:
And then each of them is dependent on the other, but because they are installed they can be invoked:
database/setup.py
migrations/setup.py
Then both are installed and migrations/env.py can call the installed package database
In order to manage all my python paths for my project and have them available as soon as I start python interpreter, I created a project.pth in the project home directory having relative paths in it,
Actually, to be read, I need to do a site.addsitedir(my_project_home_dir) each time I start the interpreter.
I tried setting PYTHONPATH or create a .pth in site-packages pointing to my project home directory, but project.pth is still not read automatically when I start the interpreter.
The only thing that works is to put my project.pth in site-packages, but by doing that, I have to transform my project relative paths to absolute paths.
So it there a master .pth file where I can specify my project home directory so I can have my project.pth located in that directory to be read automatically ?
While I know this doesn't specifically answer your question, might I suggest a different approach? The use of virtual environments (venv).
Given a project (I used pyscaffold) to create.
- Create a virtual environment at the highest level (python -m venv venv).
- Add PYTHONPATH to your venv\Scripts\activate.bat file.
- PYTHONPATH should contain at least three entries (path\to\MyTestProject; pat\to\MyTestProject\src; path\to\MyTestProject\tests). Activate your virtual environment from cmd by "venv\Scripts\activate.bat". - From within your modules, import from src (import src.mytestproject.package1.module1)
You will have your paths available to you each time you activate your virtual environment - and, you will have all the goodness associated with a virtual environment as a bonus.
MyTestProject
├── AUTHORS.rst
├── CHANGELOG.rst
├── docs
│ ├── authors.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── index.rst
│ ├── license.rst
│ ├── Makefile
│ └── _static
├── LICENSE.txt
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│ └── mytestproject
│ ├── __init__.py
│ ├── package1
│ │ ├── __init__.py
│ │ └── module1.py
│ ├── package2
│ │ ├── __init__.py
│ │ └── module2.py
│ └── skeleton.py
└── tests
├── conftest.py
└── test_skeleton.py
Finally I found 2 ways :
1) First, note that .pth files also accept lines beginning with 'import' and execute them. So the solution was to create project.pth inside site-packages importing a module in my project home dir that do actually the site.addsitedir()
2) Other possibility : use the environment variable PYTHONSTARTUP : it will execute any module you want at python startup. You can specify a module that do the site.addsitedir() or even directly add paths into sys.path
I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.
This is my first time trying to set up a vagrant environment or a python virtuelenv, so forgive me if I am missing something basic.
Right now, I ssh into my vagrant box and in the home directory I have placed my venv folder. I have run
source venv/bin/activate
From my home directory I move to /vagrant, and within here I have my project files laid out something like this:
├──project
├── LICENSE
│
├── project
│ │ ├── exceptions.py
│ │ ├── __init__.py
│ │ ├── resources
│ │ │ ├── base.py
│ │ │ ├── __init__.py
│ │ └── target
│ │ └── __init__.py
│ │ └── test.py
│ ├── README.md
My problem is I am unable to import my modules in different directories. For example, if I am in /vagrant/project/project/target/test.py and I attempt:
import project.exceptions
I will get the error
ImportError: No module named project.exceptions
If I am in the /vagrant/project/project directory and I run
import exceptions
that works fine.
I have read up on similar problems people have experienced on StackOverflow.
Based on this question: Can't import package from virtualenv I have checked that my sys.executable path is the same in both my python interpreter as well as when I run a script (home/vagrant/venv/bin/python)
Based on this question: Import error with virtualenv. I have run ~/venv/bin/python directly and attempted to import, but the import still fails.
Let me know if there is more information I can provide. Thank you.
You have two options:
You can install your project into the virtual environment, by writing a setup.py file and by calling python setup.py install. See the Python Packaging User Guide.
You can set the PYTHONPATH environment variable to point to your project, like this:
$ export PYTHONPATH=$PYTHONPATH:/vagrant/project