In order to manage all my python paths for my project and have them available as soon as I start python interpreter, I created a project.pth in the project home directory having relative paths in it,
Actually, to be read, I need to do a site.addsitedir(my_project_home_dir) each time I start the interpreter.
I tried setting PYTHONPATH or create a .pth in site-packages pointing to my project home directory, but project.pth is still not read automatically when I start the interpreter.
The only thing that works is to put my project.pth in site-packages, but by doing that, I have to transform my project relative paths to absolute paths.
So it there a master .pth file where I can specify my project home directory so I can have my project.pth located in that directory to be read automatically ?
While I know this doesn't specifically answer your question, might I suggest a different approach? The use of virtual environments (venv).
Given a project (I used pyscaffold) to create.
- Create a virtual environment at the highest level (python -m venv venv).
- Add PYTHONPATH to your venv\Scripts\activate.bat file.
- PYTHONPATH should contain at least three entries (path\to\MyTestProject; pat\to\MyTestProject\src; path\to\MyTestProject\tests). Activate your virtual environment from cmd by "venv\Scripts\activate.bat". - From within your modules, import from src (import src.mytestproject.package1.module1)
You will have your paths available to you each time you activate your virtual environment - and, you will have all the goodness associated with a virtual environment as a bonus.
MyTestProject
├── AUTHORS.rst
├── CHANGELOG.rst
├── docs
│ ├── authors.rst
│ ├── changelog.rst
│ ├── conf.py
│ ├── index.rst
│ ├── license.rst
│ ├── Makefile
│ └── _static
├── LICENSE.txt
├── README.rst
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│ └── mytestproject
│ ├── __init__.py
│ ├── package1
│ │ ├── __init__.py
│ │ └── module1.py
│ ├── package2
│ │ ├── __init__.py
│ │ └── module2.py
│ └── skeleton.py
└── tests
├── conftest.py
└── test_skeleton.py
Finally I found 2 ways :
1) First, note that .pth files also accept lines beginning with 'import' and execute them. So the solution was to create project.pth inside site-packages importing a module in my project home dir that do actually the site.addsitedir()
2) Other possibility : use the environment variable PYTHONSTARTUP : it will execute any module you want at python startup. You can specify a module that do the site.addsitedir() or even directly add paths into sys.path
Related
coming from php, the namespaces are always well defined in php, typically using psr-4 from composer
https://getcomposer.org/doc/04-schema.md#psr-4
e.g. you have a folder
src/Foo/Bar/Baz
you then define in the composer.json file that
src/Foo is where the namespace Foo starts
thereafter, all subfolders are by convention, a new sub namespace e.g.
src/Foo/Bar/Baz/MyClass.php turns into
Foo.Bar.Baz.MyClass
Say I want to drop a utility python class into my django project, that I want to use in all django "apps"
Where would I drop it, and how to properly define the namespace? what is the transparent way to understand namespaces in python?
From this document:
https://docs.djangoproject.com/en/2.2/ref/applications/#projects-and-applications
Is this snippet:
A project’s root directory (the one that contains manage.py) is usually the container for all of a project’s applications which aren’t installed separately.
You can create whatever hierarchy you chose under that directory for utilities, which can conform to python's package system:
https://docs.python.org/3/tutorial/modules.html#packages
In Python, there is a search path for modules. It can be initialized from the command line with the PYTHONPATH environment variable and accessed programmatically via sys.path.
By default, the directory of the script you use to start the Python interpreter is the first entry in the search path.
Any module or package on the python path "starts a new namespace", to use your wording. Let's assume your project called mysite has an app called polls and has this structure:
mysite
├── manage.py
├── mysite
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ ├── utils.py
│ └── wsgi.py
├── polls
│ ├── __init__.py
│ ├── models.py
│ ├── utils.py
...
If you start it with python manage.py runserver, the packages mysite and polls are available.
I'd say a good place for project-wide utils module is the project package mysite. So you do import mysite.utils in any of your apps. If it grows beyond a single file, you can turn it into a subpackage, so instead of a utils.py, you have a utils directory containing an __init__.py file.
The above structure clutters the global namespace a bit. So some people prefer a structure where all your apps are subpackages of your project package, like this:
mysite
├── manage.py
├── mysite
│ ├── __init__.py
│ ├── settings.py
│ ├── urls.py
│ ├── utils.py
│ ├── wsgi.py
│ └── apps
│ └── polls
│ ├── __init__.py
│ ├── models.py
│ ├── utils.py
...
I'm trying to pip install a GitHub project locally, outside of site-packages so that I can modify it, etc.
I've added -e git+git#github.com:Starcross/django-starcross-gallery.git#egg=gallery to my requirements.txt which brings the relevant part of my project layout to look like this:
/home/mat/venv/proj/
└── src
└── gallery
├── admin.py
├── apps.py
├── build.sh
├── django_starcross_gallery.egg-info
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── forms.py
├── __init__.py
├── LICENSE
├── MANIFEST.in
├── models.py
├── README.rst
├── settings.py
├── setup.py
├── signals.py
├── static
│ └── ...
├── templates
│ └── ...
├── tests
│ └── ...
├── tests.py
├── urls.py
└── views.py
As far as I can see the problem is that these .egg-link and .pth files like one level too deep:
lib/python3.6/site-packages/django-starcross-gallery.egg-link:
/home/mat/venv/proj/src/gallery
.
lib/python3.6/site-packages/easy-install.pth:
/home/mat/venv/proj/src/gallery
I can fix everything by either moving gallery a level deeper, or changing django-starcross-gallery.egg-link and easy-install.pth to point to src.
Is there a config parameter I can pass in requirements.txt to make this work properly? Or do I have to adjust the project layout to fit?
Since you want to modify it, why not just clone the repo. To make your interpreter able to find and use it, you have some options:
modify your sys.path, append path to the repo
create a symlink under your project directory that points to the repo
And in this way, you don't have to pip install every time you modify it.
As has been mentioned, the best way to do this is to clone the repo. This would go for most packages as pip may build extensions, and carry other actions during install aimed at using the module for production rather than editing the source.
To explain why I chose this structure, I wanted to be able to develop the package inside a Django project. As the Django docs say, the app should be placed in a separate directory, which enables setuptools to install the package correctly. There is no way I could find that would enable this to continue to work inside a project, hence the build script to move the files into a suitable directory and generate the package.
When I build or debug a particular file in my Python project (which imports a user defined package) I get an import error. How can I solve this problem?
test.py
def sum(a,b):
return a+b
test2.py
from test import sum
sum(3,4)
The above code will give an import error cannot import test.
Directory tree
├── graphs
│ ├── Dijkstra's\ Algorithm.py
│ ├── Floyd\ Warshall\ DP.py
│ ├── Kruskal's\ algorithm.py
│ ├── Prim's\ Algoritm.py
│ ├── __init__.py
│ └── graph.py
├── heap
│ ├── __init__.py
│ ├── heap.py
│ └── priority_queue.py
Trying to import in graphs;
from heap.heap import Heap
About the heap file, make sure that you are running on the project root folder.
If these test.py files are running on the same folder, try to add a __init__.py empty file on this folder.
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later (deeper) on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialisation code for the package or set the __all__ variable, described later.
I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.
This is my first time trying to set up a vagrant environment or a python virtuelenv, so forgive me if I am missing something basic.
Right now, I ssh into my vagrant box and in the home directory I have placed my venv folder. I have run
source venv/bin/activate
From my home directory I move to /vagrant, and within here I have my project files laid out something like this:
├──project
├── LICENSE
│
├── project
│ │ ├── exceptions.py
│ │ ├── __init__.py
│ │ ├── resources
│ │ │ ├── base.py
│ │ │ ├── __init__.py
│ │ └── target
│ │ └── __init__.py
│ │ └── test.py
│ ├── README.md
My problem is I am unable to import my modules in different directories. For example, if I am in /vagrant/project/project/target/test.py and I attempt:
import project.exceptions
I will get the error
ImportError: No module named project.exceptions
If I am in the /vagrant/project/project directory and I run
import exceptions
that works fine.
I have read up on similar problems people have experienced on StackOverflow.
Based on this question: Can't import package from virtualenv I have checked that my sys.executable path is the same in both my python interpreter as well as when I run a script (home/vagrant/venv/bin/python)
Based on this question: Import error with virtualenv. I have run ~/venv/bin/python directly and attempted to import, but the import still fails.
Let me know if there is more information I can provide. Thank you.
You have two options:
You can install your project into the virtual environment, by writing a setup.py file and by calling python setup.py install. See the Python Packaging User Guide.
You can set the PYTHONPATH environment variable to point to your project, like this:
$ export PYTHONPATH=$PYTHONPATH:/vagrant/project