I have the following directory structure:
├── DynamicProgramming
│ ├── 0-1_kp_problem.py
│ ├── b.py
│ ├── largest_contigous_subarray.py
│ ├── longest_common_substring.py
│ ├── min_change_for_given_money.py
│ ├── optimal_matrix_chain.py
│ ├── Readme.md
│ └── wis.py
├── helper
│ ├── a.py
│ └── __init__.py
└── Readme.md
The helper directory contains the library functions which will be used all over the code. How can I import the helper package from the scripts inside DynamicProgramming without adding it to the path?
Edit=>
I cannot move helper directory inside dynamicProgramming because there can be more than one directories using it.
You could use something like:
from ..helper import a
See python docs on packages.
If you run your code from project root folder, you are likely to succeed with import helper or import helper.a. If not, you would have to add current directory to PYTHONPATH:
$ export PYTHONPATH="."
better use project setup.py
Instead of playing with PYTHONPATH (what can be tricky business sometime), you shall create your project as python package.
You add setup.py into your project root, specify attributes of that package and build it from it.
setup.py can define multiple packages at once, but generally it is more often
using only one. For this purpose it would be better moving the helper package
into DynamicProgramming structure and import it from there.
Search for setup.py python packaging tutorials, it requires some study, but it will pay back.
Related
I'd like to figure out the cleanest and preferably self contained way to use my packages in scripts that are in a different directory to the package itself.
The example problem is as follows:
The modules in lib need to both be imported, and run as a script.
My project directory is as below and I'm having two issues:
in lib/api.py, I want to read in data_files/key.txt correctly when api.py is called or imported
in testing_script.py I want to import and use lib/get_data.py
I can't seem to find a clean way to do this, does this mean my project is structured in a non-pythonic way?
Thanks for the help.
my-project-git
├── LICENSE
├── README.md
├─── my_project
│ ├── data_files
│ │ ├── key.txt
│ │ ├── mappings.csv
│ ├── lib
│ │ ├── __init__.py
│ │ ├── api.py
│ │ └── get_data.py
│ └── test
│ ├── __init__.py
│ └── testing_script.py
├── requirements.txt
└── setup.py
As far as I know, there's isn't a pythonic way to structure your project.
This is what Kenneth Reitz recommended in 2013 and it's how I use it: https://www.kennethreitz.org/essays/repository-structure-and-python.
README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py
Inside sample (my_project in your case) you can separate into categories as you like. E.g. Utils (common functions), Database (read, write), View (user commands), etc. It depends on your project.
As for calling the modules at the same level, you should define them in the __init__ file of the top hierarchy module which is sample in this case.
For example:
__init__ in _my_project
from sample.core import a_Class
from sample.core import a_function
from sample.core import anything
then from /test/test_basic.py you do:
from sample import a_Class
# or import sample
a = a_Class() # use the class from core.py
# or a = sample.a_Class()
Take a look at the sample module repository: https://github.com/navdeep-G/samplemod
I'm trying to pip install a GitHub project locally, outside of site-packages so that I can modify it, etc.
I've added -e git+git#github.com:Starcross/django-starcross-gallery.git#egg=gallery to my requirements.txt which brings the relevant part of my project layout to look like this:
/home/mat/venv/proj/
└── src
└── gallery
├── admin.py
├── apps.py
├── build.sh
├── django_starcross_gallery.egg-info
│ ├── dependency_links.txt
│ ├── PKG-INFO
│ ├── requires.txt
│ ├── SOURCES.txt
│ └── top_level.txt
├── forms.py
├── __init__.py
├── LICENSE
├── MANIFEST.in
├── models.py
├── README.rst
├── settings.py
├── setup.py
├── signals.py
├── static
│ └── ...
├── templates
│ └── ...
├── tests
│ └── ...
├── tests.py
├── urls.py
└── views.py
As far as I can see the problem is that these .egg-link and .pth files like one level too deep:
lib/python3.6/site-packages/django-starcross-gallery.egg-link:
/home/mat/venv/proj/src/gallery
.
lib/python3.6/site-packages/easy-install.pth:
/home/mat/venv/proj/src/gallery
I can fix everything by either moving gallery a level deeper, or changing django-starcross-gallery.egg-link and easy-install.pth to point to src.
Is there a config parameter I can pass in requirements.txt to make this work properly? Or do I have to adjust the project layout to fit?
Since you want to modify it, why not just clone the repo. To make your interpreter able to find and use it, you have some options:
modify your sys.path, append path to the repo
create a symlink under your project directory that points to the repo
And in this way, you don't have to pip install every time you modify it.
As has been mentioned, the best way to do this is to clone the repo. This would go for most packages as pip may build extensions, and carry other actions during install aimed at using the module for production rather than editing the source.
To explain why I chose this structure, I wanted to be able to develop the package inside a Django project. As the Django docs say, the app should be placed in a separate directory, which enables setuptools to install the package correctly. There is no way I could find that would enable this to continue to work inside a project, hence the build script to move the files into a suitable directory and generate the package.
When I build or debug a particular file in my Python project (which imports a user defined package) I get an import error. How can I solve this problem?
test.py
def sum(a,b):
return a+b
test2.py
from test import sum
sum(3,4)
The above code will give an import error cannot import test.
Directory tree
├── graphs
│ ├── Dijkstra's\ Algorithm.py
│ ├── Floyd\ Warshall\ DP.py
│ ├── Kruskal's\ algorithm.py
│ ├── Prim's\ Algoritm.py
│ ├── __init__.py
│ └── graph.py
├── heap
│ ├── __init__.py
│ ├── heap.py
│ └── priority_queue.py
Trying to import in graphs;
from heap.heap import Heap
About the heap file, make sure that you are running on the project root folder.
If these test.py files are running on the same folder, try to add a __init__.py empty file on this folder.
The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later (deeper) on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialisation code for the package or set the __all__ variable, described later.
This is my first time trying to set up a vagrant environment or a python virtuelenv, so forgive me if I am missing something basic.
Right now, I ssh into my vagrant box and in the home directory I have placed my venv folder. I have run
source venv/bin/activate
From my home directory I move to /vagrant, and within here I have my project files laid out something like this:
├──project
├── LICENSE
│
├── project
│ │ ├── exceptions.py
│ │ ├── __init__.py
│ │ ├── resources
│ │ │ ├── base.py
│ │ │ ├── __init__.py
│ │ └── target
│ │ └── __init__.py
│ │ └── test.py
│ ├── README.md
My problem is I am unable to import my modules in different directories. For example, if I am in /vagrant/project/project/target/test.py and I attempt:
import project.exceptions
I will get the error
ImportError: No module named project.exceptions
If I am in the /vagrant/project/project directory and I run
import exceptions
that works fine.
I have read up on similar problems people have experienced on StackOverflow.
Based on this question: Can't import package from virtualenv I have checked that my sys.executable path is the same in both my python interpreter as well as when I run a script (home/vagrant/venv/bin/python)
Based on this question: Import error with virtualenv. I have run ~/venv/bin/python directly and attempted to import, but the import still fails.
Let me know if there is more information I can provide. Thank you.
You have two options:
You can install your project into the virtual environment, by writing a setup.py file and by calling python setup.py install. See the Python Packaging User Guide.
You can set the PYTHONPATH environment variable to point to your project, like this:
$ export PYTHONPATH=$PYTHONPATH:/vagrant/project
Say you want want to create a programming project that mixes C++ and Python. The Foo C++ project structure uses CMake, and a Python module is created by using Swig. The tree structure would look something like this:
├── CMakeLists.txt
├── FooConfig.cmake.in
├── FooConfigVersion.cmake.in
├── Makefile
├── README
├── foo
│ ├── CMakeLists.txt
│ ├── config.hpp.in
│ ├── foo.cpp
│ └── foo.hpp
└── swig
└── foo.i
Now you would like to make use of the Foo project within a Python project, say Bar:
├── AUTHORS.rst
├── CONTRIBUTING.rst
├── HISTORY.rst
├── LICENSE
├── MANIFEST.in
├── Makefile
├── README.rst
├── docs
│ ├── Makefile
│ ├── authors.rst
│ ├── conf.py
│ ├── contributing.rst
│ ├── history.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── make.bat
│ ├── readme.rst
│ └── usage.rst
├── bar
│ ├── __init__.py
│ └── bar.py
├── requirements.txt
├── setup.cfg
├── setup.py
├── tests
│ ├── __init__.py
│ └── test_bar.py
└── tox.ini
This structure was crated by using cookiecutter's pypackage template. A BoilerplatePP template is also available to generate a CMake C++ project using cookiecutter (no Swig part).
So now that I have the structure of both projects, and considering that the development will take place mainly in Python and the the project will be run in different systems, I need to address the following questions:
What's the best way to mix them? Should I collapse both root directories? Should I have the Foo C++ project as a directory of the Bar project or the other way around? I may be inclined to put the entire C++ structure shown above in a folder at the root level of the Python project, but I would like to know a priori any pitfalls as the CMake system is quite powerful and it may be convenient to do it the other way around.
In case I decide to put the Foo project as a directory within Bar, is the Python setuptools package as powerful as the CMake build system? I ask this because when I take a look at the Bar project, at the top level it seems there's only a bunch of scripts, but I don't know if this is the equivalent to CMake as I'm new to Python.
The Bar project outlined above has a single bar directory, but I assume that whenever this project expands, instead of having many other directories at the root level, other directories containing Python code will be placed within bar. Is this correct (in the Pythonic sense)?
I assume that a single egg will be produced from the entire project, so that it can be installed and run in many different python systems. Is the integration of the module created by the Foo project easy? I assume that this module will be created in a different directory than bar.
In order for the Python code within the bar directory, the module created by Swig has to be available, so I guess the most straightforward way to do this is to modify the environmental variable PYTHONPATH using the CMake system. Is this fine or is there a better way?
If the C++ application has no use outside the Python package that will contain it:
You can pretty safely place the C++ code within the python package that owns it. Have the "foo" directory within the "bar" directory within your example. This will make packaging the final Python module a bit easier.
If the C++ application is reusable:
I would definitely try to think of things in terms of "packages", where independent parts are self-contained. All independent parts live on the same level. If one part depends on another, you import from its corresponding "package" from the same level. This is how dependencies typically work.
I would NOT include one within the other, because one does not strictly belong to the other. What if you started a third project that needed "foo", but did not need "bar"?
I would place both "foo" and "bar" packages into the same "project" directory (and I would probably give each package it's own code repository so each package can be easily maintained and installed).