G'day,
Being a total python noob when it comes to packaging and module organisation..
Given the following (simplified) structure:
.
├── bin
│ └── fos.py
└── lib
└── drac.py
And the fact, that when installed, contents of lib folder will go somewhere into /usr/local/share/pyshared and contents of bin folder somewhere in /usr/bin, how do I persuade this whole thing to import my modules from ../lib when in VCS mode and work like it should, i.e. from modulename.drac import bla when installed, while keeping imports preferably the same?
Yes, I've read python docs on module organisation and structure, I just can't seem to wrap my head around some best practices. Asking for best practices on SO is stupid, hence this concrete example, which I have on a daily basis more or less.
Is this structure acceptable, if so, how do I organise the imports? If not, what would be the pythonic way to redo it?
Thanks!
I think you are bucking the idiom here. What you are describing is similiar to the old c ld_lib paradigm.
There is nothing wrong with a python project sourcing modules out of its own local file tree. Alternatively if your code is really that separate and your lib has a well defined API then you should package it separately and import/install it using ez_install, pip, or a setup.py
Generally if the code appears to be evolving together it best to just leave it together. Install it wherever you install your python code (opt..etc.) And symbolically link executables into /usr/local/bin
Related
It's been a while that I am struggling with imports in packages. When I develop a package, I read everywhere that it is preferable to use absolute imports in submodules of that package. I understand that and I like it more as well. But then I don't like and I also read that you shouldn't use sys.path.append('/path/to/package') to use your package in development...
So my question is, how do you develop such a package from zero, using directly absolute imports? At the moment I develop the package using relative imports, since then I am able to test the code I am writing before packaging and installing, then I change the imports once I have a release and build the package.
What is the correct way of doing such thing? In Pycharm for example you would mark the folder as 'source roor' and be able to work as if the package folder was in the path. Still I read that this is not the proper way... what am I missing? How do you develop a package while testing its code?
Your mileage may vary but this is what I usually do:
Within a package (foo), absolute (import foo.bar) or relative (import .bar) doesn't matter to me as long as it works. Sometimes, I prefer relative especially when the project is large and one day I might decide to move a number of source files into a subdirectory.
How do I test? My $PYTHONPATH usually has . in it, and my directory hierarchy is like this:
/path/to/foo_project
/setup.py
/foo
/__init__.py
/bar.py
/test
/test1.py
/test2.py
then the script in foo_project/test/test1.py will be like what you normally use the package, using import foo.bar. And when I test my code, I will be in the directory foo_project and run python test/test1.py. Since I have . in my $PYTHONPATH, it will find the directory foo and use it as a package.
So I have been fiddling around as well as conducting some serious work with Python for quite some time. Though, I still have some issues with it every once in a while.
I find it to be the most comfortable using PyCharm CE when working with Python. The typical scenario is that I just create a new virtualenv, launch PyCharm and open up my virtualenv there. And from there on out, it's like on auto-pilot, PyCharm handles all the dirty work related to which site-packages and Python runtime to use.
I always like to keep my virtualenvs clean and organized, so I often find myself semantically organizing my source code into submodules/subfolders. So whenever I want to import some code, or class, or whatever from another folder I just import it.
Imagine I have the following structure in my virtualenv:
├── bin
├── include
├── lib
└── src
├── foo.py
├── important_scripts
├── some_script.py
└── some_other_script.py
└── pip-selfcheck.json
Now, somewhere in foo.py, I want to use a function named A() that is implemented in some_script.py. The obvious way would be to add a simple line to foo.py - something like from some_script import A. Doing such works perfectly when I run and debug my code (foo.py in this case) from PyCharm.
As opposed to the the typical scenario I have described above, I wanted to do the same from the Terminal.app. So I fire up the Terminal, cd to my virtualenv and activate it. Then, what I do is, using the Python executable that is under the bin folder in my virtualenv, I try to run foo.py (At least this is what I think is the equivalent of right-clicking and running foo.py from the PyCharm window). Unfortunately, I get the error ModuleNotFoundError: No module named 'some_script'.
I think I am missing a simple detail or something. Because like I said, it works like magic when run from PyCharm.
Anyways, any advice or help will be highly appreciated. Thanks in advance.
Thanks for all the responses and references to possible solutions. While researching online, I have come across various instances of more or less the same problem that people were having while importing modules and packages. So, this is how I have just resolved it:
Under the important_scripts directory, I have included a file named __init__.py. This basically just tells the Python interpreter that this is indeed a Python package, rather than an ordinary subdirectory.
In this __init__.py, I have added in the line
from important_scripts.some_script import A
Then in the script, from which I will be importing the function A, that is, foo.py I have included the following lines:
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(__file__)))
which basically appends the virtualenv into the site-packages.
I have two python projects, one includes useful packages for file manipulation and such. They are usefull because they can be reused in any other kind of project. The second is one of these projects, requiring the use of my useful packages. Here is my projects' file structure:
Python-projects/
Useful/
package_parsing/
package_file_manipulation/
package_blabla/
Super-Application/
pkg1/
__init__.py
module1.py
module2.py
pkg2/
setup.py
MANIFEST.in
README.rst
First of, I would like to use the Useful/package_parsing package in my module Super-Application/pkg1/module1.py. Is there a more convenient way to do it other than copying the package_parsing in Super-Application project?
Depending in the first answer, that is if there is a way to link a module from a different project, how could I include such external module in a release package of my Super-Application project? I am not sure that making use of install_requires in the setup.py will do.
My main idea here is not to duplicate the Useful/package_parsing package in all of my other development projects, especially when I would like to do modifications to this useful package. I wouldn't like to update all the outdated copies in each project.
=============
EDIT 1
It appears the first part of my question cna be dealt with appending the path:
import sys
sys.path.insert(0, path/to/Useful/package_parsing)
Moreover I can simply check the available paths using:
for p in sys.path:
print p
Now for the second part, how could I include such external module in a release package, possibly using the setup.py installation file?
A project written in python with some C extensions (not using SWIG etc). I am trying to figure out how to structure my project so that:
imports work. Importing the shared objects
I don't need to change PYTHONPATH (tried figuring it out and failed).
In the future, distributing the project package will be easiest.
Current structure is, as suggested here:
Project\
docs\ # mainly documentation, right?
bin\ # empty
setup.py # setup for the project, as suggested in the above link
project\
__init__.py
module.py
tests\
bla_test.py
C\ # the package of C files
file.c
file.so
other_c_stuff.c
header.h
setup.py # setup to compile the C files and create .so files
build\ # contaisn a bunch of (hopefully) irrelevant stuf
It worked from PyDev but not from shell. Ideal answer would adress the following:
A suggested structure for the project.
How an import would be perfomed (from, say, the by modules in tests).
Should (can) I keep all the C files in a separate library?
The build of the C files is done in which of the setup.py files (should I post them here?)
Is it possible to automatically build when necessary? How?
I tried relative imports - they don't work for me for some reason.
I saw the accepted answet to this question. He says - do whatever. But I can't get the imports to work. I read this answer, but have no clue what's all the stuff he has (and I don't have). The accepted answer doesn't help me because, again, the imports fail. This blog posts gives good advice but, again, the imports!
I don't want to go into detail for a general answer, since you linked to good ones already.
Some structure, which should work for you, could look like:
Project\
build\ # directory used by setup.py
docs\ # mainly documentation, right?
setup.py # setup for the project, as suggested in the above link
project\
__init__.py
module.py
c_package\
__init__.py
file.c
file.so
other_c_stuff.c
header.h
tests\
__init__.py
test_bla.py
So within the project package and its subpackages you can use relative imports, if you build the C Extensions inplace
python setup.py build_ext --inplace
or create a setup.cfg containing
[build_ext]
inplace=True
but only use this for development and don't release it, since installation will fail.
A build automation is possible, but I don't know of any other than calling setup.py directly, whenever the C sources have changed.
Currently, I have a Python project I'm working on with a directory structure like this:
tests/
corpus/
__init__.py
tests.py
monkey/
corpus/
corpus.py
setup.py
and I want tests.py (in tests/corpus) to import corpus.py (in monkey/corpus).
I've seen many solutions that involve using relative imports and sys.path, but I've also seen people directly import using (for instance)
import monkey.corpus
How can I set up my code to be able to import anything in the root folder like this? I've seen glimpses of ideas that it might be possible through configuring setup.py. Is this true?
Thanks a bunch. My apologies for diluting this wonderful site with one more relative import-esque question. :)
Sincerely,
linuxuser
After doing some research, I found that I needed to add an empty __init__.py in the inner corpus directory and put a line in my .bashrc appending it to PYTHONPATH.
This is what my .bashrc looks like now:
...
export PYTHONPATH=$PYTHONPATH:/home/username/monkey/corpus
...
At first, it seemed unusual to have to append my .bashrc to access a library, but from what I've heard it is the typical way to edit your environment and therefore is a proper way to provide access to Python libraries.
A great resource to find info about PYTHONPATH is this blog post: http://www.stereoplex.com/blog/understanding-imports-and-pythonpath