We Keep Coding

python load modules of same package structure found in different folders

I have the following folder structure:
data_plugin_main/
/common
__init__.py
utils.py
/data
__init__.py
commmon_dal.py
plugin1
/data
__init__.py
data_plugin_dal.py
plugin2
/data
__init__.py
another_plugin_dal.py
I need to maintain this structure, but be able to write code that either imports or can reference:
data.common_dal
data.data_plugin_dal
data.another_plugin_dal
It seems like python will only import/load modules in the first data folder it encounters. so in above case, since there is in __init__.py under data_plugin_main/data/, common_dal.py is loaded. however, even if I add the other paths to the PYTHONPATH, data_plugin_dal and another_plugin_dal are not able to be imported or referenced.
Does anyone know how to solve this problem?
Thanks

If I’m understanding correctly, you may want to try Pydev environment installed freely in Eclipse.
Each module is also the directory name with a parent module that fathers them all, so in pydev if plugin_2 source needs to reference common_dal.py the import looks like from data_plugin_main import data.common_dal
So in the project explorer you would create a parent module That acts as a super __init__ and then you can create child modules as you wish. It’s kinda Javaishly inspired.

I was able to solve this using namespace-packaging: https://packaging.python.org/guides/packaging-namespace-packages/

Understand package imports

I have started learning python very recently. I do have a typescript and C# background. I was trying to understand the packages and imports statements. Following is the very basic folder structure I have created and I have few questions on this
app/
|-> pkg1/
|-> fib.py
|-> __init__.py
|- >pkg2/
|-> mul.py
|-> __init__.py
|-> app.py
ext/
|-> ext.py
|-> __init__.py
fib.py
------
from pkg2.mul import mul
// from ..pkg2.mul import mul -> Error: attempted relative import beyond top-level package
// but this should work as per docs https://docs.python.org/3/tutorial/modules.html#intra-package-references
// can someone tell why do I get this error?
def fib(n):
return mul(n) * 2
mul.py
------
def mul(n):
return n * 10
app.py
------
from pkg1.fib import fib
print(fib(1))
I have read Python treats any file as entry file.
Does python even know app.py is the entry point? No I guess.
Why does it work when I have given pkg2.mul in fib.py
Why does it work when I have given pkg1.fib in app.py
What if I want to include ext.py inside mul.py and app.py
As per folder structure, if app.py imports pkg1.fib (relative to app.py) is valid then fib.py must import mul in different way correct? why it is working for pkg2.mul? Is it actually Valid? pkg2/ is not relative to fib.py.
Am I missing something? In typescript we use relative imports and C# we use namespaces and include files but here I am pretty confused. Any help is greatly appreciated.

Your relative import attempt did not work because pkg1 and pkg2 are separate packages. Relative imports only work within one package.
Does python even know app.py is the entry point?
No. It's just some .py file sitting around in a directory.
Why does it work when I have given pkg2.mul in fib.py
Coincidence. You are in the app/ directory, and the first element of sys.path is the working directory. It likely won't work if you cd to some other directory.
Why does it work when I have given pkg1.fib in app.py
Same as above.
What if I want to include ext.py inside mul.py and app.py
You have to package ext and include it as a dependency of app.
As per folder structure, if app.py imports pkg1.fib (relative to app.py) is valid then fib.py must import mul in different way correct? Why it is working for pkg2.mul? Is it actually valid? pkg2/ is not relative to fib.py.
It's not importing relative to app.py. They are both importing from cwd (check the first element of sys.path displayed by using python -m site)

The basic issue is that your app is not package (because it is missing an __init__.py file) which means when you go two levels up to the app directory you aren't in a package any more, and thus the error stating that you are attempting to import beyond the top-level package.
See this section on intra-package references for more info on how you could restructure your package to allow what you are attempting to do. You also check out this section explaining the module search path for a little more context.

PyCharm auto-import Fails To Import Properly

I have an app structured like so:
api/
|_app/
|_ __init__.py
|_conf/
|_resources/
|_ __init__.py
|_controller.py
|_dao/
|_ __init__.py
|_thing_dao.py
|_etc...
I want to use the function get_thing_by_id in thing_dao.py inside of controller.py. In PyCharm I start to type get_thing_by_id and it prompts me to auto-import the function. The problem is it simply does from thing_dao import get_thing_by_id but when I attempt to run the app (Flask) I get ImportError: No module named 'thing_dao'. What I end up having to do is a relative import or start at the module level from app.dao.thing_dao import get_thing_by_id.
I suspect this issue is related to my project structure and less of a PyCharm bug. Is there a way I could structure my project to better hint to PyCharm how to do imports? I've looked through the configuration option for auto-imports and PyCharm and they're quite slim so I suspect that I'm doing something wrong.

I discovered I had my directories marked as a "Source Root". Unmarking them made the imports work properly.

python import path: packages with the same name in different folders

I am developing several Python projects for several customers at the same time. A simplified version of my project folder structure looks something like this:
/path/
to/
projects/
cust1/
proj1/
pack1/
__init__.py
mod1.py
proj2/
pack2/
__init__.py
mod2.py
cust2/
proj3/
pack3/
__init__.py
mod3.py
When I for example want to use functionality from proj1, I extend sys.path by /path/to/projects/cust1/proj1 (e.g. by setting PYTHONPATH or adding a .pth file to the site_packages folder or even modifying sys.path directly) and then import the module like this:
>>> from pack1.mod1 import something
As I work on more projects, it happens that different projects have identical package names:
/path/
to/
projects/
cust3/
proj4/
pack1/ <-- same package name as in cust1/proj1 above
__init__.py
mod4.py
If I now simply extend sys.path by /path/to/projects/cust3/proj4, I still can import from proj1, but not from proj4:
>>> from pack1.mod1 import something
>>> from pack1.mod4 import something_else
ImportError: No module named mod4
I think the reason why the second import fails is that Python only searches the first folder in sys.path where it finds a pack1 package and gives up if it does not find the mod4 module in there. I've asked about this in an earlier question, see import python modules with the same name, but the internal details are still unclear to me.
Anyway, the obvious solution is to add another layer of namespace qualification by turning project directories into super packages: Add __init__.py files to each proj* folder and remove these folders from the lines by which sys.path is extended, e.g.
$ export PYTHONPATH=/path/to/projects/cust1:/path/to/projects/cust3
$ touch /path/to/projects/cust1/proj1/__init__.py
$ touch /path/to/projects/cust3/proj4/__init__.py
$ python
>>> from proj1.pack1.mod1 import something
>>> from proj4.pack1.mod4 import something_else
Now I am running into a situation where different projects for different customers have the same name, e.g.
/path/
to/
projects/
cust3/
proj1/ <-- same project name as for cust1 above
__init__.py
pack4/
__init__.py
mod4.py
Trying to import from mod4 does not work anymore for the same reason as before:
>>> from proj1.pack4.mod4 import yet_something_else
ImportError: No module named pack4.mod4
Following the same approach that solved this problem before, I would add yet another package / namespace layer and turn customer folders into super super packages.
However, this clashes with other requirements I have to my project folder structure, e.g.
Development / Release structure to maintain several code lines
other kinds of source code like e.g. JavaScript, SQL, etc.
other files than source files like e.g. documents or data.
A less simplified, more real-world depiction of some project folders looks like this:
/path/
to/
projects/
cust1/
proj1/
Development/
code/
javascript/
...
python/
pack1/
__init__.py
mod1.py
doc/
...
Release/
...
proj2/
Development/
code/
python/
pack2/
__init__.py
mod2.py
I don't see how I can satisfy the requirements the python interpreter has to a folder structure and the ones that I have at the same time. Maybe I could create an extra folder structure with some symbolic links and use that in sys.path, but looking at the effort I'm already making, I have a feeling that there is something fundamentally wrong with my entire approach. On a sidenote, I also have a hard time believing that python really restricts me in my choice of source code folder names as it seems to do in the case depicted.
How can I set up my project folders and sys.path so I can import from all projects in a consistent manner if there are project and packages with identical names ?

This is the solution to my problem, albeit it might not be obvious at first.
In my projects, I have now introduced a convention of one namespace per customer. In every customer folder (cust1, cust2, etc.), there is an __init__.py file with this code:
import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
All the other __init__.py files in my packages are empty (mostly because I haven't had the time yet to find out what else to do with them).
As explained here, extend_path makes sure Python is aware there is more than one sub-package within a package, physically located elsewhere and - from what I understand - the interpreter then does not stop searching after it fails to find a module under the first package path it encounters in sys.path, but searches all paths in __path__.
I can now access all code in a consistent manner criss-cross between all projects, e.g.
from cust1.proj1.pack1.mod1 import something
from cust3.proj4.pack1.mod4 import something_else
from cust3.proj1.pack4.mod4 import yet_something_else
On a downside, I had to create an even deeper project folder structure:
/path/
to/
projects/
cust1/
proj1/
Development/
code/
python/
cust1/
__init__.py <--- contains code as described above
proj1/
__init__.py <--- empty
pack1/
__init__.py <--- empty
mod1.py
but that seems very acceptable to me, especially considering how little effort I need to make to maintain this convention. sys.path is extended by /path/to/projects/cust1/proj1/Development/code/python for this project.
On a sidenote, I noticed that of all the __init__.py files for the same customer, the one in the path that appears first in sys.path is executed, no matter from which project I import something.

You should be using the excellent virtualenv and virtualenvwrapper tools.

What happens if you accidentally import code from one customer/project in another and don't notice? When you deliver it will almost certainly fail. I would adopt a convention of having PYTHONPATH set up for one project at a time, and not try to have everything you've ever written be importable at once.
You can use a wrapper script per-project to set PYTHONPATH and start python, or use scripts to switch environments when you switch projects.
Of course some projects well have dependencies on other projects (those libraries you mentioned), but if you intend for the customer to be able to import several projects at once then you have to arrange for the names to not clash. You can only have this problem when you have multiple projects on the PYTHONPATH that aren't supposed to be used together.

Problem with python packages and tests

I have a directory struture like that:
project
| __init__.py
| project.py
| src/
| __init__.py
| class_one.py
| class_two.py
| test/
| __init__.py
| test_class_one.py
Which project.py just instantiate ClassOne and run it.
My problem is in the tests, I don't know how to import src classes. I've tried importing these ways and I got nothing:
from project.src.class_one import ClassOne
and
from ..src.class_one import ClassOne
What am I doing wrong? Is there a better directory structure?
----- EDIT -----
I changed my dir structure, it's like this now:
Project/
| project.py
| project/
| __init__.py
| class_one.py
| class_two.py
| test/
| __init__.py
| test_class_one.py
And in the test_class_one.py file I'm trying to import this way:
from project.class_one import ClassOne
And it still doesn't work. I'm not using the executable project.py inside a bin dir exactly because I can't import a package from a higher level dir. :(
Thanks. =D

It all depends on your python path. The easiest way to achieve what you're wanting to do here is to set the PYTHONPATH environment variable to where the "project" directory resides. For example, if your source is living in:
/Users/crocidb/src/project/
I would:
export PYTHONPATH=/Users/crocidb/src
and then in the test_one.py I could:
import project.src.class_one
Actually I would probably do it this way:
export PYTHONPATH=/Users/crocidb/src/project
and then this in test_one.py:
import src.class_one
but that's just my preference and really depends on what the rest of your hierarchy is. Also note that if you already have something in PYTHONPATH you'll want to add to it:
export PYTHONPATH=/Users/crocidb/src/project:$PYTHONPATH
or in the other order if you want your project path to be searched last.
This all applies to windows, too, except you would need to use windows' syntax to set the environment variables.

From Jp Calderone's excellent blog post:
Do:
name the directory something related to your project. For example, if your
project is named "Twisted", name the
top-level directory for its source
files Twisted. When you do releases,
you should include a version number
suffix: Twisted-2.5.
create a directory Twisted/bin and put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects. (Slight wrinkle:
since on Windows, the interpreter is
selected by the file extension, your
Windows users actually do want the .py
extension. So, when you package for
Windows, you may want to add it.
Unfortunately there's no easy
distutils trick that I know of to
automate this process. Considering
that on POSIX the .py extension is a
only a wart, whereas on Windows the
lack is an actual bug, if your
userbase includes Windows users, you
may want to opt to just have the .py
extension everywhere.)
If your project is expressable as a single Python source file, then put it
into the directory and name it
something related to your project. For
example, Twisted/twisted.py. If you
need multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and place
your source files in it. For example,
Twisted/twisted/internet.py.
put your unit tests in a sub-package of your package (note - this means
that the single Python source file
option above was a trick - you always
need at least one other file for your
unit tests). For example,
Twisted/twisted/test/. Of course, make
it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory called src or lib. This makes it hard
to run without installing.
put your tests outside of your Python package. This makes it hard to
run the tests against an installed
version.
create a package that only has a __init__.py and then put all your code into __init__.py. Just make a module
instead of a package, it's simpler.
try to come up with magical hacks to make Python able to import your module
or package without having the user add
the directory containing it to their
import path (either via PYTHONPATH or
some other mechanism). You will not
correctly handle all cases and users
will get angry at you when your
software doesn't work in their
environment.