We Keep Coding

Implicitly use namespace in imported modules in Python

I'm trying to make a library out of a Python project I don't own.
The project has the following directory layout:
.
├── MANIFEST.in
├── pyproject.toml
└── src
   ├── all.py
   ├── the.py
   └── sources.py
In pyproject.toml I have:
[tool.setuptools]
packages = ["mypkg"]
[tool.setuptools.package-dir]
mypkg = "src"
The problem I'm facing is that when I build and install this package I can't use it because the author is importing stuff without mypkg prefix in the various source files.
F.ex. in all.py
from the import SomeThing
Since I don't own the package I can't go modify all the sources but I still want to be able to build a library from it by just adding MANIFEST.in and pyproject.toml.
Is it possible to somehow instruct setuptools to build a package that won't litter site-packages with all the sources while still allowing them to be imported without the mypkg prefix?

It isn't possible without adding a custom import hook with the package. The hook takes the form of a module that is shipped with the package, and it must be imported before usage from your module (e.g. in src/all.py)
src/mypkgimp.py
import sys
import importlib
class MyPkgLoader(importlib.abc.Loader):
def find_spec(self, name, path=None, target=None):
# update the list with modules that should be treated special
if name in ['sources', 'the']:
return importlib.util.spec_from_loader(name, self)
return None
def create_module(self, spec):
# Uncomment if "normal" imports should have precedence
# try:
# sys.meta_path = [x for x in sys.meta_path[:] if x is not self]
# return importlib.import_module(spec.name)
# except ImportError:
# pass
# finally:
# sys.meta_path = [self] + sys.meta_path
# Otherwise, this will unconditionally shadow normal imports
module = importlib.import_module('.' + spec.name, 'mypkg')
# Final step: inject the module to the "shortened" name
sys.modules[spec.name] = module
return module
def exec_module(self, module):
pass
if not hasattr(sys, 'frozen'):
sys.meta_path = [MyPkgLoader()] + sys.meta_path
Yes, the above uses different methods described by the thread I have linked previously, as importlib have deprecated those methods in Python 3.10, refer to documentation for details.
Anyway, for the demo, put some dummy classes in the modules:
src/the.py
class SomeThing: ...
src/sources.py
class Source: ...
Now, modify src/all.py to have the following:
import mypkg.mypkgimp
from the import SomeThing
Example usage:
>>> from sources import Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sources'
>>> from mypkg import all
>>> all.SomeThing
<class 'mypkg.the.SomeThing'>
>>> from sources import Source
>>> Source
<class 'mypkg.sources.Source'>
>>> from sources import Error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Error' from 'mypkg.sources' (/tmp/mypkg/src/sources.py)
Note how the import initially didn't work, but after mypkg.all got imported, the sources import now works globally. Hence care may be needed to not shadow "real" imports and I have provided the example to import using the "default"[*] import mechanism.
If you want the module names to look different (i.e. without the mypkg. prefix), that will be a separate question, as code typically don't check for their own module name for functionality (and never mind that this actually shows how the namespace is implicitly used - changing the actual name is more akin to a module relocation, yes this can be done, but a bit more complicated and this answer is long enough as it is).
[*] "default" as in not including behaviors introduced by this custom import hook - other import hooks may do their own other weird shenanigans.

Error when importing python module from folders

I have a following directory structure:
source
source_1.py
__init__.py
source1.py has class Source defined
source1.py
class Source(object):
pass
I am able to import using this
>>> from source.source1 import Source
>>> Source
<class 'source.source1.Source'>
However when trying to import using the below method it fails.
>>> from source import *
>>> source1.Source
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'source1' is not defined
Please let me know how can we use the 2nd import ?

For importing from a package (unlike importing from a module) you need to specify what * means. To do that, in __init__.py add a line like this:
__all__ = ["source1"]
See the Python documentation for Importing * From a Package.

ipython's strange behavior with directories named "code"

I can't import anything from directories named "code".
I have the following directory structure:
In [1]: ls
code/ data/ pickles/
code_copy/ documentation/ submissions/
code_copy is an exact copy of code
In [2]: ls code_copy/
santander.py
__init__.py santander.pyc
In [3]: ls code
santander.py
__init__.py santander.pyc
However, the behavior of these two directories is different when I try to import something from them. For example, I can't import santander.py from code
In [4]: from code.
code.InteractiveConsole code.compile_command
code.InteractiveInterpreter code.interact
In [5]: from code import santander
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-20-e36a94117eb8> in <module>()
----> 1 from code import santander
ImportError: cannot import name santander
while I'm able to do it from code_copy
In [6]: from code_copy.
code_copy.santander
In [7]: from code_copy import santander
In [8]:
Is this ipython's behavior normal? Is "code" a especial name? How can this be fixed?

Importing modules from different directories

I have a problem importing a module:
It is under this directory ./dao and the code that calls it is here ./core. Schematically represented as:
rnaspace/
__init__.py
core/
__init__.py
logger.py
dao/
__init__.py
storage_configuration_reader.py
This is the error message:
Traceback (most recent call last): File "logger.py", line 21, in <module>
from rnaspace.dao.storage_configuration_reader import storage_configuration_reader ImportError: No module named rnaspace.dao.storage_configuration_reader
This file it is there /rnaspace/dao/storage_configuration_reader.py and in the same folder the __init__.py file as follows:
""" Package dao
Gathers files that access to the plateform data
"""
If I understood well this question, it should work. I think that the problem is that one is not the subdirectory of the other (or that the path is not exaclly that one), there is a way to go around it? Or need I to apply the solution to this question?
EDIT
The __init__.py file of the rnaspace folder:
import rnaspace.dao.storage_configuration_reader as scr
def update_conf(conf_path, predictors_conf_dir):
scr.update_conf(conf_path, predictors_conf_dir)

from rnaspace.dao.storage_configuration_reader import storage_configuration_reader
That is wrong because there is no "storage_configuration_reader" directory in "dao" directory
This is how it should be:
from rnaspace.dao import storage_configuration_reader
EDIT:
or this way:
import rnaspace.dao.storage_configuration_reader

I finally found the solution in another question, it is using the module imp.
I just needed to add the name of the module, and the absolute path where it was:
imp.load_source("storage_configuration_reader","./rnaspace/dao/storage_configuration_reader.py")

How to import a module in Python with importlib.import_module

I'm trying to use importlib.import_module in Python 2.7.2 and run into the strange error.
Consider the following dir structure:
a
|
+ - __init__.py
- b
|
+ - __init__.py
- c.py
a/b/__init__.py has the following code:
import importlib
mod = importlib.import_module("c")
(In real code "c"has a name.)
Trying to import a.b, yields the following error:
>>> import a.b
Traceback (most recent call last):
File "", line 1, in
File "a/b/__init__.py", line 3, in
mod = importlib.import_module("c")
File "/opt/Python-2.7.2/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named c
What am I missing?
Thanks!

For relative imports you have to:
a) use relative name
b) provide anchor explicitly
importlib.import_module('.c', 'a.b')
Of course, you could also just do absolute import instead:
importlib.import_module('a.b.c')

I think it's better to use importlib.import_module('.c', __name__) since you don't need to know about a and b.
I'm also wondering that, if you have to use importlib.import_module('a.b.c'), why not just use import a.b.c?

And don't forget to create a __init__.py with each folder/subfolder (even if they are empty)