We Keep Coding

Python: A problem with the package import from ... import * using __all__ and __init__

I have the following Python package with 2 moludes:
-pack1
|-__init__
|-mod1.py
|-mod2.py
-import_test.py
with the code:
# in mod1.py
a = 1
and
# in mod2.py
from mod1 import a
b = 2
and the __init__ code:
# in __init__.py
__all__ = ['mod1', 'mod2']
Next, I am trying to import the package:
# in import_test.py
from pack1 import *
But I get an error:
ModuleNotFoundError: No module named 'mod1'
If I remove the dependency "from mod1 import a" in mod2.py, the import goes correctly. But that dependency makes the import incorrect with that exception "ModuleNotFoundError".
???

The issue here is that from mod2 perspective the first level in which it will search for a module is in the path from which you are importing it (here I am assuming that pack1 is not in your PYTHONPATH and that you are importing it from the same directory where pack1 is contained).
This means that if pack1 is in the directory /dir/to/pack1 and you do:
from mod1 import a
Python will look for mod1 in the same directory as pack1, i.e., /dir/to/pack1.
To solve your issue it is enough to do either:
from pack1.mod1 import a
or in Python 3.5+
from .mod1 import a
As a side note, unless this is a must for you, I do not recommend designing your package to be used as from pack import *, even if __all__ exists to give you better control of your public API.

How to get parent module/package functions in namespace of child package/module in Python?

I have an extremely complicated module and I want to break out the subpackages into individual packages. My first attempt will be for the "utilities" submodule. I want to be able to import everything from the parent package example_utils.py into example_module.utils but I also want example_module.utils to have it's own functions as well.
In the end I want to be able to do the following:
import example_module as em
x = 10
y1 = em.utils.f_parent1(x)
y2 = em.utils.f_child1(x)
# and do this
from example_module.utils import f_parent1, f_child1
# and use the parent module as a standalone
from example_utils import f_parent1, f_parent2
How can I structure my child module example_module to have this functionality?**
Module utilities saved as separate standalone module example_utils.py
def f_parent1(x):
return x
def f_parent2(x,y):
return x+y
This module will be installed in my environment:
pip install path/to/example_module
Larger module (example_module) using example_utils as a dependency
# Directory structure for larger Module
example_module
|++++| __init__.py
|++++| utils
|++++| ++++ | __init__.py
|++++| ++++ | utils.py
Contents of |++++| ++++ | __init__.py
from .utils import *
Contents of |++++| ++++ | utils.py
from example_utils import *
def f_child1(x):
return x**2
Contents of |++++| __init__.py
__version__= "0.1"
__developmental__ = True
# Utilities
from .utils import utils
# =======
# Direct Exports
# =======
_submodules = ["utils"]
__all__ = sorted(__all__)
Apologies in advance if namespace is not the correct term. I get confused with namespace, scope, etc.

With suggestions from #r-ook I found that I could use getattr to grab the function by string name from the parent module. After that, I could add the function into the namespace?scope? of the child module.
example.py
from example_utils import as emu
functions_from_parent = ["f_parent1", "f_parent2"]
__all__ = {"f_child1"}
for function_name in functions_from_parent:
globals()[function_name] = getattr(emu, function_name)
__all__.add(function_name)
__all__ = sorted(__all__)
def f_child1(x):
return x**2

Import a whole folder of python files

I am making a bot in python 3 and wish it to be easily expanded so I have a central file and then one for each command. I wish to know if there is a way to import a sub-directory full of modules without importing each separately. For example:
example
├── commands
│   ├── bar.py
│   └── foo.py
└── main.py
And the code in main.pywould be something like:
import /commands/*
Thanks :D
Solution:
Import each separately with:
from commands import foo, bar
from commands import * Does not work.

If you're using python3, the importlib module can be used to dynamically import modules. On python2.x, there is the __import__ function but I'm not very familiar with the semantics. As a quick example,
I have 2 files in the current directory
# a.py
name = "a"
and
# b.py
name = "b"
In the same directory, I have this
import glob
import importlib
for f in glob.iglob("*.py"):
if f.endswith("load.py"):
continue
mod_name = f.split(".")[0]
print ("importing {}".format(mod_name))
mod = importlib.import_module(mod_name, "")
print ("Imported {}. Name is {}".format(mod, mod.name))
This will print
importing b Imported <module 'b' from '/tmp/x/b.py'>.
Name is b
importing a Imported <module 'a' from '/tmp/x/a.py'>.
Name is a

Import each separately with:
from commands import bar and
from commands import foo
from commands import * Does not work.

Python importing multiple scripts

I've been working on a python script that will require multiple libraries to be imported.
At the moment my directory structure is
program/
main.py
libs/
__init__.py
dbconnect.py
library01.py
library02.py
library03.py
My dbconnect.py which has the following contents
import psycopg2
class dbconnect:
def __init__(self):
self.var_parameters = "host='localhost' dbname='devdb' user='temp' password='temp'"
def pgsql(self):
try:
var_pgsqlConn = psycopg2.connect(self.var_parameters)
except:
print("connection failed")
return var_pgsqlConn
I am able to import and use this in my main.py using
from libs.dbconnect import dbconnect
class_dbconnect = dbconnect()
var_pgsqlConn = class_dbconnect.pgsql()
This works as expected however I am trying to import all of the library scripts each which have similar contents to bellow
def library01():
print("empty for now but this is library 01")
I have added to my __init__.py script
__all__ = ["library01", "library02"]
Then in my main.py I tried to import and use them as bellow
from libs import *
library01()
I am getting the following error
TypeError: 'module' object is not callable

I'll suppose content in your library0x.py are different (the functions/class have different names)
The best way is to import all your subfiles content in the __init__.py
# __init__.py
from .dbconnect import *
from .library01 import *
from .library02 import *
from .library03 import *
Then you can use the following :
from libs import library01, library02
If you want to restrict for some reasons importation with the wildcard (*) in your library0x.py files, you can define a __all__ variable containing all the names of the function you will import with the wildcard :
# library01.py
__all__ = ["library01"]
def a_local_function():
print "Local !"
def library01():
print "My library function"
Then, by doing from .library01 import *, only the function library01 will be import.
EDIT: Maybe i missunderstand the question : here are some ways to import the function library01 in the file library01.py :
# Example 1:
from libs.library01 import library01
library01()
# Example 2:
import libs.library01
libs.library01.library01()
# Example 3:
import libs.library01 as library01
library01.library01()

In your case library01 is a module which contains a function named library01. You import the library01 module and try to call it as a function. That's the problem. You should call the function like this:
library01.library01()

How to import classes defined in __init__.py

I am trying to organize some modules for my own use. I have something like this:
lib/
__init__.py
settings.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
In lib/__init__.py, I want to define some classes to be used if I import lib. However, I can't seem to figure it out without separating the classes into files, and import them in__init__.py.
Rather than say:
lib/
__init__.py
settings.py
helperclass.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
from lib.settings import Values
from lib.helperclass import Helper
I want something like this:
lib/
__init__.py #Helper defined in this file
settings.py
foo/
__init__.py
someobject.py
bar/
__init__.py
somethingelse.py
from lib.settings import Values
from lib import Helper
Is it possible, or do I have to separate the class into another file?
EDIT
OK, if I import lib from another script, I can access the Helper class. How can I access the Helper class from settings.py?
The example here describes Intra-Package References. I quote "submodules often need to refer to each other". In my case, the lib.settings.py needs the Helper and lib.foo.someobject need access to Helper, so where should I define the Helper class?

'lib/'s parent directory must be in sys.path.
Your 'lib/__init__.py' might look like this:
from . import settings # or just 'import settings' on old Python versions
class Helper(object):
pass
Then the following example should work:
from lib.settings import Values
from lib import Helper
Answer to the edited version of the question:
__init__.py defines how your package looks from outside. If you need to use Helper in settings.py then define Helper in a different file e.g., 'lib/helper.py'.
.
| `-- import_submodule.py
`-- lib
|-- __init__.py
|-- foo
| |-- __init__.py
| `-- someobject.py
|-- helper.py
`-- settings.py
2 directories, 6 files
The command:
$ python import_submodule.py
Output:
settings
helper
Helper in lib.settings
someobject
Helper in lib.foo.someobject
# ./import_submodule.py
import fnmatch, os
from lib.settings import Values
from lib import Helper
print
for root, dirs, files in os.walk('.'):
for f in fnmatch.filter(files, '*.py'):
print "# %s/%s" % (os.path.basename(root), f)
print open(os.path.join(root, f)).read()
print
# lib/helper.py
print 'helper'
class Helper(object):
def __init__(self, module_name):
print "Helper in", module_name
# lib/settings.py
print "settings"
import helper
class Values(object):
pass
helper.Helper(__name__)
# lib/__init__.py
#from __future__ import absolute_import
import settings, foo.someobject, helper
Helper = helper.Helper
# foo/someobject.py
print "someobject"
from .. import helper
helper.Helper(__name__)
# foo/__init__.py
import someobject

If lib/__init__.py defines the Helper class then in settings.py you can use:
from . import Helper
This works because . is the current directory, and acts as a synonym for the lib package from the point of view of the settings module. Note that it is not necessary to export Helper via __all__.
(Confirmed with python 2.7.10, running on Windows.)

You just put them in __init__.py.
So with test/classes.py being:
class A(object): pass
class B(object): pass
... and test/__init__.py being:
from classes import *
class Helper(object): pass
You can import test and have access to A, B and Helper
>>> import test
>>> test.A
<class 'test.classes.A'>
>>> test.B
<class 'test.classes.B'>
>>> test.Helper
<class 'test.Helper'>

Add something like this to lib/__init__.py
from .helperclass import Helper
now you can import it directly:
from lib import Helper

Edit, since i misunderstood the question:
Just put the Helper class in __init__.py. Thats perfectly pythonic. It just feels strange coming from languages like Java.

Yes, it is possible. You might also want to define __all__ in __init__.py files. It's a list of modules that will be imported when you do
from lib import *

Maybe this could work:
import __init__ as lib