We Keep Coding

How to setup imports and dependencies between modules in the same python package

I am writing a library in python. It has one package and 3 modules, such as
mypackage
__init__.py
utils.py
fileio.py
math.py
If I just leave the __init__.py empty, my users must figure out which functions are in which modules, which is annoying. The fact that I have three modules is an implementation detail that is not important.
Maybe I should import the main functions into the __init__.py, like this
from .utils import create_table
from .fileio import save_rows, load_rows
from .math import matrix_inverse
so that my users can just do
import mypackage as myp
rows = myp.load_rows()
Is that best practice?
What about the alternative to put ALL symbols into the __init__.py, such as
from .utils import *
from .fileio import *
from .math import *
And if there are any functions that I don't want to expose, I will prefix them with underscore. Is that better? It certainly is easier for me.
What if the fileio.py needs to call some functions in the utils.py? I could put
from .utils import *
into the fileio.py, but won't that create a circular or redundant reference? What's the best way to handle this?

Maybe I should import the main functions into the init.py, like this [...] Is that best practice?
I wouldn't say there is a "best practice", it depends on the specific case, but this is surely pretty common: you define a bunch of stuff in each module, and import the relevant ones in __init__. This is an easy way to not bother the users with remembering which submodule has the needed function, however it can get pretty annoying if you have a lot of functions to import from each module.
What about the alternative to put ALL symbols into the init.py, such as
from .utils import *
from .fileio import *
from .math import *
You most likely don't want to do this. This will import everything in user scripts, including other imported modules and internal functions. You should avoid it.
What if the fileio.py needs to call some functions in the utils.py? [...] won't that create a circular or redundant reference?
Yeah, that is something that can happen and you usually want to avoid it at all costs. If you need some functions from utils.py in fileio.py, you should import them explicitly as from .utils import x, y, z. Remember to also always use relative imports when importing things between modules of the same package (i.e. use from .utils import x, not from package.utils import x).
A good compromise between these two options you mention which solves most of the above problems (although not circular imports, you would have to avoid those yourself) would be to define an __all__ list inside each one of your modules to specify which functions should be exported when using from x import *, like this:
# utils.py
import sys
__all__ = ['user_api_one', 'user_api_two']
def user_api_one():
...
def user_api_two():
...
def internal_function():
...
If you properly define an __all__ list in all your modules, then in your __init__.py you will be able to safely do:
from .utils import *
from .fileio import *
from .math import *
This will only import relevant functions (for example user_api_one and user_api_two for utils, and not internal_function nor sys).

Why does python import module imports when importing *

Let's say I have a file where I'm importing some packages:
# myfile.py
import os
import re
import pathlib
def func(x, y):
print(x, y)
If I go into another file and enter
from myfile import *
Not only does it import func, but it also imports os, re, and pathlib,
but I DO NOT want those modules to be imported when I do import *.
Why is it importing the other packages I'm importing and how do you avoid this?

The reason
Because import imports every name in the namespace. If something has a name inside the module, then it's valid to be exported.
How to avoid
First of all, you should almost never be using import *. It's almost always clearer code to either import the specific methods/variables you're trying to use (from module import func), or to import the whole module and access methods/variables via dot notation (import module; ...; module.func()).
That said, if you must use import * from module, there are a few ways to prevent certain names from being exported from module:
Names starting with _ will not be imported by import * from .... They can still be imported directly (i.e. from module import _name), but not automatically. This means you can rename your imports so that they don't get exported, e.g. import os as _os. However, this also means that your entire code in that module has to refer to the _os instead of os, so you may have to modify lots of code.
If a module contains the name __all__: List[str], then import * will export only the names contained in that list. In your example, add the line __all__ = ['func'] to your myfile.py, and then import * will only import func. See also this answer.

from myfile import func
Here is the fix :)
When you import *, you import everything from. Which includes what yu imported in the file your source.

It has actually been discussed on Medium, but for simplification, I will answer it myself.
from <module/package> import * is a way to import all the names we can get in that specific module/package. Usually, everyone doesn't actually use import * for this reason, and rather sticked with import <module>.

Python's import essentially just runs the file you point it to import (it's not quite that but close enough). So if you import a module it will also import all the things the module imports. If you want to import only specific functions within the module, try:
from myfile import func
...which would import only myfile.func() instead of the other things as well.

How to import everything inside a folder into a single module in Python2

I simply want to take all my .py files from a single folder (I don't care about the sub-folders for now) and put them into a single module.
The use case I'm having here is that I'm writing some pretty standard object-oriented code and I'm using a single file for every class, and I don't want to have to write from myClass import myClass for every class into my __init__.py. I can't use Python3, so I'm still working with impand reloadand such.
At the moment I'm using
# this is __init__.py
import pkgutil
for loader, name, is_pkg in pkgutil.walk_packages(__path__):
if not is_pkg:
__import__(__name__ + "." + name)
and it doesn't seem to work, it includes the packages but it includes them as modules, so that I have to write MyClass.MyClass for a class that is defined in a file with it's own name. That's silly and I don't like it.
I've been searching forever and I'm just getting more confused how complicated this seemingly standard use case seems to be. Do python devs just write everything into a single file? Or do they always have tons of imports?
Is this something that should be approached in an entirely different way?

What you really want to do
To do the job you need to bind your class names to namespace of your __init__.py script.
After this step you will be able to just from YourPackageName import * and just use your classes directly. Like this:
import YourPackageName
c = YourPackageName.MyClass()
or
from YourPackageName import *
c = MyClass()
Ways to achieve this
You have multiple ways to import modules dynamically: __import__(), __all__.
But.
The only way to bind names into namespace of current module is to use from myClass import myClass statement. Static statement.
In other words, content of each of your __init__.py scripts should be looking like that:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .MyAnotherSubPackage import *
from .my_pretty_class import myPrettyClass
from .my_another_class import myAnotherClass
...
And you should know that even for a dynamic __all__:
It is up to the package author to keep this list up-to-date when a new version of the package is released.
(https://docs.python.org/2/tutorial/modules.html#importing-from-a-package)
So a clear answers to your questions:
Do python devs just write everything into a single file?
No, they don't.
Or do they always have tons of imports?
Almost. But definitely not tons. You need to import each of your modules just once (into an appropriate __init__.py scripts). And then just import whole package or sub-package at once.
Example
Let's assume that there is next package structure:
MyPackage
|---MySubPackage
| |---__init__.py
| |---pretty_class_1.py
| |---pretty_class_2.py
|---__init__.py
|---sleepy_class_1.py
|---sleepy_class_2.py
Content of the MyPackage/MySubPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .pretty_class_1 import PrettyClass1
from .pretty_class_2 import PrettyClass2
Content of the MyPackage/__init__.py:
#!/usr/bin/env python
# coding=utf-8
from .MySubPackage import *
from .sleepy_class_1 import SleepyClass1
from .sleepy_class_2 import SleepyClass2
As result, now we are able to write next code in our application:
import MyPackage
p = MyPackage.PrettyClass1()
s = MyPackage.SleepyClass2()
or
from MyPackage import *
p = PrettyClass1()
s = SleepyClass2()

Importing imports within a function - Python 2.6

I have two files, SysDump.py and libApi.py in the same folder.
In SysDump I do:
from libApi._SysDump import *
In libApi I have:
def _SysDump():
import cPickle as _cPickle
import math as _math
from zipfile import ZipFile as _ZipFile
import re as _re
However I get the error:
from libApi._SysDump import *
ImportError: No module named _SysDump
I use VS2012+PTVS to step through the code and the execution trace goes to def _SysDump() in libApi as I steop through but does not enter it. Question is how do I make this work in Python 2.6 only please?

from libApi._SysDump import *
When writing this, Python looks for a package libApi and a module in it called _SysDump. A package is equivalent to a folder and a module is a single file. From your explanations, this is not the situation you have in your case. You have a module libApi with a function _SysDump. So if anything, you could do this:
from libApi import _SysDump
So you would get a reference to the _SysDump function. Note that running that function will not give you references to all the modules you are trying to import. Inside the function, the modules will be imported and assigned to local variables. After the function ends, those references are gone.
If you want to have some module take care of all your imports, you could make a file that performs those imports and import everything from that module:
# imports.py
import cPickle as _cPickle
import math as _math
from zipfile import ZipFile as _ZipFile
import re as _re
And then:
from imports import *

Python: Is there a place when I can put default imports for all my modules?

Is there a place when I can put default imports for all my modules?

If you want default imports when using the python shell you can also set the PYTHONSTARTUP environmental variable to point to a python file that will be executed whenever you start the shell. Put all your default imports in this file.

Yes, just create a separate module and import it into yours.
Example:
# my_imports.py
'''Here go all of my imports'''
import sys
import functools
from contextlib import contextmanager # This is a long name, no chance to confuse it.
....
# something1.py
'''One of my project files.'''
from my_imports import *
....
# something2.py
'''Another project file.'''
from my_imports import *
....
Note that according to standard guidelines, from module import * should be avoided. If you're managing a small project with several files that need common imports, I think you'll be fine with from module import *, but it still would be a better idea to refactor your code so that different files need different imports.
So do it like this:
# something1.py
'''One of my project files. Takes care of main cycle.'''
import sys
....
# something2.py
'''Another project file. Main program logic.'''
import functools
from contextlib import contextmanager # This is a long name, no chance to confuse it.
....