We Keep Coding

Python Comprehension - Importing & Dunder Methods

Python Double-Underscore methods are hiding everywhere and behind everything in Python! I am curious about how this is specifically working with the interpreter.
import some_module as sm
From my current understanding:
Import searches for requested module
It binds result to the local assignment (if given)
It utilizes the __init__.py . . . ???
There seems to be something going on that is larger than my scope of understanding. I understand we use __init__() for class initialization. It is functioning as a constructor for our class.
I do not understand how calling import is then utilizing the __init__.py.
What exactly is happening when we run import?
How is __init__.py different from other dunder methods?
Can we manipulate this dunder method (if we really wanted to?)

import some_module is going to look for one of two things. It's either going to look for a some_module.py in the search path or a some_module/__init__.py. Only one of those should exist. The only thing __init__.py means when it comes to modules is "this is the module that represents this folder". So consider this folder structure.
foo/
__init__.py
module1.py
bar.py
Then the three modules available are foo (which corresponds to foo/__init__.py), foo.module1 (which corresponds to foo/module1.py), and bar (which corresponds to bar.py). By convention, foo/__init__.py will usually import important names from module1.py and reexport some of them for convenience, but this is by no means a requirement.

Call a function for every script inside a folder

Is there a way (using only python. i.e.: without a bash script nor another language code) to call a specific function in every script inside a folder without needing to import all of them explicitly.
For example, let's say that this is my structure:
main.py
modules/
module1.py
module2.py
module3.py
module4.py
and every moduleX.py has this code:
import os
def generic_function(caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
def internal_function():
print('ERROR: Someone called an internal function')
while main.py has this code:
import modules
import os
for module in modules.some_magic_function():
module.generic_function(os.path.basename(__file__))
So if I run main.py, I should get this output:
module1.py was called by main.py
module2.py was called by main.py
module3.py was called by main.py
module4.py was called by main.py
*Please note that internal_function() shouldn't be called (unlike this question). Also, I don't want to declare explicitly every module file even on a __init__.py
By the way, I don't mind to use classes for this. In fact it could be even better.

You can use exec or eval to do that. So it would go roughly this way (for exec):
def magic_execute():
import os
import glob
for pyfl in glob.glob(os.path(MYPATH, '*.py'):
with open(pyfl, 'rt') as fh:
pycode = fh.read()
pycode += '\ngeneric_function({})'.format(__file__)
exec(pycode)
The assumption here is that you are not going to import the modules at all.
Please note, that there are numerous security issues related to using exec in such a non-restricted manner. You can increase security a bit.

While sophros' approach is quickly and enough for implicitly importing the modules, you could have issues related to controlling every module or with complex calls (like having conditions for each calls). So I went with another approeach:
First I created a class with the function(s) (now methods) declared. With this I can avoid checking if the method exists as I can use the default one if I didn't declare it:
# main.py
class BaseModule:
def __init__(self):
# Any code
def generic_function(self, caller):
# This could be a Print (or default return value) or an Exception
raise Exception('generic_function wasn\'t overridden or it was used with super')
Then I created another class that extends the BaseModule. Sadly I wasn't able to get a good way for checking inherence without knowing the name of the child class so I used the same name for every module:
# modules/moduleX.py
from main import BaseModule
class GenericModule(BaseModule):
def __init__(self):
BaseModule.__init__(self)
# Any code
def generic_function(self, caller):
print('{} was called by {}'.format(os.path.basename(__file__), caller))
Finally, in my main.py, I used the importlib for importing the modules dynamically and saving an instance for each one, so I can use them later (for sake of simplicity I didn't save them in the following code, but it's easy as using a list and appending every instance on it):
# main.py
import importlib
import os
if __name__ == '__main__':
relPath = 'modules' # This has to be relative to the working directory
for pyFile in os.listdir('./' + relPath):
# just load python (.py) files except for __init__.py or similars
if pyFile.endswith('.py') and not pyFile.startswith('__'):
# each module has to be loaded with dots instead of slashes in the path and without the extension. Also, modules folder must have a __init___.py file
module = importlib.import_module('{}.{}'.format(relPath, pyFile[:-3]))
# we have to test if there is actually a class defined in the module. This was extracted from [1]
try:
moduleInstance = module.GenericModule(self)
moduleInstance.generic_function(os.path.basename(__file__)) # You can actually do whatever you want here. You can save the moduleInstance in a list and call the function (method) later, or save its return value.
except (AttributeError) as e:
# NOTE: This will be fired if there is ANY AttributeError exception, including those that are related to a typo, so you should print or raise something here for diagnosting
print('WARN:', pyFile, 'doesn\'t has GenericModule class or there was a typo in its content')
References:
[1] Check for class existence
[2] Import module dynamically
[3] Method Overriding in Python

What is the scope of imports defined in the package __init__.py file?

For a project I had to switch from C++ to python but I'm having some trouble fully understanding the __init__.py file. I found the following documentation on the __init__.py file:
https://docs.python.org/3/tutorial/modules.html
https://www.pythoncentral.io/how-to-create-a-python-package/
http://mikegrouchy.com/blog/2012/05/be-pythonic-__init__py.html
To get more familiar with the way the __init__.py file works I did some tests in both python 2.7 and 3.6.5. I used the "PythonCentral" test pacakge (see link 2) for this:
Now I understand the following about the __init__.py file:
The __init__.py file executes when a module or subpackage module is imported.
The __init__.py can be used to overload the package __all__ method.
The __init__.py can be used to define the import order
The __init__.py can be used to make classes available on package and subpackage level.
Using the different modules and classes from the scope of my main file seems to go as explained in the documentation. However when trying to use a class that is defined in another subpackage module I run into troubles. In python 2.7 created the following __init__.py file in the subanimals subpackage:
from Mammals import Mammals
print "subpackage __init__.py executed"
Following I created the following code inside the Bird.py module:
class Birds:
def __init__(self):
''' Constructor for this class. '''
# Create some member animals
self.members = ['Sparrow', 'Robin', 'Duck']
def printMembers(self):
print('Printing members of the Birds class')
for member in self.members:
print('\t%s ' % member)
test = Mammals()
test.printMembers
When running the main following main script:
from animals.subanimals import Birds
test = Birds()
test.printMembers()
I keep getting the global name Mammals not defined. I can solve this in python 2.7 by adding from Mammals import Mammals to the top of the Birds.py.
Does the __init__.py only import packages and classes on the scope level of the main script and therefore not inside the Birds.py module or am I doing something else wrong?

Answer to my question can be found in #jonrsharpe 's comment.
Each individual file must be valid on its own, defining all
appropriate imports. init.py is for aggregating multiple files in
a directory into a single module; think of it as grouping and passing
attributes up to the parent, not down to the other files in that
directory.

cross-refence link to a function/class in a different submodule

As a simple example, in the following code of 2 submodules (a.py and b.py in the same directory). The link to the same submodule function :func:`hook` works but not the link cross-referencing to a different moduel, ie, :func:`foo`. I also tried the syntax of :func:`.a.foo` - still does not work. How can I cross reference to a.foo()?
# script a.py
def foo():
'''foo func'''
# script b.py
def hook():
'''hook func'''
def spam():
'''spam func.
:func:`foo`
:func:`hook`
'''

As described in the docs:
Normally, names in these roles are searched first without any further
qualification, then with the current module name prepended, then with
the current module and class name (if any) prepended. If you prefix
the name with a dot, this order is reversed.
In that case, :func:`.a.foo` means an object named a inside of the module b. It will look for b.a.foo function.
You should try :func:`..a.foo`, which will point to b..a.foo, or just a.foo (cannot check that locally now, sorry; but I remember I was using that syntax before).
But note, that a.py & b.py should be the modules, i.e. importable under their name. If they are just the scripts, and located not in a package (no __init__.py files up to the root of the project), there is no way to cross-reference with these roles.
You can try to use :any: role — :any:`foo` — and hope that it will find the object in the general index of the described objects.

How can I uncache imported module in python?

This is really freaking me out. My project looks like this
.
|-- main.py
|-- classes
| |-- __init__.py
| |-- exceptions.py
In those exceptions.py, I initally created one exception called SecurityError:
class SecurityError(Exception):
pass
I included the file in main.py using from classes.exceptions import * and SecurityError normally works when I just raise it.
Afterwards, I also added FilesystemError, but whenever using it, I get:
global name 'FilesystemError' is not defined
I tried to delete *.pyc files in classes directory but it has no effect. Any changes to the exceptions.py take no effect, including any ridiculous ones that make the file invalid:
class FileFOO BAR BAZ systemError(Exception):
pass
class SecurityErr foo bar bazor(Exception):
pass
The program still behaves as if the file never changed since the first run. I don't think this behaviour has anything to do with intuitive programming - if there's cache, it should be in that directory.
I am running the python directly from command line:
C:\programing\python\MyProject>py -2 main.py

Confusing errors of the form global name 'XXX' is not defined typically mean that you are shadowing a standard library module. You must rename your exceptions module.