We Keep Coding

Blender Python - Force reload of module while importing all its classes

I am developing in Python/Blender, and have two needs here:
Import all the individual classes from my module (because they must each be registered with blender)
Reload the module itself each time the script is executed (to prevent caching while I'm developing the plugin and press "reload scripts")
Currently I am doing this (in __init__.py):
from importlib import reload
from .MyPlugin import *
reload(MyPlugin)
classes = [ClassA, ClassB, ClassC]
# register each class, not shown here
However the reload(MyPlugin) line causes an error: "MyPlugin is not defined".
Originally I tried reloading each of the classes instead, but it raised an error that reload expects a module.

Some colleagues helped me with an answer, what ended up working was this in __init__.py:
from importlib import reload
if "MyModule" in locals():
reload(MyModule)
else:
import MyModule
from .MyModule import *
This is detailed here: https://blenderartists.org/t/how-to-reload-add-on-code/1202715/2

this works with older Version (tested with 2.79)
Restart first!
del sys.modules['foo']
from foo import *

List project modules imported both directly and indirectly

I've been caught out by circular imports on a large project. So I'm seeking to find a way to test my code to see which of the modules in the project (and only in the project) are imported when an import statement is run. This is to inform refactoring and make sure there isn't an import somewhere deep within a package that's causing a problem.
Suppose I import project package 'agent', I want to know which project modules also get imported as a result. For instance if 'environment' and 'policy' are imported due to modules deep within the agent package containing those import statements, then I want to see just those listed. So not numpy modules listed for example as they are outside the project and so not relevant for circular dependencies.
So far I have this:
import sys
import agent # project module
for k, v in sys.modules.items():
print(f"key: {k} value: {v}")
example rows:
key: numpy.random value: <module 'numpy.random' from '/home/robin/Python/anaconda3/envs/rl/lib/python3.9/site-packages/numpy/random/__init__.py'>
key: environment value: <module 'environment' from '/home/robin/Python/Projects/RL_Sutton/Cliff/environment/__init__.py'>
This does return the modules imported both directly and indirectly but also includes a lot else such as all the components of numpy and builtins etc... If I could filter this dictionary that would solve it.
k is a str, v is <class 'module'>.
The module's __str__ method does return the module file path within it so I suppose that could be used but it's not a clean solution. I've tried looking at the documentation for sys.modules and module_type but nothing there gives a way to filter modules to the current project (that I could see).
I tried to modify the solutions for each of these without success:
How to list imported modules?
List imported modules from an imported module in Python 3
ModuleFinder also looked promising but from the limited example I couldn't see how to make path or excludes solve the problem.
Update
I didn't specify this in the original question but I'm importing modules that often look like this:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
import environment
import policy
ModuleFinder will find environment and policy even though they won't be imported at runtime and don't matter for cyclic imports. So I adapted the accepted answer below to find only runtime imports.
import agent
import sys
app_dir = '/path/to/projects_folder'
imported_module_names = []
for module_name, mod in sys.modules.items():
file = getattr(mod, '__file__', '')
if str(file).startswith(app_dir) and module_name != '__main__':
imported_module_names.append(module_name)
for module_name in sorted(imported_module_names):
print(module_name)

You can use modulefinder to run a script and inspect the imported modules. These can be filtered by using the __file__ attribute (given that you actually import these modules from the file system; don't worry about the dunder attribute, it's for consistency with the builtin module type):
from modulefinder import ModuleFinder
finder = ModuleFinder()
finder.run_script('test.py')
appdir = '/path/to/project'
modules = {name: mod for name, mod in finder.modules.items()
if mod.__file__ is not None
and mod.__file__.startswith(appdir)}
for name in modules.keys():
print(f"{name}")

You can invoke Python with the -v command line option which will print a message each time a module is initialized.

Making imports work when creating "alias" for a package

Real goal: I have a module that is common between two packages (say, bar and bar2). I want to use exact same test files for both cases so I want to change the test imports to not name the package explicitly. (Why? This can be useful during the process of extracting modules from a mega-package into separate packages.)
My idea was to add another module that imports a particular package and provides an "alias" for it. It almost worked, but I got a problem.
Initially I had:
# test.py:
from bar import some_function
If I do nothing magical, there will be two versions of test.py: one with from bar import some_function and another with from new_bar import some_function. I want to avoid this and have the test code files remain the same.
After I added indirection:
#foo.py:
import bar as baz
#test.py:
from .foo import baz # Works!
from .foo.baz import some_function # ModuleNotFoundError: No module named 'cur_dir.foo.baz'; 'cur_dir.foo' is not a package
I can make foo a package:
#foo/__init__.py:
import bar as baz
#test.py:
from .foo import baz # Works!
from .foo.baz import some_function # ModuleNotFoundError: No module named 'cur_dir.foo.baz'
The error changes a bit, but still remains.
I know that I can work around the problem by writing
# test.py:
from .foo import baz
some_function = baz.some_function
Is there any other way? I want my imports to be "normal".
Is there a way to create an "alias" for a package that can be used with the standard import mechanism?

The import statement only looks at actual modules and their paths, not at aliases inside the loaded modules. An actual module alias in Python's module registry, sys.modules, is required.
import sys
import os
sys.modules["os_alias"] = os # alias `os` to `os_alias`
import os_alias # alias import works now
from os_alias import chdir # even as from ... import ...
Once a module alias has been added to sys.modules, it is available for import in the entire application.
Note that module aliasing can lead to subtle bugs when submodules of aliased modules are loaded. In specific, if the submodules are not aliased explicitly, separate versions are created that are not identical. This means that any tests based on object identity, including isinstance(original.submodule.someclass(), alias.submodule.someclass), will fail if the versions are mixed.
To avoid this, you must alias all submodules of any aliased package.

How to import modules that import other modules without using import *

Inside the parent class basehandler.py, there are several import statements, a constant, and a class:
import os
import sys
import cgi
import json
JINJA_ENVIRONMENT = jinja2.Environment(foobar)
class BaseHandler(webapp2.RequestHandler):
pass
Another module, main.py then imports this parent module with from basehandler import *
If we use from basehandler import BaseHandler or import basehandler so as to avoid the from foo import * statement, then the modules that the parent class imports are not received and the program throws an exception.
How do I avoid from foo import * while still correctly importing the parent module with the modules that it imports?

then the modules that the parent class imports are not received and the program throws an exception.
If your program uses a module then you should import it in the module itself, not in some other module e.g., use import os in your program instead of from foo import * where foo module imports os inside.
To answer the question in the title: to make all imported modules from another module available without a wild-card import:
import inspect
import foo # your other module
# get all modules from `foo` and make the names available
globals().update(inspect.getmembers(foo, inspect.ismodule)) #XXX don't do it
It is like from foo import * that imports only modules. The same warnings against wild-card imports are applicable here.

Referencing this answer, you can define a function in basehandler.py that yields all imported modules:
import os,sys,cgi,json
import types
def get_imports():
for name, val in globals().items():
if isinstance(val, types.ModuleType):
yield val.__name__
...and then import that function in main.py and use exec to import the relevant modules:
from basehandler import BaseHandler, get_imports
for i in get_imports():
exec "import %s" % i
print os.getcwd() #this will work
Sort of a gnarly way of doing it though, is there a specific reason you're not just re-importing the list of modules in main.py?

I say there is something else wrong there -
Importing your "basehandler" module from elsewhere should just work:
the names, modules, classes and objects it needs are bound to its own namespace,
and are present in the globals dictionary of any code defined in that module -
regardless of you getting it with a from basehandler import BaseHandler statement.
Please, post your full traceback so that people can find out what is actually happening there.

How to reload python module imported using `from module import *`

I saw in this useful Q&A that one can use reload(whatever_module) or, in Python 3, imp.reload(whatever_module).
My question is, what if I had said from whatever_module import * to import? Then I have no whatever_module to refer to when I use reload(). Are you guys gonna yell at me for throwing a whole module into the global namespace? :)

I agree with the "don't do this generally" consensus, but...
The correct answer is:
from importlib import reload
import X
reload(X)
from X import Y # or * for that matter

Never use import *; it destroys readability.
Also, be aware that reloading modules is almost never useful. You can't predict what state your program will end up in after reloading a module, so it's a great way to get incomprehensible, unreproduceable bugs.

A cleaner answer is a mix of Catskul's good answer and Ohad Cohen's use of sys.modules and direct redefinition:
import sys
Y = reload(sys.modules["X"]).Y # reload() returns the new module
In fact, doing import X creates a new symbol (X) that might be redefined in the code that follows, which is unnecessary (whereas sys is a common module, so this should not happen).
The interesting point here is that from X import Y does not add X to the namespace, but adds module X to the list of known modules (sys.modules), which allows the module to be reloaded (and its new contents accessed).
More generally, if multiple imported symbols need to be updated, it is then more convenient to import them like this:
import sys
reload(sys.modules["X"]) # No X symbol created!
from X import Y, Z, T

A
from module import *
takes all “exported” objects from module and binds them to module-level (or whatever-your-scope-was-level) names. You can reload the module as:
reload(sys.modules['module'])
but that won't do you any good: the whatever-your-scope-was-level names still point at the old objects.

I've found another way to deal with reloading a module when importing like:
from directory.module import my_func
It's nice to know how do modules are being imported generally.
The module is searched in sys.modules dictionary. If it already exists in sys.modules - the module will not be imported again.
So if we would like to reload our module, we can just remove it from sys.modules and import again:
import sys
from directory.module import my_func
my_func('spam')
# output: 'spam'
# here I have edited my_func in module.py
my_func('spam') # same result as above
#output: 'spam'
del sys.modules[my_func.__module__]
from directory.module import my_func
my_func('spam') # new result
#output: 'spam spam spam spam spam'
If You would like to get reloaded module when running whole script, you could use exception handler:
try:
del sys.modules[my_func.__module__]
except NameError as e:
print("""Can't remove module that haven't been imported.
Error: {}""".format(e))
from utils.module import my_func
..........
# code of the script here

When importing using from whatever_module import whatever, whatever is counted as part of the importing module, so to reload it - you should reload your module. But just reloading your module you will still get the old whatever - from the already-imported whatever_module, so you need to reload(whatever_module), and than reload your module:
# reload(whatever_module), if you imported it
reload(sys.modules['whatever_module'])
reload(sys.modules[__name__])
if you used from whatever_module import whatever you can also consider
whatever=reload(sys.modules['whatever_module']).whatever
or
whatever=reload(whatever_module).whatever

import re
for mod in sys.modules.values():
if re.search('name', str(mod)):
reload(mod)

for python 3.7 :
from importlib import reload #import function "reload"
import YourModule #import your any modules
reload(YourModule) #reload your module
Reload function can be called from your own function
def yourFunc():
reload(YourModule)