We Keep Coding

How does import and reload actually work when dealing with packages

I have issues understanding some subtleties of the Python import system. I have condensed my doubts around a minimal example and a number of concrete and related questions detailed below.
I have defined a package in a folder called modules, whose content is an __init__.py and two regular modules, one with general functionality for the package and other with the definitions for the end user. The content is as simple as:
init.py
from .base import *
from .implementation import *
base.py
class FactoryClass():
registry = {}
#classmethod
def add_to_registry(cls, newclass):
cls.registry[newclass.__name__] = newclass
#classmethod
def getobject(cls, classname, *args, **kwargs):
return cls.registry[classname](*args, **kwargs)
class BaseClass():
def hello(self):
print(f"Hello from instance of class {type(self).__name__}")
implementation.py
from .base import BaseClass, FactoryClass
class First(BaseClass):
pass
class Second(BaseClass):
pass
FactoryClass.add_to_registry(First)
FactoryClass.add_to_registry(Second)
The user of the package will use it as:
import modules
a = modules.FactoryClass.getobject("First")
b = modules.FactoryClass.getobject("Second")
a.hello()
b.hello()
This works. The problem comes because I'm developing this, and my workflow includes adding functionality in implementation.py and then continaully test it by reloading the module. But I can not understand/predict what module I have to reload to have the functions updated. I'm making changes that have no effect and it drives me crazy (until yesterday I was working on a large .py file with all code lumped together, so I had none of these problems).
Here are some test I have done, and I'd like to understand what's happening and why.
First, I start commenting out all mentions to Second class in implementation.py (to pretend it was not yet developed):
from importlib import reload
import modules
modules.base.FactoryClass is modules.FactoryClass # returns True
modules.FactoryClass.registry # just First class is in registry
a = modules.FactoryClass.getobject("First")
b = modules.FactoryClass.getobject("Second") # raises KeyError as expected
This code and its output is pretty clear. The only thing that puzzles me is why there is a modules.base module at all (I did not import it!). Further, it is redundant as their classes point to the same objects. Why importing modules also imports modules.base and modules.implementation as separate but essentially identical objects?
Now things become interesting as I comment out, i.e. I finish developing Second, and I'd like to test it without having to restart the Python session. I have tried 3 different reloads:
reload (modules)
This does absolutely nothing. I'd expect some sort of recursivity, but as I have found in many other threats, this is the expected behavior.
Now I try to manually reload one of those "unexpected" modules:
reload (modules.implementation)
modules.base.FactoryClass is modules.FactoryClass # True
modules.FactoryClass.registry # First and Second
a = modules.FactoryClass.getobject("First")
b = modules.FactoryClass.getobject("Second") # Works as expected
This seems to be the right way to go. It updates the module contents as expected and the new functionality is usable. What puzzles me is why modules.FactoryClass has been updated (its registry) despite the fact that I did not reload the modules.base module. I'd expect this function to stay "outdated".
Finally, and starting from the just freshly uncommented version, I have tried
reload (modules.base)
modules.base.FactoryClass is modules.FactoryClass # False
modules.FactoryClass.registry # just First class is in registry
modules.base.FactoryClass.registry # empty
a = modules.base.FactoryClass.getobject("First")
b = modules.base.FactoryClass.getobject("Second") # raises KeyError
This is very odd. modules.FactoryClass is outdated (Second is unknown). modules.base.Factory is empty. Why are now modules.FactoryClass and modules.base.FactoryClass different objects?
Could someone explain why the three different versions of reload a package have so different behaviour?

You are confused about how the Python import system works, so I strongly recommend you read the corresponding documentations : the import system and importlib.reload.
A foreword : code hot-reloading in Python is tricky. I recommend to not do that if it is not required. You have seen it yourself : bugs are very tricky.
Then to your questions :
Why importing modules also imports modules.base and modules.implementation as separate but essentially identical objects?
As #Kemp answered as a comment (and I upvoted), imports are transitive. When you import a, Python will parse/compile/execute the corresponding library file. If the module does import b then Python will do it again for the b library file, and again and again. You don't see it, but when your program starts there is already a lot of things that have been imported.
Given this file :
print("nothing else")
When I set my debugger to pause before executing the print line, if I look into sys.modules I already have 338 different libraries imported : builtins (where print came from), sys, itertools, enum, json, ...
Understand that "no visible import statement" does not mean "nothing have been imported".
When you execute import a, Python will start by checking its sys.modules cache to determine if the library have already been read from disk, parsed, compiled and executed into a module object. If this library was not yet imported during this program, then Python will take the time to do all that. But because it is slow, Python optimize with a cache.
The result is a module object, that gets bind the current namespace, so that you can access it.
We can summerize it like that :
def import_library(name: str) -> Module:
if name not in sys.modules:
# cache miss
filepath = locate_library(name)
bytecode = compile_library(filepath)
module = execute(bytecode)
sys.modules[name] = module
# in any case, at this point, the module is available
return sys.modules[name]
You are thus confusing module objects with variables.
In any module you can declare variables with whatever name (but allowed by Python's grammar). And some of them will reference modules.
here is an example :
# file: main.py
import lib # create a variable named `lib` referencing the `lib` library
import lib as horse # create a variable named `horse` referencing the `lib` library
print(lib.a.number) # 14
print(horse.a.number) # 14
print(lib is horse) # True
print(lib.a.sublib.__name__) # lib.sublib
import lib.sublib
from lib import sublib
import lib.sublib as lib_sublib
print((lib.sublib is sublib, sublib is lib_sublib, lib.a.zebra is sublib)) # (True, True, True)
import sys
print(sys.modules["lib"] is lib) # True
print(sys.modules["lib.sublib"] is sublib) # True
print(lib.sublib.package_color) # blue
print(lib.sublib.color) # AttributeError: module 'lib.sublib' has no attribute 'color'
# file: lib/__init__.py
from . import a
# file: lib/a.py
from . import sublib
from . import sublib as zebra
number = 14
# file: lib/sublib/__init__.py
from .b import color as package_color
# file: lib/sublib/b.py
color = "blue"
Python offers a lot of flexibility about how to import things, what to expose, how to access. But I admit it is confusing.
Also take a look at the role of __all__ in __init__.py. Given that, you should now understand your question on subpackage naming/visibility.
reload (modules) This does absolutely nothing. I'd expect some sort of recursivity, but as I have found in many other threats, this is the expected behavior.
Given what I explained, can you now understand what it does ? And why what it does is not what you want it to do ?
Because what you want is to get modules.implementations hot-reloaded, but you ask for modules.
>>> from importlib import reload
>>> import lib
>>> lib.sublib.package_color
'blue'
>>> # I edit the "b.py" file so that the color is "red"
>>> old_lib = lib
>>> new_lib = reload(lib)
>>> lib is new_lib, lib is old_lib
(True, True)
>>> lib.sublib.package_color
'blue'
>>> lib.sublib.b.color
'red'
>>> import sys
>>> sys.modules["lib.sublib.b"].color
'red'
First, the top-level reload did not work, because what the file only did was import sublib, which hit the cache, so nothing really gets done.
You have to reload the actual module for its content to takes effect. But it does not work magically : it will create new objects (module-level definitions) and put them into the same module object, but it can't update references that may exist on the preceding module's content. That is why we see a "blue" even after the module has been reloaded : the package_color is a reference to the first version's color variable, it does not get updated when the module is reloaded. This is dangerous : there may be different copies of similar things lying around.
why modules.FactoryClass has been updated
You are reloading modules.implementation in this case. What happens is that it reloads the whole file to populate the module object, I highlighted the perceived effects :
from .base import BaseClass, FactoryClass # relative library "base" already in the `sys.modules` cache
class First(BaseClass): # was already defined, redefined
pass
class Second(BaseClass): # was not defined yed, created
pass
FactoryClass.add_to_registry(First) # overwritting the registry for "First" with the redefinition of class `First`
FactoryClass.add_to_registry(Second) # registering for "Second" the definition of class `Second`
You can see it another way :
>>> import modules
>>> from importlib import reload
>>> First_before = modules.implementation.First
>>> reload(modules.implementation)
<module 'modules.implementation' from 'C:\\PycharmProjects\\stack_overflow\\68069397\\modules\\implementation.py'>
>>> First_after = modules.implementation.First
>>> First_before_reload is First_after_reload
False
When you are reloading a module, Python will re-execute all the code, in a way that may be different than the previous time(s). Here, each time you are doing FactoryClass.add_to_registry so the FactoryClass gets updated with the (re)definitions.
Why are now modules.FactoryClass and modules.base.FactoryClass different objects?
Because you reloaded modules.base, creating a new FactoryClass class object, but the from .base import BaseClass, FactoryClass does not get reloaded, so it is still using the class object from before the reload.
Because you reloaded, you got yourselves copy of everything. The problem is that you still have lingering references to versions of before the reload.
I hope it answers your questions.
Import is not easy, but reloading is notably tricky.
If you truly desire to reload your code, then you will have to take extra extra extra care to correctly re-import everything, in the correct order (if such an order exist, if there is not too much side-effects). You need a proper script to update everything correctly, and in case it does not work perfectly, you will frequently have horrible, sad and mind-bending bugs.
But if you prefer keep your sanity, if your workflow does not require you to reload parts of the program, just close it and restart it. There is nothing wrong with that. It is the simple solution. The sane solution.
TL;DR: don't use importlib.reload if you don't know how it works exactly and understand the risks.

Pass-through/export whole third party module (using __all__?)

I have a module that wraps another module to insert some shim logic in some functions. The wrapped module uses a settings module mod.settings which I want to expose, but I don't want the users to import it from there, in case I would like to shim something there as well in the future. I want them to import wrapmod.settings.
Importing the module and exporting it works, but is a bit verbose on the client side. It results in having to write settings.thing instead of just thing.
I want the users to be able to do from wrapmod.settings import * and get the same results as if they did from mod.settings import * but right now, only from wrapmod import settings is available. How to I work around this?

If I understand the situation correctly, you're writing a module wrapmod that is intended to transform parts of an existing package mod. The specific part you're transforming is the submodule mod.settings. You've imported the settings module and made your changes to it, but even though it is available as wrapmod.settings, you can't use that module name in an from ... import ... statement.
I think the best way to fix that is to insert the modified module into sys.modules under the new dotted name. This makes Python accept that name as valid even though wrapmod isn't really a package.
So wrapmod would look something like:
import sys
from mod import settings
# modify settings here
sys.modules['wrapmod.settings'] = settings # add this line!

I ended up making a code-generator for a thin wrapper module instead, since the sys.module hacking broke all IDE integration.
from ... import mod
# this is just a pass-through wrapper around mod.settings
__all__ = mod.__all__
# generate pass-through wrapper around mod.settings; doesn't break IDE integration, unlike manual sys.modules editing.
if __name__ == "__main__":
for thing in settings.__all__:
print(thing + " = mod." + thing)
which when run as a script, outputs code that can then be appended to the end of this file.

'module' object has no attribute 'Player'

I'm trying to make a simple game, which has multiple files that need to import each other. My mob module needs to import my player module and do player = player.Player() to create a new class instance. The class is in the player.py file, but it's apparently not being recognized.
I did world = world.World() in my mob file, and that worked perfectly, so I'm confused as to why my player file won't work.
player.__file__ shows the correct path so that's not the issue.
This is what I have at the beginning of mob.py:
import world
import main
import player
world = world.World()
player = player.Player()
class Mob:
#Class definition
player.py:
import main
import world
world = world.World()
class Player:
#Definition
world.py:
import os
import main
class World:
#Definition
Not sure if this will help, but this is the main function in main.py:
if __name__ == "__main__":
console = []
player = player.Player()
movePlayer = player.Move
transformPlayer = player.Transform
goblin = mob.Mob()
world = world.World()
mapSize = world.MapSize
mainMenu()

The problem here is a circular import.
Since you haven't given us enough of the code to actually test this, it's impossible to give the exact details. If you want to see it for yourself, just add a print before and after each import, and you can see the sequence of events. But it's going to be something like this:
player depends on main. main depends on mob. So, when you try to import player, before it can run any of its top-level code, all of mob's top-level code has to run. But mob tries to create a player.Player object at top level. Since player is still waiting for import main to finish, it hasn't yet run the code to define that class. Hence the error.
The official FAQ has a nice answer on different ways to fix this. In your case, I think there's a simple answer:
Split main.py into two separate modules. Take the shared code that everyone else needs to import, and put it into, say, utils.py. Leave the top-level program code that needs to import everyone else in main.py. Change each import main to import utils. And that's it; no more circular imports. (This is effectively Matthias Ulrichs's recommendation from the FAQ, but more specific to your situation—and, as it happens, very common among projects created by novices to Python, especially those who are very skilled at languages that have separate import and implementation files.)
If this isn't doable in your case, consider moving some of the global initialization to initialization functions, which you can call after all the imports are done. (This is effectively Guido's recommendation from the FAQ.)

Python : from module import * in __init__

Here is a simple case: I want to define a module in python name robot. So, I have a folder named robot with these two files:
__init__.py:
from test import a
test.py:
a = "hello world"
Now, when I import robot in the interpreter, the robot namespace includes test and a. However, I only want it to include a. Why this odd behavior?
EDIT:
Here's a slightly more representative example of what I want to achieve:
Given the following files:
__init__.py:
from spam import a
import ham
spam.py:
a = "hello world"
ham.py:
b = "foo"
Can I have a robot namespace containing a and ham at its top level but not spam?

You have created not just a module but a package. A package contains its submodules in its namespace (once they have been imported, as you imported test here). This is as it should be, since the usual way of using packages is to provide a grouping of several modules. There's not much use to making a package with only one module (i.e., one contentful .py file) inside it.
If you just want a one-file module, just create a file called robots.py and put your code in there.
Edit: See this previous question. The answer is that you should in general not worry about excluding module names from your package namespace. The modules are supposed to be in the package namespace. If you want to add functions and stuff from submodules as well, for convenience, that's fine, but there's not really anything to be gained by "covering your tracks" and hiding the modules you imported. However, as described in the answers to that question, there are some hackish ways to approximate what you want.

Are you just asking how to import specific modules or functions?
test.py:
import robot.spam.a
import robot.ham
Don't import the entire package.

Way around Python imports within imports?

Sorry if this is a very novice question, I was just wondering one thing.
When in python and your code is split up amongst multiple files, how would you avoid a ton of imports on the same thing?
Say I have 2 files. Main, and Content.
Main:
import pygame
from pygame.locals import *
pygame.display.init()
blah
Content:
import pygame
from pygame.locals import *
pygame.display.init()
load content and stuff
pygame is imported twice and display.init is called twice. This is problematic in other places. Is there anyway to get around this, or must it just import and import and import?
One situation I can think of is this: A script that writes to a file everytime it is imported. That way, if it's imported 3 times, it gets run 3 times, therefore writing to the file 3 times.
Thanks in advance!

You misunderstand what import does. It's not the same as include. Loaded modules are singletons and their corresponding files are not evaluated twice.
That said, a well-constructed module will not have side-effects on import. That's the purpose of the if __name__=='__main__' idiom.
Do not attempt to "clean up" your imports. Import everything you need to use from within a file. You could make less use of import *, but this is purely for code readability and maintainability.

You should avoid having anything happen at import(except, see further down). a python file is a module first, so it can and should be used by other python modules. If something "happens" in the import stage of a python file, then it may happen in an undesirable way when that file is imported by another module.
Each module should just define things to be used: classes, functions, constants, and just wait for something else to use them.
Obviously, if no script ever does at import, then it's not possible for anything to actually get used and make stuff "happen". There is a special idiom for the unusual case that a module was called directly. Each python file has a variable, __name__ automatically created with the module name it was imported as. When you run a script from the command line (or however you have started it), it wasn't imported, and there's no name for it to have, and so the __name__ variable will have a special value "__main__" that indicates that it's the script being executed. You can check for this condition and act accordingly:
# main.py
import pygame
from pygame.locals import *
import content
def init():
pygame.display.init()
def stuff():
content.morestuff()
if __name__ == '__main__':
init()
stuff()
# content.py
import pygame
from pygame.locals import *
def init():
pygame.display.init()
def morestuff():
"do some more stuff"
if __name__ == '__main__':
init()
morestuff()
This way; init() and thus pygame.display.init() are only ever called once, by the script that was run by the user. the code that runs assuming that init() has already been called is broken into another function, and called as needed by the main script (whatever that happens to be)

import statements are supposed to be declarations that you're using something from another module. They shouldn't be used to cause something to happen (like writing to a file). As noted by Francis Avila, Python will try not to execute the code of a module more than once anyway.
This has the consequence that whether a given import statement will cause anything to happen is a global property of the application at runtime; you can't tell just from the import statement and the source code of the module, because it depends on whether any other code anywhere else in the project has already imported that module.
So having a module "do something" when executed is generally a very fragile way to implement your application. There is no hard and fast definition of "do something" though, because obviously the module needs to create all the things that other modules will import from it, and that may involve reading config files, possibly even writing log files, or any number of other things. But anything that seems like the "action" of your program, rather than just "setting up" things to be imported from the module, should usually not be done in module scope.