I've run into a circular import problem because I need to import only part of a package. When my program starts, I create a driver class which inherits from a class defined in a package, but another irrelevant part of that package imports the driver; I need to stop the irrelevant part of the package from running until it's needed later.
More info: I have a package of interfaces in my program. They're just parent objects with methods and attributes common to many objects in my program. They have no logical connection other than having similar purposes. They're in a package solely for my convenience; I don't want tons of .py files in the top level, and would rather sort them into subfolders.
The package looks like this:
interfaces
__init__.py
destroyinterface.py
graphicsinterface.py
And the __ init __.py looks like this:
from destroyinterface import DestroyInterface
from graphicsinterface import GraphicsInterface
I want to import DestroyInterface WITHOUT graphicsinterface.py being initialized. graphicsinterface.py imports the driver that's dependent on DestroyInterface, but I can't seem to access DestroyInterface to create the driver without graphicsinterface.py being initialized.
I don't want to remove the GraphicsInterface import from the __ init __.py because I don't want things to have to know it lives in a file called graphicsinterface.py when they import it. Including information about the structure of my packages to every single import both adds boilerplate and makes refactoring harder. I want the classes to be accessible for import directly from the interfaces module but their .py files only be initialized if I explicitly access them.
I don't want to use a lazy import of the driver in graphicsinterface.py either, both because it's messy (I only want the file being initialized when I actually need it) and because an import inside the time-sensitive methods of GraphicsInterface would slow them down.
Am I out of luck? Will I have to sort my files in a different way?
I'd recommend looking at two solutions within the graphics interface. First, if only a couple of functions need the driver, import the driver in those functions. Doing an import of an already imported module is efficient, so it should be fine to import it in the driver.
Another approach is to do something like this in the graphics interface:
driver = None # filled in when driver is imported
Then elsewhere
import driver
import interfaces.graphics
interfaces.graphics.driver = driver
So I came across a hack to fix my problem. Figured I'd share it.
My __ init __.py now looks like this:
class CrazyHack(object):
#property
def DestroyInterface(self):
import crazyhackthing.destroyinterface
return destroyinterface.DestroyInterface
#property
def GraphicInterface(self):
import crazyhackthing.graphicinterface
return graphicinterface.GraphicInterface
import sys
sys.modules["crazyhackthing"] = sys.modules["interfaces"]
sys.modules["interfaces"] = CrazyHack()
This makes any import statements from this package refer to properties of the object defined there, and thus delay the initialization of files until explicit import. No idea if this works on python 3, and it's probably an awful idea in the first place, but it works for me. May God have mercy on my soul.
Related
I've read through about ten posts on how to import local modules, and I'm still stumped on why this isn't working. I have an extremely simple module, actor.py, with a single class inside it:
class Actor(object):
def __init__(self, name, age):
self.name = name
self.age = age
I'm trying to import it into another module, scraper.py, within the same directory:
Some fixes have listed not having init.py as being a problem with local imports, so I know that's not my problem.
Initially I tried these:
import actor
and
from actor import Actor
but it tells me that actor and Actor are unresolved references. here tells me that's Python 2 syntax, and I'm using Python 3. That answer instead recommends that I do:
from .actor import Actor
When I run my program with that syntax, I get this error:
ModuleNotFoundError: No module named '__main__.actor'; '__main__' is not a package
So I go searching again, and this post tells me to remove the dot from 'actor,' but as stated before, I've tried that as well. My final guess was
from . import actor
but that yields
ImportError: cannot import name 'actor'
which I follow to here, but the answers there mention circular dependencies, and I'm certain actor and scraper have none. Am I perhaps not writing my module correctly? I can't think of any other ways to write an import statement.
edit: if it helps at all, I'm using Intellij
Try from WebScraper.actor import Actor. If this doesn't work its because your package directory is not in the PYTHONPATH. You can set that in the IntelliJ Python run configuration.
The relative import is not working for you because you are trying to run a module as a script. You can see an explanation of what is happening at https://stackoverflow.com/a/8300343/7088038. If you want relative imports to work you will have to add a __main__.py file to your module to allow it to be runnable, or execute from an external script where you use an absolute import so you don't clobber the package namespace.
One other stylistic note- usually (but not always) package names in python use all lowercase names. CamelCase is reserved for class names. So if you wanted to follow convention you would call your package webscraper and use from webscraper.actor import Actor
To import a class into your script use:
from actor import Actor
Or to import the .py entirely (including whatever imports included in it) into the namespace use:
from actor import *
I have a program consistring of several modules specifying the respective web application handlers and one, specifying the respective router.
The library I use can be found here.
Excerpt from webapp.service (there are more such modules):
from webapp.router import ROUTER
#ROUTER.route('/service/[id:int]')
class ServicePermissions(AuthenticatedService):
"""Handles service permissions."""
NODE = 'services'
NAME = 'services manager'
DESCRIPTION = 'Manages services permissions'
PROMOTE = False
webapp.router:
ROUTER = Router()
When I import the webapp.router module, the webapp.service module does obviously not run. Hence, the #ROUTER.route('/service/[id:int]') decorator is not run and my web aplication will fail with the message, that the respective route is not available.
What is the best practice in that case to run the code in webapp.service to "run" the decorators? I do not really need to import the module itself or any of its members.
As stated in the comments fot the question,
you simply have to import the modules. As for linter complaints, those are the lesser of your problems. Linters are there to help - if they get into the way, just don't listen to them.
So, the simple way just to get your things working is, at the end of your __main__.py or __init__.py, depending on your app structure, to import explicitly all the modules that make use of the view decorator.
If you have a linter, check how to silence it on the import lines - that is usually accomplished with a special comment on the import line.
Python's introspection is fantastic, but it can't find instances of a class, or subclasses, if those are defined in modules that are not imported: such a module is just a text file sitting on the disk, like any data file.
What some frameworks offer as an approach is to have a "discovery" utility that will silently import all "py" files in the project folders. That way your views can "come into existence" without explicit imports.
You could use a function like:
import os
def discover(caller_file):
caller_folder = os.path.dirname(caller_file)
for current, folders, files in os.walk(caller_folder):
if current == "__pycache__":
continue
for file in files:
if file.endswith(".py"):
__import__(os.path.join(current, file))
And call it on your main module with discover(__file__)
I have a module that wraps another module to insert some shim logic in some functions. The wrapped module uses a settings module mod.settings which I want to expose, but I don't want the users to import it from there, in case I would like to shim something there as well in the future. I want them to import wrapmod.settings.
Importing the module and exporting it works, but is a bit verbose on the client side. It results in having to write settings.thing instead of just thing.
I want the users to be able to do from wrapmod.settings import * and get the same results as if they did from mod.settings import * but right now, only from wrapmod import settings is available. How to I work around this?
If I understand the situation correctly, you're writing a module wrapmod that is intended to transform parts of an existing package mod. The specific part you're transforming is the submodule mod.settings. You've imported the settings module and made your changes to it, but even though it is available as wrapmod.settings, you can't use that module name in an from ... import ... statement.
I think the best way to fix that is to insert the modified module into sys.modules under the new dotted name. This makes Python accept that name as valid even though wrapmod isn't really a package.
So wrapmod would look something like:
import sys
from mod import settings
# modify settings here
sys.modules['wrapmod.settings'] = settings # add this line!
I ended up making a code-generator for a thin wrapper module instead, since the sys.module hacking broke all IDE integration.
from ... import mod
# this is just a pass-through wrapper around mod.settings
__all__ = mod.__all__
# generate pass-through wrapper around mod.settings; doesn't break IDE integration, unlike manual sys.modules editing.
if __name__ == "__main__":
for thing in settings.__all__:
print(thing + " = mod." + thing)
which when run as a script, outputs code that can then be appended to the end of this file.
well, as the title say, i got a group of import, all import a class, all in the same folder as the script running it:
from lvl import lvl
from get import get
from image import image
from video import vid
from video import MLStripper
from system import system
from setting import setting
from listsearch import lists
python3 doesn't have reload iirc but there is imp.reload() but it doesn't seem to work,
it just throw a error saying it not a module (it a class so it doesn't work)
after every little edit in those class that are imported, i would need to restart the script
isn't there a way to reload/reimport the class to show the effect of the edit without needing to start the script or rewriting most of the script so that imp.reload() works?
python3, linux (but prefer if it also work on window)
edit1:
example: if i use:
import system
system.system.temp()
it return:
65°C
if i change it to show °F and reload it using imp.reload
imp.reload(system)
system.system.temp()
it return:
149°F
so, it works BUT if i use
import system as _system
from system import system
system.temp()
it return:
65°C
then i change it to show °F and reload it using imp.reload
imp.reload(_system)
from system import system
system.temp()
it still return
65°C
BUT again, if i call it this way:
_system.system.temp()
it return
149°F
idk why it that but it is cause it happen in a while loop?
edit2:
file name: system.py:
before changing for test:
class system:
def temp():
temperature = open("/sys/class/thermal/thermal_zone0/temp","r").read()
temperature = temperature[:2]
return(temperature+"°C")
after changing for test:
class system:
def temp():
temperature = open("/sys/class/thermal/thermal_zone0/temp","r").read()
temperature = temperature[:2]
temperature = str(9.0 / 5.0 * int(temperature) + 32).split(".")[0]
return(temperature+"°C")
You can only reload a module:
The argument must be a module object, so it must have been successfully imported before.
In your case, you don't have any reference to the module object. You will have to import it, even if you don't want to use it for anything else, just for the sake of calling reload later.
Also, after the reload, you have re-import the names:
Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.
When you do from foo import bar, that bar is a "name external to the module", so you have to rebind it explicitly.
If you think about it, it has to work this way. There's no way reload can enumerate all objects whose definition is dependent on the previous version of the module to update them. Even if it could, there could be infinite cycles. And what would happen if the new version didn't even have a definition for a class that was in the old one? Or if the class were defined dynamically?
Looking at it another way, from foo import bar is very similar to import foo; bar = foo.bar. bar is a name in your namespace, not foo's namespace, so reload(foo) will not touch it; you need to copy the new foo.bar over again.
The easy way to solve all of these problems is to just repeat all of your from foo import bar lines after the reload.
For simple cases:
import video
from video import MLStripper
# ... later
imp.reload(video)
from video import MLStripper
However, most of your examples have an obvious naming conflict: once you from video import video, you can no longer reload(video). So, you need another reference to the video module object.
Python keeps one around for you, which you can use:
imp.reload(sys.modules['video'])
from video import MLStripper
Or, alternatively, you can use an as clause, or just an = assignment, to give it whatever name you want.
import video as _video
from video import video
# ... later
imp.reload(_video)
from video import video
From your comments and your edited question, it sounds like you have a further problem. Let's use one of the simple non-colliding cases to discuss it.
I believe you're actually doing something like this:
import video
from video import MLStripper
stripper = MLStripper('foo")
# ... later
imp.reload(video)
from video import MLStripper
The first line will successfully reload the video module, and the second will copy its MLStripper class into your globals, so any new MLStripper instances you created will be of the new type.
But that doesn't affect any existing MLStripper instances, like stripper.
Just like MLStripper was, stripper is one of those "names external to the module". But it's actually even worse. In order to adjust it, reload would have to figure out what its state would have been, had the new version of the code been in effect from the time it was created. It should be obvious that this is an unsolvable problem.
If you know the instances you want to patch up, you can deal with them in effectively the same way you dealt with the classes: just create them again:
imp.reload(video)
from video import MLStripper
stripper = MLStripper('foo")
If that's not good enough, there are three hacky possibilities that may be what you want:
Monkeypatch the methods, attributes, etc. into the instance(s) and their __class__(es).
Patch the instances' __class__ attribute directly, so anything that was inherited from the class will now be inherited from the new class.
Serialize the instances with pickle before the reload, then deserialize after.
For very simple cases, all three of these will work. For more complex cases, you will have to understand what you're doing.
Note that you can wrap a lot of this stuff up in a function, but you have to understand how locals and globals work (and how import and reload work) or you're going to end up confusing yourself.
A simpler solution is to just create "dump all state" and "load all state" functions. Then you can dump everything, quit, relaunch, and restore. The Python tutorial and the ipython docs both describe a few different ways to do this in place of using reload; it's probably worth going back and rereading those.
Access the module through sys.modules, reload that, then reassign the imported names:
imp.reload(sys.modules['lvl'])
from lvl import lvl
imp.reload(sys.modules['get'])
from get import get
etc.
All the from something import name syntax does is import something then bind name to the same object something.name refers to.
By using sys.modules you don't have to explicitly import the module again, and can reach the new definitions of the objects for rebinding after reloading.
I've done what I shouldn't have done and written 4 modules (6 hours or so) without running any tests along the way.
I have a method inside of /mydir/__init__.py called get_hash(), and a class inside of /mydir/utils.py called SpamClass.
/mydir/utils.py imports get_hash() from /mydir/__init__.
/mydir/__init__.py imports SpamClass from /mydir/utils.py.
Both the class and the method work fine on their own but for some reason if I try to import /mydir/, I get an import error saying "Cannot import name get_hash" from /mydir/__init__.py.
The only stack trace is the line saying that __init__.py imported SpamClass. The next line is where the error occurs in in SpamClass when trying to import get_hash. Why is this?
This is a pretty easy problem to encounter. What's happening is this that the interpreter evaluates your __init__.py file line-by line. When you have the following code:
import mydir.utils
def get_hash(): return 1
The interpreter will suspend processing __init__.py at the point of import mydir.utils until it has fully executed 'mydir/utils.py' So when utils.py attempts to import get_hash(), it isn't defined because the interpreter hasn't gotten to it's definition yet.
To add to what the others have said, another good approach to avoiding circular import problems is to avoid from module import stuff.
If you just do standard import module at the top of each script, and write module.stuff in your functions, then by the time those functions run, the import will have finished and the module members will all be available.
You then also don't have to worry about situations where some modules can update/change one of their members (or have it monkey-patched by a naughty third party). If you'd imported from the module, you'd still have your old, out-of-date copy of the member.
Personally, I only use from-import for simple, dependency-free members that I'm likely to refer to a lot: in particular, symbolic constants.
In absence of more information, I would say you have a circular import that you aren't working around. The simplest, most obvious fix is to not put anything in mydir/__init__.py that you want to use from any module inside mydir. So, move your get_hash function to another module inside the mydir package, and import that module where you need it.