Hello I did got into circular dependency what is not refactori-zable other than doubling code.
I have something like this (only much more complex):
myParser.py:
import sys
import main #comment this to make it runnable
def parseEvnt():
sys.stdout.write("evnt:")
main.parseCmd(1) #comment this to make it runnable
tbl.py:
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
main.py:
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
data=[1,2]
for d in data:
if(d<2):
parseCmd(d)
else:
fce = tbl.tblx[d][1]
fce()
Obvious error I'm getting is:
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 1, in <module>
import tbl
File "D:\Data\vbe\workspace\marsPython\testCircular2\tbl.py", line 1, in <module>
import myParser
File "D:\Data\vbe\workspace\marsPython\testCircular2\myParser.py", line 2, in <module>
import main
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 7, in <module>
parseCmd(d)
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 3, in parseCmd
print(tbl.tblx[d][0])
AttributeError: module 'tbl' has no attribute 'tblx'
In C I think I would just tell by declaration in tbl.py hey there is function parseEvnt(). I would not need to include myParser and there would be no circular include.
In python I do not know how to do it.
I read few threads and there is always some wise guy recommending refactorizing. But in this case parseCmd() needs to see tblx which needs to see parseEvnt() (unless function declaration) and parseEvnt() need to call parseCmd() (cos evnt contains triggering cmd and I do not want to double the decoding cmd code).
Is there way how to make it working in python?
You can frequently get away with circular dependencies as long as the modules don't try to use each other's data until all importing is complete - as a practical matter, that means referencing with namespace (from module import something is forbidden) and only using the other modules inside functions and methods (no mystuff = module.mystuff in the global space). That's because when importing starts, python puts the module name in sys.modules and won't try to import that module again.
You ran into trouble because when you run main.py, python adds __main__ to sys.modules. When the code finally came around to import main, there was no "main" in the module list and so main.py was imported again... and its top level code tried to run.
Lets rearrange your test case and throw in a few print statements to tell when import happens.
myParser.py
print(' + importing myParser')
import sys
print('import parsecmd')
import parsecmd
def parseEvnt():
sys.stdout.write("evnt:")
parsecmd.parseCmd(1)
tbl.py
print(' + importing tbl')
print('import myParser')
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
Parsecmd.py (new)
print(' + importing parsecmd')
print('import tbl')
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
main.py
print('running main.py')
print('import parsecmd')
import parsecmd
if __name__ == "__main__":
data=[1,2]
for d in data:
if(d<2):
parsecmd.parseCmd(d)
else:
fce = parsecmd.tbl.tblx[d][1]
fce()
When I run it I get
running main.py
import parsecmd
+ importing parsecmd
import tbl
+ importing tbl
import myParser
+ importing myParser
import parsecmd <-- didn't reimport parsecmd
cmd
evnt:cmd
If you're insistent on not refactoring (which is the real solution to this - not being a wise guy), you could move your problematic import into your function in myParser.py
import sys
def parseEvnt():
import main ## import moved into function
sys.stdout.write("evnt:")
main.parseCmd(1)
Again, see if you can redesign your code so such interdependencies are avoided.
The above solution is sort of a hack and won't solve future problems you might run into due to this dependency.
Circular imports should be avoided. Refactoring is required, any workaround that still requires a circular import is not a good solution.
That being said, the refactoring doesn't have to be extensive. There are at least a couple of fairly simple solutions.
Solution 1: move shared functions to a shared module
Since you want to use parseCmd from more than one place, move it to a separate file. That way both main.py and myParser.py can import the function.
Solution 2: have main pass parseCmd to parseEvnt.
First, make parseEvnt accept an argument to tell it which function to run:
# myParser.py
import sys
def parseEvnt(parseCmd):
sys.stdout.write("evnt:")
parseCmd(1)
Next, when you call myParser.parseEvnt, pass in a reference to main.parseCmd:
# main.py:
...
else:
fce = tbl.tblx[d][1]
fce(parseCmd)
There are other ways to accomplish the same thing. For example, you could add a "configure" method in myParser, and then have main.py call the configure method and pass in a reference to its parseCmd. The configure method can then store this reference in a global variable.
Another choice is to import main in the function that uses it:
main.py
import sys
def parseEvnt():
import main
sys.stdout.write("evnt:")
main.parseCmd(1)
Related
I've a main.py file with a block of code like this:
import urtc
import machine
rtc = urtc.DS3231(machine.I2C(scl=machine.Pin(0), sda=machine.Pin(2)))
from func import * #line 4
Now, func.py file which is imported on line 4 has code something like this:
def current_time():
import urtc
import machine
rtc = urtc.DS3231(machine.I2C(scl=machine.Pin(0), sda=machine.Pin(2)))
return urtc.tuple2seconds(rtc.datetime())
In main.py, I'm already importing urtc and machine and defining rtc. Is it possible to eliminate these 3 lines from function current_time():
import urtc
import machine
rtc = urtc.DS3231(machine.I2C(scl=machine.Pin(0), sda=machine.Pin(2)))
It seems redundant as I already have them in main.py global. How can I use them from main.py global instead of importing them again in function current_time()?
You should pass the urtc.DS3231 instance to the current_time function like so:
def current_time(rtc):
return urtc.tuple2seconds(rtc.datetime())
But you still need to import urtc in func.py so that urtc.tuple2seconds is available.
You should use arguments in your function, this is in fact bad design to do it the way you did.
import urtc
import machine
rtc = urtc.DS3231(machine.I2C(scl=machine.Pin(0), sda=machine.Pin(2)))
from func import *
def current_time(rtc):
return urtc.tuple2seconds(rtc.datetime())
current_time(rtc)
I would suggest that you load the dependencies in func.py (if you are not using them anywhere else in main.py, it is a better practice).
I know that if I import a module by name import(moduleName), then I can reload it with reload(moduleName)
But, I am importing a bunch of modules with a Kleene star:
from proj import *
How can I reload them in this case?
I think there's a way to reload all python modules. The code for Python 2.7 is listed below: Instead of importing the math module with an asterisk, you can import whatever you need.
from math import *
from sys import *
Alfa = modules.keys()
modules.clear()
for elem in Alfa:
str = 'from '+elem+' import *'
try:
exec(str)
except:
pass
This is a complicated and confusing issue. The method I give will reload the module and refresh the variables in the given context. However, this method will fall over if you have multiple modules using a starred import on the given module as they will retain their original values instead of updating. In generally, even having to reload a module is something you shouldn't be doing, with the exception of when working with a REPL. Modules aren't something that should be dynamic. You should consider other ways of providing the updates you need.
import sys
def reload_starred(module_name, context):
if context in sys.modules:
context = vars(sys.modules[context])
module = sys.modules[module_name]
for name in get_public_module_variables(module):
try:
del context[name]
except KeyError:
pass
module = reload(module)
context.update(get_public_module_variables(module))
def get_public_module_variables(module):
if hasattr(module, "__all__"):
return dict((k,v) for (k,v) in vars(module).items()
if k in module.__all__)
else:
return dict((k,v) for (k,v) in vars(module).items()
if not k.startswith("_"))
Usage:
reload_starred("my_module", __name__)
reload_starred("my_module", globals())
reload_starred("my_module", "another_module")
def function():
from my_module import *
...
reload_starred("my_module", locals())
I have a directory structure like this
main.py
markdown-extensions/
__init__.py
doc_extension.py
Here is my doc_extension.py (it's intended to be a bare bones markdown post processor):
from markdown.postprocessors import Postprocessor
class DocsPostProcessor(Postprocessor):
def run(self, text):
return "<h1>hello world</h1>"
class DocsExtension:
def extendMarkdown(self,md):
postProcessor = DocsPostProcessor()
postProcessor.md = md
md.postprocessors.add(postProcessor)
How do I go about importing it into my main.py? I've tried variations on the following to no avail:
import markdown-extensions.doc_extension
import markdown-extensions.*
import markdown-extensions.doc_extension
The - sign is not a valid character for a Python name (also known as identifier), whether it is a module or not. See here.
from markdown-extensions.doc_extension import *
but rather be explicit, as * will import all global variables, methods and classes. So:
from markdown-extensions.doc_extension import DocsPostProcessor, DocsExtension
*edit
And yes besides that you can't have "-"s, I mistook it for a "_".
Here are the files in this test:
main.py
app/
|- __init__.py
|- master.py
|- plugin/
|- |- __init__.py
|- |- p1.py
|- |_ p2.py
The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.
I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.
Here's the file contents:
main.py:
import app
app.test()
app.test2()
app/__init__.py:
import sys, os
from plugin import p1
def test():
print "__init__ in app is executing test"
p1.test()
def test2():
print "__init__ in app is executing test2"
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
print "The scriptdir is %s" % scriptDir
sys.path.insert(0,scriptDir)
m = __import__("p2", globals(), locals(), [], -1)
m.test()
app/master.py:
myVar = 0
app/plugin/__init__.py:
<empty file>
app/plugin/p1.py:
from .. import master
def test():
print "test in p1 is running"
print "from p1: myVar = %d" % master.myVar
app/plugin/p2.py:
from .. import master
def test():
master.myVar = 2
print "test in p2 is running"
print "from p2, myVar: %d" % master.myVar
Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:
__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 17, in test2
m = __import__("p2", globals(), locals(), [], -1)
File "....../python/test1/app/plugin/p2.py", line 1, in <module>
from .. import master
ValueError: Attempted relative import in non-package
Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)
It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.
Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the
What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)
EDIT: I found the following possible, but unsuccessful solutions:
m = __import__("p2",globals(),locals(),"plugin")
(returns the same exact error as above)
m = __import__("plugin",fromlist="p2")
(returns a reference to app.plugin, not to app.plugin.p2)
m = __import__("plugin.p2",globals(),locals())
(returns a reference to app.plugin, not to app.plugin.p2)
import importlib
m = importlib.import_module("plugin.p2")
(returns:)
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 20, in test2
m = importlib.import_module("plugin.p2")
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named plugin.p2
I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty.
You should use importlib instead
import importlib
p2 = importlib.import_module('plugin.p2')
Have you tried the following syntax:
How to use python's import function properly __import__()
It worked for me with a similar problem...
I never did find a solution, so I ended up deciding to restructure the program.
What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.
This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.
This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.
Here's basically what I did:
main.py:
import os, os.path, sys
class MyApp:
_plugins = []
def __init__(self):
self.myVar = 0
def loadPlugins(self):
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
sys.path.insert(0,scriptDir)
for plug in os.listdir(scriptDir):
if (plug[-3:].lower() == ".py"):
m = __import__(os.path.basename(plug)[:-3])
self._plugins.append(m.Plugin(self))
def runTests(self):
for p in self._plugins:
p.test()
if (__name__ == "__main__"):
app = MyApp()
app.loadPlugins()
app.runTests()
plugin/p1.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p1: myVar = %d" % self.host.myVar
plugin/p2.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p2: variable set"
self.host.myVar = 1
print "from p2: myVar = %d" % self.host.myVar
There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.
I have managed to find a solution to the problem.
By taking your example the following static import is needed to be dynamic
from .plugin import p2
the "." near plugin means there is a need to relative import and not absolute import.
I was able to do that with the following code snipset:
plugin = __import__('plugin', globals(), locals(), level=1, fromlist=['p2'])
p2 = getattr(plugin, 'p2')
level=1 Relative import parameter
fromlist Specify which sub modules to take from plugin module
As you mentioned, plugin holds the reference to 'plugin', thus additional getattr is needed to grep p2 from plugin
Inside a package I am using a few checks inside each module directory's __init__.py to see whether the environment is sane and then use from . import mod1 for nested modules (mod1 here). Each of the module directories in turn contains an __init__.py (obviously) which will import the next lower level (if any).
Some of the __init__.py contain __all__ arrays to only expose certain names upon import.
Let's say I have the following "main" script (spam.py):
import os, sys
if (sys.version_info[0] != 2) or (sys.version_info < (2,7)):
sys.exit("...")
else:
import pkgname
def main():
pass
if __name__ == '__main__':
main()
and the following pkgname/__init__.py:
import sys
if __name__ == '__main__':
sys.exit(0)
else:
from . import db
from os.path import dirname, realpath
sys.modules[__name__].__all__ = ['inipath']
sys.modules[__name__].__dict__['inipath'] = dirname(realpath(__file__)) + '.ini'
and the following pkgname/db/__init__.py:
import sys
if __name__ == '__main__':
sys.exit(0)
else:
import sqlite3
# ...
foobar = { 'spam' : 'egg' }
__all__ = ["foobar"]
will the symbol pkgname.db.foobar be visible despite the __all__ array in pkgname/__init__.py? I.e. does the __all__ array only affect the immediate module or also any lower level?
Which rules govern Python's behavior here? I tried some searches but came up empty-handed. Possibly I used the wrong searches (__all__, python, relative import)?
__all__ has no effect in the case you're describing. It only affects what happens when you do from package import *. See the documentation as well as previous questions (which I found by googling python __all__).