Inside a package I am using a few checks inside each module directory's __init__.py to see whether the environment is sane and then use from . import mod1 for nested modules (mod1 here). Each of the module directories in turn contains an __init__.py (obviously) which will import the next lower level (if any).
Some of the __init__.py contain __all__ arrays to only expose certain names upon import.
Let's say I have the following "main" script (spam.py):
import os, sys
if (sys.version_info[0] != 2) or (sys.version_info < (2,7)):
sys.exit("...")
else:
import pkgname
def main():
pass
if __name__ == '__main__':
main()
and the following pkgname/__init__.py:
import sys
if __name__ == '__main__':
sys.exit(0)
else:
from . import db
from os.path import dirname, realpath
sys.modules[__name__].__all__ = ['inipath']
sys.modules[__name__].__dict__['inipath'] = dirname(realpath(__file__)) + '.ini'
and the following pkgname/db/__init__.py:
import sys
if __name__ == '__main__':
sys.exit(0)
else:
import sqlite3
# ...
foobar = { 'spam' : 'egg' }
__all__ = ["foobar"]
will the symbol pkgname.db.foobar be visible despite the __all__ array in pkgname/__init__.py? I.e. does the __all__ array only affect the immediate module or also any lower level?
Which rules govern Python's behavior here? I tried some searches but came up empty-handed. Possibly I used the wrong searches (__all__, python, relative import)?
__all__ has no effect in the case you're describing. It only affects what happens when you do from package import *. See the documentation as well as previous questions (which I found by googling python __all__).
Related
Given the following files:
a.py
-----
class CommonClass(object):
def do_thing(self):
pass
b.py
-----
from a import CommonClass
class SubClassA(CommonClass):
def do_thing(self):
print("I am A")
class SubClassB(CommonClass):
def do_thing(self):
print("I am B")
c.py
-----
from a import CommonClass
from b import SubClassA
if __name__ == "__main__":
for member in CommonClass.__subclasses__():
member().do_thing()
I would expect only SubClassA is imported, and visible when looping through subclasses of CommonClass, but it seems SubClassB is imported as well.
I am running Python 3.8.5 and this is the output of python3 c.py:
$ python3 c.py
I am A
I am B
How can I only import the classes I want?
You did import only SubClassA to c.py. This can be tested by doing
x = SubClassB()
or
x = b.SubClassB()
Both will result in a NameError.
The thing is, that when you import a file, it is actually being ran, even when using from x import y!
This can be easily seen by adding a print("I'm from b.py") at the end of b.py and then running c.py.
This makes both SubClassA and SubClassB be subclasses of CommonClass, which you imported. So, while you don't have access to the SubClassB name, it is still a subclass of CommonClass and you can access it from there.
In general you don't have to import a module to be able to use its objects. Importing expands your namespace to include that module so you can create an object directly. You can still use this module's objects even without importing it if you acquired them in some other way (like importing a third module which returns objects from the second one).
Anyway right now, you are not really using even the imported SubClassA. If you want to "allow" certain classes to be considered only from an external source, you can create an allowed set of classes:
from a import CommonClass
from b import SubClassA
allowed_classes = {SubClassA}
if __name__ == "__main__":
for member in CommonClass.__subclasses__():
if member in allowed_classes:
member().do_thing()
Which only prints I am A
from a import CommonClass
from b import SubClassA
if __name__ == "__main__":
h = CommonClass.__subclasses__()[0]()
h.do_thing()
You can do this.
The main file is aaa.py, code as below:
import os,time
from get_newest_folder import *
def main_test():
print(newest_folder)
print(path)
if __name__ == '__main__':
get_n_folder()
main_test()
The another file is get_newest_folder.py, code as below:
import os
def get_n_folder():
path = 'D:\\Test'
os.chdir(path)
files = sorted(os.listdir(os.getcwd()), key=os.path.getmtime)
newest_folder = files[-1]
return (path, newest_folder)
if __name__ == '__main__':
get_n_folder()
I'm trying to use variable "path" and "newest_folder" from get_newest_folder.py in aaa.py, so I import "get_newest_folder". But I can't get(print) the 2 variables in aaa.py. It displays the error message as below when I run aaa.py:
NameError: name 'newest_folder' is not defined
Does anyone can correct my code or give me some suggestion?
Many thanks.
modify your aaa.py
import os,time
from get_newest_folder import *
def main_test():
path,newest_folder = get_n_folder()
print(newest_folder)
print(path)
if __name__ == '__main__':
main_test()
Inorder to access newest_folder and path, you have to declare them as global in get_newest_folder.py and import these variables in aaa.py. They are local variables currently.
But this is not an ideal way to get about things. May be you need something like this
import os,time
from get_newest_folder import *
def main_test():
_path, _newest_folder = get_n_folder()
print(_path)
print(_newest_folder)
if __name__ == '__main__':
main_test()
First of all, to answer your question, you need to modify your module:
import os
newest_folder = '' # <<<<
path = '' # <<<<
def get_n_folder():
global newest_folder # <<<<
global path # <<<<
....
# The rest is unchanged
The variables are attributes of the module now, previously they were only attributes of the function.
So that got rid of the error messages, but it just prints two blank lines!!! Why? Because newest_folder and path have not been setup, you don't call the function to populate the values!
That's one problem, the other is from get_newest_folder import *. Don't do that - it pollutes your own namespace. It gives the opportunity for a module to overwrite your own variables (namespace collision). Fine, you know this module, but will you always? Do you know very variable in every module that you import?
So:
import get_newest_folder
def main_test():
print(get_newest_folder.newest_folder)
print(get_newest_folder.path)
if __name__ == '__main__':
get_newest_folder.get_n_folder()
main_test()
But, as others have said, it is better not to share variables, if you do it means that the module implementation cannot change - it breaks encapsulation. There are a few occasions where it might be justified, but not here.
Instead, using your original get_newest_folder.py:
import os,time
import get_newest_folder
def main_test():
print(newest_folder)
print(path)
if __name__ == '__main__':
print(dir())
newest_folder, path = get_newest_folder.get_n_folder()
main_test()
Hello I did got into circular dependency what is not refactori-zable other than doubling code.
I have something like this (only much more complex):
myParser.py:
import sys
import main #comment this to make it runnable
def parseEvnt():
sys.stdout.write("evnt:")
main.parseCmd(1) #comment this to make it runnable
tbl.py:
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
main.py:
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
data=[1,2]
for d in data:
if(d<2):
parseCmd(d)
else:
fce = tbl.tblx[d][1]
fce()
Obvious error I'm getting is:
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 1, in <module>
import tbl
File "D:\Data\vbe\workspace\marsPython\testCircular2\tbl.py", line 1, in <module>
import myParser
File "D:\Data\vbe\workspace\marsPython\testCircular2\myParser.py", line 2, in <module>
import main
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 7, in <module>
parseCmd(d)
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 3, in parseCmd
print(tbl.tblx[d][0])
AttributeError: module 'tbl' has no attribute 'tblx'
In C I think I would just tell by declaration in tbl.py hey there is function parseEvnt(). I would not need to include myParser and there would be no circular include.
In python I do not know how to do it.
I read few threads and there is always some wise guy recommending refactorizing. But in this case parseCmd() needs to see tblx which needs to see parseEvnt() (unless function declaration) and parseEvnt() need to call parseCmd() (cos evnt contains triggering cmd and I do not want to double the decoding cmd code).
Is there way how to make it working in python?
You can frequently get away with circular dependencies as long as the modules don't try to use each other's data until all importing is complete - as a practical matter, that means referencing with namespace (from module import something is forbidden) and only using the other modules inside functions and methods (no mystuff = module.mystuff in the global space). That's because when importing starts, python puts the module name in sys.modules and won't try to import that module again.
You ran into trouble because when you run main.py, python adds __main__ to sys.modules. When the code finally came around to import main, there was no "main" in the module list and so main.py was imported again... and its top level code tried to run.
Lets rearrange your test case and throw in a few print statements to tell when import happens.
myParser.py
print(' + importing myParser')
import sys
print('import parsecmd')
import parsecmd
def parseEvnt():
sys.stdout.write("evnt:")
parsecmd.parseCmd(1)
tbl.py
print(' + importing tbl')
print('import myParser')
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
Parsecmd.py (new)
print(' + importing parsecmd')
print('import tbl')
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
main.py
print('running main.py')
print('import parsecmd')
import parsecmd
if __name__ == "__main__":
data=[1,2]
for d in data:
if(d<2):
parsecmd.parseCmd(d)
else:
fce = parsecmd.tbl.tblx[d][1]
fce()
When I run it I get
running main.py
import parsecmd
+ importing parsecmd
import tbl
+ importing tbl
import myParser
+ importing myParser
import parsecmd <-- didn't reimport parsecmd
cmd
evnt:cmd
If you're insistent on not refactoring (which is the real solution to this - not being a wise guy), you could move your problematic import into your function in myParser.py
import sys
def parseEvnt():
import main ## import moved into function
sys.stdout.write("evnt:")
main.parseCmd(1)
Again, see if you can redesign your code so such interdependencies are avoided.
The above solution is sort of a hack and won't solve future problems you might run into due to this dependency.
Circular imports should be avoided. Refactoring is required, any workaround that still requires a circular import is not a good solution.
That being said, the refactoring doesn't have to be extensive. There are at least a couple of fairly simple solutions.
Solution 1: move shared functions to a shared module
Since you want to use parseCmd from more than one place, move it to a separate file. That way both main.py and myParser.py can import the function.
Solution 2: have main pass parseCmd to parseEvnt.
First, make parseEvnt accept an argument to tell it which function to run:
# myParser.py
import sys
def parseEvnt(parseCmd):
sys.stdout.write("evnt:")
parseCmd(1)
Next, when you call myParser.parseEvnt, pass in a reference to main.parseCmd:
# main.py:
...
else:
fce = tbl.tblx[d][1]
fce(parseCmd)
There are other ways to accomplish the same thing. For example, you could add a "configure" method in myParser, and then have main.py call the configure method and pass in a reference to its parseCmd. The configure method can then store this reference in a global variable.
Another choice is to import main in the function that uses it:
main.py
import sys
def parseEvnt():
import main
sys.stdout.write("evnt:")
main.parseCmd(1)
Here are the files in this test:
main.py
app/
|- __init__.py
|- master.py
|- plugin/
|- |- __init__.py
|- |- p1.py
|- |_ p2.py
The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.
I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.
Here's the file contents:
main.py:
import app
app.test()
app.test2()
app/__init__.py:
import sys, os
from plugin import p1
def test():
print "__init__ in app is executing test"
p1.test()
def test2():
print "__init__ in app is executing test2"
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
print "The scriptdir is %s" % scriptDir
sys.path.insert(0,scriptDir)
m = __import__("p2", globals(), locals(), [], -1)
m.test()
app/master.py:
myVar = 0
app/plugin/__init__.py:
<empty file>
app/plugin/p1.py:
from .. import master
def test():
print "test in p1 is running"
print "from p1: myVar = %d" % master.myVar
app/plugin/p2.py:
from .. import master
def test():
master.myVar = 2
print "test in p2 is running"
print "from p2, myVar: %d" % master.myVar
Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:
__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 17, in test2
m = __import__("p2", globals(), locals(), [], -1)
File "....../python/test1/app/plugin/p2.py", line 1, in <module>
from .. import master
ValueError: Attempted relative import in non-package
Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)
It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.
Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the
What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)
EDIT: I found the following possible, but unsuccessful solutions:
m = __import__("p2",globals(),locals(),"plugin")
(returns the same exact error as above)
m = __import__("plugin",fromlist="p2")
(returns a reference to app.plugin, not to app.plugin.p2)
m = __import__("plugin.p2",globals(),locals())
(returns a reference to app.plugin, not to app.plugin.p2)
import importlib
m = importlib.import_module("plugin.p2")
(returns:)
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 20, in test2
m = importlib.import_module("plugin.p2")
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named plugin.p2
I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty.
You should use importlib instead
import importlib
p2 = importlib.import_module('plugin.p2')
Have you tried the following syntax:
How to use python's import function properly __import__()
It worked for me with a similar problem...
I never did find a solution, so I ended up deciding to restructure the program.
What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.
This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.
This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.
Here's basically what I did:
main.py:
import os, os.path, sys
class MyApp:
_plugins = []
def __init__(self):
self.myVar = 0
def loadPlugins(self):
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
sys.path.insert(0,scriptDir)
for plug in os.listdir(scriptDir):
if (plug[-3:].lower() == ".py"):
m = __import__(os.path.basename(plug)[:-3])
self._plugins.append(m.Plugin(self))
def runTests(self):
for p in self._plugins:
p.test()
if (__name__ == "__main__"):
app = MyApp()
app.loadPlugins()
app.runTests()
plugin/p1.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p1: myVar = %d" % self.host.myVar
plugin/p2.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p2: variable set"
self.host.myVar = 1
print "from p2: myVar = %d" % self.host.myVar
There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.
I have managed to find a solution to the problem.
By taking your example the following static import is needed to be dynamic
from .plugin import p2
the "." near plugin means there is a need to relative import and not absolute import.
I was able to do that with the following code snipset:
plugin = __import__('plugin', globals(), locals(), level=1, fromlist=['p2'])
p2 = getattr(plugin, 'p2')
level=1 Relative import parameter
fromlist Specify which sub modules to take from plugin module
As you mentioned, plugin holds the reference to 'plugin', thus additional getattr is needed to grep p2 from plugin
Desired directory tree:
Fibo
|-- src
| `-- Fibo.py
`-- test
`-- main.py
What I want is to call python main.py after cd'ing into test and executing main.py will run all the unit tests for this package.
Currently if I do:
import Fibo
def main():
Fibo.fib(100)
if __name__ == "__main__":
main()
I get an error: "ImportError: No module named Fibo".
But if I do:
import sys
def main():
sys.path.append("/home/tsmith/svn/usefuldsp/trunk/Labs/Fibo/src")
import Fibo
Fibo.fib(100)
if __name__ == "__main__":
main()
This seems to fix my error. And I could move forward... but this isn't a python package. This is more of a "collection of files" approach.
How would you setup your testing to work in this directory structure?
If I want to import a module that lives at a fixed, relative location to the file I'm evaluating, I often do something like this:
try:
import Fibo
except ImportError:
import sys
from os.path import join, abspath, dirname
parentpath = abspath(join(dirname(__file__), '..'))
srcpath = join(parentpath, 'src')
sys.path.append(srcpath)
import Fibo
def main():
Fibo.fib(100)
if __name__ == "__main__":
main()
If you want to be a good namespace-citizen, you could del the no longer needed symbols at the end of the except block.
Adding /home/tsmith/svn/usefuldsp/trunk/Labs/Fibo/src to your PYTHONPATH environment variable
would allow you to write
import Fibo
def main():
Fibo.fib(100)
if __name__ == "__main__":
main()
and have it import .../Fibo/src/Fibo.py correctly.
Quick and dirty way: create a symbolic link