From a main fill (or another class file), I would instantiate a class and call a method.
Knowing many languages and learning Python: this must be very easy, but could not find a good example. So, if you show me another post or answer, please feel free.
My config is:
All files are in the same folder (main.py and MathsOperations.py)
Python 3.7
IDE: Pycharm 2020.2 (just upgraded)
Starting (Running) the main.py in the folder.
The Class file:
class MathsOperations:
def __init__ (self, x, y):
self.a = x
self.b = y
def testAddition (self):
return (self.a + self.b)
def testMultiplication (self):
return (self.a * self.b)
Try-1: The main file:
from . import MathsOperations
if __name__ == '__main__':
xyz = MathsOperations(2, 3)
print( xyz.testAddition())
I get the error: ImportError: cannot import name 'MathsOperations' from '__main__'
Try-2: import in
import MathsOperations;
if __name__ == '__main__':
xyz = MathsOperations(2, 3)
print( xyz.testAddition())
Gives error: No module named MathOperations
Try-3: using dot notation with file
from .MathsOperations import MathsOperations
if __name__ == '__main__':
xyz = MathsOperations(2, 3)
print( xyz.testAddition())
Gives error: ModuleNotFoundError: No module named '__main__.MathsOperations'; '__main__' is not a package
SOLUTION: using from and import of python file
This works thanks to #Surya R and #Chad S.
from MathsOperations import MathsOperations
if __name__ == '__main__':
xyz = MathsOperations(2, 3)
print( xyz.testAddition())
This gives a WARNING in PyCharm but that one can be ignored!
to solve this you have to understand the dot in the line 1 which is important
The dot in the module name is used for relative module import
You can use more than one dot, referring not to the current package but its parent(s). This should only be used within packages, in the main module one should always use absolute module names.
let me know your folder's structure so that we could find the right import statement
incase to import a file
import file
you can use
from yourfilenmae import yourclassname
if you wanna use dot things refer this one dot syntax to import files
Related
Given the following files:
a.py
-----
class CommonClass(object):
def do_thing(self):
pass
b.py
-----
from a import CommonClass
class SubClassA(CommonClass):
def do_thing(self):
print("I am A")
class SubClassB(CommonClass):
def do_thing(self):
print("I am B")
c.py
-----
from a import CommonClass
from b import SubClassA
if __name__ == "__main__":
for member in CommonClass.__subclasses__():
member().do_thing()
I would expect only SubClassA is imported, and visible when looping through subclasses of CommonClass, but it seems SubClassB is imported as well.
I am running Python 3.8.5 and this is the output of python3 c.py:
$ python3 c.py
I am A
I am B
How can I only import the classes I want?
You did import only SubClassA to c.py. This can be tested by doing
x = SubClassB()
or
x = b.SubClassB()
Both will result in a NameError.
The thing is, that when you import a file, it is actually being ran, even when using from x import y!
This can be easily seen by adding a print("I'm from b.py") at the end of b.py and then running c.py.
This makes both SubClassA and SubClassB be subclasses of CommonClass, which you imported. So, while you don't have access to the SubClassB name, it is still a subclass of CommonClass and you can access it from there.
In general you don't have to import a module to be able to use its objects. Importing expands your namespace to include that module so you can create an object directly. You can still use this module's objects even without importing it if you acquired them in some other way (like importing a third module which returns objects from the second one).
Anyway right now, you are not really using even the imported SubClassA. If you want to "allow" certain classes to be considered only from an external source, you can create an allowed set of classes:
from a import CommonClass
from b import SubClassA
allowed_classes = {SubClassA}
if __name__ == "__main__":
for member in CommonClass.__subclasses__():
if member in allowed_classes:
member().do_thing()
Which only prints I am A
from a import CommonClass
from b import SubClassA
if __name__ == "__main__":
h = CommonClass.__subclasses__()[0]()
h.do_thing()
You can do this.
Is it possible to overwrite the definition of a class so that all calls (inside my program or other .py files) receive the overwrite? For example:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run()
# other_package.py
from package import orgClass
def run():
o = orgClass()
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
I need to alter the implementation of the orgClass for this specific program, but I don't want to change source code (of the original package; seems like a bad idea anyhow) so I figured I could just "rewrite" the class implementation in my_prog.py and that would take affect to all calls that came from the __main__.thread.
You have a few options here.
Option 1. You can modify the class object directly (monkey-patching):
import package
def new_some_method(self, x):
return x + x
package.orgClass.some_method = new_some_method
(Variants on this include monkey-patching the whole orgClass object, to similar effect.)
Option 2. You could create a replacement module, and then modify sys.path to force it to load first:
# replacement-packages/package.py
class orgClass:
def some_method(self, x):
return x + x
# my_prog.py
sys.path = ['path/to/replacement-packages'] + sys.path
import package # loads replacement-packages/package.py
(Variants on this include using path files or various shims in __init__.py, or modifying sys.modules directly.)
Option 3. You could fork the whole package project, and modify it to suit your needs. Then you could use a virtualenv or some other set-up to only install the modified package as a dependency of your my_prog program, while other programs would use the default python interpreter which would load the original package.
To the best of my knowledge, you have 2 options:
Edit the original file package.py.
Make every file that requires the new method import my_prog.py.
Since you don't want the first option, the answer is No - you cannot do that.
If I understand correctly you want to
1) import orgClass
2) adjust orgClass
#my_classes.py
import orgClass
class orgClass(orgClass):
def some_method(x):
return x+x
edit:
To then use this function use
from my_classes import orgClass
edit:
Now I see your importing is different, then it might be done using argument such as:
# package.py
class orgClass:
def some_method(x):
return x * x
# my_prog.py
import other_package
from package import orgClass
class orgClass(orgClass):
def some_method(x):
return x + x
if __name__ == '__main__':
other_package.run(orgClass)
# other_package.py
def run(o):
x = 5
print (o.some_method(x)) # would print 10 not 25; however it prints 25
I'm trying to use a custom accumulator class as per the Spark documentation. This works if I define the class locally, but when I try define it in another module and import the file using sc.addPyFile I get an ImportError.
I had the same issue when importing a helper function within an rdd.foreach, which I was able to resolve by performing the import within the foreach'd function (example to follow) as per this SO question. However, the same fix doesn't work for the custom accumulator (and I wouldn't really expect it to).
tl;dr: What is the proper way to import a custom accumulator class?
extensions/accumulators.py:
class ArrayAccumulatorParam(pyspark.AccumulatorParam):
def zero(self, initialValue):
return numpy.zeros(initialValue.shape)
def addInPlace(self, a, b):
a += b
return a
run/count.py:
from extensions.accumulators import ArrayAccumulatorParam
def main(sc):
sc.addPyFile(LIBRARY_PATH + '/import_/logs.py')
sc.addPyFile(LIBRARY_PATH + '/extensions/accumulators.py')
rdd = sc.textFile(LOGS_PATH)
accum = sc.accumulator(numpy.zeros(DIMENSIONS), ArrayAccumulatorParam())
def count(row)
import logs # This 'internal import' seems to be required to avoid ImportError for the 'logs' module
from extensions.accumulators import ArrayAccumulatorParam # Error is thrown both with and without this line
val = logs.parse(row)
accum.add(val)
rdd.foreach(count) # Throws ImportError: No module named extensions.accumulators
if __name__ == '__main__':
conf = pyspark.SparkConf().setAppName('SOME_COUNT_JOB')
sc = pyspark.SparkContext(conf=conf)
main(sc)
Error:
ImportError: No module named extensions.accumulators
Hello I did got into circular dependency what is not refactori-zable other than doubling code.
I have something like this (only much more complex):
myParser.py:
import sys
import main #comment this to make it runnable
def parseEvnt():
sys.stdout.write("evnt:")
main.parseCmd(1) #comment this to make it runnable
tbl.py:
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
main.py:
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
data=[1,2]
for d in data:
if(d<2):
parseCmd(d)
else:
fce = tbl.tblx[d][1]
fce()
Obvious error I'm getting is:
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 1, in <module>
import tbl
File "D:\Data\vbe\workspace\marsPython\testCircular2\tbl.py", line 1, in <module>
import myParser
File "D:\Data\vbe\workspace\marsPython\testCircular2\myParser.py", line 2, in <module>
import main
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 7, in <module>
parseCmd(d)
File "D:\Data\vbe\workspace\marsPython\testCircular2\main.py", line 3, in parseCmd
print(tbl.tblx[d][0])
AttributeError: module 'tbl' has no attribute 'tblx'
In C I think I would just tell by declaration in tbl.py hey there is function parseEvnt(). I would not need to include myParser and there would be no circular include.
In python I do not know how to do it.
I read few threads and there is always some wise guy recommending refactorizing. But in this case parseCmd() needs to see tblx which needs to see parseEvnt() (unless function declaration) and parseEvnt() need to call parseCmd() (cos evnt contains triggering cmd and I do not want to double the decoding cmd code).
Is there way how to make it working in python?
You can frequently get away with circular dependencies as long as the modules don't try to use each other's data until all importing is complete - as a practical matter, that means referencing with namespace (from module import something is forbidden) and only using the other modules inside functions and methods (no mystuff = module.mystuff in the global space). That's because when importing starts, python puts the module name in sys.modules and won't try to import that module again.
You ran into trouble because when you run main.py, python adds __main__ to sys.modules. When the code finally came around to import main, there was no "main" in the module list and so main.py was imported again... and its top level code tried to run.
Lets rearrange your test case and throw in a few print statements to tell when import happens.
myParser.py
print(' + importing myParser')
import sys
print('import parsecmd')
import parsecmd
def parseEvnt():
sys.stdout.write("evnt:")
parsecmd.parseCmd(1)
tbl.py
print(' + importing tbl')
print('import myParser')
import myParser
tblx = {
1:("cmd",),
2:("evnt",myParser.parseEvnt),
}
Parsecmd.py (new)
print(' + importing parsecmd')
print('import tbl')
import tbl
def parseCmd(d):
print(tbl.tblx[d][0])
main.py
print('running main.py')
print('import parsecmd')
import parsecmd
if __name__ == "__main__":
data=[1,2]
for d in data:
if(d<2):
parsecmd.parseCmd(d)
else:
fce = parsecmd.tbl.tblx[d][1]
fce()
When I run it I get
running main.py
import parsecmd
+ importing parsecmd
import tbl
+ importing tbl
import myParser
+ importing myParser
import parsecmd <-- didn't reimport parsecmd
cmd
evnt:cmd
If you're insistent on not refactoring (which is the real solution to this - not being a wise guy), you could move your problematic import into your function in myParser.py
import sys
def parseEvnt():
import main ## import moved into function
sys.stdout.write("evnt:")
main.parseCmd(1)
Again, see if you can redesign your code so such interdependencies are avoided.
The above solution is sort of a hack and won't solve future problems you might run into due to this dependency.
Circular imports should be avoided. Refactoring is required, any workaround that still requires a circular import is not a good solution.
That being said, the refactoring doesn't have to be extensive. There are at least a couple of fairly simple solutions.
Solution 1: move shared functions to a shared module
Since you want to use parseCmd from more than one place, move it to a separate file. That way both main.py and myParser.py can import the function.
Solution 2: have main pass parseCmd to parseEvnt.
First, make parseEvnt accept an argument to tell it which function to run:
# myParser.py
import sys
def parseEvnt(parseCmd):
sys.stdout.write("evnt:")
parseCmd(1)
Next, when you call myParser.parseEvnt, pass in a reference to main.parseCmd:
# main.py:
...
else:
fce = tbl.tblx[d][1]
fce(parseCmd)
There are other ways to accomplish the same thing. For example, you could add a "configure" method in myParser, and then have main.py call the configure method and pass in a reference to its parseCmd. The configure method can then store this reference in a global variable.
Another choice is to import main in the function that uses it:
main.py
import sys
def parseEvnt():
import main
sys.stdout.write("evnt:")
main.parseCmd(1)
Here are the files in this test:
main.py
app/
|- __init__.py
|- master.py
|- plugin/
|- |- __init__.py
|- |- p1.py
|- |_ p2.py
The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.
I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.
Here's the file contents:
main.py:
import app
app.test()
app.test2()
app/__init__.py:
import sys, os
from plugin import p1
def test():
print "__init__ in app is executing test"
p1.test()
def test2():
print "__init__ in app is executing test2"
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
print "The scriptdir is %s" % scriptDir
sys.path.insert(0,scriptDir)
m = __import__("p2", globals(), locals(), [], -1)
m.test()
app/master.py:
myVar = 0
app/plugin/__init__.py:
<empty file>
app/plugin/p1.py:
from .. import master
def test():
print "test in p1 is running"
print "from p1: myVar = %d" % master.myVar
app/plugin/p2.py:
from .. import master
def test():
master.myVar = 2
print "test in p2 is running"
print "from p2, myVar: %d" % master.myVar
Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:
__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 17, in test2
m = __import__("p2", globals(), locals(), [], -1)
File "....../python/test1/app/plugin/p2.py", line 1, in <module>
from .. import master
ValueError: Attempted relative import in non-package
Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)
It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.
Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the
What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)
EDIT: I found the following possible, but unsuccessful solutions:
m = __import__("p2",globals(),locals(),"plugin")
(returns the same exact error as above)
m = __import__("plugin",fromlist="p2")
(returns a reference to app.plugin, not to app.plugin.p2)
m = __import__("plugin.p2",globals(),locals())
(returns a reference to app.plugin, not to app.plugin.p2)
import importlib
m = importlib.import_module("plugin.p2")
(returns:)
Traceback (most recent call last):
File "main.py", line 4, in <module>
app.test2()
File "....../python/test1/app/__init__.py", line 20, in test2
m = importlib.import_module("plugin.p2")
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named plugin.p2
I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty.
You should use importlib instead
import importlib
p2 = importlib.import_module('plugin.p2')
Have you tried the following syntax:
How to use python's import function properly __import__()
It worked for me with a similar problem...
I never did find a solution, so I ended up deciding to restructure the program.
What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.
This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.
This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.
Here's basically what I did:
main.py:
import os, os.path, sys
class MyApp:
_plugins = []
def __init__(self):
self.myVar = 0
def loadPlugins(self):
scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
sys.path.insert(0,scriptDir)
for plug in os.listdir(scriptDir):
if (plug[-3:].lower() == ".py"):
m = __import__(os.path.basename(plug)[:-3])
self._plugins.append(m.Plugin(self))
def runTests(self):
for p in self._plugins:
p.test()
if (__name__ == "__main__"):
app = MyApp()
app.loadPlugins()
app.runTests()
plugin/p1.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p1: myVar = %d" % self.host.myVar
plugin/p2.py:
class Plugin:
def __init__(self, host):
self.host = host
def test(self):
print "from p2: variable set"
self.host.myVar = 1
print "from p2: myVar = %d" % self.host.myVar
There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.
I have managed to find a solution to the problem.
By taking your example the following static import is needed to be dynamic
from .plugin import p2
the "." near plugin means there is a need to relative import and not absolute import.
I was able to do that with the following code snipset:
plugin = __import__('plugin', globals(), locals(), level=1, fromlist=['p2'])
p2 = getattr(plugin, 'p2')
level=1 Relative import parameter
fromlist Specify which sub modules to take from plugin module
As you mentioned, plugin holds the reference to 'plugin', thus additional getattr is needed to grep p2 from plugin