This question is related to these other posts on SO, yet the solutions suggested therein do not seem to work for my case. In short, my problem can be illustrated by the following example. I have an Algebra class where by the method triPower I aim at computing the power of a trinomial, i.e. (a+b+c)**n for many n values with fixed a, b, c. To do so, I thought of creating a method _triPower(a,b,c,n) and pass it to my pool.map() function by functools.partial(_triPower,...) where I fix a, b, c and leave n as the only parameter, since I am working in Python 2.7 and map from the multiprocessing module wants only one argument function (see otherwise this post). The code is the following:
from __future__ import division
import numpy as np
import functools as fntls
import multiprocessing as mp
import multiprocessing.pool as mppl
# A couple of classes introduced to allow multiple processes to have their own daemons (parallelization)
class NoDaemonProcess(mp.Process):
# make 'daemon' attribute always return False
def _get_daemon(self):
return False
def _set_daemon(self, value):
pass
daemon = property(_get_daemon, _set_daemon)
# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(mppl.Pool):
Process = NoDaemonProcess
# Sample class where I want a method to run a parallel loop
class Algebra(object):
def __init__(self,offset):
self.offset = offset
def trinomial(self,a,b,c):
return a+b+c
def _triPower(self,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
return self.offset + self.trinomial(a,b,c)**n
def triPower(self,n):
pls = MyPool(4)
vals = pls.map(fntls.partial(self._triPower,a=1.,b=0.,c=1.),n)
print vals
# Testing
if __name__ == "__main__":
A = Algebra(0.)
A.triPower(np.arange(0.,10.,0.01))
The above does not work and produces (as expected from this post) the error:
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed
Hence, following the same post, I tried to define _triPower as a global function i.e.
def _triPower(alg,a,b,c,n):
"""This is the method that I want to run in parallel from the next method"""
return alg.trinomial(a,b,c)**n
and then editing Algebra.triPower(...) according to:
def triPower(self,n):
pls = MyPool(4)
vals = pls.map(fntls.partial(_triPower, alg=self, a=1., b=0., c=1.), n)
print vals
and this latter instead gives some weird TypeError like:
TypeError: _triPower() got multiple values for keyword argument 'alg'
On the other hand the suggestion to try to make the methods serializable by VeryPicklableObject as in this other post seems also not to work and this package appears dead by now (as of 05/2019). So what am I doing wrong and how can I make my computation run in parallel?
Related
I'm trying to run a basic MATLAB script that defines a class, and get that class object returned to python. I don't know MATLAB well, and am very new to Oct2Py so I may be completely misunderstanding how to do this. Any help would be greatly appreciated.
Here is the Matlab file (taken from here)
classdef BasicClass
properties
Value {mustBeNumeric}
end
methods
function r = roundOff(obj)
r = round([obj.Value],2);
end
function r = multiplyBy(obj,n)
r = [obj.Value] * n;
end
end
end
And I call this in the python script with the following
from oct2py import octave
octave.addpath(r'C:\Users\i13500020\.spyder-py3\IST')
oclass = octave.class_example(nout=1)
when I run this I get a warning that prints four times and then an error message
First:
warning: struct: converting a classdef object into a struct overrides the access restrictions defined for properties. All properties are returned, including private and protected ones.
And then:
TypeError: 'NoneType' object is not iterable
I don't have any trouble running the roundtrip example from the Oct2Py page, so I know my installation is fine
I've written a small work around to use custom matlab classes with oct2py.
Currently, this approach supports only access to member functions (and not attributes) of the Matlab class as this was what I needed:
from oct2py import octave
class MatlabClass():
_counter = 0
def __init__(self, objdef) -> None:
"""Use matlab object as python class.
Args:
objdef (str): Class initialization as string.
"""
MatlabClass._counter += 1
self.name = f"object_for_python{MatlabClass._counter}"
octave.eval(f"{self.name} = {objdef};")
def __getattr__(self, item):
"""Maps values to attributes.
Only called if there *isn't* an attribute with this name
"""
def f(*args):
call = f"{self.name}.{item}({','.join([str(arg) for arg in args])});"
return octave.eval(call)
return f
use this class as followed:
param = 0.24 # random value you might need for class initialization
oclass = MatlabClass(f"BasicClass({param})")
x = oclass.roundOff()
y = oclass.multiplyBy(2)
Note: you might need an init function in your octave code to run set your Value variable.
I have a method like this in Python :
def test(a,b):
return a+b, a-b
How can I run this in a background thread and wait until the function returns.
The problem is the method is pretty big and the project involves GUI, so I can't wait until it's return.
In my opinion, you should besides this thread run another thread that checks if there is result. Or Implement callback that is called at the end of the thread. However, since you have gui, which as far as I know is simply a class -> you can store result into obj/class variable and check if the result came.
I would use mutable variable, which is sometimes used. Lets create special class which will be used for storing results from thread functions.
import threading
import time
class ResultContainer:
results = [] # Mutable - anything inside this list will be accesable anywher in your program
# Lets use decorator with argument
# This way it wont break your function
def save_result(cls):
def decorator(func):
def wrapper(*args,**kwargs):
# get result from the function
func_result = func(*args,**kwargs)
# Pass the result into mutable list in our ResultContainer class
cls.results.append(func_result)
# Return result from the function
return func_result
return wrapper
return decorator
# as argument to decorator, add the class with mutable list
#save_result(ResultContainer)
def func(a,b):
time.sleep(3)
return a,b
th = threading.Thread(target=func,args=(1,2))
th.daemon = True
th.start()
while not ResultContainer.results:
time.sleep(1)
print(ResultContainer.results)
So, in this code, we have class ResultContainer with list. Whatever you put in it, you can easily access it from anywhere in the code (between threads and etc... exception is between processes due to GIL). I made decorator, so you can store result from any function without violating the function. This is just example how you can run threads and leave it to store result itself without you taking care of it. All you have to do, is to check, if the result arrived.
You can use global variables, to do the same thing. But I dont advise you to use them. They are ugly and you have to be very careful when using them.
For even more simplicity, if you dont mind violating your function, you can just, without using decorator, just push result to class with list directly in the function, like this:
def func(a,b):
time.sleep(3)
ResultContainer.results.append(tuple(a,b))
return a,b
If everything is an object in python then why does the following not work?
class Hello(object):
def hello(self):
print(x)
h = Hello()
h.hello.x = "hello world"
When doing this I get:
AttributeError: 'instancemethod' object has no attribute 'value'
A way which I can achieve this is by using partial however I am not sure what the effects of that would be on an object. Is there another way to achieve this?
from functools import partial
class Hello(object):
def hello(self, x):
print(x)
h = Hello()
newMethod = partial(h.hello, "helloworld")
newMethod()
Okay, from the comments below, people want a "real" scenario one example which can be considered as the following:
file.txt contains the following list
["ItemID1", "ItemID2", "ItemID3", "ItemID4"]
def BindMethods(self):
# ["ItemID1", "ItemID2", "ItemID3", "ItemID4"]
items = readFromFile("file.txt")
self.bind(AN_EVENT, GenericMethod)
def GenericMethod(self, i, event):
# The person calling this method knows that the method
# only accepts a single argument, not two.
# I am attempting to preserve the interface which I do not have control over
data = generateDataFromID(i)
table[i] = data
The idea here being that you bind a method to an event whereby the caller is expecting the method of a specific signature i.e. a single input argument.
But at the point of binding you want to pass in some extra information down to the handler. You don't know how much information you want to pass down since it is variable.
A expanded/compile time version of the same thing would be something like this:
file.txt contains the following list
["ItemID1", "ItemID2", "ItemID3", "ItemID4"]
def BindMethods(self):
# ["ItemID1", "ItemID2", "ItemID3", "ItemID4"]
items = readFromFile("file.txt")
self.bind(AN_EVENT, self.Item1Method)
self.bind(AN_EVENT, self.Item2Method)
self.bind(AN_EVENT, self.Item3Method)
......
def Item1Method(self, event):
data = generateDataFromID("Item1")
table["Item1ID"] = data
def Item2Method(self, event):
data = generateDataFromID("Item2")
table["Item2ID"] = data
This answer was referring to the question before it was editet; it might not be appropriate anymore.
Actually, there is a solution to this question. In python 3, you can directly assign attributes to bound methods, whereas in python 2.7 it is not that straightforward.
Look at a class definition like this:
class Hello(object):
def hello(self):
print(self.hello.x)
In regular use (instantiate and subsequently call the instance method hello()), the implementation would fail due to the missing attribute x:
h = Hello()
h.hello()
>>> AttributeError: 'function' object has no attribute 'x'
Changing the code a bit (Warning: Hack.) to something like this:
class Hello(object):
def hello(self):
if not hasattr(self.hello, 'x'):
self.hello.__dict__['x'] = 0
print(self.hello.x)
Would let you use the class implementation just as usual:
h = Hello()
h.hello()
>>> 0
print(h.hello.x)
>>> 0
Thus, you can bind attributes to bound methods, but the way you tried it only works in python 3.
Whatever this is good for, you may want to ask yourself whether it is the best way to achieve the desired behavior.
After your edit, this seems to me as if you seek a possibility to route certain events to certain functions, right?
Instead of binding new attributes to existing instance methods, you should think about setting up a clean and extendable event routing system.
A Basic Routing Core
While usually application of the singleton pattern always is a good point to ask oneself whether one is following the right concept, it totally makes sense here. The core of your routing is something similar to this:
def singleton(cls):
instance = cls()
instance.__call__ = lambda: instance
return instance
#singleton
class EventRouter(object):
def __init__(self):
self._routes = dict()
def bind(self, key, func):
self._routes[key] = func
def route(self, key):
if key in self._routes.keys():
return self._routes[key]()
else:
raise KeyError('Unbound routing key: {}'.format(key))
The EventRouter is a singleton class that can bind functions to certain keys and is furthermore able to route occurring events to their desired destination functions.
You now kan bind an event to a desired function during runtime:
def my_event_function():
print('An event was handled.')
EventRouter.bind('E_MYEVT', my_event_function)
# If you are iterating over events (for example drawn from a file),
# you can direct them to their functions:
for e in ['E_MYEVT', 'E_WHATEVT', 'E_NOEVT']:
EventRouter.route(e)
A Routing Decorator
If required, you can set up another decorator that can be used to directly bind methods to a certain event type:
def eventroute(key):
def register_method(call):
EventRouter.bind(key, call)
return register_method
#eventroute('E_MYEVT')
def handle_myevt():
print('Handling event: E_MYEVT')
#eventroute('E_WHATEVT')
def handle_whatevt():
print('Handling event: E_WHATEVT')
if __name__ == '__main__':
for e in ['E_MYEVT', 'E_WHATEVT']:
EventRouter.route(e)
>>> Handling event: E_MYEVT
>>> Handling event: E_WHATEVT
Passing Arguments to the Route Endpoints
If you want to pass arguments to the endpoints, just extend the calls by optional keyword arguments **kwargs when forwarding the call:
In EventRouter class, change route method to:
def route(self, key, **kwargs):
if key in self._routes.keys():
return self._routes[key](**kwargs)
else:
raise KeyError('Unbound routing key: {}'.format(key))
For your endpoints, set the desired arguments. In this example, I use the argument filename:
#eventroute(...)
def handle_...(filename):
print('Handling event: ...')
print(filename)
And when dispatching your events, provide the required argument as kwarg:
events = ['E_MYEVT', 'E_WHATEVT']
for e in events:
EventRouter.route(e, filename='whatever.jpg')
>>> Handling event: E_MYEVT
>>> whatever.jpg
>>> Handling event: E_WHATEVT
>>> whatever.jpg
Where to go
This is a brief overview of just a very basic setup.
You can extend your EventRouter to be arbitrarily complex, e.g. add routing keys in a multi-layered hierarchy, or the like. Furthermore, you can add more arguments to your methods. You can even think about extending the eventroute decorator to define the kwargs that shall be passed to the endpoint function to keep the function signatures as generic as possible.
Might seem like overkill, but structures like these are pretty neat and can be extended by anyone who imports your library/script and wants to add her own endpoints for individual events.
The x in the first version is not defined: the way you write the code, it should be a global variable; the way you use it, it should be an attribute of the object. You want to do something like
class Hello(object):
def __init__(self, x):
self.x = x
def hello(self):
print(self.x)
h = Hello("Hello World")
For example, this method:
import pygame
import mapas
class ContenedoresPorCapa():
def __init__(self, layers[]):
self.mapasCapas = len(self.mapas.capas)
for i in range(mapasCapas):
return layers[i]
You don't need the [] in the parameter list:
import pygame
import mapas
class ContenedoresPorCapa:
def __init__(self, layers):
self.mapasCapas = len(self.mapas.capas)
for i in range(mapasCapas):
return layers[i]
Also, note that returning values from __init__ methods of classes (constructors) is meaningless (and will produce a runtime TypeError, as Aशwini चhaudhary kindly pointed out).
Also the () at the end of the class definition is redundant.
Furthermore, this for loop will only iterate once and then return immediately:
for i in range(mapasCapas):
return layers[i]
Unless you describe what your goal is, it seems impossible to offer you any further guidance (other than strongly recommending to read something like http://learnpythonthehardway.org).
I want to process a large for loop in parallel, and from what I have read the best way to do this is to use the multiprocessing library that comes standard with Python.
I have a list of around 40,000 objects, and I want to process them in parallel in a separate class. The reason for doing this in a separate class is mainly because of what I read here.
In one class I have all the objects in a list and via the multiprocessing.Pool and Pool.map functions I want to carry out parallel computations for each object by making it go through another class and return a value.
# ... some class that generates the list_objects
pool = multiprocessing.Pool(4)
results = pool.map(Parallel, self.list_objects)
And then I have a class which I want to process each object passed by the pool.map function:
class Parallel(object):
def __init__(self, args):
self.some_variable = args[0]
self.some_other_variable = args[1]
self.yet_another_variable = args[2]
self.result = None
def __call__(self):
self.result = self.calculate(self.some_variable)
The reason I have a call method is due to the post I linked before, yet I'm not sure I'm using it correctly as it seems to have no effect. I'm not getting the self.result value to be generated.
Any suggestions?
Thanks!
Use a plain function, not a class, when possible. Use a class only when there is a clear advantage to doing so.
If you really need to use a class, then given your setup, pass an instance of Parallel:
results = pool.map(Parallel(args), self.list_objects)
Since the instance has a __call__ method, the instance itself is callable, like a function.
By the way, the __call__ needs to accept an additional argument:
def __call__(self, val):
since pool.map is essentially going to call in parallel
p = Parallel(args)
result = []
for val in self.list_objects:
result.append(p(val))
Pool.map simply applies a function (actually, a callable) in parallel. It has no notion of objects or classes. Since you pass it a class, it simply calls __init__ - __call__ is never executed. You need to either call it explicitly from __init__ or use pool.map(Parallel.__call__, preinitialized_objects)