I'm trying to create threads to run a class method. However, when I try to pass one class to another, it tries to initialize the class and never gets threaded.
I'm taking a list of tuples and trying to pass that list to the cfThread class, along with the class method that I want to use. From here, I'd like to create a separate thread to run the classes method and take action on one of tuples from the list. The REPLACEME is a placeholder because the class is looking for a tuple but I don't have one to pass to it yet. My end goal is to be able to pass a target (class / function) to a thread class that can create it's own queue and manage the threads without having to manually do it.
Below is a simple example to hopefully do a better job of explaining what I'm trying to do.
#!/bin/python3.10
import concurrent.futures
class math:
def __init__(self, num) -> None:
self.num = num
def add(self):
return self.num[0] + self.num[1]
def sub(self):
return self.num[0] - self.num[1]
def mult(self):
return self.num[0] * self.num[1]
class cfThread:
def __init__(self, target, args):
self.target = target
self.args = args
def run(self):
results = []
with concurrent.futures.ThreadPoolExecutor(10) as execute:
threads = []
for num in self.args:
result = execute.submit(self.target, num)
threads.append(result)
for result in concurrent.futures.as_completed(threads):
results.append(result)
return results
if __name__ == '__main__':
numbers = [(1,2),(3,4),(5,6)]
results = cfThread(target=math(REPLACEME).add(), args=numbers).run()
print(results)
target has to be a callable; you want to wrap your call to add in a lambda expression.
results = cfThread(target=lambda x: math(x).add(), args=numbers)
Related
Given this example code where we have a series of log processors, I can't help feeling there ought to be a more pythonic/efficient way of deciding which log processor to use to process some data:
class Component1ErrorLogProcessor:
def process(logToProcess):
# Do something with the logs
pass
class Component2ErrorLogProcessor:
def process(logToProcess):
# Do something with the logs
pass
class LogProcessor:
def __init__(self):
self.component1 = Component1ErrorLogProcessor()
self.component2 = Component2ErrorLogProcessor()
def process_line(self, line, component):
if component == "Component1Log-" or component == "[Component1]":
self.component1.process_errors(line)
elif component == "Component2Log-" or component == "[Component2]":
self.component2.process_errors(line)
I'd personally use the idea of registry, so you map each class to component names.
There are a bunch of different ways to go about this, here's a quick example by using a base class:
class ComponentLogProcessor(object):
_Mapping = {}
#classmethod
def register(cls, *component_names):
for name in component_names:
cls._Mapping[name] = cls
#classmethod
def cls_from_component(cls, component):
return cls._Mapping[component]
class Component1ErrorLogProcessor(ComponentLogProcessor):
def process(logToProcess):
# Do something with the logs
pass
Component1ErrorLogProcessor.register('Component1Log-', '[Component1]')
class Component2ErrorLogProcessor(ComponentLogProcessor):
def process(logToProcess):
# Do something with the logs
pass
Component2ErrorLogProcessor.register('Component2Log-', '[Component2]')
class LogProcessor:
def process_line(self, line, component):
ComponentLogProcessor.cls_from_component(component).process_errors(line)
I recently started to work with Python's classes, since I need to work with it through the use of OTree, a Python framework used for online experiment.
In one file, I define the pages that I want to be created, using classes. So essentially, in the OTree system, each class corresponds to a new page. The thing is, all pages (so classes) are basically the same, at the exception to some two parameters, as shown in the following code:
class Task1(Page):
form_model = 'player'
form_fields = ['Envie_WordsList_Toy']
def is_displayed(self):
return self.round_number == self.participant.vars['task_rounds'][1]
def vars_for_template(player):
WordsList_Toy= Constants.WordsList_Toy.copy()
random.shuffle(WordsList_Toy)
return dict(
WordsList_Toy=WordsList_Toy
)
#staticmethod
def live_method(player, data):
player.WTP_WordsList_Toy = int(data)
def before_next_page(self):
self.participant.vars['Envie_WordsList_Toy'] = self.player.Envie_WordsList_Toy
self.participant.vars['WTP_WordsList_Toy'] = self.player.WTP_WordsList_Toy
So here, the only thing that would change would be the name of the class, as well as the suffix of the variable WordsList_ used throughout this code, which is Toy.
Naively, what I tried to do is to define a function that would take those two parameters, such as this:
def page_creation(Task_Number,name_type):
class Task+str(Task_Number)(Page):
form_model = 'player'
form_fields = ['Envie_WordsList_'+str(name_type)]
def is_displayed(self):
return self.round_number == self.participant.vars['task_rounds'][1]
def vars_for_template(player):
WordsList_+str(name_type) = Constants.WordsList+str(name_type).copy()
random.shuffle(WordsList_+str(name_type))
return dict(
WordsList_+str(name_type)=WordsList_+str(name_type)
)
#staticmethod
def live_method(player, data):
player.WTP_WordsList_+str(name_type) = int(data)
def before_next_page(self):
self.participant.vars['Envie_WordsList_+str(name_type)'] = self.player.Envie_WordsList_+str(name_type)
self.participant.vars['WTP_WordsList_+str(name_type)'] = self.player.WTP_WordsList_+str(name_type)
Obviously, it does not work since I have the feeling that it is not possible to construct variables (or classes identifier) this way. I just started to really work on Python some weeks ago, so some of its aspects might escape me still. Could you help me on this issue? Thank you.
You can generate dynamic classes using the type constructor:
MyClass = type("MyClass", (BaseClass1, BaseClass2), {"attr1": "value1", ...})
Thus, according to your case, that would be:
cls = type(f"Task{TaskNumber}", (Page, ), {"form_fields": [f"Envive_WordList_{name_type}"], ...})
Note that you still have to construct your common methods like __init__, is_displayed and so on, as inner functions of the class factory:
def class_factory(*args, **kwargs):
...
def is_displayed(self):
return self.round_number == self.participant.vars['task_rounds']
def vars_for_template(player):
...
# Classmethod wrapping is done below
def live_method(player, data):
...
cls = type(..., {
"is_displayed": is_displayed,
"vars_for_template": vars_for_template,
"live_method": classmethod(live_method),
...,
}
#classmethod could be used as a function - {"live_method": classmethod(my_method)}
I have a script that takes a list of metrics as an input, and then fetches those metrics from the database to perform various operations with them.
My problem is that different clients get different subsets of the metrics, but I don't want to write a new IF block every time we add a new client. So right now, I have a large IF block that calls different functions based on whether the corresponding metric is in the list. What is the most elegant or Pythonic way of handling this?
Setup and function definitions:
clientOne = ['churn','penetration','bounce']
clientTwo = ['engagement','bounce']
def calcChurn(clientId):
churn = cursor.execute(sql to get churn)
[...]
return churn
def calcEngagement(clientId):
engagement = cursor.execute(sql to get engagement)
[...]
return engagement
Imagine three other functions in a similar format, so there is one function that corresponds to each unique metric. Now here is the block of code in the script that takes the list of metrics:
def scriptName(client, clientId):
if churn in client:
churn = calcChurn(clientId)
if engagement in client:
engagement = calcEngagement(clientId)
if penetration in client:
[...]
Generally, you'd create a mapping of names to functions and use that to calculate the stuff you want:
client_action_map = {
'churn': calcChurn,
'engagement': calcEngagement,
...
}
def scriptName(actions, clientId):
results = {}
for action in actions:
results[action] = client_action_map[action](clientId)
return results
You can create a class with static methods and use getattr to get the correct method. It's similar to what mgilson suggests but you essentially get the dict creation for free:
class Calculators:
#staticmethod
def calcChurn():
print("called calcChurn")
#staticmethod
def calcEngagement():
print("called calcEngagement")
#staticmethod
def calcPenetration():
print("called calcPenetration")
stats = ["churn", "engagement", "penetration", "churn", "churn", "engagement", "undefined"]
def capitalise(str):
return str[0].upper() + str[1:]
for stat in stats:
try:
getattr(Calculators, "calc" + capitalise(stat))()
except AttributeError as e:
print("Unknown statistic: " + stat)
called calcChurn
called calcEngagement
called calcPenetration
called calcChurn
called calcChurn
called calcEngagement
Unknown statistic: undefined
Perhaps it might make sense to encapsulate the required calls inside an object.
If it makes sense for your clients to be object and especially if many clients call the same set of functions to obtain metrics, then you could create a set of Client sub classes, which call a predefined set of the functions to obtain metrics.
It's a bit heavier than the mapping dict.
''' Stand alone functions for sql commands.
These definitions however dont really do anything.
'''
def calc_churn(clientId):
return 'result for calc_churn'
def calc_engagement(clientId):
return 'result for calc_engagement'
''' Client base object '''
class Client(object):
''' Base object allows list of functions
to be stored in client subclasses'''
def __init__(self, id):
self.id = id
self.metrics = []
self.args = []
def add_metrics(self, metrics, *args):
self.metrics.extend(metrics)
self.args = args
def execute_metrics(self):
return {m.__name__: m(*self.args) for m in self.metrics}
''' Specific sub classes '''
class Client1(Client):
def __init__(self, id):
''' define which methods are called for this class'''
super(Client1, self).__init__(id)
self.add_metrics([calc_churn], id)
class Client2(Client):
def __init__(self, id):
''' define which methods are called for this class'''
super(Client2, self).__init__(id)
self.add_metrics([calc_churn, calc_engagement], id)
''' create client objects and '''
c1 = Client1(1)
c2 = Client2(2)
for client in [c1, c2]:
print client.execute_metrics()
The result you will get from execute_metrics is a dict mapping the function name to its results for that client.
I want to use a list throughout a program I am writing. Basically, it is a list full of tuples with information regarding different people, each person's information (name, phone, address, etc) is stored as in a tuple. I define this list through an initial function, but i need to use this in my interaction function as well as others.
My question is, is it possible for me to use this list without defining it as a global variable?
def load_friends(filename):
"""imports filename as a list of tuples using the import command"""
import csv
with open(filename, 'Ur')as filename:
friends_list = list(tuple(x) for x in csv.reader(filename, delimiter=','))
def add_friend(friend_info, friends_list):
"""appends the friend_info tupple to the list friends_list"""
new_list = friends_list.append(friends_info)
def interact():
"""interaction function: accepts user input commands"""
while True:
command = raw_input('Command: ')
I should also mention that there is a command to parse the use inputs to perform the functions. Would this affect the use of the list?
You could declare list inside the first function that calls it and return it from there, latter functions should receive this list as an argument then.
def func1():
my_list=[]
"""Do stuff
"""
return list
def func2(my_list):
"""Do stuff with my_list
"""
return
def func3(my_list):
"""Do stuff with my_list
"""
return
def main():
"""First we retrieve the list from func1,
func2/3 get it passed to them as an argument
"""
foo=func1
func2(foo)
func3(foo)
if __name__ == "__main__":
main()
You could do the following:
# you can define the list here, it's global but doesn't require the keyword
my_list_globally = []
def func1(the_list):
pass
def func2(the_list):
pass
def func3(the_list):
pass
# you can use a hub function to pass the list into things that need it
def main():
my_list = []
func1(my_list)
func2(my_list)
func3(my_list)
if __name__ == "__main__":
main()
I don't quite understand the last part of your question but one of those 2 ways will be what you need.
Yes. Pass the "list of friends" back and forth between functions as an argument.
load_friends() would become
def load_friends(filename):
import csv
with open(filename, 'Ur') as f:
return map(tuple, csv.reader(f, delimiter=","))
add_friend() is close, but that assignment to new_list is unnecessary, because list.append() mutates the existing list in place:
def add_friend(friend_info, friend_list):
friend_list.append(friend_info)
would suffice.
interact() would also have a friends_list argument.
def interact(friends_list):
#interaction stuff here...
and you could call it like so:
interact(load_friends("myfile.csv"))
Classes are useful for this kind of thing, and easy to use:
class PersInfo:
def setName(self, name):
self._name = name
def getName(self):
return self._name
def setNumber(self, number):
self._phNumber = number
def getNumber(self):
return self._phNumber
def setAddr(self, address):
self._address = address
def getAddr(self)
return self._address
def main():
# Read in data from your CSV Here
infoList = ([])
for person in person_list: # Assuming person is a tuple here
foo = PersInfo()
foo.setName(person[index1])
foo.setNumber(person[index2])
foo.setAddr(person[index3])
infoList.append(foo)
# To access the info...
for person in infoList:
print(person.getName())
print(person.getNumber())
print(person.getAddr())
You do end up with the list being "global," sort of. It is in the main() function, where the PersInfo objects are being instantiated. This may be more than you wanted, but in the long run it is a good way to organize your code and keep it readable.
Also, you could build the infoList I made directly where you are creating person_list.
To be specific in my case, the class Job has a number of Task objects on which it operates.
import tasker
class Job(object):
_name = None
_tasks = []
_result = None
def __init__(self, Name):
self._name = Name
def ReadTasks(self):
# read from a Json file and create a list of task objects.
def GetNumTasks(self):
return len(self._tasks)
def GetNumFailedTasks(self):
failTaskCnt = 0
for task in self._tasks:
if task.IsTaskFail():
failTaskCnt += 1
To make GetNumFailedTasks more succinct, I would like to use a filter, but I am not sure what is the correct way to provide filter with IsTaskFail as the first parameter.
In case, this is a duplicate, please mark it so, and point to the right answer.
You can use a generator expression with sum:
failTaskCnt = sum(1 for task in self._tasks if task.IsTaskFail())