This question already has answers here:
How to use multiprocessing pool.map with multiple arguments
(23 answers)
Closed 5 years ago.
I have a function to be called from multiprocessing pool.map with multiple arguments.
from multiprocessing import Pool
import time
def printed(num,num2):
print 'here now '
return num
class A(object):
def __init__(self):
self.pool = Pool(8)
def callme(self):
print self.pool.map(printed,(1,2),(3,4))
if __name__ == '__main__':
aa = A()
aa.callme()
but it gives me following error
TypeError: printed() takes exactly 2 arguments (1 given)
I have tried solutions from other answers here but they are not working for me.
How can i resolve it and what is the reason for this problem (I did not get the pickle POV)
You should be giving args in array
from multiprocessing import Pool
import time
def printed(*args):
print 'here now '
return args[0][0]
class A(object):
def __init__(self):
self.pool = Pool(8)
def callme(self):
print self.pool.map(printed,[(1,2),(3,4)])
if __name__ == '__main__':
aa = A()
aa.callme()
Related
This question already has answers here:
python decorator TypeError missing 1 required positional argument
(3 answers)
How to create a decorator that can be used either with or without parameters?
(16 answers)
Closed 2 years ago.
This is meant to calculate execution time and log / print it:
def timer(logger=None):
def decorator(func):
def wrapper(*args, **kwargs):
start_time = perf_counter()
result = func(*args, **kwargs)
total_time = perf_counter() - start_time
message = f'{func.__name__} Time: ' f'{total_time} seconds'
if logger is not None:
logger.info(message)
else:
print(message)
return result
return wrapper
return decorator
#timer
def decorated():
print("Running ...")
if __name__ == '__main__':
decorated()
I'm expecting this to print time, instead it complains about a missing argument:
TypeError: decorator() missing 1 required positional argument: 'func'
However, when I do:
#timer(None)
def decorated():
print("Running ...")
or:
#timer(logger=None)
def decorated():
print("Running ...")
It works! what is this nonsense?
Running ...
decorated Time: 2.3044999999999316e-05 seconds
Notes:
To those marking my question as a duplicate, can you explain why
other decorators work just fine?
In the example below, lru_cache accepts maxsize and typed parameters and works without explicit calls as well.
Example:
from functools import lru_cache
#lru_cache
def decorated():
print("Running ...")
Out:
Running ...
I am trying to figure out how to pass a string as an argument in Python threading.Thread. This problem has been encountered before: Python Threading String Arguments
Is there a better way to pass in a string? There has to be a more obvious way, and I am just too new to coding to figure it out.
Code Block A
import threading
def start_my_thread():
my_thread = threading.Thread(target=my_func, args="string")
my_thread.start()
def my_func(input):
print(input)
Result:
TypeError: my_func() takes 1 positional argument but 6 were given
Code Block B
import threading
def start_my_thread():
my_thread = threading.Thread(target=my_func, args=("string",))
my_thread.start()
def my_func(input):
print(input)
Result: string
you can inherit Thread,and define my_func as run method.and create a new instance.
import threading
class MyThread(threading.Thread):
def __init__(self,string):
super().__init__()
self.string = string
def run(self):
print(self.string)
# def start_my_thread():
# my_thread = threading.Thread(target=my_func, args=("string",))
# my_thread.start()
# def my_func(input):
# print(input)
if __name__ == "__main__":
MyThread("hello").start()
I'm new with all the multiprocessing stuff and my current program doesnt work. I read the last hours about the problem and I tried a lot, the method in or out of the class and within a different class and it didnt work.
import multiprocessing as mp
class A:
#staticmethod
def multi():
a = [1,2,3]
b = 4
prepared = list()
for x in a:
prepared.append((x, b))
pool = mp.Pool(mp.cpu_count()-1)
result = pool.starmap(method, prepared)
pool.close()
pool.join()
print(result)
def method(a, x):
return (a-x, a+x)
if __name__ == "__main__":
a = A()
a.multi()
This is just an example how my class/method structure looks like (and this one does work, even though I changed nothing in the multiprocessing part).
This is the exception I get:
AttributeError: Can't pickle local object 'FeatureExtracter.<locals>.feature_extracter_fwd'
It would be nice if someone knows the solution or at least why the method cant be pickled.
import multiprocessing as mp
class A:
#staticmethod
def multi():
b = 4
return [(x, b) for x in [1,2,3]]
def method(a,x): return (a-x, a+x)
if __name__ == "__main__":
with mp.Pool(mp.cpu_count() - 1) as p:
result = p.starmap(method, A().multi())
print(result)
Here is an example:
import multiprocessing
def function():
for i in range(10):
print(i)
if __name__ == '__main__':
p = multiprocessing.Pool(5)
p.map(function, )
yields the error: TypeError: map() missing 1 required positional argument: 'iterable'
The function does not need any input, so I wish to not artificially force it to. Or does multiprocessing need some iterable?
The following code returns / prints nothing. Why?
import multiprocessing
def function():
for i in range(10):
print(i)
if __name__ == '__main__':
p = multiprocessing.Pool(5)
p.map(function, ())
If you are only trying to perform a small number of tasks, it may be better to use Process for reasons described here.
This site provides an excellent tutorial on use of Process() which i have found helpful. Here is an example from the tutorial using your function():
import multiprocessing
def function():
for i in range(10):
print(i)
if __name__ == '__main__':
jobs = []
for i in range(5):
p = multiprocessing.Process(target=function)
jobs.append(p)
p.start()
If you have no arguments to pass in, you don't have to use map. You can simply use multiprocessing.Pool.apply instead:
import multiprocessing
def function():
for i in range(10):
print(i)
if __name__ == '__main__':
p = multiprocessing.Pool(5)
p.apply(function)
I'm using Python's multiprocessing library to process a list of inputs with the built-in map() method. Here's the relevant code segment:
subp_pool = Pool(self.subprocesses)
cases = subp_pool.map(self.get_case, input_list)
return cases
The function to be run in parallel is self.get_case(), and the list of inputs is input_list.
I wish to print a progress prompt to the standard output in the following format:
Working (25/100 cases processed)
How can I update a local variable inside the class that contains the Pool, so that whenever a subprocess finishes, the variable is incremented by 1 (and then printed to the standard output)?
There's no way to do this using multiprocessing.map, because it doesn't alert the main process about anything until it's completed all its tasks. However, you can get similar behavior by using apply_async in tandem with the callback keyword argument:
from multiprocessing.dummy import Pool
from functools import partial
import time
class Test(object):
def __init__(self):
self.count = 0
self.threads = 4
def get_case(self, x):
time.sleep(x)
def callback(self, total, x):
self.count += 1
print("Working ({}/{}) cases processed.".format(self.count, total))
def do_async(self):
thread_pool = Pool(self.threads)
input_list = range(5)
callback = partial(self.callback, len(input_list))
tasks = [thread_pool.apply_async(self.get_case, (x,),
callback=callback) for x in input_list]
return [task.get() for task in tasks]
if __name__ == "__main__":
t = Test()
t.do_async()
Call the print_data() from the get_case() method and you are done.
from threading import Lock
Class A(object):
def __init__(self):
self.mutex = Lock()
self.count = 0
def print_data(self):
self.mutex.acquire()
try:
self.count += 1
print('Working (' + str(self.count) + 'cases processed)')
finally:
self.mutex.release()