So in those past days I have been trying to figure out how to even start getting at thread and I finally got it to work! The issue I am having now is that I want them to run simultaneously. I'm hearing different things such as GIL won't work. Some say it works with multiprocessing etc etc.. However I turned here to see if its possible and whats the say to do it?
Basically my code looks right now like this:
The code with Thread
def start(config):
NameUrl = config["Url"]
myNote = config["My-Note"]
checkoutNames(Nameurl, MyNote)
if __name__ == '__main__':
with open('config.json', 'r', encoding='UTF-8') as json_data:
config = json.load(json_data)
threads = []
for i, e in enumerate(config):
threads.append(threading.Thread(target=start, args=(config[i] or e)))
As you can see in if __name__ == '__main__': That's where the thread is at this moment. However what this does at this moment is that it first does thread 1, when its done it turns to thread 2 and goes on like that and my wish is to turn it into concurrently/simultaneously if that is even possible?
EDIT
CODE
if __name__ == '__main__':
with open('config.json', 'r', encoding='UTF-8') as json_data:
config = json.load(json_data)
jobs = []
for i, e in enumerate(config):
c = (config[i] or e)
p = multiprocessing.Process(target=start, args=(c))
jobs.append(p)
p.start()
The error iam getting with the code above:
[<Process(Process-1, initial)>]
<Process(Process-1, initial)>
{'Email': '123o#gmail.com', 'PersonNumber': '4234', 'ZipCode': '1241234', 'Name': 'Guess', 'LastName': 'TheyKnow'}
[<Process(Process-1, started)>, <Process(Process-2, initial)>]
<Process(Process-2, initial)>
{'Email': 'Hello#hotmail.com', 'PersonNumber': '1234', 'ZipCode': '56431', 'Name': 'Stack', 'LastName': 'Overflow'}
Process Process-1:
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
TypeError: start() takes 1 positional argument but 16 were given
Process Process-2:
Traceback (most recent call last):
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
TypeError: start() takes 1 positional argument but 16 were given
I think you need to fix the args parameter:
...
jobs = []
for i, e in enumerate(config):
c = (config[i] or e)
p = multiprocessing.Process(target=start, args=(c,))
jobs.append(p)
p.start()
Note the syntax used in the args parameter:
args=(c,)) # <=== (c,) means that we are assigning the tuple (c,) to args
# which gets translated into start(c)
Related
I have looked at this question to get started and it works just fine How can I recover the return value of a function passed to multiprocessing.Process?
But in my case I would like to write a small tool, that would connect to many computers and gather some statistics, each stat would be gathered within a Process to make it snappy. But as soon as I try to wrap up the multiprocessing command in a class for a machine then it fails.
Here is my code
import multiprocessing
import pprint
def run_task(command):
p = subprocess.Popen(command, stdout = subprocess.PIPE, universal_newlines = True, shell = False)
result = p.communicate()[0]
return result
MACHINE_NAME = "cptr_name"
A_STAT = "some_stats_A"
B_STAT = "some_stats_B"
class MachineStatsGatherer():
def __init__(self, machineName):
self.machineName = machineName
manager = multiprocessing.Manager()
self.localStats = manager.dict() # creating a shared ressource for the sub processes to use
self.localStats[MACHINE_NAME] = machineName
def gatherStats(self):
self.runInParallel(
self.GatherSomeStatsA,
self.GatherSomeStatsB,
)
self.printStats()
def printStats(self):
pprint.pprint(self.localStats)
def runInParallel(self, *fns):
processes = []
for fn in fns:
process = multiprocessing.Process(target=fn, args=(self.localStats))
processes.append(process)
process.start()
for process in processes:
process.join()
def GatherSomeStatsA(self, returnStats):
# do some remote command, simplified here for the sake of debugging
result = "Windows"
returnStats[A_STAT] = result.find("Windows") != -1
def GatherSomeStatsB(self, returnStats):
# do some remote command, simplified here for the sake of debugging
result = "Windows"
returnStats[B_STAT] = result.find("Windows") != -1
def main():
machine = MachineStatsGatherer("SOMEMACHINENAME")
machine.gatherStats()
return
if __name__ == '__main__':
main()
And here is the error message
Traceback (most recent call last):
File "C:\Users\mesirard\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "C:\Users\mesirard\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "d:\workdir\trunks6\Tools\VTKAppTester\Utils\NXMachineMonitorShared.py", line 45, in GatherSomeStatsA
returnStats[A_STAT] = result.find("Windows") != -1
TypeError: 'str' object does not support item assignment
Process Process-3:
Traceback (most recent call last):
File "C:\Users\mesirard\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 297, in _bootstrap
self.run()
File "C:\Users\mesirard\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "d:\workdir\trunks6\Tools\VTKAppTester\Utils\NXMachineMonitorShared.py", line 50, in GatherSomeStatsB
returnStats[B_STAT] = result.find("Windows") != -1
TypeError: 'str' object does not support item assignment
The issue is coming from this line
process = multiprocessing.Process(target=fn, args=(self.localStats))
it should have a extra comma at the end of args like so
process = multiprocessing.Process(target=fn, args=(self.localStats,))
The issue was posted several times here in stackoverflow, but none of them could help me as my case is very specific.
I have a function Myfunction(p1,p2,p3,p4) which need 4 parameters to run.
I need to run Myfunction on multiprocessing way.
I use concurrent.futures.ProcessPoolExecutor to do this job.
I know how to pass a list as an argument in the executor.map(Myfunction, argument)
But this time, I have a list of tuples which is the list of my 4 arguments for my multiprocessing.
Here is my code:
def Myfunction(p_udid,p_systemPort,p_deviceName, p_version, p_os):
desired_caps = {}
desired_caps['platformName'] = p_os
desired_caps['platformVersion'] = p_version
desired_caps['deviceName'] = p_deviceName
desired_caps['udid'] = p_udid
desired_caps['noReset'] = 'true'
if __name__ == '__main__':
list=[('41492968379078','4730','S6S5IN3G','6','Android'),('53519716736397','4731','S6S5IN3G','6','Android'),('0123456789ABCDEF','4732','20','7','Android')]
with concurrent.futures.ProcessPoolExecutor() as executor:
multiprocesses = executor.map(Myfunction, list)
Of course I get an error :
concurrent.futures.process._RemoteTraceback: """ Traceback (most
recent call last): File
"C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py",
line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs) File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py",
line 198, in _process_chunk
return [fn(*args) for args in chunk] File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py",
line 198, in
return [fn(*args) for args in chunk] TypeError: Myfunction() missing 4 required positional arguments: 'p_systemPort',
'p_deviceName', 'p_version', and 'p_os' """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File
"E:/DropboxBACKUP14112018/Cff/Python/project_GITHUB/test2.py", line
24, in
for multiprocess in multiprocesses: File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures\process.py",
line 483, in _chain_from_iterable_of_lists
for element in iterable: File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py",
line 598, in result_iterator
yield fs.pop().result() File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py",
line 435, in result
return self.__get_result() File "C:\Users\Nino\AppData\Local\Programs\Python\Python37\lib\concurrent\futures_base.py",
line 384, in __get_result
raise self._exception TypeError: Myfunction() missing 4 required positional arguments: 'p_systemPort', 'p_deviceName', 'p_version', and
'p_os
I tried different stuff from all the answers similar to my issue, but I didn't succeed.
Can someone help me please?
Here's a way you can make it work:
def Myfunction(*args):
p_udid,p_systemPort,p_deviceName, p_version, p_os = args[0]
desired_caps = {}
desired_caps['platformName'] = p_os
desired_caps['platformVersion'] = p_version
desired_caps['deviceName'] = p_deviceName
desired_caps['udid'] = p_udid
desired_caps['noReset'] = 'true'
return desired_caps
def cpu_tasks(func, *args):
# set chunksize to be even
with ProcessPoolExecutor() as tp:
result = tp.map(func, chunksize=10, *args)
return list(result)
if __name__ == '__main__':
lst=[('41492968379078','4730','S6S5IN3G','6','Android'),('53519716736397','4731','S6S5IN3G','6','Android'),('0123456789ABCDEF','4732','20','7','Android')]
ans = cpu_tasks(Myfunction, *[lst])
print(ans)
[{'platformName': 'Android',
'platformVersion': '6',
'deviceName': 'S6S5IN3G',
'udid': '41492968379078',
'noReset': 'true'},
{'platformName': 'Android',
'platformVersion': '6',
'deviceName': 'S6S5IN3G',
'udid': '53519716736397',
'noReset': 'true'},
{'platformName': 'Android',
'platformVersion': '7',
'deviceName': '20',
'udid': '0123456789ABCDEF',
'noReset': 'true'}]
I have a python program like below.
from multiprocessing import Lock, Process, Queue, current_process
import time
lock = Lock()
def do_job(tasks_to_accomplish, tasks_that_are_done):
while not tasks_to_accomplish.empty():
task = tasks_to_accomplish.get()
print(task)
lock.acquire()
tasks_that_are_done.put(task + ' is done by ' + current_process().name)
lock.release()
time.sleep(1)
return True
def main():
number_of_task = 10
number_of_processes = 4
tasks_to_accomplish = Queue()
tasks_that_are_done = Queue()
processes = []
for i in range(number_of_task):
tasks_to_accomplish.put("Task no " + str(i))
# creating processes
for w in range(number_of_processes):
p = Process(target=do_job, args=(tasks_to_accomplish, tasks_that_are_done))
processes.append(p)
p.start()
# completing process
for p in processes:
p.join()
# print the output
while not tasks_that_are_done.empty():
print(tasks_that_are_done.get())
return True
if __name__ == '__main__':
main()
Sometimes program run perfectly but sometimes it gets stuck and doesn't complete. When quit manually, it produces following error.
$ python3 multiprocessing_example.py
Task no 0
Task no 1
Task no 2
Task no 3
Task no 4
Task no 5
Task no 6
Task no 7
Task no 8
Task no 9
^CProcess Process-1:
Traceback (most recent call last):
File "multiprocessing_example.py", line 47, in <module>
main()
File "multiprocessing_example.py", line 37, in main
p.join()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 121, in join
res = self._popen.wait(timeout)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 51, in wait
return self.poll(os.WNOHANG if timeout == 0.0 else 0)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 29, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "multiprocessing_example.py", line 9, in do_job
task = tasks_to_accomplish.get()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/queues.py", line 94, in get
res = self._recv_bytes()
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
buf = self._recv_bytes(maxlength)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
chunk = read(handle, remaining)
KeyboardInterrupt
Can someone tell me what is the issue with the program? I am using python 3.6.
Note: Lock is not needed around a Queue.
lock.acquire()
tasks_that_are_done.put(task + ' is done by ' + current_process().name)
lock.release()
Queue
The Queue class in this module implements all the required locking semantics.
Question: ... what is the issue with the program?
You are using Queue.empty() and Queue.get(),
such leads to Deadlock on calling join() because there is no guarantee that the empty() State don't change until get()
was reaching.
Deadlock prone:
while not tasks_to_accomplish.empty():
task = tasks_to_accomplish.get()
Instead of using empty/get, Pair use for instance:
import queue
while True:
try:
task = tasks_to_accomplish.get_nowait()
except queue.Empty:
break
else:
# Handle task here
...
tasks_to_accomplish.task_done()
I met some behavior of Python multiprocessing which I cannot understand...
For example:
from multiprocessing import Pool
import time
import sys
def f(x):
time.sleep(10)
print(x)
return x * x
def f2(x, f):
time.sleep(10)
print(x, file=f)
return x * x
if __name__ == '__main__':
p = Pool(5)
for t in range(10):
p.apply_async(f, args=(t,))
p.close()
p.join() # Here it blocks and prints the number, which is normal.
p = Pool(5)
for t in range(10):
p.apply_async(f2, args=(t, sys.stdout))
p.close()
p.join() # Here it does not block and nothing happends(no output at all)...
The output is:
3
1
0
2
4
5
9
6
7
8
I know that we have to use something like shared variables to pass to the function when using multiprocessing and apply_async, but what will happen if I pass a normal variable to a function used in apply_async?
The multiprocessing.Pool executes your logic in a separate process. If the logic raises and exception, the Pool will return it to the caller.
In your code you are not collecting the output of your functions, therefore you don't notice the real issue.
Try to modify your code as follows:
p = Pool(5)
for t in range(10):
task = p.apply_async(f2, args=(t, sys.stdout))
task.get()
You will then get the actual exception which was raised within f2:
Traceback (most recent call last):
File "asd.py", line 24, in <module>
p.apply_async(f2, args=(t, sys.stdout)).get()
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
put(task)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 50, in dumps
cls(buf, protocol).dump(obj)
TypeError: cannot serialize '_io.TextIOWrapper' object
It turns out that sys.stdout is not picklable. Which, in this case, is not an issue as sys.stdout unique per process. You can avoid passing it over the function and just use it as is within f2.
I test python multiprocessing and write simple program:
from multiprocessing import Process
from time import sleep
def f(name):
print 'hello', name
x=1
while True:
x+=1
sleep(1)
print 'subprocess '+str(x)
if x==10:
quit()
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
x=1
while True:
x+=1
sleep(0.1)
print 'main process '+str(x)
if x==100:
quit()
Its work, but I has little error:
Traceback (most recent call last):
File "ttt.py", line 17, in <module>
p.start()
File "/usr/lib64/python2.6/multiprocessing/process.py", line 104, in start
self._popen = Popen(self)
File "/usr/lib64/python2.6/multiprocessing/forking.py", line 99, in __init__
code = process_obj._bootstrap()
File "/usr/lib64/python2.6/multiprocessing/process.py", line 242, in _bootstrap
sys.stderr.write(e.args[0] + '\n')
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
Use sys.exit() instead of quit(). The latter is meant to be used only in the interactive interpreter.
As Kevin noted, you can use return in f to exit the function normally. This would be perhaps more appropriate.