I am working on the integration of two different framework, say Main_process1.py and Main_process2.py. Consider, Main_process1.py have a main() and Main_process2.py have another main().
So, i have changed the main() as main1() and main2() respectively for Main_process1.py and Main_process2.py and created a new file like overall_Main.py.
The new file overall_Main.py will have two process, one starting main1() in Main_process1.py and main2() in Main_process2.py.
Basically, what's required is, an overall main process has to be created for two different main processes. Please give some suggestion
This is, like, the very basics of Python Multiprocessing:
from multiprocessing import Process
from Main_process1 import main1
from Main_process2 import main2
if __name__ == '__main__':
p1 = Process(target=main1, args=(list your args for main1 here,))
p2 = Process(target=main2, args=(list your args for main2 here,))
p1.start()
p2.start()
#your other main code
p1.join()
p2.join()
Related
Suppose i have two functions
def funct1():
##does something
def funct2():
##does something
I want to use them in another function with multiprocessing like so:
def my_funct
##does something
if __name__ == '__main__':
p1 = Process(target = funct1)
p2 = Process(target = funct2)
p1.start()
p2.start()
##more code
p1.terminate()
p2.terminate()
return something
Basically i want to start and end processes inside my function but its not working properly. What would be the correct way to do this?
On platforms that use the spawn method to create new processes it becomes necessary to place the process-creation code that exists at the global scope within an if __name__ == '__main__': block to prevent the newly created child process from trying to re-execute recursively the process-creation code since all code at the global level is re-executed in order to initialize memory (e.g. function definitions and global variables) for the process. Putting such a test within a function or method, which by definition would not be at global scope, would not normally make too much sense.
In your case there must be some code in the main script being executed (where __name__ would be '__main__' unless that script is launched as a module with the -m Python flag) that invokes directly or indirectly your my_funct function. It is that code that should be placed in a if __name__ == '__main__': block. For example:
def my_funct
##does something
p1 = Process(target = funct1)
p2 = Process(target = funct2)
p1.start()
p2.start()
##more code
p1.terminate()
p2.terminate()
return something
def function main():
# Do some work
...
# Call my_funct, which creates new child processes:
print(my_funct())
# Do some more work
...
if __name__ == '__main__':
# The following function invocation is at global scope in the main script
# and invokes code that will ultimately be creating new child processes:
main()
If the if __name__ == '__main__': test were instead moved to where you had it originally, then function main would be invoked as part of the initialization of memory for the new processes that were created in function my_funct. But I am sure you would not want main or any part of my_funct to be re-executed, which would happen with this move.
Note
I should add that any code at global scope that you do not want or do not need to be re-executed as part of memory initialization for the new child process should be placed within a if __name__ == '__main__': block , not just process-creation code. Also note that if your my_funct is imported from some module, then __name__ would not be '__main__' to begin with.
I am quite new to python and the multi-processing module. I want to know how to make the process skip the beginning so it doesn't repeat it. Any help would be appreciated :)
print("Doing something!!!")
Code:
import multiprocessing
print("Doing something!!!")
def stuff():
print("Doing stuff")
if __name__ == '__main__':
p1 = multiprocessing.Process(target=stuff)
p1.start()
Output:
Doing something!!!
Doing something!!!
Doing stuff
Desired output:
Doing something!!!
Doing stuff
See the multiprocessing Programming Guidelines in the documentation.
On systems that use the "spawn" or "forkserver" methods of creating processes:
Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).
Your script is being imported into every process, so it will run any global code in all processes. Just move anything global into the if __name__ == '__main__': section:
import multiprocessing
def stuff():
print("Doing stuff")
if __name__ == '__main__':
print("Doing something!!!")
p1 = multiprocessing.Process(target=stuff)
p1.start()
This insures the function stuff() will be imported and defined in every process, but your print will only run once in the main process.
What I'd like to do is the following program to print out:
Running Main
Running Second
Running Main
Running Second
[...]
Code:
from multiprocessing import Process
import time
def main():
while True:
print('Running Main')
time.sleep(1)
def second():
while True:
print('Running Second')
time.sleep(1)
p1 = Process(main())
p2 = Process(second())
p1.start()
p2.start()
But it doesn't have the desired behavior. Instead it just prints out:
Running Main
Running Main
[...]
I suspect my program doesn't work because of the while statement?
Is there any way I can overcome this problem and have my program print out what I mentioned no matter what I execute in my function?
The issue here seems to be when you make the process vars. I suspect the reason for why the process inclusively runs the first function is because of syntax. My interpretation is that instead of creating a process out of a function you are making a process that executes a function exclusively.
When you want to create Process object you want to avoid using this
p1 = Process(target=main())
and rather write
p1 = Process(target=main)
That also means if you want to include any input for the function you will have to
p1 = Process(target=main, args=('hi',))
I have multiple data files that I process using python Pandas libraries. Each file is processed one by one, and only one logical processor is used when I look at Task manager (it is at ~95%, and the rest are within 5%)
Is there a way to process data files simultaneously?
If so, is there a way to utilize the other logic processors to do that?
(Edits are welcome)
If your file names are in a list, you could use this code:
from multiprocessing import Process
def YourCode(filename, otherdata):
# Do your stuff
if __name__ == '__main__':
#Post process files in parallel
ListOfFilenames = ['file1','file2', ..., 'file1000']
ListOfProcesses = []
Processors = 20 # n of processors you want to use
#Divide the list of files in 'n of processors' Parts
Parts = [ListOfFilenames[i:i + Processors] for i in xrange(0, len(ListOfFilenames), Processors)]
for part in Parts:
for f in part:
p = multiprocessing.Process(target=YourCode, args=(f, otherdata))
p.start()
ListOfProcesses.append(p)
for p in ListOfProcesses:
p.join()
You can process the different files in different threads or in different processes.
The good thing of python is that its framework provides tools for you to do this:
from multiprocessing import Process
def process_panda(filename):
# this function will be started in a different process
process_panda_import()
write_results()
if __name__ == '__main__':
p1 = Process(target=process_panda, args=('file1',))
# start process 1
p1.start()
p2 = Process(target=process_panda, args=('file2',))
# starts process 2
p2.start()
# waits if process 2 is finished
p2.join()
# waits if process 1 is finished
p1.join()
The program will start 2 child-processes, which can be used do process your files.
Of cource you can do something similar with threads.
You can find the documentation here:
https://docs.python.org/2/library/multiprocessing.html
and here:
https://pymotw.com/2/threading/
I trying to play around with multi-threading so I can better at it, but for some weird reason, my code doesn't want to follow the commands. It's suppose to go into a while loop and print, but it doesn't, and it's also not raising any errors, so which lines is the mistake on?
#!/usr/bin/env python
#
#
#
import random
import thread
import time
import sys
import os
def DisplayA(name,wait):
while True:
print 'Display: 1';time.sleep(wait)
def DisplayB(name,wait):
while True:
print 'Display: 2';time.sleep(wait)
def DisplayC(name,wait):
while True:
print 'Display: 3';time.sleep(wait)
thread.start_new_thread(DisplayA,('Display1',2))
thread.start_new_thread(DisplayB,('Display2',3))
thread.start_new_thread(DisplayC,('Display3',5))
Add this to the bottom:
while True:
pass
The problem is that you're running off the bottom of your main program. This terminates the entire execution session.
Quick and short solution:
while True:
time.sleep(1)
Do not use pass in the while loop, because it eats CPU.
Expensive way of doing nothing.
If you want a more general solution, then you can import Tread from threading, then you can use join:
from threading import Thread
...
p1 = Thread(name="A", target=DisplayA, args=('Display1',2))
p2 = Thread(name="B", target=DisplayB, args=('Display2',3))
p3 = Thread(name="C", target=DisplayC, args=('Display3',5))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
This solution works also if the threads do not run endless, and your program can continue after the threads have finished.
You can either do what Prune here suggested, or you can suspend the main thread after initiating DisplayA, DisplayB and DisplayC.