I have two python function which I want to run in parallel. I don't want sub_task function to wait for main_task function.
from threading import Thread
from multiprocessing import Process
from time import sleep,time
def main_task():
while True:
sleep(2)
print('main task running')
def sub_task():
while True:
sleep(5)
print('sub task running')
When I used thread this way, I can see output
q=Thread(target = main_task).start()
s=Thread(target = sub_task).start()
But when I used Process this way, I cannot see ouput
q=Process(target = main_task).start()
s=Process(target = sub_task).start()
So what is wrong with the implementation.
Related
I was thinking to use multiprocess package to run a function in parallel but I need to pass a different value of parameter every run (every 1 sec).
e.g.)
def foo(list):
while True:
<do something with list>
sleep(1000)
def main():
process = multiprocess.Process(target=foo, args=(lst))
process.start()
<keep updating lst>
This will cause a foo() function running with the same parameter value over and over. How can I work around in this scenario?
Armed with the knowledge of what you're actually trying to do, i.e.
The foo function does an http post call to save some logs (batch) to the storage. The main function is getting text logs (save log to the batch) while running a given shell script. Basically, I'm trying to do batching for logging.
the answer is to use a thread and a queue for message passing (multiprocessing.Process and multiprocessing.Queue would also work, but aren't really necessary):
import threading
import time
from queue import Queue
def send_batch(batch):
print("Sending", batch)
def save_worker(queue: Queue):
while True:
batch = queue.get()
if batch is None: # stop signal
break
send_batch(batch)
queue.task_done()
def main():
batch_queue = Queue()
save_thread = threading.Thread(target=save_worker, args=(batch_queue,))
save_thread.start()
log_batch = []
for x in range(42): # pretend this is the shell script outputting things
message = f"Message {x}"
print(message)
log_batch.append(message)
time.sleep(0.1)
if len(log_batch) >= 7: # could also look at wallclock
batch_queue.put(log_batch)
log_batch = []
if log_batch:
batch_queue.put(log_batch) # send the last batch
print("Script stopped, waiting for worker to finish")
batch_queue.put(None) # stop signal
save_thread.join()
if __name__ == "__main__":
main()
import threading
import time
def run_every_second(param):
# your function code here
print(param)
# create a list of parameters to pass to the function
params = [1, 2, 3, 4]
# create and start a thread for each parameter
for param in params:
t = threading.Thread(target=run_every_second, args=(param,))
t.start()
time.sleep(1)
# wait for all threads to complete
for t in threads:
t.join()
This will create a new thread for each parameter, and each thread will run the run_every_second function with the corresponding parameter. The threads will run concurrently, so the functions will be executed in parallel. There will be a 1-second time lapse between the start of each thread.
I'm testing multiprocessing using apply_async.
However, it looks like each apply_async is called from MainProcess and it's not exactly asynchronous. Each function is called only after previous one is finished. I'm not sure what I'm missing here.
I'm using Windows with Python 3.8, so it's using the spawn method to create processes.
import os
import time
from multiprocessing import Pool, cpu_count, current_process
from threading import current_thread
def go_to_sleep():
pid = os.getpid()
thread_name = current_thread().name
process_name = current_process().name
print(f"{pid} Process {process_name} and {thread_name} going to sleep")
time.sleep(5)
def apply_async():
pool = Pool(processes=cpu_count())
print(f"Number of procesess {len(pool._pool)}")
for i in range(20):
pool.apply_async(go_to_sleep())
pool.close()
pool.join()
def main():
apply_async()
if __name__ == "__main__":
start_time = time.perf_counter()
main()
end_time = time.perf_counter()
print(f"Elapsed run time: {end_time - start_time} seconds.")
Output:
Number of procesess 8
26776 Process MainProcess and MainThread going to sleep
26776 Process MainProcess and MainThread going to sleep
26776 Process MainProcess and MainThread going to sleep
The problem is that your code is not actually calling the specified function in the process pool, it is calling it in the main thread, and passing the result of calling it to pool.apply_async.
That is, instead of calling pool.apply_async(go_to_sleep()), you should call pool.apply_async(go_to_sleep). You need to pass the function that should be called to Pool.apply_async - you should not call the function when you call Pool.apply_async.
I have written a simple code like below. This is just a model of another, much more complicated problem. Here is a simple function "task submit" addint tasks in the queue, its aim is to continiously seek tasks deligated by used since user can create new tasks after the code has been launched. I have a worker, behaving like doing something, just a simple worker function. Then I call ThreadPoolExecutor, call "task submit" with queue argument. Then I start adding tasks pulled from queue. But it happens the code doest terminate even when only main thread (which is my program itself) remains in the pool of threads. Cant understand why even shutdown doesnt work.
from concurrent.futures import ThreadPoolExecutor as Tpe
import time
import random
import queue
import threading
def task_submit(q):
for i in range(7):
threading.currentThread().setName('task_submit')
new_task = random.randint(10, 20)
q.put_nowait(new_task)
print(f' {i} new task with argument {new_task} has been added to queue')
time.sleep(5)
def worker(t):
threading.currentThread().setName(f'worker {t}')
print(f'{threading.currentThread().getName()} started')
time.sleep(t)
print(f'{threading.currentThread().getName()} FINISHED!')
with Tpe(max_workers=4) as executor:
q = queue.Queue(maxsize=100)
q_thread = executor.submit(task_submit, q)
tasks = []
while True:
time.sleep(10)
print('\n\n------------NEW CYCLE----------------\n\n')
if not q.empty():
print(threading.enumerate())
tasks.append(executor.submit(worker, q.get()))
else:
print('is queue empty?', q.empty())
print(f'active threads: {threading.active_count()}')
print(threading.enumerate())
executor.shutdown(wait=True)
I want to execute a task after certain time, so I have tried a countdown timer with a condition of being finished (when countdown variable = 0, the task is performed). The thing is that I don't want to stop the execution of the main program while performing the countdown. I have tried this:
import time
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
return num_of_secs
So, I run my code setting a number of seconds to the countdown, and when this countdown reaches the 0 value, a task must be executed. Using this code (it uses a while), when I call my function "countdown" it stops the execution of the main program, so it is the same as a big time.sleep. I want to carry out this countdown in the background, without stopping other actions until the countdown finishes and the task starts.
Thank you
Another alternative is by using threading.
I've got a simple example here with 2 Threads where the working thread is waiting for the countdown thread to finish and starting. The Main is still working fine.
import threading
import time
def do_something():
countdown_thread.join()
print("Starting Task")
time.sleep(3)
print("Finished Task")
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
print(num_of_secs)
if __name__ == '__main__':
countdown_thread = threading.Thread(target=countdown, args=(3,))
work_thread = threading.Thread(target=do_something)
countdown_thread.start()
work_thread.start()
while True:
print("Main doing something")
time.sleep(1)
Example picture for multithreading: Sequential vs Threading
Usually python only has a single program flow, so every instruction needs to complete before the next one can get executed.
For your case you need asynchronicity, with e.g. asyncio.sleep(5) as a separate task in the same event loop.
import asyncio
async def sleeper():
print('Holding...')
await asyncio.sleep(5)
print('Doing Work!')
async def work():
print('Doing work')
print('while')
print('the other guy is sleeping')
async def main():
await asyncio.gather(sleeper(), work())
asyncio.run(main())
The most common and easiest way to implement this would be with a Timer object from the threading library. It would go as follows:
import threading
import time
i = 0
done = False
def show_results():
print("results from GPIO readings")
print("=)")
global done
done = True # signal end of while loop
def read_GPIO():
print("reading GPIO...")
t = threading.Timer(60, show_results) # task will trigger after 60 seconds
t.start()
# your while loop would go here
read_GPIO() # do work
while not done:
print("waiting", i) # doing work while waiting for timer
time.sleep(1)
i += 1
pass
Notice that the time library is used only for illustrative purposes. You could also start the timer recursively to check periodically GPIOs and print results or trigger an event. For more information on the threading library or the Timer object check the docs
I want to create processes without waiting for other processes finish which they can't because they are in an infinite loop.
import time
from multiprocessing import Process
def child_function(param1, param2):
print(str(param1 * param2))
while True:
print("doing some stuff")
time.sleep(3)
def main_function():
print("Initializing some things.")
for _ in range(10):
Process(target=child_function(3, 5)).start()
if __name__ == '__main__':
main_function()
This code only starts one process and waits for it to finish. How can I do this?
Edit: Comment answer works fine and the answer below also works fine but for creating thread. Thank you everyone.
Try this Python module Threading
import time
import threading
def child_function(param1, param2):
print(str(param1 * param2))
while True:
print("doing some stuff")
time.sleep(3)
def main_function():
print("Initializing some things.")
for _ in range(10):
x = threading.Thread(target=child_function, args=(3,5, ))
x.start()
main_function()
Explanation: as already mentioned in the comments, notice that we are passing the function as opposed to calling it via the thread constructor, Also you can compare Threading vs Multiprocessing and use whichever best suits the project.