I'm making a virtual assistant using Python.
For that I need one main thread running continuously which is required for Speech recognition and want to run other threads like actions after detecting speech to run in background.
For the tasks like timer, I want to make it running in background while the main thread is running, so that I can perform other tasks even when the timer is running... and after reaching the time it should return it as tts to main thread
current structure I'm using is
main.py
-> class Main()
->Running logger in background // which is meant to exit with mainLoop
-> and Command() loop for speech recognition continuously
->`which links to Brain.py to timer.py
A few words about multithreading vs multiprocessing:
In multithreading you start a thread in the current process. Python runs (through the global interpreter lock) threads in short sequential order, never really in parallel. The upside is that threads can access the same variables (i.e. share memory).
On the other side in multiprocessing you run a new process (in the OS it appears as a separate program). They can really run in parallel but sharing variables is a lot more tricky (and much slower as well).
For your use case it seems that not two things are "CPU bound" i.e. it will not be the case that two things will need the CPU for 100% at the same time. In this situation, multithreading is probably the better solution, that is you should go for James Lim's solution.
If you still want to go for multiprocessing, then the following code could be your basic setup for timer. For the speech recognition function it would be accordingly (espeically the part about returning the list should be sufficient for returning tts from the speech recognition):
import multiprocessing
import time
def timer_with_return(seconds, return_list):
time.sleep(seconds)
return_list.append('foo')
if __name__ == "__main__":
# variables by manager are shared among process
# e.g. for return values
manager = multiprocessing.Manager()
timer_return = manager.list()
timer = multiprocessing.Process(target=timer_with_return, args=(3, timer_return))
timer.start()
while True:
time.sleep(1)
if not timer.is_alive():
break
print("timer is still running")
timer.join() # make sure the process is really finished
print("timer finished, return value is {}".format(timer_return))
Running this produces:
timer is still running
timer is still running
timer is still running
timer finished, return value is ['foo']
Related
In this minimal example, I expect the program to print foo and fuu as the tasks are scheduled.
import asyncio
async def coroutine():
while True:
print("foo")
async def main():
asyncio.create_task(coroutine())
#asyncio.run_coroutine_threadsafe(coroutine(),asyncio.get_running_loop())
while True:
print("fuu")
asyncio.run(main())
The program will only write fuu.
My aim is to simply have two tasks executing concurrently, but it seems like the created task is never scheduled.
I also tried using run_coroutine_threadsafe with no success.
If I add await asyncio.sleep(1) to the main. The created task takes the execution and the program will only write foo.
What I am supposed to do to run two tasks simultaneously using asyncio ?
I love this question and the explanation of this question tells how asyncio and python works.
Spoilers - It works similar to Javascript single threaded runtime.
So, let's look at your code.
You only have one main thread running which would be continuously running since python scheduler don't have to switch between threads.
Now, the thing is, your main thread that creates a task actually creates a coroutine(green threads, managed by python scheduler and not OS scheduler) which needs main thread to get executed.
Now the main thread is never free, since you have put while True, it is never free to execute anything else and your task never gets executed because python scheduler never does the switching because it is busy executing while True code.
The moment you put sleep, it detects that the current task is sleeping and it does the context switching and your coroutine kicks in.
My suggestion. if your tasks are I/O heavy, use tasks/coroutines and if they are CPU heavy which is in your case (while True), create either Python Threads or Processes and then the OS scheduler will take care of running your tasks, they will get CPU slice to run while True.
I am using Threading module in python. How to know how many max threads I can have on my system?
I am using Threading module in python. How to know how many max
threads I can have on my system?
There doesn't seem to be a hard-coded or configurable MAX value that I've ever found, but there is definitely a limit. Run the following program:
import threading
import time
def mythread():
time.sleep(1000)
def main():
threads = 0 #thread counter
y = 1000000 #a MILLION of 'em!
for i in range(y):
try:
x = threading.Thread(target=mythread, daemon=True)
threads += 1 #thread counter
x.start() #start each thread
except RuntimeError: #too many throws a RuntimeError
break
print("{} threads created.\n".format(threads))
if __name__ == "__main__":
main()
I suppose I should mention that this is using Python 3.
The first function, mythread(), is the function which will be executed as a thread. All it does is sleep for 1000 seconds then terminate.
The main() function is a for-loop which tries to start one million threads. The daemon property is set to True simply so that we don't have to clean up all the threads manually.
If a thread cannot be created Python throws a RuntimeError. We catch that to break out of the for-loop and display the number of threads which were successfully created.
Because daemon is set True, all threads terminate when the program ends.
If you run it a few times in a row you're likely to see that a different number of threads will be created each time. On the machine from which I'm posting this reply, I had a minimum 18,835 during one run, and a maximum of 18,863 during another run. And the more you fiddle with the code, as in, the more code you add to this in order to experiment or find more information, you'll find the fewer threads can/will be created.
So, how to apply this to real world.
Well, a server may need the ability to start a triple-digit number of threads, but in most other cases you should re-evaluate your game plan if you think you're going to be generating a large number of threads.
One thing you need to consider if you're using Python: if you're using a standard distribution of Python, your system will only execute one Python thread at a time, including the main thread of your program, so adding more threads to your program or more cores to your system doesn't really get you anything when using the threading module in Python. You can research all of the pedantic details and ultracrepidarian opinions regarding the GIL / Global Interpreter Lock for more info on that.
What that means is that cpu-bound (computationally-intensive) code doesn't benefit greatly from factoring it into threads.
I/O-bound (waiting for file read/write, network read, or user I/O) code, however, benefits greatly from multithreading! So, start a thread for each network connection to your Python-based server.
Threads can also be great for triggering/throwing/raising signals at set periods, or simply to block out the processing sections of your code more logically.
I am developing a Python script in which I am sampling data from a BLE device at a rate of about 50-200Hz. As of right now, I am doing this synchronously:
while True:
if time_now > time_before + (1/sample_rate):
do_stuff()
This works great, except for the fact that it is blocking the thread and other applications completely (if I want to combine with a Qt GUI, e.g.). What is the correct way to go about this issue?
Can I easily implement a multithreading setup, where each "sampler" (while-loop) gets its own thread?
Should I implement timer operations, and how do I make sure the script is not killed while waiting for a new sample?
My problem is similar to this, which, however, is for C#.
If the sampling itself doesn't take much time, you could use a QTimer and do the sampling in a slot on timeout. If it takes a lot of time blocking on I/O and not executing python code, you should probably use a Thread for the polling and send the result to the main thread using a signal.
If the sampling uses a lot of time executing python code, you are out of luck with most python implementations because of the GIL (Global Interpreter Lock). In most python implementations only one thread can actively execute python code. So real parallelism in pyhton is often done by creating new processes instead of new threads.
I love the Javascript setInterval pattern, my guess is this is more like what you want.
import threading
def setInterval(func,time):
e = threading.Event()
while not e.wait(time):
func()
def foo():
print "do poll here"
# using
setInterval(foo,5)
https://stackoverflow.com/a/39842247/1598412
I am writing a program that creates two new processes and must wait for them both to finish before continuing. How does one launch both processes and have the program wait for both to exit? Consider the pseudocode:
I currently have:
create_process("program1.exe").wait()
create_process("program2.exe").wait()
This is enificent as program2 can run concurently with program1.
create_process("program1.exe")
create_process("program2.exe").wait()
This may be wrong as program1 may take longer than program2.
I'm interested in a general solution, I bet there's algorithms or design patterns invented to deal with this stuff. But to add context to the question, I'm writing a Python script that calls pgsql2shp.exe twice, to export two tables from the database to the local machine and then preform an intersection. This script is written in Python 2.7 and uses subprocess.popen
How about using Threading?
If you spin up a couple of threads, each thread can run independently and you can join the threads when they are completed.
Try some code like this: (This code is heavily commented so that you can follow what's going on)
# Import threading
import threading
# Create a handler class.
# Each instance will run in it's own independent thread
class ThreadedHandler (threading.Thread):
# This method is called when you call threadInstance.start()
def run(self):
# Run your sub process and wait for it
# How you run your process is up to you
create_process(self.programName).wait()
# Create a new thread object
thread1 = ThreadedHandler()
# Set your program name so that when the thread is started
# the correct process is run
thread1.programName = 'program1.exe'
# Start the thread
thread1.start()
# Again, create a new thread object for the 2nd program
thread2 = ThreadedHandler()
# Set the program name
thread2.programName = 'program2.exe'
# Start the thread
thread2.start()
# At this point, each program is running independently in separate threads
# Each thread will wait for their respective sub process to complete
# Here, we join both threads. (Wait for both threads to complete)
thread1.join()
thread2.join()
# When we get here, both of our programs are finished and they both ran in parallel
here's some example code
while True: #main-loop
if command_received:
thread = Thread(target = doItNOW)
thread.start()
......
def doItNOW():
some_blocking_operations()
my problem is I need "some_blocking_operations" to start IMMEDIATELY (as soon as command_received is True).
but since they're blocking i can't execute them on my main loop
and i can't change "some_blocking_operations" to be non-blocking either
for "IMMEDIATELY" i mean as soon as possible, not more than 10ms delay.
(i once got a whole second of delay).
if it's not possible, a constant delay would also be acceptable. (but it MUST be constant. with very few milliseconds of error)
i'm currently working on a linux system (Ubuntu, but it may be another one in the future. always linux)
a python solution would be amazing.. but a different one would be better than nothing
any ideas?
thanks in advance
from threading import Thread
class worker(Thread):
def __init__(self, someParameter=True):
Thread.__init__(self)
# This is how you "send" parameters/variables
# into the thread on start-up that the thread can use.
# This is just an example in case you need it.
self.someVariable = someParameter
self.start() # Note: This makes the thread self-starting,
# you could also call .start() in your main loop.
def run():
# And this is how you use the variable/parameter
# that you passed on when you created the thread.
if self.someVariable is True:
some_blocking_operations()
while True: #main-loop
if command_received:
worker()
This is a non-blocking execution of some_blocking_operations() in a threaded manner. I'm not sure if what you're looking for is to actually wait for the thread to finish or not, or if you even care?
If all you want to do is wait for a "command" to be received and then to execute the blocking operations without waiting for it, verifying that it completes, then this should work for you.
Python mechanics of threading
Python will only run in one CPU core, what you're doing here is simply running multiple executions on overlapping clock invervals in the CPU. Meaning every other cycle in the CPU an execution will be made in the main thread, and the other your blocking call will get a chance to run a execution. They won't actually run in parallel.
There are are some "You can, but..." threads.. Like this one:
is python capable of running on multiple cores?