Hi while i'm programming i had to do a choice :
while not cont_flag :
pass
and using a Event object :
if not const_flag.is_set() :
const_flag.wait()
i want to know if there is a difference in performance between the two methode
There is. The first method is called busy waiting and is very different from blocking. In busy waiting, the CPU is being used constantly as the while loop is executed. In blocking, the thread is actually suspended until a wake-up condition is met.
See also this discussion:
What is the difference between busy-wait and polling?
The first one is referred to as busy waiting, it will eat up 100% of the CPU time while waiting. It's a way better practice to have some signalling mechanism to communicate events (e.g. something's done).
Python only allows a single thread to execute at a time, regardless of how many cpus your system may have. If multiple threads are ready to run, python will switch among them periodically. If you "busy wait" as in your first example, that while loop will eat up much of the time that your other threads could use for their work. While the second solution is far superior, if you end up using the first one, add a modest sleep to it.
while not cont_flag:
time.sleep(.1)
Related
This behavior seems really odd to me. I'm running a main pygame loop in which I process the fastevent queue, and I have a separate thread running that actually runs the game. The odd thing is that if I add a short sleep statement within my main loop, the game thread executes much faster. Here's the code for the main loop:
exited = False
while not exited:
if launcher.is_game_running:
launcher.game.grm.board.update_turn_timer()
# Run the event loop so pygame doesnt crash. Note: This loop CANNOT be
# allowed to hang. It must execute quickly. If any on_click function
# is going to stall and wait for user input, it had better process the
# fastevent queue iteslf.
# TODO(?) I don't know why, but having this sleep in here *speeds up*
# the game execution by a SIGNIFICANT factor. Like 10x. As far
# as I can tell, the value in the sleep can be anything small.
time.sleep(0.001)
for event in pygame.fastevent.get():
if event.type == pygame.QUIT:
exited = True
break
# Handle clicks, mouse movement, keyboard, etc
launcher.handle_event(event)
if len(launcher.delayed_on_click_effects) > 0:
launcher.delayed_on_click_effects.popleft()()
I'm really at a loss here, I don't see how adding that sleep could possibly speed up the execution of the other thread. Any ideas? I know this code snippet isn't enough to know what's going on in the other thread and such. I would post more code, but I have so little idea about what's going on here that I don't know which parts of my codebase are actually relevant. Can post more if anyone has suggestions.
I wasn't planning on worrying about this too much, but now a new change I've introduced is slowing my runtime back down again. Without knowing what's actually going on, it's hard to figure out how to get the runtime back where it was.
Thanks Thomas - I had no idea the GIL was even a thing, but yes, it looks like my issue is that certain threads are CPU-intensive and are not releasing the GIL frequently enough for the other threads.
I had noticed that I could replace the time.sleep(0.001) in my main loop with a print statement, and I would get the same speedup effect on the other thread. This makes sense if what that sleep is doing is releasing the GIL, because prints also release the GIL.
The "new change I've introduced" that I mentioned in the post was adding more threads (which handled message passing between the game client and a server). So my suspicion is that one of these new threads is CPU-intensive and is not releasing the GIL, thus partially starving the game thread.
To try to debug this, I added a bunch of print statements wherever I was creating new threads just to make sure I understood how many I had. And it turns out, these print statements fixed the runtime issues. So apparently, one of the places where I just added a print statement was within a thread that was hogging the GIL. The new print statement releases the GIL, allowing the game thread to run.
So my takeaways from this are:
The GIL exists (good to know)
Only one thread can actually execute at a time
If you want a thread to "wait and let other threads do things", then you should release the GIL with an i/o call (print, socket.recv, etc) or with time.sleep()
Threads should not "wait" by, eg, executing a while loop and checking for some condition to be true. This will hog the GIL and slow down other threads (unless you make sure to release the GIL each iteration of the loop with a sleep)
I am creating a multi-threaded program, in which I want only 1 thread at a time to go in the critical section where is creates a socket and send some data and all the other to wait for that variable to clear.
I tried threading.Events but later realized that on set() it will notify all the threads waiting. While I only wanted to notify one.
Tried locks(acquire and release). It suited my scenario well but I got to know that lock contention for a long time is expensive. After acquiring the lock my thread was performing many functions and hence resulted in holding the lock for long.
Now I tried threading.conditions. Just wanted to know if acquiring and holding the condition for a long time, is it not a good practice as it also uses locks.
Can anyone suggest a better approach to my problem statement.
I would use an additional thread dedicated to sending. Use a Queue where the other threads put their Send-Data. The socket-thread gets items from the queue in a loop and sends them one after the other.
As long as the queue is empty, .get blocks and the send-thread sleeps.
The "producer" threads have no waiting time at all, they just put their data in the queue and continue.
There is no concern about possible deadlock conditions.
To stop the send-thread, put some special item (e.g. None) in the queue.
To enable returning of values, put a tuple (send_data, return_queue) in the send-queue. when a result is ready, return it by putting it in the return_queue.
I have some python code I'm writing that's interfacing real-world hardware. It's replacing a hardware PLC. What I'm planning is when an event trigger happens to kick off multiple threads to effect certain 'on' actions, then go to sleep for a set interval, and then perform the corresponding 'off' actions. For example: at trigger, spawn a thread that turns the room lights on. Then go to sleep for 20 minutes. Then turn the lights off and terminate the thread.
However, I will have situations where the event trigger re-occurs. In that scenario I want the entire sequence to start over. My original plan was to use threads with unique names, so if a trigger occurs, check if the 'lights' thread exists, if if does kill it, and then re-spawn a new 'lights' thread. But in researching around these parts, it seems like people are suggesting that killing a thread is a Very Bad Thing to do.
So what would a better approach be to handling my situation? Note that in my example I only talked about one thread- but in reality there will be many different threads controlling many different devices.
This is python 3.x on a Rapberry Pi running raspbian, using rpi.gpio to monitor my input triggers and an I2C relay board for my output devices in case any of that info is useful.
Thanks!
the reason for not killing off threads is that it's easy to do it in a way that doesn't give the code any chance to "clean up" appropriately. i.e. finally blocks not run, resources leaked, etc…
there are various ways to get around this, you could wait on an Event as suggested by #Jérôme, treating a timeout as a signal to carry on
asyncio is another alternative as Cancelled exceptions tend to get used to clean up nicely and don't have the problems associated with killing native threads
I am trying wait for any of multiple multiprocessing events at the same time, so I came up with code like this:
if e1.wait(timeout) or e2.wait(timeout):
# this part will be reached if either of both
# events is set or the wait timed out
It works like the comment says. But how does this work? Is the if polling bot methods all the time? Or is it called as soon as one event gets set?
Bonus question: Is there some clever way to adjust the code to wait for any number of events, i.e. a list of events? if True in [e1.wait(timeout),e2.wait(timeout)] does not work as expected.
It only waits for the first one. This is due to python's support of short circuiting.
Wait on a thread or process is blocking, so it will block the current thread for going future until the timeout or the thread has finished. The semantics of if in Python is short circuit, which means that if the first one returns true, then the second one will not be called - simonzack said.
Waiting on a number of threads would be kinda hard to implement and maintain for a variety of threads. I would suggest you to use Message passing, and get each process to send a message to a Queue when it is finished. This way you could just check if the queue was of ´len(n)´, where ´n´ is the number of threads/processes. see more here Queues in multiprocessing
I have some Python code which uses threading.Timer to implement a 60-second timeout for an operation.
The problem is that this code runs in a job-control environment where it may get pre-empted by a higher priority job. In this case it will be sent SIGSTOP, and then some time later, SIGCONT. I need a way to somehow notice that this has happened and reset the timeout: obviously the operation hasn't really timed out if it's been suspended for the whole 60 seconds.
I tried to add a signal handler for SIGCONT but this seems to get executed after the code provided to threading.Timer has been executed.
Is there some way to achieve this?
A fairly simple answer that occurred to me after posting this is to simply break up the timer into multiple sub-timers, e.g. having 10 6-second timers instead where each one starts the next one in a chain. That way, if I get suspended, I only lose one of the timers and still get most of the wait before timing out.
This is of course not foolproof, especially if I get repeatedly suspended and restarted, but it's easy to do and seems like it might be good enough.
You need to rethink what you're asking for; a timeout reflects elapsed time (wall time); you want to know the time used by your process.
Fortunately you can measure this with getrusage: http://docs.python.org/library/resource.html
You'll still need to set a timeout; when it returns, measure the increase in user or system time usage since the start of the operation and terminate the operation if it exceeds the limit, else reschedule the timeout appropriately.
If your application is multi-threaded, the docs says that:
only the main thread can set a new signal handler, and the main thread will be the only one to receive signals
Make sure you are handling your signals from the main thread.