We Keep Coding

How to run tasks periodically without interrupting the whole program

I have a program that constantly runs if it receives an input, it'll do a task then go right back to awaiting input. I'm attempting to add a feature that will ping a gaming server every 5 minutes, and if the results every change, it will notify me. Problem is, if I attempt to implement this, the program halts at this function and won't go on to the part where I can then input. I believe I need multithreading/multiprocessing, but I have no experience with that, and after almost 2 hours of researching and wrestling with it, I haven't been able to figure it out.
I have tried to use the recursive program I found here but haven't been able to adapt it properly, but I feel this is where I was closest. I believe I can run this as two separate scripts, but then I have to pipe the data around and it would become messier. It would be best for the rest of the program to keep everything on one script.
'''python
def regular_ping(IP):
last_status = None
while True:
present_status = ping_status(IP) #ping_status(IP) being another
#program that will return info I
#need
if present_status != last_status:
notify_output(present_status) #notify_output(msg) being a
#program that will notify me of
# a change
last_status = present_status
time.sleep(300)
'''
I would like this bit of code to run on its own, notifying me of a change (if there is one) every 5 minutes, while the rest of my program also runs and accepts inputs. Instead, the program stops at this function and won't run past it. Any help would be much appreciated, thanks!

You can use a thread or a process for this. But since this is not a CPU bound operation, overhead of dedicating a process is not worth it. So a thread would be enough. You can implement it as follows:
import threading
thread = threading.Thread(target=regular_ping, args=(ip,))
thread.start()
# Rest of the program
thread.join()

A Process to check if Infinite Loop is still running in Python3

I am unable to grasp this with the help of Programming concepts in general with the following scenario:
Note: All Data transmission in this scenario is done via UDP packets using socket module of Python3
I have a Server which sends some certain amount of data, assume 300 Packets over a WiFi Channel
At the other end, I have a receiver which works on a certain Decoding process to decode the data. This Decoding Process is kind of Infinite Loop which returns Boolean Value true or false at every iteration depending on certain aspects which can be neglected as of now
a Rough Code Snippet is as follows:Python3
incomingPacket = next(bringNextFromBuffer)
if decoder.consume_data(incomingPacket):
# this if condition is inside an infinite loop
# unless the if condition becomes True keep
# keep consuming data in a forever for loop
print("Data has been received")
Everything as of moment works since the Server and Client are in proximity and the data can be decoded. But in practical scenarios I want to check the loop that is mentioned above. For instance, after a certain amount of time, if the above loop is still in the Forever (Infinite) state I would like to send out something back to the server to start the data sending again.
I am not much clear with multithreading concept, but can I use a thread over here in this scenario?
For Example:
Thread a Process for a certain amount of time and keep checking the decoder.consume_data() function and if the time expires and the output is still False can I then send out a kind of Feedback to the server using struct.pack() over sockets.
Of course the networking logic, need NOT be addressed as of now. But is python capable of MONITORING THIS INFINITE LOOP VIA A PARALLEL THREAD OR OTHER CONCEPT OF PROGRAMMING?
Caveats
Unfortunately the Receiver in question is a dumb receiver i.e. No user control is specified. Only thing Receiver can do is decode the data and perhaps send a Feedback to the Server stating whether the data is received or not and that is possible only when the above mentioned LOOP is completed.
What is a possible solution here?
(Would be happy to share more information on request)

Yes you can do this. Roughly it'll look like this:
from threading import Thread
from time import sleep
state = 'running'
def monitor():
while True:
if state == 'running':
tell_client()
sleep(1) # to prevent too much happening here
Thread(target=monitor).start()
while state == 'running':
receive_data()

Interact with long running python process

I have a long running python process running headless on a raspberrypi (controlling a garden) like so:
from time import sleep
def run_garden():
while 1:
/* do work */
sleep(60)
if __name__ == "__main__":
run_garden()
The 60 second sleep period is plenty of time for any changes happening in my garden (humidity, air temp, turn on pump, turn off fan etc), BUT what if i want to manually override these things?
Currently, in my /* do work */ loop, i first call out to another server where I keep config variables, and I can update those config variables via a web console, but it lacks any sort of real time feel, because it relies on the 60 second loop (e.g. you might update the web console, and then wait 45 seconds for the desired effect to take effect)
The raspberryPi running run_garden() is dedicated to the garden and it is basically the only thing taking up resources. So i know i have room to do something, I just dont know what.
Once the loop picks up the fact that a config var has been updated, the loop could then do exponential backoff to keep checking for interaction, rather than wait 60 seconds, but it just doesnt feel like that is a whole lot better.
Is there a better way to basically jump into this long running process?

Listen on a socket in your main loop. Use a timeout (e.g. of 60 seconds, the time until the next garden update should be performed) on your socket read calls so you get back to your normal functionality at least every minute when there are no commands coming in.
If you need garden-tending updates to happen no faster than every minute you need to check the time since the last update, since read calls will complete significantly faster when there are commands coming in.

Python's select module sounds like it might be helpful.
If you've ever used the unix analog (for example in socket programming maybe?), then it'll be familiar.
If not, here is the select section of a C sockets reference I often recommend. And here is what looks like a nice writeup of the module.
Warning: the first reference is specifically about C, not Python, but the concept of the select system call is the same, so the discussion might be helpful.
Basically, it allows you to tell it what events you're interested in (for example, socket data arrival, keyboard event), and it'll block either forever, or until a timeout you specify elapses.
If you're using sockets, then adding the socket and stdin to the list of events you're interested in is easy. If you're just looking for a way to "conditionally sleep" for 60 seconds unless/until a keypress is detected, this would work just as well.
EDIT:
Another way to solve this would be to have your raspberry-pi "register" with the server running the web console. This could involve a little bit extra work, but it would give you the realtime effect you're looking for.
Basically, the raspberry-pi "registers" itself, by alerting the server about itself, and the server stores the address of the device. If using TCP, you could keep a connection open (which might be important if you have firewalls to deal with). If using UDP you could bind the port on the device before registering, allowing the server to respond to the source address of the "announcement".
Once announced, when config. options change on the server, one of two things usually happen:
A) You send a tiny "ping" (in the general sense, not the ICMP host detection protocol) to the device alerting it that config options have changed. At this point the host would immediately request the full config. set, acquiring the update with it.
B) You send the updated config. option (or maybe the entire config. set) back to the device. This decreases the number of messages between the device and server, but would probably take more work as it seems like more a deviation from your current setup.

Why not use an event based loop instead of sleeping for a certain amount of time.
That way your loop will only run when a change is detected, and it will always run when a change is detected (which is the point of your question?).
You can do such a thing by using:
python event objects
Just wait for one or all of your event objects to be triggered and run the loop. You can also wait for X events to be done, etc, depending if you expect one variable to be updated a lot.
Or even a system like:
broadcasting events

How to break out of loop in subprocess in python

I feel like this should be an easy solution but it's the end of the day and I'm brain-dead.
I am currently spawning a couple of processes, one process is receiving and storing data to a file. Another is parsing the data and third is waiting for user input to know when to stop the storing of data.
What I need to know how to do is breakout of my while loop. I'd like to not use global variables set by the parent process but if that is required I can do that.
Right now my code looks something like this:
while(packetReceived < totalToReceive):
data, addr = sock.recvfrom(packetSize)
My thoughts were something like this:
breakout = 0
while(packetReceived < totalToReceive || breakout != 0):
data, addr = sock.recvfrom(packetSize)
but then I need to set breakout somehow. Any help would be greatly appreciated.

You can't share state just by having a global variable in the parent process. This may appear to work, but it only works sometimes; it's neither reliable nor predictable. Except that on Windows, it reliably and predictably never works; each child will always have its own independent copy of the flag, and therefore you will never quit.
If you really want to do this by sharing a variable, see Sharing state between processes in the docs, but the short version is: You create a multiprocessing.Value. And then you use a multiprocessing.Condition to protect that value against races, because otherwise, there's no guarantee that the child processes will ever see a change from the parent.
Of course you can fake this by, e.g. creating an mmap of minimum size and just using m[0] as a flag and m.flush() instead of the condition, but that's not really any simpler.
The alternative way to do this is to use a multiprocessing.Pipe or similar to pass a "shut down now" message. The child processes can each spawn a thread to block on the pipe, or you can toss the pipe and your socket into a select together, or all the other usual tricks.
There may be another, simpler option in this case: don't use multiprocessing in the first place. Clearly your background task is not CPU-bound, since it's just looping around reading from a socket, so why not just threading?
Also, it strikes me that you might be able to simplify your design in other ways, which could remove this problem entirely. Do you need a file between the reading and processing jobs instead of, say, a queue, or even just a direct sequential pipeline? Can you toss the user input and the socket into the same event loop (plain old select if user input is stdin and you don't care about Windows; use a QSocket instead of a socket.socket if user input is a Qt GUI; twisted if you're willing to learn twisted; etc.). Or, is there real user input, or just "quit now" (or "shut down the socket and process the remaining messages now"), which you could handle with ^C?

Instead of using a multiprocessing variable, consider checking for the presence of "poison pill" to break out of your loop.
For example, change:
data, addr = sock.recvfrom(packetSize)
to something like:
received = sock.recvfrom(packetSize)
if received is None:
break
data, addr = received
You can signal the process to break out of its loop by sending it a None value. I'm not sure if your sock can send/receive None, but the general idea is the same.

How to run two functions simultaneously

I am running test but I want to run 2 functions at the same time. I have a camera and I am telling it to move via suds, I am then logging into the camera via SSH to check the speed the camera is set to. When I check the speed the camera has stopped so no speed is available. Is there a way I can get these functions to run at the same time to test the speed of the camera. Sample code is below:
class VerifyPan(TestAbsoluteMove):
def runTest(self):
self.dest.PanTilt._x=350
# Runs soap move command
threading.Thread(target = SudsMove).start()
self.command = './ptzpanposition -c 0 -u degx10'
# Logs into camera and checks speed
TestAbsoluteMove.Ssh(self)
# Position of the camera verified through Ssh (No decimal point added to the Ssh value)
self.assertEqual(self.Value, '3500')
I have now tried the threading module as mentioned below. The thread does not run in sync with the function TestAbsoluteMove.Ssh(). Is there any other code I need to make this work.
I have looked at putting arguments into the thread statement that state the thread runs when the Ssh() function. Does anyone know what to enter in this statement?
Sorry if I haven't explained correctly. The 'SudsMove' function moves the camera and the 'Ssh' function logs into the camera and checks the speed the camera is currently moving at. The problem is that by the time the 'Ssh' function logs in the camera has stopped. I need both functions to run in parallel so I can check the camera speed while it is still moving.
Thanks

Import the threading module and run SudsMove() like so:
threading.Thread(target = SudsMove).start()
That will create and start a background thread which does the movement.
ANSWER TO EDITED QUESTION:
As far as I understand this, TestAbsoluteMove.Ssh(self) polls the speed once and stores the result in self.Value?! And you're testing the expected end tilt/rotation/position with self.assertEqual(self.Value, '3500')?!
If that's correct, you should wait for the camera to start its movement. You could probably poll the speed in a certain interval:
# Move camera in background thread
threading.Thread(target = SudsMove).start()
# What does this do?
self.command = './ptzpanposition -c 0 -u degx10'
# Poll the current speed in an interval of 250 ms
import time
measuredSpeedsList = []
for i in xrange(20):
# Assuming that this call will put the result in self.Value
TestAbsoluteMove.Ssh(self)
measuredSpeedsList.append(self.Value)
time.sleep(0.25)
print "Measured movement speeds: ", measuredSpeedsList
The movement speed will be the biggest value in measuredSpeedsList (i.e. max(measuredSpeedsList)). Hope that makes sense...

If you want to use the common Python implementation (CPython), you can certainly use the multiprocessing module, which does wonders (you can pass non-pickleable arguments to subprocesses, kill tasks,…), offers an interface similar to that of threads, and does not suffer from the Global Interpreter Lock.
The downside is that subprocesses are spawned, which takes more time than creating threads; this should only be a problem if you have many, many short tasks. Also, since data is passed (via serialization) between processes, large data both takes a long time to pass around and ends up having a large memory footprint (as it is duplicated between each process). In situations where each task takes a "long" time and the data in and out of each task is not too large, the multiprocessing module should be great.

There can only be one thread running at the same time. This has been answered extensively here. One solution will be to use two separate processes. The above answer provides some tips.

If you can get your code to run under Jython or IronPython, then you can run several threads simultaneously; they don't have that goofy "Global Interpreter Lock" thing of CPython.