I have a program that constantly runs if it receives an input, it'll do a task then go right back to awaiting input. I'm attempting to add a feature that will ping a gaming server every 5 minutes, and if the results every change, it will notify me. Problem is, if I attempt to implement this, the program halts at this function and won't go on to the part where I can then input. I believe I need multithreading/multiprocessing, but I have no experience with that, and after almost 2 hours of researching and wrestling with it, I haven't been able to figure it out.
I have tried to use the recursive program I found here but haven't been able to adapt it properly, but I feel this is where I was closest. I believe I can run this as two separate scripts, but then I have to pipe the data around and it would become messier. It would be best for the rest of the program to keep everything on one script.
'''python
def regular_ping(IP):
last_status = None
while True:
present_status = ping_status(IP) #ping_status(IP) being another
#program that will return info I
#need
if present_status != last_status:
notify_output(present_status) #notify_output(msg) being a
#program that will notify me of
# a change
last_status = present_status
time.sleep(300)
'''
I would like this bit of code to run on its own, notifying me of a change (if there is one) every 5 minutes, while the rest of my program also runs and accepts inputs. Instead, the program stops at this function and won't run past it. Any help would be much appreciated, thanks!
You can use a thread or a process for this. But since this is not a CPU bound operation, overhead of dedicating a process is not worth it. So a thread would be enough. You can implement it as follows:
import threading
thread = threading.Thread(target=regular_ping, args=(ip,))
thread.start()
# Rest of the program
thread.join()
Related
I am designing a trading strategy to work with binance API as my research project. The following is the basic scheme of program
The program needs to wait to receive a line of data (kline) from the API. The time needs to be different for different pairs and at different time of the day (which is set in a separate trading logic file). In short, the waiting time is not fixed (hence I am using threading.Condition() with .wait() and .notify() functions)
Once the data is available, condition.notify() comes into action and technical analysis is performed on the line received
Because I want the program to run forever, the thread is started in a while loop
However, during my testing phase, after receiving around 12000 lines of data, the script gave me the following error
File "/usr/lib/python3.8/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
I am thinking the system ran out of memory because it is creating a new thread during every iteration of the loop. Is there a better way to make the program wait to receive data? or to better manage the thread responsible for waiting?
During the testing phase, I am reading the data from csv files line by line.
tc_kline_received = threading.Condition()
for file in list_of_files:
with open(file, "r") as csv_file:
file_reader = reader(csv_file)
for line in file_reader:
btc_busd_5_min.append(line)
The following function pops one line from the list and stores in a separate variable. It is also used to for the thread responsible for waiting
def collect_kline():
global btc_busd_5_min
global kline_btc_busd_5_min
global klines_dict
kline_btc_busd_5_min = btc_busd_5_min.pop(0)
klines_dict["btc_busd_5_min"] = kline_btc_busd_5_min
with tc_kline_received:
tc_kline_received.notify()
Below is the never ending while loop.
while True:
with tc_kline_received:
t_collect_kline = threading.Thread(target=collect_kline)
t_collect_kline.start()
tc_kline_received.wait()
t_collect_kline.join()
insert_kline_to_db(klines_dict)
create_ta_db(klines_dict)
The last two functions are to create a sqlite db with technical analysis.
I am thinking the system ran out of memory because it is creating a new thread during every iteration of the loop.
The loop creates a new thread on every iteration, but it cannot get past the t_collect_kline.join() line until the new thread is finished, so there will never be more than one "collect_kline()" thread running at any given moment in time. If your program is running out of memory, it's not because of too many threads.
Unfortunately, I don't know what half of the lines in that program actually do, so I can't say what might be using up a lot of memory.
About those threads though...
...Each time your main loop creates a new thread, it then does nothing until the thread is finished. That doesn't seem useful. The whole point of creating a new thread is that the new thread can do one thing, while the original thread does some other thing. But, your original thread does nothing else. It never makes any sense to create a new thread if the very next thing you do is wait for it to end.
You could make your program smaller, simpler, and probably faster if you'd just do this in your main loop instead:
while True:
global btc_busd_5_min
global klines_dict
klines_dict["btc_busd_5_min"] = btc_busd_5_min.pop(0)
insert_kline_to_db(klines_dict)
create_ta_db(klines_dict)
But like I said, if you've got an out-of-memory problem, then this change isn't going to fix it. It will only change when and where the out-of-memory condition gets reported.
My script has to run over a day and its core cycle runs 2-3 times per a minute. I used multiprocessing to give a command simultaneously and each of them will be terminated/join within one cycle.
But in reality I found the software end up out of swap memory or computer freezing situation, I guess this is caused by accumulated processes. I can see on another session while running program, python PID abnormally increasing by time. So I just assume this must be something process thing. What I don't understand is how it happens though I made sure each cycle's process has to be finished on that cycle before proceed the next one.
so I am guessing, actual computing needs more time to progress 'terminate()/join()' job, so I should not "reuse" same object name. Is this proper guessing or is there other possibility?
def function(a,b):
try:
#do stuff # audio / serial things
except:
return
flag_for_2nd_cycle=0
for i in range (1500): # main for running long time
#do something
if flag_for_2nd_cycle==1:
while my_process.is_alive():
if (timecondition) < 30: # kill process if it still alive
my_process.terminate()
my_process.join()
flag_for_2nd_cycle=1
my_process=multiprocessing.process(target=function, args=[c,d])
my_process.start()
#do something and other process jobs going on, for example
my_process2 = multiprocessing.process() ##*stuff
my_process2.terminate()
my_process2.join()
Based on your comment, you are controlling three projectors over serial ports.
The simplest way to do that would be to simply open three serial connections (using pySerial). Then run a loop where you check for available data each of the connections and if so, read and process it. Then you send commands to each of the projectors in turn.
Depending on the speed of the serial link you might not need more than this.
I'm making a personal assistant like Google Assistant or Siri, and I want the user to be able to set reminders. For example, if they type "Remind me to wash the dishes at 5pm" I would like it to pop up later and remind them. However I also want code to be able to run while waiting, so you could set multiple reminders or check the weather.
time.sleep simply stops the program. I'm pretty sure there's a way to do it with threads but I'm not sure how. Please help!
Python threading has a Timer which does exactly what you ask for:
from datetime import datetime
from threading import Timer
def create_notification(time, name):
delay = (time - datetime.now()).total_seconds()
Timer(delay, show_notification, args=[name]).start()
def show_notification(name):
print(f'notification: {name}!')
create_notification(datetime(2034, 1, 1), 'Hello future!')
One thing to watch out for is this approach creates a single thread for each event (which doesn't scale well for lots of events). This also suffers from the problem that if the user closes your program, your program crashes, computer shuts down, power loss, etc. you lose all of your notifications. If you need to handle this, then save them to a file. If you need the notifications to show up even when your program isn't running look into solutions provided by the OS like cronjobs.
I have a Python script running selenium framework in a command line and running continuously to control Chrome do background data processing and monitoring. My simplified code structure is as following
while True:
Do_task1() # a blocking function
Do_task2() # another blocking fucntion
... # calling many blocking functions below, including wait(), time.sleep()
I need a way to interrupt the script anytime and anywhere to pause, terminate safely and give commands. What are the best ways to do this?
I've thought and tried of several ways but I am not exactly sure how to approach it:
I tried this:
if msvcrt.kbhit():
key = msvcrt.getch().decode("utf-8").lower()
if key == "j":
self.setting1 = True
elif key == "k":
self.setting2 = True
in the while loop, but it has to pass through bunch of blocking calls before reacting to my keypresses. And the command isn't exactly accepting my keyboard input in real time, that is I'll enter a character input, and I think it will be in the background buffer, then once the code execution reaches the code above, it starts to do stuff.
For terminating the script, I just do Ctrl-C in the CMD window, which I don't think it's the best way, and I should properly end the program ending background processes like Chromedriver.
I thought of having a GUI which runs somehow asynchronously and I can have buttons for pausing, and terminating the script. But I am not exactly sure how to approach it, or if this is a good idea to even try. Any suggestion is welcomed.
Use a script monitoring/workflow monitoring framework like AirBnB's Airflow or Luigi. I had only done brief research on this.
A related question but I don't need to return exactly where it's left off
I usually use try and except to do this
while True:
try:
Do_Task1()
Do_Task2()
except KeyBoardInterrrupt:
break
I currently have a python script that does exactly what I need it to do, however every now and then the script will hang and the only way to restart it is by killing the script and relaunching it.
I was wondering if there was a way to put in a few commands that will restart it lets say everytime it hangs or when a specific message appears or even just restart it on a timer eg:every 50 seconds.
I cannot provide the code through here, but I can provide it if we talk in private.
I am willing to pay you a bit of money if your fix does work.
please email me at stackoverflow1#shaw.ca
Thanks!
Edit: I see, ok - then is it possible to provide me with some codes which it will restart on a specific timer?
Edit2: Ok thanks everyone for their comments - I will get in touch with the person who built it to see if they can rewrite it from scratch to include a timer.
Cheers.
Feel free to pay me if you want, although it is by no means necessary.
Here:
import time
import threading
import os
def restart():
time.sleep(50)
os.execv('/full/path/to/this/script', ['second argument', 'third argument'])
def main():
t = threading.Thread(target=restart, args=(), name='reset')
t.start()
# ... The rest of your code.
If you have any buffers open that you care about (such as stdout) you'll want to flush them right before the call to execv up there.
I haven't tested this code, because I don't have a python interpreter handy at the moment, but I'd be surprised if it didn't work. That call to execv replaces the current context, so you don't get an increasingly deep hierarchy of child processes. All I'm doing, in case you're curious and want to know what magic phrase to google, is setting a "timer interrupt handler". For the pedants, no, I recognize this thing isn't directly handling any interrupts.
The numeric argument to sleep is in seconds. I would simply request that you not use my code in malware, unless it is for research purposes. I'm particular that way.
edit: Additionally, a lot of it was taken from here.