I wrote proxy checker and I need to automate checking for every 5 minutes. My while True loop skips the functions. When I have first iteration - functions are running, but with next iterations they are skipped, just printing START!!!!!! and END---- in my console.
if __name__ == "__main__":
while True:
print("START!!!!!!!!!!")
http_proxy_check()
save_proxy('http')
print("END------------------")
time.sleep(2)
http_proxy_check function:
def http_proxy_check():
for proxy in http_proxyList:
print(f"Checking {http_proxyList.index(proxy)} of {len(http_proxyList)} element")
thread = Thread(target=checkIp_http,args=(proxy.strip(), ))
if len(threads)<=1200 :
thread.start()
threads.append(thread)
else:
threads.clear()
for thread in threads:
thread.join()
You are directly trying to run subthreads. Why are you not using multiprocess?
Also you can try to "timeloop" library for periodic tasks;
https://pypi.org/project/timeloop/
pip install timeloop
import time
from timeloop import Timeloop
from datetime import timedelta
tl = Timeloop()
#tl.job(interval=timedelta(minutes=5))
def sample_job_every_5m():
print "5m job current time : {}".format(time.ctime())
while True: # infinite loop
your_code_that_process() # put your code that do stuff once and exit
time.sleep(5*60) # 300 seconds of sleeping, make it wait 300sec
if check_for_abort_if_so_break(): # you can make flag that you trigger on error or by input
# so you can exit out of infinite loop
break # exit infinite loop (not so infinite anymore)
Related
I am trying to make an attendance system and right now, I want to create a random password, and put it on a countdown so that as it runs out, the student can't use the code anymore. However, when I try to run it, it only displays the password and the countdown, and only asks for input after the timer runs out.
I have attempted to use a for loop as well as the multiprocessing module to no avail. I suspect that the error is located somewhere around my use of the threads.
import threading
#create code and timer
Thread1 = threading.Thread(target=generateCodeandTimer(600))
# make input
Thread2 = threading.Thread(target=attend)
# Start the thread
Thread1.start()
# Start the thread
Thread2.start()
But for reference, this is my full code:
import string
import random
import time
import sys
import threading
code = ""
def generateCodeandTimer(s):
global code
code = ''.join((random.choice(string.ascii_lowercase + string.digits) for x in range(6)))
print("Attendance code:", code)
while s != -1:
mins = s // 60
secs = s % 60
countdown = '{:02d}:{:02d}'.format(mins, secs)
sys.stdout.write('\r' + countdown)
time.sleep(1)
s -= 1
if s==-1:
print()
print("Code expired")
def attend():
print()
studentinput = input("Please enter the code")
if studentinput == code:
print()
print("Your attendance has been taken")
else:
print()
print("Wrong code!")
#create code and timer
Thread1 = threading.Thread(target=generateCodeandTimer(600))
# make input
Thread2 = threading.Thread(target=attend)
# Start the thread
Thread1.start()
# Start the thread
Thread2.start()
In this line:
Thread1 = threading.Thread(target=generateCodeandTimer(600))
you are actually calling the function generateCodeandTimer. The target keyword requires a function object, but this code calls the function and then passes the result as the target of the thread.
The second time you started a thread, you got it right:
Thread2 = threading.Thread(target=attend)
Note the difference: target=attend passes the function object attend because you do not CALL the function. If you had written target=attend(), you would have called the function and passed its result as the target.
The solution is found in the documentation for the Thread constructor. Change the first thread creation to this:
Thread1 = threading.Thread(target=generateCodeandTimer, args=(600,))
The comma after 600 is necessary because the args= keyword requires a tuple.
Your program will now run as you intend. You will discover some other problems - for example, the program won't exit immediately when the user types in the password. But I will let you figure those out, or ask more questions if you run into trouble.
I want to execute a task after certain time, so I have tried a countdown timer with a condition of being finished (when countdown variable = 0, the task is performed). The thing is that I don't want to stop the execution of the main program while performing the countdown. I have tried this:
import time
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
return num_of_secs
So, I run my code setting a number of seconds to the countdown, and when this countdown reaches the 0 value, a task must be executed. Using this code (it uses a while), when I call my function "countdown" it stops the execution of the main program, so it is the same as a big time.sleep. I want to carry out this countdown in the background, without stopping other actions until the countdown finishes and the task starts.
Thank you
Another alternative is by using threading.
I've got a simple example here with 2 Threads where the working thread is waiting for the countdown thread to finish and starting. The Main is still working fine.
import threading
import time
def do_something():
countdown_thread.join()
print("Starting Task")
time.sleep(3)
print("Finished Task")
def countdown(num_of_secs):
while(num_of_secs):
time.sleep(1)
num_of_secs -= 1
print(num_of_secs)
if __name__ == '__main__':
countdown_thread = threading.Thread(target=countdown, args=(3,))
work_thread = threading.Thread(target=do_something)
countdown_thread.start()
work_thread.start()
while True:
print("Main doing something")
time.sleep(1)
Example picture for multithreading: Sequential vs Threading
Usually python only has a single program flow, so every instruction needs to complete before the next one can get executed.
For your case you need asynchronicity, with e.g. asyncio.sleep(5) as a separate task in the same event loop.
import asyncio
async def sleeper():
print('Holding...')
await asyncio.sleep(5)
print('Doing Work!')
async def work():
print('Doing work')
print('while')
print('the other guy is sleeping')
async def main():
await asyncio.gather(sleeper(), work())
asyncio.run(main())
The most common and easiest way to implement this would be with a Timer object from the threading library. It would go as follows:
import threading
import time
i = 0
done = False
def show_results():
print("results from GPIO readings")
print("=)")
global done
done = True # signal end of while loop
def read_GPIO():
print("reading GPIO...")
t = threading.Timer(60, show_results) # task will trigger after 60 seconds
t.start()
# your while loop would go here
read_GPIO() # do work
while not done:
print("waiting", i) # doing work while waiting for timer
time.sleep(1)
i += 1
pass
Notice that the time library is used only for illustrative purposes. You could also start the timer recursively to check periodically GPIOs and print results or trigger an event. For more information on the threading library or the Timer object check the docs
I am trying to add a progress bar based on elapsed time in Python2.7. Basically, I am executing a bash script from python that takes about 90 minutes to complete, but the underlying application being called doesn't provide information that often. There are about 45 minutes where the user wouldn't see anything happening and I don't want them to kill the script thinking the process has frozen. Because I know approximately how long the executable takes to finish, I want to add a progress bar based on the elapsed time. Here is what I have so far:
def sys_call(self, cmd, log_file="prog.log"):
# setup toolbar and start timer
start = datetime.datetime.now()
stdo = sys.stdout
msg = "elapsed {}".format(datetime.timedelta())
stdo.write(msg)
stdo.write("\b"*len(msg)) # go to beginning of line
stdo.flush() # flush buffer
# initiate command and monitor
log = open(log_file, 'w')
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
for line in iter(process.stdout.readline, ''):
log.write(line.decode('utf-8'))
# display toolbar
msg = "elapsed {}".format(datetime.datetime.now() - start)
stdo.write(msg)
stdo.write("\b"*len(msg)) # go to beginning of line
stdo.flush() # flush buffer
log.close()
I am looking for a way to update the progress in the terminal based on the elapsed time. The line loop above stops looping for long periods of time after the script is executed because the application stops sending anything to stdout, so my elapsed time stops updating which is misleading. I can take care of calculating and printing a percentage for the user to see the process is not frozen, but I'm not sure how to run the elapsed time code in a separate thread.
I looked into progressbar and tqdm, but unfortunately, I cannot use third-party packages due to requirements outside my control.
I followed the blow snippet from this link
# Python program killing
# threads using stop
# flag
import threading
import time
def run(stop):
while True:
print('thread running')
if stop():
break
def main():
stop_threads = False
t1 = threading.Thread(target = run, args =(lambda : stop_threads, ))
t1.start()
time.sleep(1)
stop_threads = True
t1.join()
print('thread killed')
main()
Ultimately, here is what I came up with:
import datetime
import time
def thread_progressbar(self, stop):
"""Progress Bar Thread
This function is to be used with `threading`. Reference example.
Note, ensure the thread is a daemon otherwise CTRL+C will not
work and if this occurs, you will have to kill the thread from
the command line even though `main()` has already terminated.
Example
```
thread_stop = False
t = threading.Thread(target=thread_progressbar, args=(lambda:thread_stop,))
t.daemon = True # make daemon to kill thread if main terminates
t.start() # start executing thread
<do something else>
thread_stop = True # break out of inf loop
t.join() # kill thread
```
"""
start = datetime.datetime.now()
while True:
# get elapsed time and stay at 99 if we exceed expected time
elapsed = datetime.datetime.now() - start
percent = int(100.0*(elapsed.total_seconds()/self.expect_time.total_seconds()))
percent = 99 if percent >= 100 else percent
# display toolbar
sys.stdout.write("[{}%](elapsed {})".format(percent, elapsed))
sys.stdout.write("\b"*len(msg)) # go to beginning of line
sys.stdout.flush() # flush buffer
# use stop to determine when we are done
if stop():
elapsed = datetime.datetime.now() - start
sys.stdout.write("[100%](elapsed {})\n".format(elapsed))
break
time.sleep(1)
Use '\r' to move cursor back to beginning of the line after every time you print the bar, and just loop the bar until the task is done making sure to have '\r' at the end of every iteration.
This is an example of how that would be used
import time
for x in range(101):
print(f'{x}% done',end='\r')
time.sleep(1)
I need help please, I use schedule.every().day.at("17:40").do(my_function) and I would like my program to run normally and when the schedule.every().day.at("17:40").do(my_function) arrives, it executes the associated function but then it comes back in my loop and wait for another day etc.... I dont know how to do it because i think schedule.every().day.at("17:40").do(my_function) need
while1:
schedule.run_pending()
time.sleep(1)
So i dont know how to changes this 3 lignes to make my programme work.
Thanks!
You would have to run it in separated threading or multiprocessing.
But first you should check documentation because I found in Common Questions:
How to continuously run the scheduler without blocking the main thread?
They created class Scheduler which put it in thread and you need to run run_continuously()
But I use it to created shorter example
import schedule
import time
import threading
# --- functions ---
stop_running = threading.Event() # to control loop in thread
def run_continuously(scheduler, interval=1):
#print('starting loop in thread')
while not stop_running.is_set():
schedule.run_pending()
time.sleep(interval)
#print('stoping loop in thread')
def job():
print("I'm working...")
# --- main ---
schedule.every(1).minutes.do(job)
# run schedule in thread
schedule_in_thread = threading.Thread(target=run_continuously, args=(schedule,))
schedule_in_thread.start()
# run other code
#print('starting main loop')
try:
while True:
print("other code")
time.sleep(3)
except KeyboardInterrupt as ex:
print('stoping', ex)
#print('stoping main loop')
# stop schedule in thread
stop_running.set() # to stop loop in `run_continuously`
schedule_in_thread.join() # wait until thread finish
I use try/except with KeyboardInterrupt only to gracefully stop program when I press Ctrl+C - and code may stop thread.
I want to execute FUNCTION periodically every 60 seconds, but I don't want to execute FUNCTION again IF the previous run has not completed yet. If the previous run completes in e.g. 120s then I want to execute a new FUNCTION call straight away. If previous run completed in e.g. 10s then I want to wait 50s before I execute a new FUNCTION call.
Please see my implementation below.
Can I achieve it with e.g. subprocess.run or some timeloop library so that the implementation would be much cleaner?
import time
def hello(x):
# some logic here
# execution could take any time between e.g. <10s, 120s>
def main(ii):
while True:
start = int(time.time())
try:
val = next(ii)
except StopIteration as ex:
return None
else:
hello(val)
run_time_diff = int(time.time()) - start
if run_time_diff < 60:
time.sleep(60 - run_time_diff)
ii = iter(list[[...],[...],...[...]])
main(ii=ii)
maybe apsheduler could help you. But if your job wil run more then waiting time, it could be skipped. In this case you can increase number of workers.
import datetime
import time
from apscheduler.schedulers.background import BackgroundScheduler
scheduler = BackgroundScheduler()
def some_job():
time.sleep(5)
print(f"{datetime.datetime.utcnow()}: Every 10 seconds")
job = scheduler.add_job(some_job, 'interval', seconds=10, max_instances=1)
scheduler.start()
try:
while True:
time.sleep(1)
finally:
scheduler.shutdown()