Use of python progress bar in script - python

I want to show progress bar in my script because it takes a lots of time to execute while working on huge files.I gone through the python progressbar module
and examples also its good and very intresting to use but as per examples all values are predefine .As we can't guess the max execution time of programm or function.So i am not able to figure out how should i use progress bar function in my sctipt
for data in files:
crawl_data(data)
this is the crawl_data function which take time so how can i set the progress bar values
pbar = ProgressBar(widgets=[Percentage(), Bar()], maxval=300).start()
for i in range(300):
time.sleep(0.01)
pbar.update(i+1)
pbar.finish()
how can i define this range and maxval values in above lines of code.

This is what I got working.
Stdout
Working: | Elapsed Time: 0:00:10
Python
import time
import progressbar
import threading
def crawl_data(data):
# sleep for 10 seconds
time.sleep(10)
# end of def crawl_data
def main():
data = 'some data'
widgets = ['Working: ', progressbar.AnimatedMarker(), ' ',
progressbar.Timer()]
pbar = progressbar.ProgressBar(widgets=widgets)
# creating thread to run crawl_data()
thread = threading.Thread(target=crawl_data,
args=(data,))
thread.daemon = True
# starting thread and progress bar
thread.start()
pbar.start()
i = 1
# continuous loop until crawl_data thread is not alive
while True:
# update every second
time.sleep(1)
pbar.update(i)
if not thread.is_alive():
pbar.finish()
break
# end of if thread is not alive
i += 1
# end of continuous loop until crawl_data thread is not alive
# prints a new line
print
# end of def main
# run main
main()

If you can't guess the execution time, a progress bar is worthless (remember most of the old MS progress bars?). You are probably looking for something like a activity indicator. Since web2.0 it is common to use something rotating.

Related

How do I thread these functions?

I am trying to code an alarm in python that has 6 functions that need to multithread. 5 of these are alarms and one of them displays the time. The threads need to start and stop whenever the menu option is selected and when the alarm rings. The Display thread is the only thread that keeps going until the program stops. My current code for the alarm looks like this (I've removed a lot for the sake of clarity)
class TAlarm1 (threading.Thread):
def Alarm1():
while True:
#code which keeps running until the time is equal to the input given (expected to thread)
thread1 = threading.Thread(target=TAlarm1)
thread1.start()
def AlarmSelector():
print("Select an Alarm") #5 alarms will be added however each one accomplishes the same task. all of them need to run simultaneously
choice = int(input())
if choice == 1:
ala = TAlarm1()
ala.Alarm1()
if choice == 6:
DisplayTime() #goes back to displaying time
Whenever I run this code, the program displays no errors however it does not run the code in TAlarm1().
How can I solve this problem?
While your intent isn't clear to me. Here is how you can subclass Thread with its run method overridden and start it conditionally.
import threading
class TAlarm1 (threading.Thread):
def run(self):
n =4
while True:
#code which keeps running until the time is equal to the input given (expected to thread)
print(n,end=' | ')
n -= 1
if n < 0:
break
print()
t1 = TAlarm1()
if True:
t1.start()
A thread can only be started once so you have to make a new one every time you need it to run.
>>> t = TAlarm1()
>>> t.start()
4 | 3 | 2 | 1 | 0 |
>>> t.start()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python38\lib\threading.py", line 848, in start
raise RuntimeError("threads can only be started once")
RuntimeError: threads can only be started once
>>> t = TAlarm1()
>>> t.start()
4 | 3 | 2 | 1 | 0 |
>>>
The target parameter of Thread takes a callable. A class is a callable, but calling it just creates an instance of the class. Pass it a function instead:
import threading
def Alarm1():
print('Alarm1 called')
thread1 = threading.Thread(target=Alarm1)
thread1.start()
There are two basic ways of implementing threaded code in Python. You seem to have half of each.
The first implementation model is to put the logic to run in the thread into a function, then pass that function as the target argument when you create a threading.Thread instance:
import threading
import time
def worker(n):
for i in range(n):
print(i)
time.sleep(0.5)
my_thread = threading.Thread(target=worker, args=(10,))
my_thread.start()
# do other stuff in the main thread, if desired
my_thread.join()
The other implementation approach is to subclass threading.Thread and put the code to be run in the thread inside of the run method (or in other methods called from run). This is especially useful if your thread code has some complicated state and you want to be able to use additional methods to manipulate that state while the thread is running:
class MyThread(threading.Thread):
def __init__(self, n):
super().__init__()
self.n = n
self.unpaused = threading.Event()
self.unpaused.set() # we start unpaused
def run(self):
for i in range(self.n):
self.unpaused.wait() # block if we're paused
print(i)
time.sleep(0.5)
def pause(self):
self.unpaused.clear()
def unpause(self):
self.unpaused.set()
my_thread = MyThread(10)
my_thread.start()
# an example of inter-thread communication, we pause and unpause our thread using its methods
time.sleep(2)
my_thread.pause()
time.sleep(2)
my_thread.unpause()
my_thread.join()

Updating Popup.Animated to play gif until external task is completed (PYSimpleGUI)

I am looking to create a UI that displays an animated popup while another task is being carried out. That will exit upon completion. I am using PYSimpleGUI and am using the example listed here to base my work off. I can get a single frame of the animation to display once I start the code and exit upon completion of the task but can't get it to play the entire gif. Code:
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
#time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
#sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s'%work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
#for i in range(200000):
work_id = work_id+1 if work_id < 19 else 0
#while True:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
#if message == None:
#break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
sg.PopupAnimated(None)
#window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
#window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program'
)
You've got your call to popup_animated inside of an "if" statement that is only executed once.
You must call popup_animated for every frame you wish to show. It's not spun off as a task that works in the background.
This change to your code will keep the animation going as long as there as background tasks running.
import queue
import threading
import time
import PySimpleGUI as sg
# ############################# User callable CPU intensive code #############################
# Put your long running code inside this "wrapper"
# NEVER make calls to PySimpleGUI from this thread (or any thread)!
# Create one of these functions for EVERY long-running call you want to make
def long_function_wrapper(work_id, gui_queue):
# LOCATION 1
# this is our "long running function call"
# time.sleep(10) # sleep for a while as a simulation of a long-running computation
x = 0
while True:
print(x)
time.sleep(0.5)
x = x + 1
if x == 5:
break
# at the end of the work, before exiting, send a message back to the GUI indicating end
gui_queue.put('{} ::: done'.format(work_id))
# at this point, the thread exits
return
def the_gui():
gui_queue = queue.Queue() # queue used to communicate between the gui and long-running code
layout = [[sg.Text('Multithreaded Work Example')],
[sg.Text('This is a Test.', size=(25, 1), key='_OUTPUT_')],
[sg.Text(size=(25, 1), key='_OUTPUT2_')],
[sg.Button('Go'), sg.Button('Exit')], ]
window = sg.Window('Multithreaded Window').Layout(layout)
# --------------------- EVENT LOOP ---------------------
work_id = 0
while True:
event, values = window.Read(timeout=100) # wait for up to 100 ms for a GUI event
if event is None or event == 'Exit':
# sg.PopupAnimated(None)
break
if event == 'Go': # clicking "Go" starts a long running work item by starting thread
window.Element('_OUTPUT_').Update('Starting long work %s' % work_id)
# LOCATION 2
# STARTING long run by starting a thread
thread_id = threading.Thread(target=long_function_wrapper, args=(work_id, gui_queue,), daemon=True)
thread_id.start()
# for i in range(200000):
work_id = work_id + 1 if work_id < 19 else 0
# while True:
# if message == None:
# break
# --------------- Read next message coming in from threads ---------------
try:
message = gui_queue.get_nowait() # see if something has been posted to Queue
except queue.Empty: # get_nowait() will get exception when Queue is empty
message = None # nothing in queue so do nothing
# if message received from queue, then some work was completed
if message is not None:
# LOCATION 3
# this is the place you would execute code at ENDING of long running task
# You can check the completed_work_id variable to see exactly which long-running function completed
completed_work_id = int(message[:message.index(' :::')])
window.Element('_OUTPUT2_').Update('Finished long work %s' % completed_work_id)
work_id -= 1
if not work_id:
sg.PopupAnimated(None)
if work_id:
sg.PopupAnimated(sg.DEFAULT_BASE64_LOADING_GIF, background_color='white', time_between_frames=100)
# window['_GIF_'].update_animation(sg.DEFAULT_BASE64_LOADING_GIF, time_between_frames=100)
# window.read(timeout = 1000)
# if user exits the window, then close the window and exit the GUI func
window.Close()
############################# Main #############################
if __name__ == '__main__':
the_gui()
print('Exiting Program')

Let Python keep loop statement running and check the condition every 3 seconds

I want to keep the loop conditional statement running, but do not always check the conditions.
For example, if the condition is true, then in the next 3 seconds, the loop's conditional statement will run, and then check the condition after the 3rd second, then repeat this process.
I don't want to wait or sleep for three seconds, I want my loop to do work for three seconds. And then check if it should continue for another three as mentioned by #RemcoGerlich
while if_active() == True: #check the condition every 3 seconds`
try: # it will keep running in 3 seconds if if_active() is true
with open(masterpath, 'r') as f:
s = f.read()
exec(s)
Here's a fun and async way. Just for fun, with a demo for activateing
import signal, os
import time
def handler(signum, frame):
for i in range(3):
print("foo bar")
time.sleep(0.1)
signal.alarm(3)
# Set the signal handler and a 5-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(3)
while True:
try:
active = not active
if not active:
signal.alarm(0)
time.sleep(60)
except KeyboardInterrupt as interrupt:
# demonstrating activate, with ctrl+c
signal.alarm(3)
You can track when the last time was that you did the check, and only re-do the check if three seconds have passed.
from datetime import datetime, timedelta
INTERVAL = timedelta(minutes=3)
last_checked = datetime.now() - INTERVAL
while True:
now = datetime.now()
if last_checked <= (now - INTERVAL):
if not if_active():
break
last_checked = now
# do your thing here
pass
This could use some refactoring but the idea should work.
You can use a command like sleep to avoid running continuously. You can see a more explained answer in this thread: How can I make a time delay in Python?

How to use Timer Thread with Python

I am writing a Ryu application(Python) in which I have if else statement. If a condition satisfies for the first time, then it should start the timer till 10 seconds, within these 10 seconds there will be other packets arriving as well matching the same condition but I don't want to start timer every time a condition is satisfied(within these 10 seconds). In short, the timer should run in parallel.
Here is the code snippet I used for thread.
Every time I run this and send multiple packets then multiple threads start whereas I want only one thread to run till 10 seconds
def timeit():
time.sleep(10)
aggr()
return
def aggr():
self.no_of_data=len(self.iot_data)
self.ip_proto=proto
self.ip_saddr=source
self.ip_daddr=destination
ip_head= pack('!BBHHHBBH16s16s' , self.ip_ihl_ver, self.ip_tos, self.ip_tot_len, self.ip_id, self.ip_frag_off, self.ip_ttl,self.ip_check,self.ip_proto, self.ip_saddr, self.ip_daddr)
total_pkts= pack('!I', self.no_of_data)
print "TOTALLLL,,,,",self.no_of_data
ip_head="{" + ip_head + "}"
total_pkts="{" + total_pkts + "}"
s='$'
data = s.join(self.iot_data)
data="$" + data
pckt= ip_head + total_pkts + data
self.iot_data = []
print "BUFFER: ", self.iot_data
self.iot_data_size = 0
self.start_time = time.time()
self.logger.info("packet-out %s" % (repr(pckt),))
out_port = ofproto.OFPP_FLOOD
actions = [parser.OFPActionOutput(out_port)]
out = parser.OFPPacketOut(datapath=datapath,
buffer_id=ofproto.OFP_NO_BUFFER,
in_port=in_port, actions=actions,
data=pckt)
print "out--->" , out
datapath.send_msg(out)
thread1 = threading.Thread(target=timeit)
thread1.start()
if proto == 150 and total_len < 1500:
if not thread1.isAlive():
thread1.run()
print "ifff"
data = msg.data
#print " # stores the packet data"
self.iot_data.append(data)
#print "# increment size counter"
self.iot_data_size += total_len
#elapsed_time = time.time() - self.start_time
print "ELAPSED: ", elapsed_time
print "BUFFER: ", self.iot_data
After 10 seconds, again timer should start when the first packet arrives and it should run parallel with the same code.
I am so much confused with this. Please anyone help.
I hope this is clear if not I am sorry please ask for the clarification.
Thank you
Indeed, you have to go with multi-threading (might be achieved without it but it would certainly be a pain in the ass). The idea is to run a thread that will run a function that sleeps for 10 seconds and returns. After this function returns, the thread will be set as inactive, until we run it the next time.
By knowing that we can write the following code. All details and explanations are written as comments for easier reference.
import time
import threading
packet_groups = [] # Groups of packets inside 10 seconds.
group = [] # Temporary group that will get stored into packet_groups.
# The function that will count 10 seconds:
def timeit():
sleep(10)
return
# Do something with packets.
def packet_handler():
...
# Put all your code inside another function that does not create
# new thread each time. Create a thread in main and then run this function.
def get_packets(thread1):
... # get packets
if dst == 'some_address':
# Check if thread is alive. If it is alive, a counter is running.
# If it is not alive, then we must start the counter by running
# thread.
if not thread1.isAlive():
thread1.run()
packet_handler(packet, True)
else:
packet_handler(packet, False)
if __name__ == '__main__':
# Create thread.
thread1 = threading.Thread(target=timeit)
# Start the thread. This is done only once per each thread.
thread1.start()
get_packets(thread1)
Now since you mentioned that you want to group the packets inside these 10 seconds blocks, you can implement packet_handler() like this:
def packet_handler(packet, new):
# If we started new thread and the group isn't empty, we must
# append group to packet_groups (that is all groups) and reset
# the group to only contain current packet
if new and group != []:
packet_groups.append(group)
group = [packet]
return
# If group isn't new, we are still inside 10 seconds block. We
# just append the current packet to this block.
if not new:
group.append(packet)
If you want to be able to print or in any other way be able to show the timer, you can't sleep for 10 seconds because if you sleep for 10 seconds, nothing will be done in between. In such case you want to change timeit() to something like this:
def timeit():
for i in range(10):
print 'Time remaining: {0}s'.format(10-i)
sleep(1)
return

How do I schedule a one-time script to run x minutes from now? (alternative for 'at')

(Background: I'd like to control a light source with a motion sensor. Light should turn off x minutes after last detected motion. The framework is in place, scheduling is what remains to be done.)
Currently, when motion is detected the light gets turned on and a job to turn it off in 'now + x minutes' is scheduled. Whenever motion is detected during the x minutes the job gets removed from the queue and a new one is set up, extending effectively the time the light stays on.
I tried the "at" command but job handling is quite clunky. Whenever a job is removed from the queue an email gets sent. I looked at the Python crontab module but it would need much additional programming (handling relative time, removing old cronjobs, etc.) and seems to be slower.
What are my alternatives (bash, python, perl)?
-- Edit: My python skills are at beginner level, here's what I put together:
#!/usr/bin/env python2.7
# based on http://raspi.tv/2013/how-to-use-interrupts-with-python-on-the-raspberry-pi-and-rpi-gpio-part-2
# more than 160 seconds without activity are required to re-trigger action
import time
from subprocess import call
import os
import RPi.GPIO as GPIO
PIR = 9 # data pin of PIR sensor (in)
LED = 7 # positive pin of LED (out)
timestamp = '/home/pi/events/motiontime' # file to store last motion detection time (in epoch)
SOUND = '/home/pi/events/sounds/Hello.wav' # reaction sound
# GPIO setup
GPIO.setmode(GPIO.BCM)
GPIO.setup(PIR,GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
GPIO.setup(LED,GPIO.OUT)
# function which gets called when motion is reported (sensor includes own delay-until-hot again
# and sensibility settings
def my_callback(channel):
now = time.time() # store current epoch time in variable 'now'
f = open(timestamp, "r")
then = float(f.readline()) # read last detection time from file
difference = now - then # calculate time that has passed
call(['/home/pi/bin/kitchenlights.sh', '-1']) # turn light on
call(['/home/pi/bin/lighttimer.sh']) # schedule at job to turn lights off
if difference > 160: # if more than 160 seconds without activity have passed then...
GPIO.output(LED, True) # turn on LED
if not os.path.isfile("/home/pi/events/muted"): # check if system is muted, else
call(['/usr/bin/mplayer', '-really-quiet', '-noconsolecontrols', SOUND]) # play sound
GPIO.output(LED, False) # turn of LED
f = open(timestamp, "w")
f.write(repr(now)) # update timestamp
f.close()
else: # when less than 160 seconds have passed do nothing and
f = open(timestamp, "w")
f.write(repr(now)) # update timestamp (thus increasing the interval of silence)
f.close()
GPIO.add_event_detect(PIR, GPIO.RISING,callback=my_callback,bouncetime=100) # add rising edge detection on a channel
while True:
time.sleep(0.2)
pass
Now that questions come in I think I could put a countdown in the while loop, right? How would that work?
I would approach this with the threading module. To do this, you'd set up the following thread class:
class CounterThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.count = 0
self.start()
def run(self):
while self.count < COUNTLIMIT:
time.sleep(0.1)
self.count += 0.1
#Call function to turn off light here
return
def newSig(self):
self.count = 0
This is a thread which everytime it recieves a new signal (the thread's newSig function is called), the counter restarts. If the COUNTLIMIT is reached (how long you want to wait in seconds), then you call the function to turn off the light.
Here's how you'd incorporate this into your code:
import threading
from subprocess import call
import os
import time
import RPi.GPIO as GPIO
PIR = 9 # data pin of PIR sensor (in)
LED = 7 # positive pin of LED (out)
SOUND = '/home/pi/events/sounds/Hello.wav' # reaction sound
COUNTLIMIT = 160
countThread = None
WATCHTIME = 600 #Run for 10 minutes
# GPIO setup
GPIO.setmode(GPIO.BCM)
GPIO.setup(PIR,GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
GPIO.setup(LED,GPIO.OUT)
#------------------------------------------------------------
class CounterThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.count = 0
self.start()
def run(self):
call(['/home/pi/bin/kitchenlights.sh', '-1']) # turn light on
while self.count < COUNTLIMIT:
time.sleep(0.1)
self.count += 0.1
call(['/home/pi/bin/kitchenlights.sh', '-0'])
threadKiller()
return
def newSig(self):
self.count = 0
#------------------------------------------------------------
def my_callback(channel):
'''function which gets called when motion is reported (sensor includes own delay-until-hot again and sensibility settings'''
global countThread
try:
countThread.newSig()
except:
countThread = CounterThread()
#------------------------------------------------------------
def threadKiller():
global countThread
countThread = None
#------------------------------------------------------------
def main():
GPIO.add_event_detect(PIR, GPIO.RISING,callback=my_callback,bouncetime=100) # add rising edge detection on a channel
t = 0
while t < WATCHTIME:
t += 0.1
time.sleep(0.1)
#------------------------------------------------------------
if __name__ == "__main__": main()
I don't have any way to test this, so please let me know if there is anything that breaks. Since you said you're new to Python I made a few formatting changes to make your code a bit prettier. These things are generally considered to be good form, but are optional. However, you need to be careful about indents, because as you have them in your question, your code should not run (it will throw an IndentError)
Hope this helps

Categories