I am writing a Ryu application(Python) in which I have if else statement. If a condition satisfies for the first time, then it should start the timer till 10 seconds, within these 10 seconds there will be other packets arriving as well matching the same condition but I don't want to start timer every time a condition is satisfied(within these 10 seconds). In short, the timer should run in parallel.
Here is the code snippet I used for thread.
Every time I run this and send multiple packets then multiple threads start whereas I want only one thread to run till 10 seconds
def timeit():
time.sleep(10)
aggr()
return
def aggr():
self.no_of_data=len(self.iot_data)
self.ip_proto=proto
self.ip_saddr=source
self.ip_daddr=destination
ip_head= pack('!BBHHHBBH16s16s' , self.ip_ihl_ver, self.ip_tos, self.ip_tot_len, self.ip_id, self.ip_frag_off, self.ip_ttl,self.ip_check,self.ip_proto, self.ip_saddr, self.ip_daddr)
total_pkts= pack('!I', self.no_of_data)
print "TOTALLLL,,,,",self.no_of_data
ip_head="{" + ip_head + "}"
total_pkts="{" + total_pkts + "}"
s='$'
data = s.join(self.iot_data)
data="$" + data
pckt= ip_head + total_pkts + data
self.iot_data = []
print "BUFFER: ", self.iot_data
self.iot_data_size = 0
self.start_time = time.time()
self.logger.info("packet-out %s" % (repr(pckt),))
out_port = ofproto.OFPP_FLOOD
actions = [parser.OFPActionOutput(out_port)]
out = parser.OFPPacketOut(datapath=datapath,
buffer_id=ofproto.OFP_NO_BUFFER,
in_port=in_port, actions=actions,
data=pckt)
print "out--->" , out
datapath.send_msg(out)
thread1 = threading.Thread(target=timeit)
thread1.start()
if proto == 150 and total_len < 1500:
if not thread1.isAlive():
thread1.run()
print "ifff"
data = msg.data
#print " # stores the packet data"
self.iot_data.append(data)
#print "# increment size counter"
self.iot_data_size += total_len
#elapsed_time = time.time() - self.start_time
print "ELAPSED: ", elapsed_time
print "BUFFER: ", self.iot_data
After 10 seconds, again timer should start when the first packet arrives and it should run parallel with the same code.
I am so much confused with this. Please anyone help.
I hope this is clear if not I am sorry please ask for the clarification.
Thank you
Indeed, you have to go with multi-threading (might be achieved without it but it would certainly be a pain in the ass). The idea is to run a thread that will run a function that sleeps for 10 seconds and returns. After this function returns, the thread will be set as inactive, until we run it the next time.
By knowing that we can write the following code. All details and explanations are written as comments for easier reference.
import time
import threading
packet_groups = [] # Groups of packets inside 10 seconds.
group = [] # Temporary group that will get stored into packet_groups.
# The function that will count 10 seconds:
def timeit():
sleep(10)
return
# Do something with packets.
def packet_handler():
...
# Put all your code inside another function that does not create
# new thread each time. Create a thread in main and then run this function.
def get_packets(thread1):
... # get packets
if dst == 'some_address':
# Check if thread is alive. If it is alive, a counter is running.
# If it is not alive, then we must start the counter by running
# thread.
if not thread1.isAlive():
thread1.run()
packet_handler(packet, True)
else:
packet_handler(packet, False)
if __name__ == '__main__':
# Create thread.
thread1 = threading.Thread(target=timeit)
# Start the thread. This is done only once per each thread.
thread1.start()
get_packets(thread1)
Now since you mentioned that you want to group the packets inside these 10 seconds blocks, you can implement packet_handler() like this:
def packet_handler(packet, new):
# If we started new thread and the group isn't empty, we must
# append group to packet_groups (that is all groups) and reset
# the group to only contain current packet
if new and group != []:
packet_groups.append(group)
group = [packet]
return
# If group isn't new, we are still inside 10 seconds block. We
# just append the current packet to this block.
if not new:
group.append(packet)
If you want to be able to print or in any other way be able to show the timer, you can't sleep for 10 seconds because if you sleep for 10 seconds, nothing will be done in between. In such case you want to change timeit() to something like this:
def timeit():
for i in range(10):
print 'Time remaining: {0}s'.format(10-i)
sleep(1)
return
Related
I want to start thread multiple times, but only when it is not running.
There is a simple model what I am trying:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
if not H.isAlive():
H.start()
print(threading.active_count())
Also what i don't understand is that:
When I run program it prints: 0. Then after 3 seconds it prints:1 and so on after every 3 second it is increased by 1.
But I thought it would print: 0. Then after 3 second it would print: 1. And then immediately increasing fast.
Because after starting first thread it would immediately start the next one and so on. why does this happen?
How not to start a thread 'up', if it's already running?
Not sure if I got your question completely, but here are some thoughts.
When I run your code I get an increasing number of active threads, as you are creating a new thread every time, checking its status (which will always be not alive) and then starting it.
What you want to do instead is to check the status of the last run thread and if that's not alive start a new one. In order to do that you should create a new thread if the old one is done:
def up (x, r):
time.sleep(3)
r['h'] = x + 1
def main():
hum = {'h' : 0}
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
while True:
# print(hum['h'])
if not H.isAlive():
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
print(threading.active_count())
What happens in your code:
Print the value of hum['h']
Create a thread (note you create it, you are not starting it yet)
Set the value of a property
If the thread is not started then start it
Print the count of active threads (active, NOT started)
Since you replace the H variable every time, you'll have a new thread every time that gets immediately started.
If you add a print that says "starting" in the if for the is alive, you'll see that it gets called every time.
You can use join() to wait for the thread to finish:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
while True:
print(hum['h'])
H = threading.Thread(target = up, args=(hum['h'],hum))
H.daemon=True
H.start()
H.join()
print(threading.active_count())
If you don't want to wait, you can just save the current running thread in a variable and check it in the loop:
import threading
import time
def up (x, r):
time.sleep(3)
r['h'] = x + 1
hum = {'h' : 0}
current_thread = None
while True:
print(hum['h'])
if current_thread is None:
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()
elif not current_thread.isAlive():
current_thread = threading.Thread(target = up, args=(hum['h'],hum))
current_thread.daemon=True
current_thread.start()
I have a program written in python which goes as follows:
for i in range(4954):
........
save("/Downloads/pos_" + str(i) + ".h5")
The fact is the program running time increases exponentially(we observed it using time.time())
So what I need is run the program for 10 min and then re run the program.
But I just need to change the i in for loop to the number at which it is stopped.
I can do a cron job, but what should I do to change i value?
You could use marshal.dump(https://docs.python.org/3/library/marshal.html) to save the value when stopping the program, and then loading it via marshal.load when starting.
Honestly, I think it would be a much better approach to fix the root cause of the problem, i.e. solving the exponential run time.
If each iteration (each i value) in the loop is independent with each other, you can parallelize it with ProcessPoolExecutor:
import concurrent.futures
def downloader(i):
...
save("/Downloads/pos_" + str(i) + ".h5")
def main():
i_range = list(range(4954))
with concurrent.futures.ProcessPoolExecutor(10) as executor: # create 10 workers (subprocesses)
for number, result in zip(i_range , executor.map(downloader, i_range)):
print('File %d is scheduled to download!' % (number))
if __name__ == '__main__':
main()
This requires Python3.
Consider using a generator:
import time
def gen(n):
for i in range(n):
yield n
g = gen(4954)
time_limit = 600
while some_condition:
# whatever condition required for you to continually restart this section
start = time.time()
while time.time() - start < time_limit:
... some code...
try:
save("/Downloads/pos_" + str(next(g)) + ".h5")
except StopIteration:
print('end of iterations')
Where g will remember the iteration it's on and continue the next time you call next(g).
Here my solution:
import os
import time
import threading
class test_class(object):
def __init__(self):
self.iter_number = 0
self.new_iter_number = 0
self.iter_file_dir = "/tmp_file.txt" # tmp file where sto the new iteration number remaining
self.end_iteration = False # Flag to stop the iteration
self.delay = 1 # Seconds that means 10 minutes
with open(self.iter_file_dir, "r") as file:
self.iter_number = int(file.readline())
self.run()
def for_function(self):
for i in range(self.iter_number):
save("/Downloads/pos_" + str(i) + ".h5")
if(self.end_iteration):
self.new_iter_number = self.iter_number - i
break
def run(self):
threading.Thread(name="Iteration_Thread", target=self.for_function).start()
time_remaining = self.delay-time.time()%self.delay
time.sleep(time_remaining)
self.end_iteration = True
# Save the file with the new value
with open(self.iter_file_dir, 'w') as f:
f.write(str(self.new_iter_number))
test_class()
The idea behind this code is simple:
Create a tmp_file.txt where you want, and write in them the number of iteration that you need in the for.
Create a timer that sleep for X minutes and after that set a flag that terminate the for loop.
After that save in the tmp_file.txt the new calculated iteration number
Each time that the cron run the program it will load the new iteration value. Obviously when the iteration are finished in the file the 0 is written, so no more iteration were performed.
I am launching concurrent threads doing some stuff:
concurrent = 10
q = Queue(concurrent * 2)
for j in range(concurrent):
t = threading.Thread(target=doWork)
t.daemon = True
t.start()
try:
# process each line and assign it to an available thread
for line in call_file:
q.put(line)
q.join()
except KeyboardInterrupt:
sys.exit(1)
At the same time I have a distinct thread counting time:
def printit():
threading.Timer(1.0, printit).start()
print current_status
printit()
I would like to increase (or decrease) the amount of concurrent threads for the main process let's say every minute. I can make a time counter in the time thread and make it do things every minute but how to change the amount of concurrent threads in the main process ?
Is it possible (and if yes how) to do that ?
This is my worker:
def UpdateProcesses(start,processnumber,CachesThatRequireCalculating,CachesThatAreBeingCalculated,CacheDict,CacheLock,IdleLock,FileDictionary,MetaDataDict,CacheIndexDict):
NewPool()
while start[processnumber]:
IdleLock.wait()
while len(CachesThatRequireCalculating)>0 and start[processnumber] == True:
CacheLock.acquire()
try:
cacheCode = CachesThatRequireCalculating[0] # The list can be empty if an other process takes the last item during the CacheLock
CachesThatRequireCalculating.remove(cacheCode)
print cacheCode,"starts processing by",processnumber,"process"
except:
CacheLock.release()
else:
CacheLock.release()
CachesThatAreBeingCalculated.append(cacheCode[:3])
Array,b,f = TIPP.LoadArray(FileDictionary[cacheCode[:2]])#opens the dask array
Array = ((Array[:,:,CacheIndexDict[cacheCode[:2]][cacheCode[2]]:CacheIndexDict[cacheCode[:2]][cacheCode[2]+1]].compute()/2.**(MetaDataDict[cacheCode[:2]]["Bit Depth"])*255.).astype(np.uint16)).transpose([1,0,2]) #slices and calculates the array
f.close() #close the file
if CachesThatAreBeingCalculated.count(cacheCode[:3]) != 0: #if not, this cache is not needed annymore (the cacheCode is removed bij wavelengthchange)
CachesThatAreBeingCalculated.remove(cacheCode[:3])
try: #If the first time the object if not aivalable try a second time
CacheDict[cacheCode[:3]] = Array
except:
CacheDict[cacheCode[:3]] = Array
print cacheCode,"done processing by",processnumber,"process"
if start[processnumber]:
IdleLock.clear()
This is how I start them:
self.ProcessLst = [] #list with all the processes who calculate the caches
for processnumber in range(min(NumberOfMaxProcess,self.processes)):
self.ProcessTerminateLst.append(True)
for processnumber in range(min(NumberOfMaxProcess,self.processes)):
self.ProcessLst.append(process.Process(target=Proc.UpdateProcesses,args=(self.ProcessTerminateLst,processnumber,self.CachesThatRequireCalculating,self.CachesThatAreBeingCalculated,self.CacheDict,self.CacheLock,self.IdleLock,self.FileDictionary,self.MetaDataDict,self.CacheIndexDict,)))
self.ProcessLst[-1].daemon = True
self.ProcessLst[-1].start()
I close them like this:
for i in range(len(self.ProcessLst)): #For both while loops in the processes self.ProcessTerminateLst[i] must be True. So or the process is now ready to be terminad or is still in idle mode.
self.ProcessTerminateLst[i] = False
self.IdleLock.set() #Makes sure no process is in Idle and all are ready to be terminated
I would use a pool. a pool has a max number of threads it uses at the same time, but you can apply inf number of jobs. They stay in the waiting list until a thread is available. I don't think you can change number of current processes in the pool.
I want to show progress bar in my script because it takes a lots of time to execute while working on huge files.I gone through the python progressbar module
and examples also its good and very intresting to use but as per examples all values are predefine .As we can't guess the max execution time of programm or function.So i am not able to figure out how should i use progress bar function in my sctipt
for data in files:
crawl_data(data)
this is the crawl_data function which take time so how can i set the progress bar values
pbar = ProgressBar(widgets=[Percentage(), Bar()], maxval=300).start()
for i in range(300):
time.sleep(0.01)
pbar.update(i+1)
pbar.finish()
how can i define this range and maxval values in above lines of code.
This is what I got working.
Stdout
Working: | Elapsed Time: 0:00:10
Python
import time
import progressbar
import threading
def crawl_data(data):
# sleep for 10 seconds
time.sleep(10)
# end of def crawl_data
def main():
data = 'some data'
widgets = ['Working: ', progressbar.AnimatedMarker(), ' ',
progressbar.Timer()]
pbar = progressbar.ProgressBar(widgets=widgets)
# creating thread to run crawl_data()
thread = threading.Thread(target=crawl_data,
args=(data,))
thread.daemon = True
# starting thread and progress bar
thread.start()
pbar.start()
i = 1
# continuous loop until crawl_data thread is not alive
while True:
# update every second
time.sleep(1)
pbar.update(i)
if not thread.is_alive():
pbar.finish()
break
# end of if thread is not alive
i += 1
# end of continuous loop until crawl_data thread is not alive
# prints a new line
print
# end of def main
# run main
main()
If you can't guess the execution time, a progress bar is worthless (remember most of the old MS progress bars?). You are probably looking for something like a activity indicator. Since web2.0 it is common to use something rotating.
I've read a lot of posts about using threads, subprocesses, etc.. A lot of it seems over complicated for what I'm trying to do...
All I want to do is stop executing a function after X amount of time has elapsed.
def big_loop(bob):
x = bob
start = time.time()
while True:
print time.time()-start
This function is an endless loop that never throws any errors or exceptions, period.
I"m not sure the difference between "commands, shells, subprocesses, threads, etc.." and this function, which is why I'm having trouble manipulating subprocesses.
I found this code here, and tried it but as you can see it keeps printing after 10 seconds have elapsed:
import time
import threading
import subprocess as sub
import time
class RunCmd(threading.Thread):
def __init__(self, cmd, timeout):
threading.Thread.__init__(self)
self.cmd = cmd
self.timeout = timeout
def run(self):
self.p = sub.Popen(self.cmd)
self.p.wait()
def Run(self):
self.start()
self.join(self.timeout)
if self.is_alive():
self.p.terminate()
self.join()
def big_loop(bob):
x = bob
start = time.time()
while True:
print time.time()-start
RunCmd(big_loop('jimijojo'), 10).Run() #supposed to quit after 10 seconds, but doesn't
x = raw_input('DONEEEEEEEEEEEE')
What's a simple way this function can be killed. As you can see in my attempt above, it doesn't terminate after 20 seconds and just keeps on going...
***OH also, I've read about using signal, but I"m on windows so I can't use the alarm feature.. (python 2.7)
**assume the "infinitely running function" can't be manipulated or changed to be non-infinite, if I could change the function, well I'd just change it to be non infinite wouldn't I?
Here are some similar questions, which I haven't able to port over their code to work with my simple function:
Perhaps you can?
Python: kill or terminate subprocess when timeout
signal.alarm replacement in Windows [Python]
Ok I tried an answer I received, it works.. but how can I use it if I remove the if __name__ == "__main__": statement? When I remove this statement, the loop never ends as it did before..
import multiprocessing
import Queue
import time
def infinite_loop_function(bob):
var = bob
start = time.time()
while True:
time.sleep(1)
print time.time()-start
print 'this statement will never print'
def wrapper(queue, bob):
result = infinite_loop_function(bob)
queue.put(result)
queue.close()
#if __name__ == "__main__":
queue = multiprocessing.Queue(1) # Maximum size is 1
proc = multiprocessing.Process(target=wrapper, args=(queue, 'var'))
proc.start()
# Wait for TIMEOUT seconds
try:
timeout = 10
result = queue.get(True, timeout)
except Queue.Empty:
# Deal with lack of data somehow
result = None
finally:
proc.terminate()
print 'running other code, now that that infinite loop has been defeated!'
print 'bla bla bla'
x = raw_input('done')
Use the building blocks in the multiprocessing module:
import multiprocessing
import Queue
TIMEOUT = 5
def big_loop(bob):
import time
time.sleep(4)
return bob*2
def wrapper(queue, bob):
result = big_loop(bob)
queue.put(result)
queue.close()
def run_loop_with_timeout():
bob = 21 # Whatever sensible value you need
queue = multiprocessing.Queue(1) # Maximum size is 1
proc = multiprocessing.Process(target=wrapper, args=(queue, bob))
proc.start()
# Wait for TIMEOUT seconds
try:
result = queue.get(True, TIMEOUT)
except Queue.Empty:
# Deal with lack of data somehow
result = None
finally:
proc.terminate()
# Process data here, not in try block above, otherwise your process keeps running
print result
if __name__ == "__main__":
run_loop_with_timeout()
You could also accomplish this with a Pipe/Connection pair, but I'm not familiar with their API. Change the sleep time or TIMEOUT to check the behaviour for either case.
There is no straightforward way to kill a function after a certain amount of time without running the function in a separate process. A better approach would probably be to rewrite the function so that it returns after a specified time:
import time
def big_loop(bob, timeout):
x = bob
start = time.time()
end = start + timeout
while time.time() < end:
print time.time() - start
# Do more stuff here as needed
Can't you just return from the loop?
start = time.time()
endt = start + 30
while True:
now = time.time()
if now > endt:
return
else:
print end - start
import os,signal,time
cpid = os.fork()
if cpid == 0:
while True:
# do stuff
else:
time.sleep(10)
os.kill(cpid, signal.SIGKILL)
You can also check in the loop of a thread for an event, which is more portable and flexible as it allows other reactions than brute killing. However, this approach fails if # do stuff can take time (or even wait forever on some event).