I am trying to emulated the generation of a sequence of numbers as shown in the code below. I want to execute this at regular intervals without time drift and without keeping track of the number of times serial_sequence() has been called, i.e. without using the variable num.
How can this be done?
In the example code, delay2 is always 0.1, but it should be slightly less since the event loop is busy elsewhere between calls to serial_sequence(). Thus the elapsed time is over 2.0 seconds.
import asyncio
import time
rx_data = list()
EOT = 20
async def serial_sequence():
'''async generator function that simulates incoming sequence of serial data'''
num = 0
tprev = time.time()
while True:
dt = time.time() - tprev
delay2 = 0.1 - dt #Want to avoid time drift
print('dt: {}'.format(dt))
print('delay2: {}'.format(delay2))
await asyncio.sleep(delay2) #simulated IO delay
tprev = time.time()
num += 1
yield num
async def read_serial1():
gen = serial_sequence()
while(True):
data = await gen.__anext__()
rx_data.append(data)
print('read_serial1:', data)
if data == EOT:
break
return rx_data
async def main():
start = time.time()
task1 = asyncio.create_task(read_serial1())
await(task1)
stop = time.time()
print('Elapsed: {}'.format(stop-start))
if __name__ == '__main__':
asyncio.run(main())
The code in the while loop itself needs some time to compute. The timedrift in your example accumulates because you base dt on the time of the previous timestep tprev.
You could instead use the absolute starting time of serial_sequence as point of reference like so:
async def serial_sequence():
'''async generator function that simulates incoming sequence of serial data'''
num = 0
starttime = time.time()
while True:
dt = (time.time() - starttime) % 0.1
delay2 = 0.1 - dt #Want to avoid time drift
print('dt: {}'.format(dt))
print('delay2: {}'.format(delay2))
await asyncio.sleep(delay2) #simulated IO delay
tprev = time.time()
num += 1
yield num
Compare the accumulation by changing EOT. I.e. Doubling EOT results in double the time drift for your solution, while it is approximately constant for this one.
My answer is largely based on this answer to a very similar question.
Related
I have a program that I created using threads, but then I learned that threads don't run concurrently in python and processes do. As a result, I am trying to rewrite the program using multiprocessing, but I am having a hard time doing so. I have tried following several examples that show how to create the processes and pools, but I don't think it's exactly what I want.
Below is my code with the attempts I have tried. The program tries to estimate the value of pi by randomly placing points on a graph that contains a circle. The program takes two command-line arguments: one is the number of threads/processes I want to create, and the other is the total number of points to try placing on the graph (N).
import math
import sys
from time import time
import concurrent.futures
import random
import multiprocessing as mp
def myThread(arg):
# Take care of imput argument
n = int(arg)
print("Thread received. n = ", n)
# main calculation loop
count = 0
for i in range (0, n):
x = random.uniform(0,1)
y = random.uniform(0,1)
d = math.sqrt(x * x + y * y)
if (d < 1):
count = count + 1
print("Thread found ", count, " points inside circle.")
return count;
# end myThread
# receive command line arguments
if (len(sys.argv) == 3):
N = sys.argv[1] # original ex: 0.01
N = int(N)
totalThreads = sys.argv[2]
totalThreads = int(totalThreads)
print("N = ", N)
print("totalThreads = ", totalThreads)
else:
print("Incorrect number of arguments!")
sys.exit(1)
if ((totalThreads == 1) or (totalThreads == 2) or (totalThreads == 4) or (totalThreads == 8)):
print()
else:
print("Invalid number of threads. Please use 1, 2, 4, or 8 threads.")
sys.exit(1)
# start experiment
t = int(time() * 1000) # begin run time
total = 0
# ATTEMPT 1
# processes = []
# for i in range(totalThreads):
# process = mp.Process(target=myThread, args=(N/totalThreads))
# processes.append(process)
# process.start()
# for process in processes:
# process.join()
# ATTEMPT 2
#pool = mp.Pool(mp.cpu_count())
#total = pool.map(myThread, [N/totalThreads])
# ATTEMPT 3
#for i in range(totalThreads):
#total = total + pool.map(myThread, [N/totalThreads])
# p = mp.Process(target=myThread, args=(N/totalThreads))
# p.start()
# ATTEMPT 4
# with concurrent.futures.ThreadPoolExecutor() as executor:
# for i in range(totalThreads):
# future = executor.submit(myThread, N/totalThreads) # start thread
# total = total + future.result() # get result
# analyze results
pi = 4 * total / N
print("pi estimate =", pi)
delta_time = int(time() * 1000) - t # calculate time required
print("Time =", delta_time, " milliseconds")
I thought that creating a loop from 0 to totalThreads that creates a process for each iteration would work. I also wanted to pass in N/totalThreads (to divide the work), but it seems that processes take in an iterable list rather than an argument to pass to the method.
What is it I am missing with multiprocessing? Is it at all possible to even do what I want to do with processes?
Thank you in advance for any help, it is greatly appreciated :)
I have simplified your code and used some hard-coded values which may or may not be reasonable.
import math
import concurrent.futures
import random
from datetime import datetime
def myThread(arg):
count = 0
for i in range(0, arg[0]):
x = random.uniform(0, 1)
y = random.uniform(0, 1)
d = math.sqrt(x * x + y * y)
if (d < 1):
count += 1
return count
N = 10_000
T = 8
_start = datetime.now()
with concurrent.futures.ThreadPoolExecutor() as executor:
futures = {executor.submit(myThread, (int(N / T),)): _ for _ in range(T)}
total = 0
for future in concurrent.futures.as_completed(futures):
total += future.result()
_end = datetime.now()
print(f'Estimate for PI = {4 * total / N}')
print(f'Run duration = {_end-_start}')
A typical output on my machine looks like this:-
Estimate for PI = 3.1472
Run duration = 0:00:00.008895
Bear in mind that the number of threads you start is effectively managed by the ThreadPoolExecutor (TPE) [ when constructed with no parameters ]. It makes decisions about the number of threads that can run based on your machine's processing capacity (number of cores etc). Therefore you could, if you really wanted to, set T to a very high number and the TPE will block execution of any new threads until it determines that there is capacity.
I'm trying to learn how to do parallel programming in python. I wrote a simple int square function and then ran it in serial, multi-thread, and multi-process:
import time
import multiprocessing, threading
import random
def calc_square(numbers):
sq = 0
for n in numbers:
sq = n*n
def splita(list, n):
a = [[] for i in range(n)]
counter = 0
for i in range(0,len(list)):
a[counter].append(list[i])
if len(a[counter]) == len(list)/n:
counter = counter +1
continue
return a
if __name__ == "__main__":
random.seed(1)
arr = [random.randint(1, 11) for i in xrange(1000000)]
print "init completed"
start_time2 = time.time()
calc_square(arr)
end_time2 = time.time()
print "serial: " + str(end_time2 - start_time2)
newarr = splita(arr,8)
print 'split complete'
start_time = time.time()
for i in range(8):
t1 = threading.Thread(target=calc_square, args=(newarr[i],))
t1.start()
t1.join()
end_time = time.time()
print "mt: " + str(end_time - start_time)
start_time = time.time()
for i in range(8):
p1 = multiprocessing.Process(target=calc_square, args=(newarr[i],))
p1.start()
p1.join()
end_time = time.time()
print "mp: " + str(end_time - start_time)
Output:
init completed
serial: 0.0640001296997
split complete
mt: 0.0599999427795
mp: 2.97099995613
However, as you can see, something weird happened and mt is taking the same time as serial and mp is actually taking significantly longer (almost 50 times longer).
What am I doing wrong? Could someone push me in the right direction to learn parallel programming in python?
Edit 01
Looking at the comments, I see that perhaps the function not returning anything seems pointless. The reason I'm even trying this is because previously I tried the following add function:
def addi(numbers):
sq = 0
for n in numbers:
sq = sq + n
return sq
I tried returning the addition of each part to a serial number adder, so at least I could see some performance improvement over a pure serial implementation. However, I couldn't figure out how to store and use the returned value, and that's the reason I'm trying to figure out something even simpler than that, which is just dividing up the array and running a simple function on it.
Thanks!
I think that multiprocessing takes quite a long time to create and start each process. I have changed the program to make 10 times the size of arr and changed the way that the processes are started and there is a slight speed-up:
(Also note python 3)
import time
import multiprocessing, threading
from multiprocessing import Queue
import random
def calc_square_q(numbers,q):
while q.empty():
pass
return calc_square(numbers)
if __name__ == "__main__":
random.seed(1) # note how big arr is now vvvvvvv
arr = [random.randint(1, 11) for i in range(10000000)]
print("init completed")
# ...
# other stuff as before
# ...
processes=[]
q=Queue()
for arrs in newarr:
processes.append(multiprocessing.Process(target=calc_square_q, args=(arrs,q)))
print('start processes')
for p in processes:
p.start() # even tho' each process is started it waits...
print('join processes')
q.put(None) # ... for q to become not empty.
start_time = time.time()
for p in processes:
p.join()
end_time = time.time()
print("mp: " + str(end_time - start_time))
Also notice above how I create and start the processes in two different loops, and then finally join with the processes in a third loop.
Output:
init completed
serial: 0.53214430809021
split complete
start threads
mt: 0.5551605224609375
start processes
join processes
mp: 0.2800724506378174
Another factor of 10 increase in size of arr:
init completed
serial: 5.8455305099487305
split complete
start threads
mt: 5.411392450332642
start processes
join processes
mp: 1.9705185890197754
And yes, I've also tried this in python 2.7, although Threads seemed slower.
I am creating a program that will measure the execution times of various sorting algorithms (Selection, Bubble, Merge, and Tree sort).
The list sizes used for the test cases should start at 10,000, and go up by 10,000 for each test until the execution time for the test exceeds 60 seconds.
And that is my issue.
I have this probably very wrong (and ugly) code that I have created (I am currently testing with just the Bubble Sort).
import random
import time
def bubbleSort(a_list):
for passnum in range(len(a_list)-1,0,-1):
for i in range(passnum):
if a_list[i]>a_list[i+1]:
temp = a_list[i]
a_list[i] = a_list[i+1]
a_list[i+1] = temp
a_list = []
for i in range(10000):
a_list.append(random.randrange(0,10000))
start = time.perf_counter()
bubbleSort(a_list)
end = time.perf_counter()
elapsed = end - start
print("{0:.8f}".format(elapsed, "\n"))
print(a_list)
if elapsed <= 60:
for i in range(len(a_list), len(a_list)+10000):
a_list.append(random.randrange(len(a_list)+10000))
start = time.perf_counter()
bubbleSort(a_list)
end = time.perf_counter()
elapsed = end - start
print("{0:.8f}".format(elapsed, "\n"))
print(a_list)
else:
#it'll quit
I'm sorry for the ignorance that is very apparent. So above was my first reaction. Then I came up with this loop:
start = time.perf_counter()
while start <= 60:
for i in range(len(a_list)+10000):
a_list.append(random.randrange(len(a_list)+10000))
bubbleSort(a_list)
end = time.perf_counter()
elapsed = end - start
print("{0:.8f}".format(elapsed, "\n"))
print(a_list)
I would be very grateful if someone can give me a push in the right direction and help me think of the logic behind it. Thank you much in advance.
First, collapse some of the code to improve readability:
Element switch now uses the Python idiom a, b = b, a
Build the list with a comprehension, not a loop
parametrize the list size; increment each time through the loop.
Do you really need 8 decimal places for the execution time?
Code:
import random
import time
def bubbleSort(a_list):
for passnum in range(len(a_list)-1,0,-1):
for i in range(passnum):
if a_list[i] > a_list[i+1]:
a_list[i], a_list[i+1] = a_list[i+1], a_list[i]
elapsed = 0
size = 0
size_inc = 10000
print("Size\tTime")
while elapsed < 60:
# Add 10,000 numbers to the list
size += size_inc
a_list = [random.randrange(0,size) for i in range(size)]
start = time.perf_counter()
bubbleSort(a_list)
end = time.perf_counter()
elapsed = end - start
print(size, "\t{0:.8f}".format(elapsed, "sec.\n"))
Output:
Size Time
10000 12.05934826
20000 47.99201040
30000 111.39582218
I want to parallelize a task using python, so I read about pool.map, where data is divided into multiple chunks and processed by each process (thread).
I have a huge dictionary(2 million words) and a text file of sentences, the idea is to divide sentences into words and match each word to the exiting dictionary and do further processing based on the return result. Before doing that ,I wrote a dummy program to check the functionality of pool.map but it is not working as expected (i.e single process takes less time than multiple process) (I am using process and thread interchangeably because I think every thread is nothing but a process here)
def add_1(x):
return (x*x+x)
def main():
iter = 10000000
num = [i for i in xrange(iter)]
threads = 4
pool = ThreadPool(threads)
start = time.time()
results = pool.map(add_1,num,iter/threads)
pool.close()
pool.join()
end = time.time()
print('Total Time Taken = %f')% (end-start)
Total Time Taken:
Thread 1 Total Time Taken = 2.252361
Thread 2 Total Time Taken = 2.560798
Thread 3 Total Time Taken = 2.938640
Thread 4 Total Time Taken = 3.048179
Just using pool = ThreadPool()
def main:
num = [i for i in xrange(iter)]
#pool = ThreadPool(threads)
pool = ThreadPool()
start = time.time()
#results = pool.map(add_1,num,iter/threads)
results = pool.map(add_1,num)
pool.close()
pool.join()
end = time.time()
print('Total Time Taken = %f')% (end-start)
Total Time Taken = 3.031125
Normal for loop execution:
def main():
iter = 10000000
start = time.time()
for k in xrange(iter):
add_1(k)
end = time.time()
print ('Total Time normally = %f') % (end-start)
Total Time normally = 1.087591
Config:
I am using python 2.7.6
I want to compute how many times my computer can do counter += 1 in one second. A naive approach is the following:
from time import time
counter = 0
startTime = time()
while time() - startTime < 1:
counter += 1
print counter
The problem is time() - startTime < 1 may be considerably more expensive than counter += 1.
Is there a way to make a less "clean" 1 sec sample of my algorithm?
The usual way to time algorithms is the other way around: Use a fixed number of iterations and measure how long it takes to finish them. The best way to do such timings is the timeit module.
print timeit.timeit("counter += 1", "counter = 0", number=100000000)
Note that timing counter += 1 seems rather pointless, though. What do you want to achieve?
Why don't you infer the time instead? You can run something like:
from datetime import datetime
def operation():
counter = 0
tbeg = datetime.utcnow()
for _ in range(10**6):
counter += 1
td = datetime.utcnow() - tbeg
return (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10**6)/10.0**6
def timer(n):
stack = []
for _ in range(n):
stack.append(operation()) # units of musec/increment
print sum(stack) / len(stack)
if __name__ == "__main__":
timer(10)
and get the average elapsed microseconds per increment; I get 0.09 (most likely very inaccurate). Now, it is a simple operation to infer that if I can make one increment in 0.09 microseconds, then I am able to make about 11258992 in one second.
I think the measurements are very inaccurate, but maybe is a sensible approximation?
I have never worked with the time() library, but according to that code I assume it counts seconds, so what if you do the /sec calculations after ctrl+C happens? It would be something like:
#! /usr/bin/env python
from time import time
import signal
import sys
#The ctrl+C interruption function:
def signal_handler(signal, frame):
counts_per_sec = counter/(time()-startTime)
print counts_per_sec
exit(0)
signal.signal(signal.SIGINT, signal_handler)
counter = 0
startTime = time()
while 1:
counter = counter + 1
Of course, it wont be exact because of the time passed between the last second processed and the interruption signal, but the more time you leave the script running, the more precise it will be :)
Here is my approach
import time
m = 0
timeout = time.time() + 1
while True:
if time.time() > timeout:
break
m = m + 1
print(m)