We Keep Coding

Code runs slower with Multiprocessing.pool.map than without

This code simulates loading a CSV, parsing it and loading it int a pandas dataframe. I would like to parallelize this problem so that it runs faster, but my pool.map implementation is actually slower than the serial implementation.
The csv is read as one big string and first split into lines and then split into values. It is an irregularly formatted csv with recurring headers so I cannot use the pandas read_csv. At least not that I know how.
My idea was to simply read in the file as string, split the long string into four parts (one for each core) and then process each chunk separately in parallel. This it turns out is slower than the serial version.
from multiprocessing import Pool
import datetime
import pandas as pd
def data_proc(raw):
pre_df_list = list()
for item in (i for i in raw.split('\n') if i and not i.startswith(',')):
if ' ' in item and ',' in item:
key, freq, date_observation = item.split(' ')
date, observation = date_observation.split(',')
pre_df_list.append([key, freq, date, observation])
return pre_df_list
if __name__ == '__main__':
raw = '\n'.join([f'KEY FREQ DATE,{i}' for i in range(15059071)]) # instead of loading csv
start = datetime.datetime.now()
pre_df_list = data_proc(raw)
df = pd.DataFrame(pre_df_list, columns=['KEY','FREQ','DATE','VAL'])
end = datetime.datetime.now()
print(end - start)
pool = Pool(processes=4)
start = datetime.datetime.now()
len(raw.split('\n'))
number_of_tasks = 4
chunk_size = int((len(raw) / number_of_tasks))
beginning = 0
multi_list = list()
for i in range(1,number_of_tasks+1):
multi_list.append(raw[beginning:chunk_size*i])
beginning = chunk_size*i
results = pool.imap(data_proc, multi_list)
# d = results[0]
pool.close()
pool.join()
# I haven'f finished conversion to dataframe since previous part is not working yet
# df = pd.DataFrame(d, columns=['SERIES_KEY','Frequency','OBS_DATE','val'])
end = datetime.datetime.now()
print(end - start)
EDIT: the serial version finishes in 34 seconds and the parallel after 53 seconds on my laptop. When I started working on this, my initial assumption was that I would be able to get it down to 10-ish seconds on a 4 core machine.
It looks like the parallel version I posted never finishes. I changed the pool.map call to pool.imap and now it works again. Note it has to be ran from the command line, not Spyder.

In General:
Multiprocessing isn't always the best way to do things. It takes overhead to create and manage the new processes, and to synchronize their output. In a relatively simple case, like parsing 150M lines of small text, you may or may not get a big time savings from using the multiprocessor.
There are a lot of other confounding variables - process load on the machine, number of processors, any access to I/O being spread across the processor (that one is not a problem in your specific case), the potential to fill up memory and then deal with page swaps... The list can keep growing. There are times multiprocessing is the ideal, and there are times where it makes matters worse. (In my production code, I have one place where I left a comment: "Using multiprocesing here took 3x longer than not. Just using regular map...")
Your specific case
However, without knowing your exact system specifications, I think that you should have a performance improvment from properly done multiprocessing. You may not; this task may be small enough that it's not worth the overhead. However, there are several issues with your code that will result in your multiprocessing path taking longer. I'll call out the ones that catch my attention
len(raw.split('\n'))
This line is very expensive, and accomplishes nothing. It goes through every line of your raw data, splits it, takes the length of the result, then throws out the split data and the len. You likely want to do something like:
splitted = raw.split('\n')
splitted_len = len(splitted) # but I'm not sure where you need this.
This would save the split data, so you could make use of it later, in your for loop. As it is right now, your for loop operates on raw, which has not been split. So instead of, e.g., running on [first_part, second_part, third_part, fourth_part], you're running on [all_of_it, all_of_it, all_of_it, all_of_it]. This, of course, is a HUGE part of your performance degradation - you're doing the same work x4!
I expect, if you take care of splitting on \n outside of your processing, that's all you'll need to get an improvement from multiprocessing. (Note, you actually don't need any special processing for 'serial' vs. 'parallel' - you can test it decently by using map instead of pool.map.)
Here's my take at re-doing your code. It moves the line splitting out of the data_proc function, so you can focus on whether splitting the array into 4 chunks gives you any improvement. (Other than that, it makes each task into a well-defined function - that's just style, to help clarify what's testing where.)
from multiprocessing import Pool
import datetime
import pandas as pd
def serial(raw):
pre_df_list = data_proc(raw)
return pre_df_list
def parallel(raw):
pool = Pool(processes=4)
number_of_tasks = 4
chunk_size = int((len(raw) / number_of_tasks))
beginning = 0
multi_list = list()
for i in range(1,number_of_tasks+1):
multi_list.append(raw[beginning:chunk_size*i])
beginning = chunk_size*i
results = pool.map(data_proc, multi_list)
pool.close()
pool.join()
pre_df_list = []
for r in results:
pre_df_list.append(r)
return pre_df_list
def data_proc(raw):
# assume raw is pre-split by the time you're here
pre_df_list = list()
for item in (i for i in if i and not i.startswith(',')):
if ' ' in item and ',' in item:
key, freq, date_observation = item.split(' ')
date, observation = date_observation.split(',')
pre_df_list.append([key, freq, date, observation])
return pre_df_list
if __name__ == '__main__':
# don't bother with the join, since we would need it in either case
raw = [f'KEY FREQ DATE,{i}' for i in range(15059071)] # instead of loading csv
start = datetime.datetime.now()
pre_df_list = serial(raw)
end = datetime.datetime.now()
print("serial time: {}".format(end - start))
start = datetime.datetime.now()
pre_df_list = parallel(raw)
end = datetime.datetime.now()
print("parallel time: {}".format(end - start))
# make the dataframe. This would happen in either case
df = pd.DataFrame(pre_df_list, columns=['KEY','FREQ','DATE','VAL'])

workarounds for (pseudo-)decorating a for loop in python?

Of course I know this is not directly possible as in Python, as read in
Statement decorators
but still I would like to find a way to programmatically turn (on and off) a loop as:
for i in range(L[:]):
# do stuff
into
for i in range(L[0:N])):
# estimate how much time it
# took to run the loop over a subset N element of the list
for i in range(L):
# do stuff on the full list
Is there any pythonic way to do so?

Let's assume L is a list
import time
def timed_loop(L,subset,timeout):
start = time.time()
for i in L[0:subset]:
do_something()
stop = time.time()
if stop-start < timeout:
for i in L:
do_something_else()

Need to restart loop

Still a NOOB in Python. Get stuck many times.
Script runs 3 sequencies, one after the other, each for 20 seconds.
Each sequence has a while loop. and a time out statement.
Then it starts the next loop, and so on till the end of end of the
3rd loop. Then it quits. I would like to start again from the top.
I probably have too many while loops.
#!/usr/bin/env python
# Import required libraries
import time
# More setup
# Choose a matrix to use
mat = mat1
t_end = time.time() + 20
#Start loop
while time.time() < t_end:
# code
# loop timeout
# 2 more loops follow just like first one, except matrix becomes
mat = mat2
mat = mat3

As others have already commented, you should do any repetitive tasks within a function. In Python, functions are defined using the "def" keyword. Using a function, it could be done as follows:
import time
# Replace these dummy assignments with real code
mat1 = "blah"
mat2 = "rhubarb"
mat3 = "custard"
def processMatrix(matrix, seconds=20):
t_end = time.time() + seconds
while time.time() < t_end:
pass # 'pass' does nothing - replace with your real code
processMatrix(mat1)
processMatrix(mat2)
processMatrix(mat3)
Note that I've also included the time/seconds as a parameter in the function. This gives you more flexibility, in case you wanted to run for different times for testing or different times for each matrix, etc. However, I've done it with a default value of 20 so that you don't need to include it in the function call. If you do want to override the default you could call, eg,
processMatrix(mat1, 5)
instead of,
processMatrix(mat1)

Python test class stuck on 'Instantiating tests'

My test class produces the required plot, but I have to manually stop execution every time - the console continues to show 'Instantiating tests'; can anyone spot why the execution never halts? Any tips on increasing my code's 'Pythonic-ness' would also be appreciated!
(Python 3.5 running in PyCharm CE 2016.2.1 on Mac OS X 10.11.6)
# A program to test various sorting algorithms. It generates random lists of various sizes,
# sorts them, and then tests that:
# a. the list is in ascending order
# b. the set of elements is the same as the original list
# c. record the time taken
import random
import timeit
from unittest import TestCase
import matplotlib.pyplot as plt
from Sorter import insertionsort, mergesort, quicksort
class TestSort(TestCase):
def test_sorting(self):
times_insertionsort = [] # holds the running times for insertion sort
times_quicksort = [] # holds the running times for quick sort
times_mergesort = [] # holds the running times for merge sort
lengths = [] # holds the array lengths
# determine the number of lists to be created
for i in range(0, 13):
# initialise a new empty list
pre_sort = []
# determine the list's length, then populate the list with 'random' data
for j in range(0, i * 100):
pre_sort.append(random.randint(0, 1000))
# record the length of the list
lengths.append(len(pre_sort))
# record the time taken by quicksort to sort the list
start_time = timeit.default_timer()
post_quicksort = quicksort(pre_sort)
finish_time = timeit.default_timer()
times_quicksort.append((finish_time - start_time) * 1000)
# record the time taken by insertionsort to sort the list
start_time = timeit.default_timer()
post_insertionsort = insertionsort(pre_sort)
finish_time = timeit.default_timer()
times_insertionsort.append((finish_time - start_time) * 1000)
# record the time taken by mergesort to sort the list
start_time = timeit.default_timer()
post_mergesort = mergesort(pre_sort)
finish_time = timeit.default_timer()
times_mergesort.append((finish_time - start_time) * 1000)
# check that:
# a. the list is in ascending order
# b. the set of elements is the same as the original list
for k in range(0, len(pre_sort) - 1):
self.assertTrue(post_insertionsort[k] in pre_sort)
self.assertTrue(post_insertionsort[k] <= post_insertionsort[k + 1])
self.assertTrue(post_mergesort[k] == post_insertionsort[k])
self.assertTrue(post_mergesort[k] == post_quicksort[k])
# plot the results
plt.plot(lengths, times_insertionsort, 'r')
plt.plot(lengths, times_quicksort, 'g')
plt.plot(lengths, times_mergesort, 'b')
plt.xlabel('List size')
plt.ylabel('Execution time (ms)')
plt.show()

You need to close the window where the plot is shown in order for the script (your test in this case) to continue/complete/finish. From the docs, emphasis mine:
When you want to view your plots on your display, the user interface backend will need to start the GUI mainloop. This is what show() does. It tells matplotlib to raise all of the figure windows created so far and start the mainloop. Because this mainloop is blocking by default (i.e., script execution is paused), you should only call this once per script, at the end. Script execution is resumed after the last window is closed. Therefore, if you are using matplotlib to generate only images and do not want a user interface window, you do not need to call show (see Generate images without having a window appear and What is a backend?).
Or don't use show() for the tests and just generate an image which you can verify against later.

I had the same error in Django. I found one of the migrations weren't applied.

Using time.time() to time a function often return 0 seconds

I have to time the implementation I did of an algorithm in one of my classes, and I am using the time.time() function to do so. After implementing it, I have to run that algorithm on a number of data files which contains small and bigger data sets in order to formally analyse its complexity.
Unfortunately, on the small data sets, I get a runtime of 0 seconds even if I get a precision of 0.000000000000000001 with that function when looking at the runtimes of the bigger data sets and I cannot believe that it really takes less than that on the smaller data sets.
My question is: Is there a problem using this function (and if so, is there another function I can use that has a better precision)? Or am I doing something wrong?
Here is my code if ever you need it:
import sys, time
import random
from utility import parseSystemArguments, printResults
...
def main(ville):
start = time.time()
solution = dynamique(ville) # Algorithm implementation
end = time.time()
return (end - start, solution)
if __name__ == "__main__":
sys.argv.insert(1, "-a")
sys.argv.insert(2, "3")
(algoNumber, ville, printList) = parseSystemArguments()
(algoTime, solution) = main(ville)
printResults(algoTime, solution, printList)
The printResults function:
def printResults(time, solution, printList=True):
print ("Temps d'execution = " + str(time) + "s")
if printList:
print (solution)

The solution to my problem was to use the timeit module instead of the time module.
import timeit
...
def main(ville):
start = timeit.default_timer()
solution = dynamique(ville)
end = timeit.default_timer()
return (end - start, solution)

Don't confuse the resolution of the system time with the resolution of a floating point number. The time resolution on a computer is only as frequent as the system clock is updated. How often the system clock is updated varies from machine to machine, so to ensure that you will see a difference with time, you will need to make sure it executes for a millisecond or more. Try putting it into a loop like this:
start = time.time()
k = 100000
for i in range(k)
solution = dynamique(ville)
end = time.time()
return ((end - start)/k, solution)
In the final tally, you then need to divide by the number of loop iterations to know how long your code actually runs once through. You may need to increase k to get a good measure of the execution time, or you may need to decrease it if your computer is running in the loop for a very long time.