I'm working to speed up this chunk of code. It's currently taking multiple days to complete two compare two dataframes [192184 rows x 256 columns] by [7739 rows x 256 columns]. This is a relatively small use-case and its way too slow already. I'd like to compare all rows of dataframe 1 to all rows of dataframe 2. This is done in order to rank the rows in DF2 by similarity to each row of DF1.
There is no need to keep this in pandas format, my end goal is just to write the output to a csv.
For simplicity, I'll write short example inputs, and the expected example output.
Input:
DF1:
A 1 10 7 8 3
B 1 0 1 0 1
C 6 1 0 0 0
DF2:
D 4 0 0 0 1
E 1 0 0 0 2
F 2 8 8 5 2
Output:
D E F
A 0.113690 0.209633 0.971074
B 0.700140 0.774597 0.546019
C 0.956943 0.441129 0.259129
My current code:
import pandas
from scipy import spatial
#set up test case
a = [1,10,7,8,3]
b = [1,0,1,0,1]
c = [6,1,0,0,0]
d = [4,0,0,0,1]
e = [1,0,0,0,2]
f = [2,8,8,5,2]
DF1 = pd.DataFrame([a,b,c])
DF1.index = ["A","B","C"]
DF2 = pd.DataFrame([d,e,f])
DF2.index = ["D","E","F"]
#declare variables
a_dict = {}
rows_list = list(DF2.index.values)
#something is wrong with this meat
for line in DF1.iterrows():
temp = []
for row in DF2.iterrows():
temp.append(1- spatial.distance.cosine(line[1],row[1]))
a_dict[line[0]] = temp
#write final output
final_output = pd.DataFrame.from_dict(a_dict, orient="index", columns = rows_list)
For speeding this up, I've read that itertuples should be faster, but I don't think that addresses the main issue which is either the nested for loop or the spatial.distance.cosine test itself. I've read about different functions to run this test, but honestly don't know how to apply them in this scenario.
Also, this is my first question on here - feel free to give me tips for next time :)
Let's start to see how we could have accelerated the same code. Just as an exercise about how to avoid some for loops.
Tho, spoiler alert, we won't use that algorithm at the end anyway, so it is just an exercise.
Vectorization of your code
What you are doing here is calling spatial.distance.cosine for every combination of rows. Hence the two nested for loops. Let's remove the inner for loop, by using np.vectorize. Whose purpose is to apply the same function to all items of a vector, and return the vector of result. A bit like map, but for numpy.
Example:
def f(a,b):
return a*b
fv=np.vectorize(f)
fv(np.array([1,2,3]), np.array([4,5,6]))
#array([ 4, 10, 18])
fv(np.array([1,2,3]), 4)
#array([ 4, 8, 12])
fv(np.array([[1,2],[3,4]]), np.array([5,6]))
#array([[ 5, 12],
# [15, 24]])
So, fv is the vectorized version of f. You recognize the behavior of * on numpy arrays. Not a coincidence.
np.vectorize goes down any dimension needed to find scalars, in each parameters, and apply the original function on them. But in your case, we don't want it to do that. Because we don't want it to iterate through all 256 columns, and then apply spatial.distance.cosine on individual scalars.
But there is a remedy to that. We can say specify during vectorization, what is the type of the "atoms", what is the signature of f.
sdcv=np.vectorize(spatial.distance.cosine, signature="(n),(n)->()")
creates a vectorized version of cosine expecting to find, in both arguments, arrays of the same size, or arrays of such arrays, or arrays of arrays of such arrays, etc.
So, now, if you call
sdcv(line[1], DF2.values)
Since line[1] is an array of scalars (5 in your example. 256 in your real use case), and DF2.values is an array of arrays of scalars, what this does is calling cosine(line[1], x) for x being each line of DF2.values. And return the len(DF2)` results as an array.
Hence an optimization
sdcv=np.vectorize(spatial.distance.cosine, signature="(n),(n)->()")
def mvec():
#declare variables
a_dict = {}
rows_list = list(DF2.index.values)
for line in DF1.iterrows():
temp = 1-sdcv(line[1], DF2.values)
a_dict[line[0]] = temp
#write final output
return pd.DataFrame.from_dict(a_dict, orient="index", columns = rows_list)
We still have a loop. But the inner loop is now implicit, and done by vectorization.
Second layer
We can do the same with the outer loop
sdcv2=np.vectorize(sdcv, signature="(n),(m,n)->(m)")
def mvec2():
return pd.DataFrame(1-sdcv2(DF1.values, DF2.values))
This time, we tell vectorize that we want 1st argument to be a row, and 2nd argument to be a whole dataframe. And result of to be a list of cosine. So, what the previous vectorization usage was doing.
So, if we call sdcv2 with DF1 and DF2, since DF1 is an array of rows, it will call sdcv for each row of DF1, with second argument being DF2 (it doesn't iterate on DF2, because DF2 is already the expected 2d array).
And since a call to sdcv, with DF2 as second argument, returns an array for each row, we get as a result an array of array.
Linalg solution
Now, that was just an exercise on vectorization. Because, there is a detail in your code, and therefore in both of mine. It is the 1- before spatial.distance.cosine.
But spatial.distance.cosine(u,v) definition is 1-u.v/(||u||.||v||)
So what you really want is u.v/(||u||.||v||)
For the u.v part, it is very easy. There is a well known operation in linear algebra that computes the scalar product of all possible combinations of a set of rows. That is simply matrix multiplication. Or almost so: you need to transpose the 2nd matrix (since matrix multiplication results in the array of all combination of a scalar product between a row of 1st matrix and a column of second matrix).
So, in your case,
DF1.values # DF2.values.T
is matrix of n×n scalar products between rows of DF1 and rows of DF2.
The u.v for u being a row of DF1 and v being a row of DF2
We need to divide that by norms of u and v in each cell.
To be more accurate, in all cells of the first line, u was the first row of DF1. In all cells of 2nd line, u was the 2nd row of DF1. Etc.
So we need to divide the kth line of DF1.values#DF2.values.T by norm of kth line of DF1.
We can compute the norms of all lines of DF1 easily
np.linalg.norm(DF1.values, axis=1)
is an 1D array of the norm of all lines of DF1.
And
np.linalg.norm(DF1.values, axis=1).reshape(-1,1)
is the same result, but in the form of a column
So
(DF1.values#DF2.values.T)/np.linalg.norm(DF1.values,axis=1).reshape(-1,1)
is a matrix of u.v/||u|| for u each row of DF1, and v each row of DF2.
Likewise, line of norms of rows of DF2 is np.linalg.norm(DF2,axis=1)
So
(DF1.values#DF2.values.T)/np.linalg.norm(DF1.values,axis=1).reshape(-1,1)/np.linalg.norm(DF2.values, axis=1)
Is a matrix made of u.v/||u||/||v|| for u each row of DF1 and v each row of DF2.
Exactly what you wanted.
Hence my linalg method
def mnp():
d=(DF1.values#DF2.values.T) / np.linalg.norm(DF1.values,axis=1).reshape(-1,1) / np.linalg.norm(DF2.values, axis=1)
return pd.DataFrame(d, columns=DF2.index, index=DF1.index)
Timings
With dataframes of 500 columns and 3 rows (At first, I just increased the number of rows so that vectorization has a chance to be usefull, but still keeping a 3x3 matrix that I can visually control)
Method
Timing (ms)
Yours
2.9259588550194167
Vec
2.3712358399643563
Vec2
1.5134614360285923
NP
0.19741979095852003
So, 20% gain for vectorization. ×2 for double vectorization. And ×15 gain for linalg.
But 3 rows is not realistic. Now that we know it works, let increase the number of rows
With 100 rows (so 30x more, and it is a O(N²) algorithm, even with constant column size)
Method
Timing (ms)
Yours
1968.870209006127
Vec
904.9228489748202
Vec2
807.5719091983046
NP
5.865840002661571
This times, advantage of vectorization, and moreover, of using global numpy operations, is more visible. ×2 for vectorization (2nd layer this time is less impressive. Because outer loop is never the one that works the more). And ×335 for NP.
On my machine, for 256 columns, and 1000 rows for DF1 and 7739 for DF2, it takes 4 seconds (and that is a very slow machine. Pretty sure, with vectorization, that my Mac would do it way under 1 second, but I don't have it at hand)
You've 192184 rows in DF1, you say, not 1000. But since you have "only" 7739 for DF2, at this point it is linear: it would take only 192 times more to have the all 192184 rows than 1000. So a few minutes.
What might be more a problem with that many data, is memory. Because of storage of result matrix itself.
But that is not a problem. You can slice DF1 by batches of 1000 or 10000 rows. And you'll get a 1000x7739 matrix each time. The real result is just the concatenation of those matrix.
Edit
A variant I've tried, but expecting that improvement would be negligible, is using einsum. Not a game changer, but not as negligible I was expected.
einsum is a way to specify many arrays iterative sum operation. For example, matrix multiplication A#B is the equivalent of np.einsum('ij,jk',A,B).
So, cross scalar of every rows, would be np.einsum('ij,kj', A, B).
Which is, again, the same as matrix multiplication, with a transposition of second argument.
And just avoiding this .T is not that negligible apparently.
So, code for this methond
def mein():
d=np.einsum('ij,kj', DF1.values, DF2.values) / np.linalg.norm(DF1.values,axis=1).reshape(-1,1) / np.linalg.norm(DF2.values, axis=1)
return pd.DataFrame(d, columns=DF2.index, index=DF1.index)
It is the exact same as before. But DF1.values#DF2.values.T has been changed into np.einsum('ij,kj',DF1.values,DF2.values)
And update of timings with this
Method
Timing (ms)
Yours
1968.87
Vec
904.92
Vec2
807.57
NP
5.87
einsum
5.41
So, not a revolution, but still, and optimization.
I'm looking for help with the Pandas .corr() method.
As is, I can use the .corr() method to calculate a heatmap of every possible combination of columns:
corr = data.corr()
sns.heatmap(corr)
Which, on my dataframe of 23,000 columns, may terminate near the heat death of the universe.
I can also do the more reasonable correlation between a subset of values
data2 = data[list_of_column_names]
corr = data2.corr(method="pearson")
sns.heatmap(corr)
That gives me something that I can use--here's an example of what that looks like:
What I would like to do is compare a list of 20 columns with the whole dataset. The normal .corr() function can give me a 20x20 or 23,000x23,000 heatmap, but essentially I would like a 20x23,000 heatmap.
How can I add more specificity to my correlations?
Thanks for the help!
Make a list of the subset that you want (in this example it is A, B, and C), create an empty dataframe, then fill it with the desired values using a nested loop.
df = pd.DataFrame(np.random.randn(50, 7), columns=list('ABCDEFG'))
# initiate empty dataframe
corr = pd.DataFrame()
for a in list('ABC'):
for b in list(df.columns.values):
corr.loc[a, b] = df.corr().loc[a, b]
corr
Out[137]:
A B C D E F G
A 1.000000 0.183584 -0.175979 -0.087252 -0.060680 -0.209692 -0.294573
B 0.183584 1.000000 0.119418 0.254775 -0.131564 -0.226491 -0.202978
C -0.175979 0.119418 1.000000 0.146807 -0.045952 -0.037082 -0.204993
sns.heatmap(corr)
After working through this last night, I came to the following answer:
#datatable imported earlier as 'data'
#Create a new dictionary
plotDict = {}
# Loop across each of the two lists that contain the items you want to compare
for gene1 in list_1:
for gene2 in list_2:
# Do a pearsonR comparison between the two items you want to compare
tempDict = {(gene1, gene2): scipy.stats.pearsonr(data[gene1],data[gene2])}
# Update the dictionary each time you do a comparison
plotDict.update(tempDict)
# Unstack the dictionary into a DataFrame
dfOutput = pd.Series(plotDict).unstack()
# Optional: Take just the pearsonR value out of the output tuple
dfOutputPearson = dfOutput.apply(lambda x: x.apply(lambda x:x[0]))
# Optional: generate a heatmap
sns.heatmap(dfOutputPearson)
Much like the other answers, this generates a heatmap (see below) but it can be scaled to allow for a 20,000x30 matrix without computing the correlation between the entire 20,000x20,000 combinations (and therefore terminating much quicker).
Usually the calculation of correlation coefficients pairwise for all variables make most sense. pd.corr() is convenience function to calculate the correlation coefficient pairwise (and for all pairs).
You can do it with scipy also only for specified pairs within a loop.
Example:
d=pd.DataFrame([[1,5,8],[2,5,4],[7,3,1]], columns=['A','B','C'])
One pair in pandas could be:
d.corr().loc['A','B']
-0.98782916114726194
Equivalent in scipy:
import scipy.stats
scipy.stats.pearsonr(d['A'].values,d['B'].values)[0]
-0.98782916114726194
I have a time series data(1000 data points) with following column names :
X , Y , Z , A , B .
I want to generate 10 segments each of 100 data points with 3 channels, where the first channel contains the column X,A,B , the second channel Y,A,B and the third channel Z,A,B?
How can I accomplish this in python?
Numpy
To rearrange the time series into the 10 segments, you can simply use np.reshape.
Example data of shape (XYZAB, timepoints):
a = np.random.randint(0,10,(5,1000))
print a.shape
>> (5L, 1000L)
Reshaping into the ten segments, resulting in (XYZAB, segments, timepoints):
b = np.reshape(a,(5,10,100))
print b.shape
>> (5L, 10L, 100L)
At this point, it may not be desirable to create what you call 'channels', as you would triplicate parts of your data (A and B) without really making it easier to access that data. You could access e.g. XAB simply like this:
xab = b[(0,3,4),:,:]
If you absolutely need the channels as individual copies, you can simply get them like this:
c = np.array([b[(0,3,4),:,:],
b[(1,3,4),:,:],
b[(2,3,4),:,:]])
print c.shape
>> (3L, 3L, 10L, 100L)
Which results in an array of shape (channel,column,segment,timepoints), where column refers to the original column names (e.g. (X,A,B) for channel 0).
Pandas
Just saw the pandas tag on your question, so...
df = pd.DataFrame(a.T, columns=list('XYZAB'))
Split into segments of 100 time points as a list of dfs:
segments = []
for group, segment in df.groupby(np.arange(len(df)) // 100):
segments.append(segment)
Or, even better, just create a new column that indicates which segment each row belongs to:
df['segment'] = df.apply(lambda x : x.name // 100, axis=1)
At this point it's probably again best not to triplicate your data and instead use the df as it is. You can easily apply operations per time segment using df.groupby(['segment']), while selecting columns of interest by standard column selection, e.g.
df.groupby(['segment'])['X','A','B'].mean()
to get the per-segment mean of columns X, A and B.
Of course you can create e.g. a list or dict of 'channels' in this way, if you really need it.
channels = {'XAB':df[['segment','X','A','B']],
'YAB':df[['segment','Y','A','B']],
'ZAB':df[['segment','Z','A','B']]}
And you can make this into a pandas Panel:
pnl = pd.Panel(channels)
The best data structure to use depends on your particular use-case, but in general I would avoid using Panels and stick with either the 2D df or the 3D array (i.e. b).
I'm starting to learn Pandas and am trying to find the most Pythonic (or panda-thonic?) ways to do certain tasks.
Suppose we have a DataFrame with columns A, B, and C.
Column A contains boolean values: each row's A value is either true or false.
Column B has some important values we want to plot.
What we want to discover is the subtle distinctions between B values for rows that have A set to false, vs. B values for rows that have A is true.
In other words, how can I group by the value of column A (either true or false), then plot the values of column B for both groups on the same graph? The two datasets should be colored differently to be able to distinguish the points.
Next, let's add another feature to this program: before graphing, we want to compute another value for each row and store it in column D. This value is the mean of all data stored in B for the entire five minutes before a record - but we only include rows that have the same boolean value stored in A.
In other words, if I have a row where A=True and time=t, I want to compute a value for column D that is the mean of B for all records from time t-5 to t that have the same A=True.
In this case, how can we execute the groupby on values of A, then apply this computation to each individual group, and finally plot the D values for the two groups?
I think #herrfz hit all the high points. I'll just flesh out the details:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
sin = np.sin
cos = np.cos
pi = np.pi
N = 100
x = np.linspace(0, pi, N)
a = sin(x)
b = cos(x)
df = pd.DataFrame({
'A': [True]*N + [False]*N,
'B': np.hstack((a,b))
})
for key, grp in df.groupby(['A']):
plt.plot(grp['B'], label=key)
grp['D'] = pd.rolling_mean(grp['B'], window=5)
plt.plot(grp['D'], label='rolling ({k})'.format(k=key))
plt.legend(loc='best')
plt.show()