I am working with a large array of 1's and need to systematically remove 0's from sections of the array. The large array is comprised of many smaller arrays, for each smaller array I need to replace its upper and lower triangles with 0's systematically. For example we have an array with 5 sub arrays indicated by the index value (all sub-arrays have the same number of columns):
0 1 2
0 1.0 1.0 1.0
1 1.0 1.0 1.0
1 1.0 1.0 1.0
2 1.0 1.0 1.0
2 1.0 1.0 1.0
2 1.0 1.0 1.0
3 1.0 1.0 1.0
3 1.0 1.0 1.0
3 1.0 1.0 1.0
3 1.0 1.0 1.0
4 1.0 1.0 1.0
4 1.0 1.0 1.0
4 1.0 1.0 1.0
4 1.0 1.0 1.0
4 1.0 1.0 1.0
I want each group of rows to be modified in its upper and lower triangle such that the resulting matrix is:
0 1 2
0 1.0 1.0 1.0
1 1.0 1.0 0.0
1 0.0 1.0 1.0
2 1.0 0.0 0.0
2 0.0 1.0 0.0
2 0.0 0.0 1.0
3 1.0 0.0 0.0
3 1.0 1.0 0.0
3 0.0 1.0 1.0
3 0.0 0.0 1.0
4 1.0 0.0 0.0
4 1.0 1.0 0.0
4 1.0 1.0 1.0
4 0.0 1.0 1.0
4 0.0 0.0 1.0
At the moment I am using only numpy to achieve this resulting array, but I think I can speed it up using Pandas grouping. In reality my dataset is very large almost 500,000 rows long. The numpy code is below:
import numpy as np
candidateLengths = np.array([1,2,3,4,5])
centroidLength =3
smallPaths = [min(l,centroidLength) for l in candidateLengths]
# This is the k_values of zeros to delete. To be used in np.tri
k_vals = list(map(lambda smallPath: centroidLength - (smallPath), smallPaths))
maskArray = np.ones((np.sum(candidateLengths), centroidLength))
startPos = 0
endPos = 0
for canNo, canLen in enumerate(candidateLengths):
a = np.ones((canLen, centroidLength))
a *= np.tri(*a.shape, dtype=np.bool, k=k_vals[canNo])
b = np.fliplr(np.flipud(a))
c = a*b
endPos = startPos + canLen
maskArray[startPos:endPos, :] = c
startPos = endPos
print(maskArray)
When I run this on my real dataset it takes nearly 5-7secs to execute. I think this is down to this massive for loop. How can I use pandas groupings to achieve a higher speed? Thanks
New Answer
def tris(n, m):
if n < m:
a = np.tri(m, n, dtype=int).T
else:
a = np.tri(n, m, dtype=int)
return a * a[::-1, ::-1]
idx = np.append(df.index.values, -1)
w = np.append(-1, np.flatnonzero(idx[:-1] != idx[1:]))
c = np.diff(w)
df * np.vstack([tris(n, 3) for n in c])
0 1 2
0 1.0 1.0 1.0
1 1.0 1.0 0.0
1 0.0 1.0 1.0
2 1.0 0.0 0.0
2 0.0 1.0 0.0
2 0.0 0.0 1.0
3 1.0 0.0 0.0
3 1.0 1.0 0.0
3 0.0 1.0 1.0
3 0.0 0.0 1.0
4 1.0 0.0 0.0
4 1.0 1.0 0.0
4 1.0 1.0 1.0
4 0.0 1.0 1.0
4 0.0 0.0 1.0
Old Answer
I define some helper triangle functions
def tris(n, m):
if n < m:
a = np.tri(m, n, dtype=int).T
else:
a = np.tri(n, m, dtype=int)
return a * a[::-1, ::-1]
def tris_df(df):
n, m = df.shape
return pd.DataFrame(tris(n, m), df.index, df.columns)
Then
df * df.groupby(level=0, group_keys=False).apply(tris_df)
0 1 2
0 1.0 1.0 1.0
1 1.0 1.0 0.0
1 0.0 1.0 1.0
2 1.0 0.0 0.0
2 0.0 1.0 0.0
2 0.0 0.0 1.0
3 1.0 0.0 0.0
3 1.0 1.0 0.0
3 0.0 1.0 1.0
3 0.0 0.0 1.0
4 1.0 0.0 0.0
4 1.0 1.0 0.0
4 1.0 1.0 1.0
4 0.0 1.0 1.0
4 0.0 0.0 1.0
Related
This is my dataframe
import pandas as pd
import numpy as np
data = {'c1':[-1,-1,1,1,np.nan,1,1,1,1,1,np.nan,-1],\
'c2':[1,1,1,-1,1,1,-1,-1,1,-1,1,np.nan]}
index = pd.date_range('2000-01-01','2000-03-20', freq='W')
df = pd.DataFrame(index=index, data=data)
>>> df
c1 c2
2000-01-02 -1.0 1.0
2000-01-09 -1.0 1.0
2000-01-16 1.0 1.0
2000-01-23 1.0 -1.0
2000-01-30 NaN 1.0
2000-02-06 1.0 1.0
2000-02-13 1.0 -1.0
2000-02-20 1.0 -1.0
2000-02-27 1.0 1.0
2000-03-05 1.0 -1.0
2000-03-12 NaN 1.0
2000-03-19 -1.0 NaN
and this is a cumulative sum by month
df2 = df.groupby(df.index.to_period('m')).cumsum()
>>> df2
c1 c2
2000-01-02 -1.0 1.0
2000-01-09 -2.0 2.0
2000-01-16 -1.0 3.0
2000-01-23 0.0 2.0
2000-01-30 NaN 3.0
2000-02-06 1.0 1.0
2000-02-13 2.0 0.0
2000-02-20 3.0 -1.0
2000-02-27 4.0 0.0
2000-03-05 1.0 -1.0
2000-03-12 NaN 0.0
2000-03-19 0.0 NaN
what I need more is to ignore the increment if it is more than 3 or less than 0, something like this function
def cumsum2(arr, low=-float('Inf'), high=float('Inf')):
arr2 = np.copy(arr)
sm = 0
for index, elem in np.ndenumerate(arr):
if not np.isnan(elem):
sm += elem
if sm > high:
sm = high
if sm < low:
sm = low
arr2[index] = sm
return arr2
the desired result is
c1 c2
2000-01-02 0.0 1.0
2000-01-09 0.0 2.0
2000-01-16 1.0 3.0
2000-01-23 2.0 2.0
2000-01-30 2.0 3.0
2000-02-06 1.0 1.0
2000-02-13 2.0 0.0
2000-02-20 3.0 0.0
2000-02-27 3.0 1.0
2000-03-05 1.0 0.0
2000-03-12 1.0 1.0
2000-03-19 0.0 1.0
I tried to use apply and lambda but doesn't work and it's slow for large dataframe.
df.groupby(df.index.to_period('m')).apply(lambda x: cumsum2(x, 0, 3))
What's wrong? Is there a faster way?
You can try accumulate from itertools and use a custom function to clip values between 0 and 3:
from itertools import accumulate
lb = 0 # lower bound
ub = 3 # upper bound
def cumsum2(dfm):
def clip(bal, val):
return np.clip(bal + val, lb, ub)
return list(accumulate(dfm.to_numpy(), clip, initial=0))[1:]
out = df.fillna(0).groupby(df.index.to_period('m')).transform(cumsum2)
Output:
>>> out
c1 c2
2000-01-02 0.0 1.0
2000-01-09 0.0 2.0
2000-01-16 1.0 3.0
2000-01-23 2.0 2.0
2000-01-30 2.0 3.0
2000-02-06 1.0 1.0
2000-02-13 2.0 0.0
2000-02-20 3.0 0.0
2000-02-27 3.0 1.0
2000-03-05 1.0 0.0
2000-03-12 1.0 1.0
2000-03-19 0.0 1.0
In such sophisticated case we can resort to pandas.Series.rolling with window of size 2 piping each window to a custom function to keep each interim accumulation within a certain threshold:
def cumsum_tsh(x, low=-float('Inf'), high=float('Inf')):
def f(w):
w[-1] = min(high, max(low, w[0] if w.size == 1 else w[0] + w[1]))
return w[-1]
return x.apply(lambda s: s.rolling(2, min_periods=1).apply(f))
res = df.fillna(0).groupby(df.index.to_period('m'), group_keys=False)\
.apply(lambda x: cumsum_tsh(x, 0, 3))
c1 c2
2000-01-02 0.0 1.0
2000-01-09 0.0 2.0
2000-01-16 1.0 3.0
2000-01-23 2.0 2.0
2000-01-30 2.0 3.0
2000-02-06 1.0 1.0
2000-02-13 2.0 0.0
2000-02-20 3.0 0.0
2000-02-27 3.0 1.0
2000-03-05 1.0 0.0
2000-03-12 1.0 1.0
2000-03-19 0.0 1.0
I've tried various solutions, for some reason the fastest is manipulating single columns of frames created by groupby.
This is the code if it can be useful to anyone
def cumsum2(frame, low=-float('Inf'), high=float('Inf')):
for col in frame.columns:
sm = 0
xs = []
for e in frame[col]:
sm += e
if sm > high:
sm = high
if sm < low:
sm = low
xs.append(sm)
frame[col] = xs
return frame
res = df.fillna(0).groupby(df.index.to_period('m'), group_keys=False)\
.apply(cumsum2,0,3)
I've spent a little while on this and got an answer but seems a little convoluted so curious if people have a better solution.
Given a list I want a table indicating all the possible combinations between the elements.
sample_list = ['a', 'b', 'c', 'd']
(pd.concat(
[
pd.DataFrame(
[dict.fromkeys(i, 1) for i in combinations(sample_list, j)]
) for j in range(len(sample_list)+1)
]).
fillna(0).
reset_index(drop = True)
)
With the result, as desired:
a b c d
0 0.0 0.0 0.0 0.0
1 1.0 0.0 0.0 0.0
2 0.0 1.0 0.0 0.0
3 0.0 0.0 1.0 0.0
4 0.0 0.0 0.0 1.0
5 1.0 1.0 0.0 0.0
6 1.0 0.0 1.0 0.0
7 1.0 0.0 0.0 1.0
8 0.0 1.0 1.0 0.0
9 0.0 1.0 0.0 1.0
10 0.0 0.0 1.0 1.0
11 1.0 1.0 1.0 0.0
12 1.0 1.0 0.0 1.0
13 1.0 0.0 1.0 1.0
14 0.0 1.0 1.0 1.0
15 1.0 1.0 1.0 1.0
For learning purposes would like to know better solutions.
Thanks
Check Below code
import itertools
import pandas as pd
sample_list = ['a', 'b', 'c', 'd']
pd.DataFrame(list(itertools.product([0, 1], repeat=len(sample_list))), columns=sample_list)
Output:
I have a dataframe that looks like this:
iso3 prod_level alloc_key cell5m x y rec_type tech_type unit whea_a ... acof_pct_prod rcof_pct_prod coco_pct_prod teas_pct_prod toba_pct_prod bana_pct_prod trof_pct_prod temf_pct_prod vege_pct_prod rest_pct_prod
35110 IND IN16011 9243059 3990418 74.875000 13.041667 P A mt 0.0 ... 1.0 1.0 1.0 1.0 1.0 0.958586 0.449218 1.0 1.0 0.004520
35109 IND IN16011 9243058 3990417 74.791667 13.041667 P A mt 0.0 ... 1.0 1.0 1.0 1.0 1.0 0.970957 0.459725 1.0 1.0 0.009037
35406 IND IN16003 9283093 4007732 77.708333 12.708333 P A mt 0.0 ... 1.0 1.0 1.0 1.0 1.0 0.883868 1.000000 1.0 1.0 0.012084
35311 IND IN16011 9273062 4003381 75.125000 12.791667 P A mt 0.0 ... 1.0 1.0 1.0 1.0 1.0 0.942550 0.381430 1.0 1.0 0.015024
35308 IND IN16011 9273059 4003378 74.875000 12.791667 P A mt 0.0 ... 1.0 1.0 1.0 1.0 1.0 0.991871 0.887494 1.0 1.0 0.017878
I want to set all values that are greater than 0.9 in columns that end in 'prod' to zero. I can select only those columns like this:
cols2=[col for col in df.columns if col.endswith('_prod')]
df[cols2]
whea_pct_prod rice_pct_prod maiz_pct_prod barl_pct_prod pmil_pct_prod smil_pct_prod sorg_pct_prod pota_pct_prod swpo_pct_prod cass_pct_prod ... acof_pct_prod rcof_pct_prod coco_pct_prod teas_pct_prod toba_pct_prod bana_pct_prod trof_pct_prod temf_pct_prod vege_pct_prod rest_pct_prod
35110 1.0 0.958721 0.359063 1.0 1.0 1.000000 1.0 1.0 1.00000 0.992816 ... 1.0 1.0 1.0 1.0 1.0 0.958586 0.449218 1.0 1.0 0.004520
35109 1.0 0.878148 0.200283 1.0 1.0 1.000000 1.0 1.0 1.00000 0.993140 ... 1.0 1.0 1.0 1.0 1.0 0.970957 0.459725 1.0 1.0 0.009037
35406 1.0 0.996354 0.980844 1.0 1.0 0.274348 1.0 1.0 0.99945 1.000000 ... 1.0 1.0 1.0 1.0 1.0 0.883318 1.000000 1.0 1.0 0.012084
35311 1.0 0.570999 0.341217 1.0 1.0 1.000000 1.0 1.0 1.00000 0.997081 ... 1.0 1.0 1.0 1.0 1.0 0.942550 0.381430 1.0 1.0 0.015024
35308 1.0 0.657520 0.161771 1.0 1.0 1.000000 1.0 1.0 1.00000 0.991491 ... 1.0 1.0 1.0 1.0 1.0 0.991871 0.887494 1.0 1.0 0.017878
Now, when I try and set the values greater than 0.9 to be zero, it does not work.
df[cols2][df[cols2]>0.9]=0
What should I be doing instead?
You can use df.where(cond, other) to replace the values with other where cond == False.
df[cols2] = df[cols2].where(df[cols]<=0.9, other=0)
I'm trying to drop a feature which if float and a number of missing values is higher than certain number.
I've tried:
# Define threshold to 1/6
threshold = 0.1667
# Drop float > threshold
for f in data:
if data[f].dtype==float & data[f].isnull().sum() / data.shape[0] > threshold: del data[f]
..which raises an error:
TypeError: unsupported operand type(s) for &: 'type' and
'numpy.float64'
Help would be aprreciated.
Use DataFrame.select_dtypes for only floats columns, check missing values and get mean - sum/count and add another non floats columns by Series.reindex, last filter by inverse condition > to <= by boolean indexing:
np.random.seed(2019)
df = pd.DataFrame(np.random.choice([np.nan,1], p=(0.2,0.8),size=(10,10))).assign(A='a')
print (df)
0 1 2 3 4 5 6 7 8 9 A
0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
1 1.0 1.0 NaN 1.0 NaN 1.0 NaN 1.0 1.0 1.0 a
2 1.0 1.0 1.0 1.0 1.0 NaN 1.0 NaN 1.0 1.0 a
3 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 NaN 1.0 a
4 1.0 NaN 1.0 1.0 1.0 1.0 1.0 NaN 1.0 1.0 a
5 1.0 1.0 1.0 1.0 1.0 1.0 NaN 1.0 1.0 1.0 a
6 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
7 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
8 1.0 NaN 1.0 1.0 1.0 1.0 NaN 1.0 1.0 1.0 a
9 NaN 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 NaN a
threshold = 0.1667
df1 = df.select_dtypes(float).isnull().mean().reindex(df.columns, fill_value=False)
df = df.loc[:, df1 <= threshold]
print (df)
0 2 3 4 5 8 9 A
0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
1 1.0 NaN 1.0 NaN 1.0 1.0 1.0 a
2 1.0 1.0 1.0 1.0 NaN 1.0 1.0 a
3 1.0 1.0 1.0 1.0 1.0 NaN 1.0 a
4 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
5 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
6 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
7 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
8 1.0 1.0 1.0 1.0 1.0 1.0 1.0 a
9 NaN 1.0 1.0 1.0 1.0 1.0 NaN a
Given an example dataframe:
import pandas as pd
import numpy as np
values = np.array([
[0, 0.5, 1, 0, 0, 3],
[1, 0, 0, 1, 1, 0 ],
[0, 0.5, 0, 0, 2, 1],
[0, 0, 0, 0, 4, 0],
])
indexes= 'a','b','c','d'
columns='ab','bc','cd','de','ef','fg'
df = pd.DataFrame(index=indexes,columns=columns, data=values)
print(df)
which looks like:
ab bc cd de ef fg
a 0.0 0.5 1.0 0.0 0.0 3.0
b 1.0 0.0 0.0 1.0 1.0 0.0
c 0.0 0.5 0.0 0.0 2.0 1.0
d 0.0 0.0 0.0 0.0 4.0 0.0
desired output:
ab bc cd de ef fg
a 0.0 0.5 1.0 0.0 0.0 3.0
b 1.0 0.0 0.0 1.0 1.0 0.0
c 0.0 0.5 0.0 0.0 2.0 1.0
d 0.0 0.0 0.0 0.0 4.0 0.0
e Nan Nan Nan NAn 7.0 4.0
Is it somehow possible to add a row where are displayed only the sums of the two last columns? (Of course, below the respective columns)
Thanks for your attention!
Edit: ohhh. Thanks for the clarification. You create a new row and assign that with the sum of the last two columns. The iloc indexer is of the format [row,col]. So we want : all rows but only the final two columns -2:.
df.loc['e'] = df.iloc[:,-2:].sum()
Result:
>>> df
ab bc cd de ef fg
a 0.0 0.5 1.0 0.0 0.0 3.0
b 1.0 0.0 0.0 1.0 1.0 0.0
c 0.0 0.5 0.0 0.0 2.0 1.0
d 0.0 0.0 0.0 0.0 4.0 0.0
e NaN NaN NaN NaN 7.0 4.0
Old answer:
I assume you mean the last two rows...
You can use pd.concat here
pd.concat([df,df.iloc[-2,:] + df.iloc[-1:]])
Result:
>>> pd.concat([df,df.iloc[-2,:] + df.iloc[-1:]])
ab bc cd de ef fg
a 0.0 0.5 1.0 0.0 0.0 3.0
b 1.0 0.0 0.0 1.0 1.0 0.0
c 0.0 0.5 0.0 0.0 2.0 1.0
d 0.0 0.0 0.0 0.0 4.0 0.0
d 0.0 0.5 0.0 0.0 6.0 1.0
You can use loc
df.loc['sum_c_d'] = df[-2:].sum()
ab bc cd de ef fg
a 0 0.5 1 0 0 3
b 1 0.0 0 1 1 0
c 0 0.5 0 0 2 1
d 0 0.0 0 0 4 0
sum_c_d 0 0.5 0 0 6 1