This question already has an answer here:
Pandas Dataframe, change values on "diagonal" (where index-value is equal to column-name)
(1 answer)
Closed 6 years ago.
I have a dataframe with binary values. I want to change values to zero where row and column name is same. What is the easiest way to do this?
tmp_dict= {'x':[0,0,0,0,0],'y':[1,0,1,0,0],'z':[1,1,1,1,1],'p':[0,1,1,0,1],'q':[0,1,1,1,1]}
mydf= pd.DataFrame(tmp_dict, columns=['x','y','z','p','q'], index=['x','y','z','p','q'])
mydf
x y z p q
x 0 1 1 0 0
y 0 0 1 1 1
z 0 1 1 1 1
p 0 0 1 0 1
q 0 0 1 1 1
Desired dataframe:
x y z p q
x 0 1 1 0 0
y 0 0 1 1 1
z 0 1 0 1 1
p 0 0 1 0 1
q 0 0 1 1 0
You can loop through the columns and set the corresponding row(s) to zero using loc.
for col in mydf:
mydf.loc[col, col] = 0
Related
I have the following dataframe:
1 2 3 4 5 6 7 8 9
0 0 0 1 0 0 0 0 0 1
1 0 0 0 0 1 1 0 1 0
2 1 1 0 1 1 0 0 1 1
...
I want to get for each row the longest sequence of value 0 in the row.
so, the expected results for this dataframe will be an array that looks like this:
[5,4,2,...]
as on the first row, maximum sequenc eof value 0 is 5, ect.
I have seen this post and tried for the beginning to get this for the first row (though I would like to do this at once for the whole dataframe) but I got errors:
s=df_day.iloc[0]
(~s).cumsum()[s].value_counts().max()
TypeError: ufunc 'invert' not supported for the input types, and the
inputs could not be safely coerced to any supported types according to
the casting rule ''safe''
when I inserted manually the values like this:
s=pd.Series([0,0,1,0,0,0,0,0,1])
(~s).cumsum()[s].value_counts().max()
>>>7
I got 7 which is number of total 0 in the row but not the max sequence.
However, I don't understand why it raises the error at first, and , more important, I would like to run it on the end on the while dataframe and per row.
My end goal: get the maximum uninterrupted occurance of value 0 in a row.
Vectorized solution for counts consecutive 0 per rows, so for maximal use max of DataFrame c:
#more explain https://stackoverflow.com/a/52718619/2901002
m = df.eq(0)
b = m.cumsum(axis=1)
c = b.sub(b.mask(m).ffill(axis=1).fillna(0)).astype(int)
print (c)
1 2 3 4 5 6 7 8 9
0 1 2 0 1 2 3 4 5 0
1 1 2 3 4 0 0 1 0 1
2 0 0 1 0 0 1 2 0 0
df['max_consecutive_0'] = c.max(axis=1)
print (df)
1 2 3 4 5 6 7 8 9 max_consecutive_0
0 0 0 1 0 0 0 0 0 1 5
1 0 0 0 0 1 1 0 1 0 4
2 1 1 0 1 1 0 0 1 1 2
Use:
df = df.T.apply(lambda x: (x != x.shift()).astype(int).cumsum().where(x.eq(0)).dropna().value_counts().max())
OUTPUT
0 5
1 4
2 2
The following code should do the job.
the function longest_streak will count the number of consecutive zeros and return the max, and you can use apply on your df.
from itertools import groupby
def longest_streak(l):
lst = []
for n,c in groupby(l):
num,count = n,sum(1 for i in c)
if num==0:
lst.append((num,count))
maxx = max([y for x,y in lst])
return(maxx)
df.apply(lambda x: longest_streak(x),axis=1)
import pandas as pd
df = pd.DataFrame({'Org1': [1,1,1,1,2,2,2,2,3,3,3,4,4,4],
'Org2': ['x','x','y','y','z','y','z','z','x','y','y','z','x','x'],
'Org3': ['a','a','b','b','c','b','c','c','a','b','b','c','a','a'],
'Value': [0,0,3,1,0,1,0,5,0,0,0,1,1,1]})
df
For each unique set of "Org1, Org2, Org3" and based on the "Value"
The first non zero "value" should have "FLAG" = 1 and others = 0
If all "value" are 0 then one of the row's "FLAG" = 1 and others = 0
If "value" are all NON ZERO in a Column then first instance to have FLAG = 1 and others 0
I was using the solutions provided in
Flag the first non zero column value with 1 and rest 0 having multiple columns
One difference is in the above Point 2 isnt covered
"If all "value" are 0 then one of the row's "FLAG" = 1 and others = 0"
You can modify linked solution with remove .where:
m = df['Value'].ne(0)
idx = m.groupby([df['Org1'],df['Org2'],df['Org3']]).idxmax()
df['FLAG'] = df.index.isin(idx).astype(int)
print (df)
Org1 Org2 Org3 Value FLAG
0 1 x a 0 1
1 1 x a 0 0
2 1 y b 3 1
3 1 y b 1 0
4 2 z c 0 0
5 2 y b 1 1
6 2 z c 0 0
7 2 z c 5 1
8 3 x a 0 1
9 3 y b 0 1
10 3 y b 0 0
11 4 z c 1 1
12 4 x a 1 1
13 4 x a 1 0
How to implement :
t=np.where(<exists at least 1 zero in the same column of t>,t,np.zeros_like(t))
in the "pythonic" way?
this code should set all column to zero in t if t has at least 1 zero in that column
Example :
1 1 1 1 1 1
0 1 1 1 1 1
1 1 0 1 0 1
should turn to
0 1 0 1 0 1
0 1 0 1 0 1
0 1 0 1 0 1
any is what you need
~(arr == 0).any(0, keepdims=True) * arr
0 1 0 1 0 1
0 1 0 1 0 1
0 1 0 1 0 1
this code should set all column to zero in t if t has at least 1 zero
in that column
The simplest way to do this particular task:
t * t.min(0)
A more general way to do it (in case you have an array with different values and the condition is: if a column has at least one occurrence of some_value, then set that column to some_value).
cond = (arr == some_value).any(0)
arr[:, cond] = some_value
This question already has answers here:
How can I pivot a dataframe?
(5 answers)
Closed 3 years ago.
I have dataframe which looks as this:
FIRST SECOND
1 a
1 b
1 c
1 b
2 a
2 k
3 r
3 r
3 r
And I need to get matrix as this, which represent count of repetition of each word for every number:
FIRST a b c k r
1 1 2 1 0 0
2 1 0 0 1 0
3 0 0 0 0 3
Can anyone help me with this? :)
This works:
pd.concat([df.FIRST, pd.get_dummies(df.SECOND)],1).groupby('FIRST').sum()
Use pivot_table with aggfunc='count'
pd.pivot_table(df, values = 'SECOND',
columns = df['SECOND'],
index = df['FIRST'],
aggfunc ='count',
fill_value = 0)
Outputs
SECOND a b c k r
FIRST
1 1 2 1 0 0
2 1 0 0 1 0
3 0 0 0 0 3
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
i want to count number of consecutive zeros in my Dataframe shown below, help please
DEC JAN FEB MARCH APRIL MAY consecutive zeros
0 X X X 1 0 1 0
1 X X X 1 0 1 0
2 0 0 1 0 0 1 2
3 1 0 0 0 1 1 3
4 0 0 0 0 0 1 5
5 X 1 1 0 0 0 3
6 1 0 0 1 0 0 2
7 0 0 0 0 1 0 4
For each row, you want cumsum(1-row) with reset at every point when row == 1. Then you take the row max.
For example
ts = pd.Series([0,0,0,0,1,1,0,0,1,1,1,0])
ts2 = 1-ts
tsgroup = ts.cumsum()
consec_0 = ts2.groupby(tsgroup).transform(pd.Series.cumsum)
consec_0.max()
will give you 4 as needed.
Write that in a function and apply to your dataframe
Here's my two cents...
Think of all the other non-zero elements as 1, then you will have a binary code. All you need to do now is find the 'largest interval' where there's no bit flip starting with 0.
We can write a function and 'apply' with lambda
def len_consec_zeros(a):
a = np.array(list(a)) # convert elements to `str`
rr = np.argwhere(a == '0').ravel() # find out positions of `0`
if not rr.size: # if there are no zeros, return 0
return 0
full = np.arange(rr[0], rr[-1]+1) # get the range of spread of 0s
# get the indices where `0` was flipped to something else
diff = np.setdiff1d(full, rr)
if not diff.size: # if there are no bit flips, return the
return len(full) # size of the full range
# break the array into pieces wherever there's a bit flip
# and the result is the size of the largest chunk
pos, difs = full[0], []
for el in diff:
difs.append(el - pos)
pos = el + 1
difs.append(full[-1]+1 - pos)
# return size of the largest chunk
res = max(difs) if max(difs) != 1 else 0
return res
Now that you have this function, call it on every row...
# join all columns to get a string column
# assuming you have your data in `df`
df['concated'] = df.astype(str).apply(lambda x: ''.join(x), axis=1)
df['consecutive_zeros'] = df.concated.apply(lambda x: len_consec_zeros(x))
Here's one approach -
# Inspired by https://stackoverflow.com/a/44385183/
def pos_neg_counts(mask):
idx = np.flatnonzero(mask[1:] != mask[:-1])
if len(idx)==0: # To handle all 0s or all 1s cases
if mask[0]:
return np.array([mask.size]), np.array([0])
else:
return np.array([0]), np.array([mask.size])
else:
count = np.r_[ [idx[0]+1], idx[1:] - idx[:-1], [mask.size-1-idx[-1]] ]
if mask[0]:
return count[::2], count[1::2] # True, False counts
else:
return count[1::2], count[::2] # True, False counts
def get_consecutive_zeros(df):
arr = df.values
mask = (arr==0) | (arr=='0')
zero_count = np.array([pos_neg_counts(i)[0].max() for i in mask])
zero_count[zero_count<2] = 0
return zero_count
Sample run -
In [272]: df
Out[272]:
DEC JAN FEB MARCH APRIL MAY
0 X X X 1 0 1
1 X X X 1 0 1
2 0 0 1 0 0 1
3 1 0 0 0 1 1
4 0 0 0 0 0 1
5 X 1 1 0 0 0
6 1 0 0 1 0 0
7 0 0 0 0 1 0
In [273]: df['consecutive_zeros'] = get_consecutive_zeros(df)
In [274]: df
Out[274]:
DEC JAN FEB MARCH APRIL MAY consecutive_zeros
0 X X X 1 0 1 0
1 X X X 1 0 1 0
2 0 0 1 0 0 1 2
3 1 0 0 0 1 1 3
4 0 0 0 0 0 1 5
5 X 1 1 0 0 0 3
6 1 0 0 1 0 0 2
7 0 0 0 0 1 0 4