I have the next DataFrame:
pd.DataFrame(['a','a','a','a','b','b','b','c','c','a'])
I need to create a column considering the variation on the other column.
Following this result:
Letter
Number
a
1
a
0
a
0
a
0
b
1
b
0
b
0
c
1
c
0
a
1
Every time the letter change, I need to put a 1.
shift
I'm assuming that df is what OP provided
df = pd.DataFrame(['a','a','a','a','b','b','b','c','c','a'])
Then reasigned the first column to a series letter
letter = df.iloc[:, 0]
pd.DataFrame({
'Letter': letter,
'Number': letter.shift().ne(letter).astype(int)
})
Letter Number
0 a 1
1 a 0
2 a 0
3 a 0
4 b 1
5 b 0
6 b 0
7 c 1
8 c 0
9 a 1
Related
I have a pandas data frame as follows
A
B
C
D
...
Z
and another data frame in which every column has zero or more letters as follows:
Letters
A,C,D
A,B,F
A,H,G
A
B,F
None
I want to match the two dataframes to have something like this
A
B
C
D
...
Z
1
0
1
1
0
0
make example and desired output for answer
Example:
data = ['A,C,D', 'A,B,F', 'A,E,G', None]
df = pd.DataFrame(data, columns=['letter'])
df :
letter
0 A,C,D
1 A,B,F
2 A,E,G
3 None
get_dummies and groupby
pd.get_dummies(df['letter'].str.split(',').explode()).groupby(level=0).sum()
output:
A B C D E F G
0 1 0 1 1 0 0 0
1 1 1 0 0 0 1 0
2 1 0 0 0 1 0 1
3 0 0 0 0 0 0 0
I have a dataframe where one of the columns has its items separated with commas. It looks like:
Data
a,b,c
a,c,d
d,e
a,e
a,b,c,d,e
My goal is to create a matrix that has as header all the unique values from column Data, meaning [a,b,c,d,e]. Then as rows a flag indicating if the value is at that particular row.
The matrix should look like this:
Data
a
b
c
d
e
a,b,c
1
1
1
0
0
a,c,d
1
0
1
1
0
d,e
0
0
0
1
1
a,e
1
0
0
0
1
a,b,c,d,e
1
1
1
1
1
To separate column Data what I did is:
df['data'].str.split(',', expand = True)
Then I don't know how to proceed to allocate the flags to each of the columns.
Maybe you can try this without pivot.
Create the dataframe.
import pandas as pd
import io
s = '''Data
a,b,c
a,c,d
d,e
a,e
a,b,c,d,e'''
df = pd.read_csv(io.StringIO(s), sep = "\s+")
We can use pandas.Series.str.split with expand argument equals to True. And value_counts each rows with axis = 1.
Finally fillna with zero and change the data into integer with astype(int).
df["Data"].str.split(pat = ",", expand=True).apply(lambda x : x.value_counts(), axis = 1).fillna(0).astype(int)
#
a b c d e
0 1 1 1 0 0
1 1 0 1 1 0
2 0 0 0 1 1
3 1 0 0 0 1
4 1 1 1 1 1
And then merge it with the original column.
new = df["Data"].str.split(pat = ",", expand=True).apply(lambda x : x.value_counts(), axis = 1).fillna(0).astype(int)
pd.concat([df, new], axis = 1)
#
Data a b c d e
0 a,b,c 1 1 1 0 0
1 a,c,d 1 0 1 1 0
2 d,e 0 0 0 1 1
3 a,e 1 0 0 0 1
4 a,b,c,d,e 1 1 1 1 1
Use the Series.str.get_dummies() method to return the required matrix of 'a', 'b', ... 'e' columns.
df["Data"].str.get_dummies(sep=',')
If you split the strings into lists, then explode them, it makes pivot possible.
(df.assign(data_list=df.Data.str.split(','))
.explode('data_list')
.pivot_table(index='Data',
columns='data_list',
aggfunc=lambda x: 1,
fill_value=0))
Output
data_list a b c d e
Data
a,b,c 1 1 1 0 0
a,b,c,d,e 1 1 1 1 1
a,c,d 1 0 1 1 0
a,e 1 0 0 0 1
d,e 0 0 0 1 1
You could apply a custom count function for each key:
for k in ["a","b","c","d","e"]:
df[k] = df.apply(lambda row: row["Data"].count(k), axis=1)
I have a dataframe, it's in one hot format:
dummy_data = {'a': [0,0,1,0],'b': [1,1,1,0], 'c': [0,1,0,1],'d': [1,1,1,0]}
data = pd.DataFrame(dummy_data)
Output:
a b c d
0 0 1 0 1
1 0 1 1 1
2 1 1 0 1
3 0 0 1 0
I am trying to get the occurrence matrix from dataframe, but if I have columns name in list instead of one hot like this:
raw = [['b','d'],['b','c','d'],['a','b','d'],['c']]
unique_categories = ['a','b','c','d']
Then I am able to find the occurrence matrix like this:
df = pd.DataFrame(raw).stack().rename('val').reset_index().drop(columns='level_1')
df = df.loc[df.val.isin(unique_categories)]
df = df.merge(df, on='level_0').query('val_x != val_y')
final = pd.crosstab(df.val_x, df.val_y)
adj_matrix = (pd.crosstab(df.val_x, df.val_y)
.reindex(unique_categories, axis=0).reindex(unique_categories, axis=1)).fillna(0)
Output:
val_y a b c d
val_x
a 0 1 0 1
b 1 0 1 3
c 0 1 0 1
d 1 3 1 0
How to get the occurrence matrix directly from one hot dataframe?
You can have some fun with matrix math!
u = np.diag(np.ones(df.shape[1], dtype=bool))
df.T.dot(df) * (~u)
a b c d
a 0 1 0 1
b 1 0 1 3
c 0 1 0 1
d 1 3 1 0
I have a pandas dataframe consiting of one column containing a string seperated by "/" I would like split these seperated strings into new columns denoted by a boolean (if they exist)
d = {'col1': ["A/B/C", "B/C", "D/B/A", "C/B"]}
dataFrame = pd.DataFrame(data=d)
col1
0 A/B/C
1 B/C
2 D/B/A
3 C/B
the result would be as following:
d = {'A': [1, 0, 1, 0], 'B':[1,1,1,1], 'C':[1,1,0,1], 'D':[0,0,1,0]}
dataFrame = pd.DataFrame(data=d)
A B C D
0 1 1 1 0
1 0 1 1 0
2 1 1 0 1
3 0 1 1 0
I have attempted with pandas.Series.str.split and pandas.pivot but nothing quite returns the result I am looking for. Any help or nudges in the right direction, would be highly appreciated!
Use pandas.Series.str.get_dummies
df.col1.str.get_dummies('/')
A B C D
0 1 1 1 0
1 0 1 1 0
2 1 1 0 1
3 0 1 1 0
Setup
d = {'col1': ["A/B/C", "B/C", "D/B/A", "C/B"]}
df = pd.DataFrame(data=d)
I'm trying perform a specific operation on a dataframe.
Given the following dataframe:
df1 = pd.DataFrame({
'id': [0, 1, 2, 1, 3, 0],
'letter': ['a','b','c','b','b','a'],
'status':[0,1,0,0,0,1]})
id letter status
0 a 0
1 b 1
2 c 0
1 b 0
3 b 0
0 a 1
I'd like to create another dataframe which contains rows from df1 based on the following restriction.
If 2 or more rows have the same id and letter, then return whichever row has a status of 1. All other rows must be copied over.
The resulting dataframe should look like this:
id letter status
0 a 1
1 b 1
2 c 0
3 b 0
Any help is greatly appreciated. Thank you
this should work:
>>> fn = lambda obj: obj[obj.status == 1] if any(obj.status == 1) else obj
>>> df.groupby(['id', 'letter'], as_index=False).apply(fn)
id letter status
5 0 a 1
1 1 b 1
2 2 c 0
4 3 b 0
[4 rows x 3 columns]
sort by status first and then use groupby
In [1932]: df.sort_values(by='status').groupby('id', as_index=False).last()
Out[1932]:
id letter status
0 0 a 1
1 1 b 1
2 2 c 0
3 3 b 0