We Keep Coding

nunique compare two Pandas dataframe with duplicates and pivot them

My input:
df1 = pd.DataFrame({'frame':[ 1,1,1,2,3,0,1,2,2,2,3,4,4,5,5,5,8,9,9,10,],
'label':['GO','PL','ICV','CL','AO','AO','AO','ICV','PL','TI','PL','TI','PL','CL','CL','AO','TI','PL','ICV','ICV'],
'user': ['user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1','user1']})
df2 = pd.DataFrame({'frame':[ 1, 1, 2, 3, 4,0,1,2,2,2,4,4,5,6,6,7,8,9,10,11],
'label':['ICV','GO', 'CL','TI','PI','AO','GO','ICV','TI','PL','ICV','TI','PL','CL','CL','CL','AO','AO','PL','ICV'],
'user': ['user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2','user2']})
df_c = pd.concat([df1,df2])
I trying compare two df, frame by frame, and check if label in df1 existing in same frame in df2. And make some calucation with result (pivot for example)
That my code:
m_df = df1.merge(df2,on=['frame'],how='outer' )
m_df['cross']=m_df.apply(lambda row: 'Matched'
if row['label_x']==row['label_y']
else 'Mismatched', axis='columns')
pv_m_unq= pd.pivot_table(m_df,
columns='cross',
index='label_x',
values='frame',
aggfunc=pd.Series.nunique,fill_value=0,margins=True)
pv_mc = pd.pivot_table(m_df,
columns='cross',
index='label_x',
values='frame',
aggfunc=pd.Series.count,fill_value=0,margins=True)
but this creates a some problem:
first, I can calqulate "simple" total (column All) of matched and missmatched as descipted in picture, or its "duplicated" as AO in pv_m or wrong number as in CL in pv_m_unq
and second, I think merge method as I use int not clever way, because I get if frame+label repetead in df(its happens often), in merged df I get number row in df1 X number of rows in df2 for this specific frame+label
I think maybe there is a smarter way to compare df and pivot them?

You got the unexpected result on margin total because the margin is making use the same function passed to aggfunc (i.e. pd.Series.nunique in this case) for its calculation and the values of Matched and Mismatched in these 2 rows are both the same as 1 (hence only one unique value of 1). (You are currently getting the unique count of frame id's)
Probably, you can achieve more or less what you want by taking the count on them (including margin, Matched and Mismatched) instead of the unique count of frame id's, by using pd.Series.count instead in the last line of codes:
pv_m = pd.pivot_table(m_df,columns='cross',index='label_x',values='frame', aggfunc=pd.Series.count, margins=True, fill_value=0)
Result
cross Matched Mismatched All
label_x
AO 0 1 1
CL 1 0 1
GO 1 1 2
ICV 1 1 2
PL 0 2 2
All 3 5 8
Edit
If all you need is to have the All column being the sum of Matched and Mismatched, you can do it as follows:
Change your code of generating pv_m_unq without building margin:
pv_m_unq= pd.pivot_table(m_df,
columns='cross',
index='label_x',
values='frame',
aggfunc=pd.Series.nunique,fill_value=0)
Then, we create the column All as the sum of Matched and Mismatched for each row, as follows:
pv_m_unq['All'] = pv_m_unq['Matched'] + pv_m_unq['Mismatched']
Finally, create the row All as the sum of Matched and Mismatched for each column and append it as the last row, as follows:
row_All = pd.Series({'Matched': pv_m_unq['Matched'].sum(),
'Mismatched': pv_m_unq['Mismatched'].sum(),
'All': pv_m_unq['All'].sum()},
name='All')
pv_m_unq = pv_m_unq.append(row_All)
Result:
print(pv_m_unq)
Matched Mismatched All
label_x
AO 1 3 4
CL 1 2 3
GO 1 1 2
ICV 2 4 6
PL 1 5 6
TI 2 3 5
All 8 18 26

You can use isin() function like this:
df3 =df1[df1.label.isin(df2.label)]

Pandas: add number of unique values to other dataset (as shown in picture):

I need to add the number of unique values in column C (right table) to the related row in the left table based on the values in common column A (as shown in the picture):
thank you in advance

Groupby column A in second dataset and calculate count of each unique value in column C. merge it with first dataset on column A. Rename column C to C-count if needed:
>>> count_df = df2.groupby('A', as_index=False).C.nunique()
>>> output = pd.merge(df1, count_df, on='A')
>>> output.rename(columns={'C':'C-count'}, inplace=True)
>>> output
A B C-count
0 2 22 3
1 3 23 2
2 5 21 1
3 1 24 1
4 6 21 1

Use DataFrameGroupBy.nunique with Series.map for new column in df1:
df1['C-count'] = df1['A'].map(df2.groupby('A')['C'].nunique())

This may not be the most effective way of doing this, so if your databases are too big be careful.
Define the following function:
def c_value(a_value, right_table):
c_ids = []
for index, row in right_table.iterrows():
if row['A'] == a_value:
if row['C'] not in c_ids:
c_ids.append(row['C'])
return len(c_ids)
For this function I'm supposing that the right_table is a pandas.Dataframe.
Now, you do the following to build the new column (assuming that the left table is a pandas.Dataframe):
new_column = []
for index, row in left_table.iterrows():
new_column.append(c_value(row['A'],right_table))
left_table["C-count"] = new_column
After this, the left_table Dataframe should be the one dessired (as far as I understand what you need).

Select highest member of close coordinates saved in pandas dataframe

I have a dataframe that has following columns: X and Y are Cartesian coordinates and Value is the value of element at these coordinates. What I want to achieve is to select only one coordinates out of n that are close to other, lets say coordinates are close if distance is lower than some value m, so the initial DF looks like this (example):
data = {'X':[0,0,0,1,1,5,6,7,8],'Y':[0,1,4,2,6,5,6,4,8],'Value':[6,7,4,5,6,5,6,4,8]}
df = pd.DataFrame(data)
X Y Value
0 0 0 6
1 0 1 7
2 0 4 4
3 1 2 5
4 1 6 6
5 5 5 5
6 6 6 6
7 7 4 4
8 8 8 8
distance is count with following function:
def countDistance(lat1, lon1, lat2, lon2):
#use basic knowledge about triangles - values are in meters
distance = sqrt(pow(lat1-lat2,2)+pow(lon1-lon2,2))
return distance
lets say if we want to m<=3, the output dataframe would look like this:
X Y Value
1 0 1 7
4 1 6 6
8 8 8 8
What is to be done:
rows 0,1,3 are close, highest value is in row 1, continue
rows 2 and 4 (from original df) are close, keep row 4
rows 5,6,7 are close, keep row 6
left over row 6 is close to row 8, keep row 8, has higher value
So I need to go through dataframe row by row, check the rest, select best match and then continue. I can't think about any simple method how to achieve this, this cant be use case of drop_duplicates, since they are not duplicates, but looping over the whole DF will be very inefficient. One method I could think about was to loop just once, for each of rows finds close ones (probably apply countdistance()), select the best fitting row and replace rest with its values, in the end use drop_duplicates. The other idea was to create a recursive function that would create a new DF, then while original df will have rows select first, find close ones, best match append to new DF, remove first row and all close from original DF and continue until empty, then return same function with new DF as to remove possible uncaught close points.
These ideas are all kind of inefficient, is there a nice and efficient pythonic way to achieve this?

For now, I have created simple code with recursion, the code works but is most likely not optimal.
def recModif(self,df):
#columns=['','X','Y','Value']
new_df = df.copy()
new_df = new_df[new_df['Value']<0] #create copy to work with
changed = False
while not df.empty: #for all the data
df = df.reset_index(drop=True) #need to reset so 0 is always accessible
x = df.loc[0,'X'] #first row x and y
y = df.loc[0,'Y']
df['dist'] = self.countDistance(x,y,df['X'],df['Y']) #add column with distances
select = df[df['dist']<10] #number of meters that two elements cant be next to other
if(len(select.index)>1): #if there is more than one elem close
changed = True
#print(select,select['Value'].idxmax())
select = select.loc[[select['Value'].idxmax()]] #get the highest one
new_df = new_df.append(pd.DataFrame(select.iloc[:,:3]),ignore_index=True) #add it to new df
df = df[df['dist'] >= 10] #drop the elements now
if changed:
return self.recModif(new_df) #use recursion if possible overlaps
else:
return new_df #return new df if all was OK

How can I find a combination of rows from a table where each column sums a specific number (or range)?

I have a table with three columns. Let's say the first row is filled with some people names. The second and the third are numbers representing the value they spent. I want to build another table with a subset of those people where the sum from each column of this new table gives a specific value. How can I do that in Python?
Example: This is my table
Col1 Col2 Col3
John 10 100
Andrew 5 50
Martha 8 20
Ana 2 5
Let's say I wanted a combination where the second column sum is 20 and the third is 125. The result would be:
Col1 Col2 Col3
John 10 100
Martha 8 20
Ana 2 5
Note: Of course, sometimes might be impossible to get exactly the sum. If the code accepts some approximation, like from 0,9X to 1,1X, being X the sum I want, it would be just fine.
Also, I don't need to get a specific number of rows. It can be a combination of 2, 3,...,n.

This is the algorithmic task - to find the combination of values that match the needed criteria. For not complex tasks you can use the following script which removes row by row in the dataframe and checks if the column's sum combination matches needed criteria. However, the script should be elaborated in case you want to continue removing rows (i.e. removing two rows if after trying to remove one row the match was not found). Here the specific algorithm should be implemented (i.e. which exact two rows to remove and in which order?) and there could be a very large number of combinations depending on the complexity of your data.
#sample dataframe
d = {'Column1': ["John", "Andrew", "Martha", "Ana"], 'Column2': [10, 5, 8, 2], 'Column3': [100, 50, 20, 5]}
df = pd.DataFrame(data=d)
#count the sum of each column
totalColumn2 = df['Column2'].sum()
totalColumn3 = df['Column3'].sum()
#function to check if sums of columns match the requrements
def checkRequirements():
if totalColumn2 == 20 and totalColumn3 == 125: #vsums of each column
return True
else:
return False
#iterating through dataframe, removing rows and checking the match
ind = 0
for i, row in df.iterrows():
df1 = df.drop(df.index[ind])
totalColumn2 = df1['Column2'].sum()
totalColumn3 = df1['Column3'].sum()
checkRequirements()
if checkRequirements() is True:
print(df1)
break
ind = ind+1

Extending on #stanna's solution: We can create all possible combinations of the rows to be dropped using iterables.combinations() and check if our requirements are satisfied
def checkRequirements(sum1, sum2):
if sum1 == 20 and sum2 == 125:
return True
else:
return False
# first check if the df as a whole satisfy the requirement
if checkRequirements(df['Col2'].sum(), df['Col3'].sum()) == True:
print(df)
else:
# create multiple combination of rows and drop them and check if they satisfy the requriement
for r in range(1, len(df.index)):
drop_list = list(combinations(list(df.index), r))
for idx in drop_list:
temp_df = df.drop(list(idx))
if checkRequirements(temp_df['Col2'].sum(), temp_df['Col3'].sum()) == True:
print(temp_df)
break
Output:
Col1 Col2 Col3
0 John 10 100
2 Martha 8 20
3 Ana 2 5
Remove the break stmt at the end if you want to print all the matching subsets

Change the row values in pandas based on lookup

I have two pandas dataframe. One Contains actual data and second contains row index which i need to replace with some value.
Df1 : Input record
A B record_id record_type
0 12342345 10 011 H
1 65767454 20 012 I
2 78545343 30 013 I
3 43455467 40 014 I
Df2 :Information contains which row index need to change(e.g :here it is #)
Column1 Column2 Column3 record_id
0 1 2 4 011
1 1 2 None 012
2 1 2 4 013
3 1 2 None 014
Output Result:
A B record_id record_type
0 # # 011 #
1 # # 012 I
2 # # 013 #
3 # # 014 I
So based on record_id lookup and want to change corresponding row index value.
Here (1 2 4 011) present in Df2 contains information saying we want to modify row index first ,second and forth for particular record whose id is 011 from Df1.
So in output result we replace row value for record id 011 for row index 1,2,4 and populate value as #.
please suggest any other approach to do same in pandas.

First, you can do some preprocessing to make life easier. Set the index to be record_id and then rename column3 from df2 to be record_type. Now the dataframes have identical index and column names and makes for easy automatic alignment.
df1 = df1.set_index('record_id')
df2 = df2.set_index('record_id')
df2 = df2.rename(columns={'Column3':'record_type'})
df2 = df2.replace('None', np.nan)
Then we can fill in missing values of df2 with d2 and then make all the original non-missing values '#'.
df2.fillna(df1).where(df2.isnull()).fillna('#')
Column1 Column2 record_type
record_id
11 # # #
12 # # I
13 # # #
14 # # I

Here (1 2 4 011) present in Df2 contains information saying we want to modify row index first, second and forth for particular record whose id is 011 from Df1.
This makes no sense to me -- the row with record_id = 011 does not itself have further rows (of which you seem to want to choose the first, second, fourth). Please complete the output values with the exact results you expect.
In any case, I came across the same problem as in the title, and solved it like this:
Assuming you have a DataFrame df and three equally long vectors rsel, csel (for row/column selectors) and val (say, of length N), and would like to do the equivalent of
df.lookup(rsel, csel) = val
Then, the following code will work (at least) for pandas v.0.23 and python 3.6, assuming that rsel does not contain duplicates!
Warning: this is not really suited for large datasets, because it initialises a full square matrix of the dimensions of shape (N, N)!
import pandas as pd
import numpy as np
from functools import reduce
def coalesce(df, ltr=True):
if not ltr:
df = df.iloc[:, ::-1] # flip left to right
# use iloc as safeguard against non-unique column names
list_of_series = [df.iloc[:, i] for i in range(len(df.columns))]
# this is like a SQL coalesce
return reduce(lambda interm, x: interm.combine_first(x), list_of_series)
# column names generally not unique!
square = pd.DataFrame(np.diag(val), index=rsel, columns=csel)
# np.diag creates 0s everywhere off-diagonal; set them to nan
square = square.where(np.diag([True] * len(rsel)))
# assuming no duplicates in rsel; this is empty
upd = pd.DataFrame(index=rsel, columns=sorted(csel.unique()))
# collapse square into upd
upd = upd.apply(lambda col: coalesce(square[square.columns == col.name]))
# actually update values
df.update(upd)
PS. If you know that you only have strings as column names, then square.filter(regex=col.name) is much faster than square[square.columns == col.name].