I have two csv files with 200 columns each. The two files have the exact same numbers in rows and columns. I want to compare each columns separately.
The idea would be to compare column 1 value of file "a" to column 1 value of file "b" and check the difference and so on for all the numbers in the column (there are 100 rows) and write out a number that in how many cases were the difference more than 3.
I would like to repeat the same for all the columns. I know it should be a double for loop but idk exactly how. Probably 2 for loops but have no idea how to do that...
Thanks in advance!
import pandas as pd
dk = pd.read_csv('C:/Users/D/1_top_a.csv', sep=',', header=None)
dk = dk.dropna(how='all')
dk = dk.dropna(how='all', axis=1)
print(dk)
dl = pd.read_csv('C:/Users/D/1_top_b.csv', sep=',', header=None)
dl = dl.dropna(how='all')
dl = dl.dropna(how='all', axis=1)
print(dl)
rows=dk.shape[0]
print(rows)
for i
print(dk._get_value(0,0))
df1 = pd.DataFrame(dict(cola=[1,2,3,4], colb=[4,5,6,7]))
df2 = pd.DataFrame(dict(cola=[1,2,4,5], colb=[9,7,8,9]))
for label, content in df1.items():
diff = df1[label].compare(df2[label])
if diff.shape[0] >= 3:
print(f'Found {diff.shape[0]} diffs in {label}')
print(diff.to_markdown())
Out:
Found 4 diffs in colb
| | self | other |
|---:|-------:|--------:|
| 0 | 4 | 9 |
| 1 | 5 | 7 |
| 2 | 6 | 8 |
| 3 | 7 | 9 |
Related
I am new to pandas dataframes, so I apologies in case there's an easy or even built-in way to do so.
Let's say I have a dataframe df with 3 columns A (a string), B (a float) and C (a bool). Values of column A are not unique. B is a random number and rows with same A value can have different values of B. Columns C is True if the value of A is repeated in the dataset.
An example
| | A | B | C |
|---|-----|-----|-------|
| 0 | cat | 10 | True |
| 1 | dog | 10 | False |
| 2 | cat | 20 | True |
| 3 | bee | 100 | False |
(The column C is actually redundant and could be obtained with df['C']=df['A'].duplicated(keep=False))
What I want to obtain is a dataframe were, for duplicated entries of A (C==True), only the row with the highest B value is kept.
I know how to get the list of rows with maximum value of B:
df.loc[df[df['C']].groupby('A')['B'].idxmax()] #is this the best way actually?
but what I want is the opposite: filter df so to get only the entries not duplicated (C==False) and the duplicated ones with the highest B.
One possibility could be to concatenate df[~df['C']] and the previous table but is it the best way actually?
One approach:
res = df.iloc[df.groupby("A")["B"].idxmax()]
print(res)
Output
A B C
3 bee 100 False
2 cat 20 True
1 dog 10 False
I am trying to clean my data frame but I just want to remove special characters for just one column. (Please refer the figure below)
df1
| A | B | C |
|---------|----––|––----|
| Ags(1) | 5 | 4 |
| Cdmx(2) | 6 | 6 |
|Leon(4) | 90 | 45 |
|
What I want to remove is just the numbers and special characters of the column A
This is what I tried:
df = re.sub('[^A-Za-z0-9]+', '', df1["A"])
>> expected string or bytes-like object
I would try to use a lambda with the apply function on the wanted column.
df1["A"] = df1["A"].apply(lambda x: re.sub('[^A-Za-z0-9]+', '', x))
You can also use .str.extract() to keep the part you want (vs replace, which eliminates the part you don't want):
from io import StringIO
import pandas as pd
data = ''' A B C
Ags(1) 5 4
Cdmx(2) 6 6
Leon(4) 90 45
'''
df = pd.read_csv(StringIO(data), sep='\s\s+', engine='python')
df['A'] = df['A'].str.extract(r'(\w+)', expand=False)
print(df)
A B C
0 Ags 5 4
1 Cdmx 6 6
2 Leon 90 45
I am new to Python and Stackoverflow, so please bear with me. I have a large datafile of roughly 140k rows stored as a csv. The file is split up into sections based on age groups, ie. 16-24, 24-50 etc. At every break there are information lines about the age and etnicity of the subjects. After loading the csv into pandas, I tried to break up the dataframe into several smaller ones by dividing on the information lines of the age groups using iloc. Now I have a list of dataframes. I can access each dataframe in the list, no problem, however (I guess due to the information lines) pandas displays all information in one column. Is there a way to format the output and make pandas display the column headers and put the information lines into the header above the column headers? I'm sorry if this is not very clear, please feel free to suggest any edits.
The data in the csv looks something like this:
0 Some information
1 Some information
2 Some information
3
4
5 a | b | c | d |
6 a | 1 | 1 | 1 |
7 a | 1 | 1 | 1 |
8 a | 1 | 1 | 1 |
9
10 Some information
11 Some information
12 Some information
13
14
15 a | b | c | d |
16 a | 1 | 1 | 1 |
17 a | 1 | 1 | 1 |
18 a | 1 | 1 | 1 |
I used iloc to break this up on the information lines by row index.
36065,43278,50491,57704,
64917,72130,79343,86556,
93769,100982,108195,115408,
122621,129834,137047]
l_mod = [0] + l + [max(l)+1]
list_of_dfs = [mydata_df.iloc[l_mod[n]:l_mod[n+1]] for n in range(len(l_mod)-1)]
when accessing I used: df1_df=list_of_dfs[1]
The output is currently as follows:
0
--------------------
1 a,b,c
2 a,1,1,
I hope this makes sense, please suggest edits and I'll do my best to explain.
You can try df[0].str.split(',', expand=True), which expands your dataframe based on every split on a comma. Then you can assign the new column names to it, since it will give column names [0, 1, 2, 3.. etc]
I'm stuck with a little problem with python and regular expressions.
I got a pandas table with records with a different
different order of construction, see below.
+----------------------------------------------+
| Total |
+----------------------------------------------+
| Total Price: 4 x 2 = 8 |
| Total Price 200 Price_per_piece 10 Amount 20 |
+----------------------------------------------+
I want to separate the records in the ‘Total’ column to 3 other columns like below.
Do I need first to split those columns in 2 subset and to do different regular expressions or do you guys have some other solutions/ideas?
+-------+-----------------+--------+
| Total | Price_per_piece | Amount |
+-------+-----------------+--------+
| 8 | 4 | 2 |
| 200 | 10 | 20 |
+-------+-----------------+--------+
Try this one:
dtotal = ({"Total":["Total Price: 4 x 2 = 8","Total Price 200 Price_per_piece 10 Amount 20"]})
dt = pd.DataFrame(dtotal)
data = []
for item in dt['Total']:
regex = re.findall(r"(\d+)\D+(\d+)\D+(\d+)",item)
regex = (map(list,regex))
data.append(list(map(int,list(regex)[0])))
dftotal = pd.DataFrame(data, columns=['Total','Price_per_piece','Amount'])
print(dftotal)
Output:
Total Price_per_piece Amount
0 4 2 8
1 200 10 20
I got the following problem which I got stuck on and unfortunately cannot resolve by myself or by similar questions that I found on stackoverflow.
To keep it simple, I'll give a short example of my problem:
I got a Dataframe with several columns and one column that indicates the ID of a user. It might happen that the same user has several entries in this data frame:
| | userID | col2 | col3 |
+---+-----------+----------------+-------+
| 1 | 1 | a | b |
| 2 | 1 | c | d |
| 3 | 2 | a | a |
| 4 | 3 | d | e |
Something like this. Now I want to known the number of rows that belongs to a certain userID. For this operation I tried to use df.groupby('userID').size() which in return I want to use for another simple calculation, like division whatsover.
But as I try to save the results of the calculation in a seperate column, I keep getting NaN values.
Is there a way to solve this so that I get the result of the calculations in a seperate column?
Thanks for your help!
edit//
To make clear, how my output should look like. The upper dataframe is my main data frame so to say. Besides this frame I got a second frame looking like this:
| | userID | value | value/appearances |
+---+-----------+----------------+-------+
| 1 | 1 | 10 | 10 / 2 = 5 |
| 3 | 2 | 20 | 20 / 1 = 20 |
| 4 | 3 | 30 | 30 / 1 = 30 |
So I basically want in the column 'value/appearances' to have the result of the number in the value column divided by the number of appearances of this certain user in the main dataframe. For user with ID=1 this would be 10/2, as this user has a value of 10 and has 2 rows in the main dataframe.
I hope this makes it a bit clearer.
IIUC you want to do the following, groupby on 'userID' and call transform on the grouped column and pass 'size' to identify the method to call:
In [54]:
df['size'] = df.groupby('userID')['userID'].transform('size')
df
Out[54]:
userID col2 col3 size
1 1 a b 2
2 1 c d 2
3 2 a a 1
4 3 d e 1
What you tried:
In [55]:
df.groupby('userID').size()
Out[55]:
userID
1 2
2 1
3 1
dtype: int64
When assigned back to the df aligns with the df index so it introduced NaN for the last row:
In [57]:
df['size'] = df.groupby('userID').size()
df
Out[57]:
userID col2 col3 size
1 1 a b 2
2 1 c d 1
3 2 a a 1
4 3 d e NaN