We Keep Coding

Is there any way to adjust the space among precision, recall, f1 in the sklearn's classification report?

My class names are lengthy. So, when I save the classification report from sklearn in a txt file, the format looks messy. For example:
precision recall f1-score support
カップ 0.96 0.94 0.95 69
セット 0.70 0.61 0.65 23
パウチ 0.96 0.92 0.94 53
ビニール容器 0.53 0.47 0.50 19
ビン 0.91 0.90 0.90 305
プラ容器（ヤクルト型） 0.69 0.80 0.74 25
プラ容器（ヨーグルト型） 0.94 0.53 0.68 32
ペットボトル 0.98 0.98 0.98 1189
ペットボトル（ミニ） 0.71 0.74 0.72 23
ボトル缶 0.93 0.89 0.91 96
ポーション 0.80 0.52 0.63 23
箱（飲料） 0.76 0.77 0.76 134
紙パック（Pキャン） 0.86 0.69 0.76 35
紙パック（キャップ付き） 0.93 0.80 0.86 126
紙パック（ゲーブルトップ） 0.85 0.93 0.88 54
紙パック（ブリックパック） 0.84 0.90 0.87 277
紙パック（円柱型） 0.90 0.56 0.69 16
缶 0.89 0.96 0.92 429
accuracy 0.91 2928
macro avg 0.84 0.77 0.80 2928
weighted avg 0.91 0.91 0.91 2928
Is there any way to adjust the space among different metrics so that the alignment keeps same for all rows and columns?
I have looked into the sklearn.metrics.classification_report parameters, but I have not found anything to adjust spacing.

How to write a function with DataFrame in pandas/Python?

Supposed we have a df with a sum() value in the below DataFrame, thanks so much for #jezrael 's answer here, but we have many different df like below DataFrame with different columns, is it possible to add those three line code in a function?
df.columns=['value_a','value_b','name','up_or_down','difference']
# from here
df.loc['sum'] = df[['value_a','value_b','difference']].sum()
df1 = df[['value_a','value_b','difference']].sum().to_frame().T
df = pd.concat([df1, df], ignore_index=True)
# end here
df
value_a value_b name up_or_down difference
project_name
sum 27.56 25.04 -1.31
2021-project11 0.43 0.48 2021-project11 up 0.05
2021-project1 0.62 0.56 2021-project1 down -0.06
2021-project2 0.51 0.47 2021-project2 down -0.04
2021-porject3 0.37 0.34 2021-porject3 down -0.03
2021-porject4 0.64 0.61 2021-porject4 down -0.03
2021-project5 0.32 0.25 2021-project5 down -0.07
2021-project6 0.75 0.81 2021-project6 up 0.06
2021-project7 0.60 0.60 2021-project7 down 0.00
2021-project8 0.85 0.74 2021-project8 down -0.11
2021-project10 0.67 0.67 2021-project10 down 0.00
2021-project9 0.73 0.73 2021-project9 down 0.00
2021-project11 0.54 0.54 2021-project11 down 0.00
2021-project12 0.40 0.40 2021-project12 down 0.00
2021-project13 0.76 0.77 2021-project13 up 0.01
2021-project14 1.16 1.28 2021-project14 up 0.12
2021-project15 1.01 0.94 2021-project15 down -0.07
2021-project16 1.23 1.24 2021-project16 up 0.01
2022-project17 0.40 0.36 2022-project17 down -0.04
2022-project_11 0.40 0.40 2022-project_11 down 0.00
2022-project4 1.01 0.80 2022-project4 down -0.21
2022-project1 0.65 0.67 2022-project1 up 0.02
2022-project2 0.75 0.57 2022-project2 down -0.18
2022-porject3 0.32 0.32 2022-porject3 down 0.00
2022-project18 0.91 0.56 2022-project18 down -0.35
2022-project5 0.84 0.89 2022-project5 up 0.05
2022-project19 0.61 0.48 2022-project19 down -0.13
2022-project6 0.77 0.80 2022-project6 up 0.03
2022-project20 0.63 0.54 2022-project20 down -0.09
2022-project8 0.59 0.55 2022-project8 down -0.04
2022-project21 0.58 0.54 2022-project21 down -0.04
2022-project10 0.76 0.76 2022-project10 down 0.00
2022-project9 0.70 0.71 2022-project9 up 0.01
2022-project22 0.62 0.56 2022-project22 down -0.06
2022-project23 2.03 1.74 2022-project23 down -0.29
2022-project12 0.39 0.39 2022-project12 down 0.00
2022-project24 1.35 1.55 2022-project24 up 0.20
project25 0.45 0.42 project25 down -0.03
project26 0.53 NaN project26 down NaN
project27 0.68 NaN project27 down NaN
Can I add a function with conditions like below, and our other df values can use the function directly?
def sum_handler(x):
if .......
return .....
elif .......
return .....
else
return .....
Thanks so much for any advice

You could try a different approach for summing up your dataframe like shown in this answer.
df.loc['Total'] = df.sum(numeric_only=True, axis=0)
Since this is a one line of code, there would be no need to create a custom function to do this. But for future referrence, you can create a custom function and apply it to a dataframe like this:
import pandas as pd
def double_columns(df: pd.DataFrame, columns: list[str]):
""" Doubles chosen columns of a dataframe """
df[columns] = df[columns] * 2
return df
df = pd.DataFrame({'col1': [1,2], 'col2': [2,3]})
df = double_columns(df, ['col1'])
print(df)
would return
col1 col2
0 2 2
1 4 3

How to use np.genfromtxt and fill in missing columns?

I am trying to use np.genfromtxt to load a data that looks something like this into a matrix:
0.79 0.10 0.91 -0.17 0.10 0.33 -0.90 0.10 -0.19 -0.00 0.10 -0.99 -0.06 0.10 -0.42 -0.66 0.10 -0.79 0.21 0.10 0.93 0.79 0.10 0.91 -0.72 0.10 0.25 0.64 0.10 -0.27 -0.36 0.10 -0.66 -0.52 0.10 0.92 -0.39 0.10 0.43 0.63 0.10 0.25 -0.58 0.10 -0.03 0.59 0.10 0.02 -0.69 0.10 0.79 0.30 0.10 0.09 0.70 0.10 0.67 -0.04 0.10 -0.65 -0.07 0.10 0.70 -0.06 0.10 0.08 7 566 112 32 163 615 424 543 424 422 490 47 499 595 94 515 163 535
0.79 0.10 0.91 -0.17 0.10 0.33 -0.90 0.10 -0.19 -0.00 0.10 -0.99 -0.06 0.10 -0.42 -0.66 0.10 -0.79 0.21 0.10 0.93 0.79 0.10 0.91 -0.72 0.10 0.25 0.64 0.10 -0.27 -0.36 0.10 -0.66 -0.52 0.10 0.92 -0.39 0.10 0.43 0.63 0.10 0.25 -0.58 0.10 -0.03 0.59 0.10 0.02 -0.69 0.10 0.79 0.30 0.10 0.09 0.70 0.10 0.67 -0.04 0.10 -0.65 -0.07 0.10 0.70 -0.06 0.10 0.08 263 112 32 30 163 366 543 457 424 422 556 55 355 485 112 515 163 509 112 535
0.79 0.10 0.91 -0.17 0.10 0.33 -0.90 0.10 -0.19 -0.00 0.10 -0.99 -0.06 0.10 -0.42 -0.66 0.10 -0.79 0.21 0.10 0.93 0.79 0.10 0.91 -0.72 0.10 0.25 0.64 0.10 -0.27 -0.36 0.10 -0.66 -0.52 0.10 0.92 -0.39 0.10 0.43 0.63 0.10 0.25 -0.58 0.10 -0.03 0.59 0.10 0.02 -0.69 0.10 0.79 0.30 0.10 0.09 0.70 0.10 0.67 -0.04 0.10 -0.65 -0.07 0.10 0.70 -0.06 0.10 0.08 311 112 32 543 457 77 639 355 412 422 509 112 535 163 77 125 30 412 422 556 55 355 485 112 515
Suppose I want to import data into a matrix of size (4, 5). If not all rows have 5 columns, when it imports the matrix it should replace those columns without 5 rows with "". For example, if the data were simpler, it would look like this:
1,2,3,4,5
6,7,8,9,10
11,12,13,14,15
16,"","","",""
Thus, I want the number of columns to be imported to match that of the max row column count, and if a row doesn't have that many columns, it will fill it with "". I am reading from a file called "data.txt".
This is what I have tried so far:
trainData = np.genfromtxt('data.txt', usecols = range(0, 5), invalid_raise=False, missing_values = "", filling_values="")
However, it gives errors saying:
Line #4 (got 1 columns instead of 5)
How can I solve this?
Thanks!

Pandas has more robust readers and you can use the DataFrame methods to handle the missing values.
You'll have to figure out how many columns to use first:
columns = max(len(l.split()) for l in open('data.txt'))
To read the file:
import pandas
df = pandas.read_table('data.txt',
delim_whitespace=True,
header=None,
usecols=range(columns),
engine='python')
To convert to a numpy array:
import numpy
a = numpy.array(df)
This will fill in NaNs in the blank positions. You can use .fillna() to get other values for blanks.
filled = numpy.array(df.fillna(999))

You need to modify the filling_values argument to np.nan (which is considered of type float so you won't have the string conversion issue) and specify the delimiter to be comma since by default genfromtxt expects only white space as delimiters:
trainData = np.genfromtxt('data.txt', usecols = range(0, 5), invalid_raise=False, missing_values = "", filling_values=np.nan, delimiter=',')

I managed to figure out a solution.
df = pandas.DataFrame([line.strip().split() for line in open('data.txt', 'r')])
data = np.array(df)

With the copy-n-paste of the 3 big lines, this pandas reader works:
In [149]: pd.read_csv(BytesIO(txt), delim_whitespace=True,header=None,error_bad_
...: lines=False,names=list(range(91)))
Out[149]:
0 1 2 3 4 5 6 7 8 9 ... 81 82 \
0 0.79 0.1 0.91 -0.17 0.1 0.33 -0.9 0.1 -0.19 -0.0 ... 515 163
1 0.79 0.1 0.91 -0.17 0.1 0.33 -0.9 0.1 -0.19 -0.0 ... 515 163
2 0.79 0.1 0.91 -0.17 0.1 0.33 -0.9 0.1 -0.19 -0.0 ... 125 30
83 84 85 86 87 88 89 90
0 535 NaN NaN NaN NaN NaN NaN NaN
1 509 112.0 535.0 NaN NaN NaN NaN NaN
2 412 422.0 556.0 55.0 355.0 485.0 112.0 515.0
_.values to get the array.
The key is specifying a big enough names list. Pandas can fill incomplete lines, while genfromtxt requires explicit delimiters.

Creating a new column in pandas quantile using Quantile function

I want to create a column Quantile, for each date. Calculated the Quantile for each unique value Sales value. Ie Category always corresponds to the same number in sales for each particular date.
I have dataframe which is indexed by date. There are many dates and multiple of the same dates. Example of the subset of df for 1 day:
Category Sales Ratio 1 Ratio 2
11/19/2016 Bar 300 0.46 0.96
11/19/2016 Bar 300 0.56 0.78
11/19/2016 Bar 300 0.43 0.96
11/19/2016 Bar 300 0.47 0.94
11/19/2016 Casino 550 0.92 0.12
11/19/2016 Casino 550 0.43 0.74
11/19/2016 Casino 550 0.98 0.65
11/19/2016 Casino 550 0.76 0.67
11/19/2016 Casino 550 0.79 0.80
11/19/2016 Casino 550 0.90 0.91
11/19/2016 Casino 550 0.89 0.31
11/19/2016 Café 700 0.69 0.99
11/19/2016 Café 700 0.07 0.18
11/19/2016 Café 700 0.75 0.59
11/19/2016 Café 700 0.07 0.64
11/19/2016 Café 700 0.14 0.42
11/19/2016 Café 700 0.30 0.67
11/19/2016 Pub 250 0.64 0.09
11/19/2016 Pub 250 0.93 0.37
11/19/2016 Pub 250 0.69 0.42
I want a code which adds a new column called Quantile which calculates for each date the 0.5 quantile of unique Sales. Key thing to note is Sales is always the same for a category for a particular date (things change as dates change).
Example of a solution: df['Quantile'] = df.Sales.groupby(df.index).transform(lambda x: x.quantile(q=0.5, axis=0, interpolation='midpoint'))
However this would not suffice (even if it worked). For this example (for this one date), In the new column df['Quantile'], all values would be the same for a partcular date.
For this date the calculation would use 300, 550, 700 and 250 for the quantile.
Therefore the final df would look like this:
Category Sales Ratio 1 Ratio 2 Quantile
11/19/2016 Bar 300 0.46 0.96 425
11/19/2016 Bar 300 0.56 0.78 425
11/19/2016 Bar 300 0.43 0.96 425
11/19/2016 Bar 300 0.47 0.94 425
11/19/2016 Casino 550 0.92 0.12 425
11/19/2016 Casino 550 0.43 0.74 425
11/19/2016 Casino 550 0.98 0.65 425
11/19/2016 Casino 550 0.76 0.67 425
11/19/2016 Casino 550 0.79 0.80 425
11/19/2016 Casino 550 0.90 0.91 425
11/19/2016 Casino 550 0.89 0.31 425
11/19/2016 Café 700 0.69 0.99 425
11/19/2016 Café 700 0.07 0.18 425
11/19/2016 Café 700 0.75 0.59 425
11/19/2016 Café 700 0.07 0.64 425
11/19/2016 Café 700 0.14 0.42 425
11/19/2016 Café 700 0.30 0.67 425
11/19/2016 Pub 250 0.64 0.09 425
11/19/2016 Pub 250 0.93 0.37 425
11/19/2016 Pub 250 0.69 0.42 425
If I was to do Quantile of all Sales for a particular date without looking at only one element of each category I would get something like 550 (which I do not want).
Key thing is I would like the code to be simple, and reasonably fast (as date is quite big). Also interpolation has to be midpoint.

It seems you need drop_duplicates:
df['Quantile'] = df.Sales.groupby(df.index)
.transform(lambda x: x.drop_duplicates().quantile())
print (df)
Category Sales Ratio 1 Ratio 2 Quantile
11/19/2016 Bar 300 0.46 0.96 425
11/19/2016 Bar 300 0.56 0.78 425
11/19/2016 Bar 300 0.43 0.96 425
11/19/2016 Bar 300 0.47 0.94 425
11/19/2016 Casino 550 0.92 0.12 425
11/19/2016 Casino 550 0.43 0.74 425
11/19/2016 Casino 550 0.98 0.65 425
11/19/2016 Casino 550 0.76 0.67 425
11/19/2016 Casino 550 0.79 0.80 425
11/19/2016 Casino 550 0.90 0.91 425
11/19/2016 Casino 550 0.89 0.31 425
11/19/2016 Cafe 700 0.69 0.99 425
11/19/2016 Cafe 700 0.07 0.18 425
11/19/2016 Cafe 700 0.75 0.59 425
11/19/2016 Cafe 700 0.07 0.64 425
11/19/2016 Cafe 700 0.14 0.42 425
11/19/2016 Cafe 700 0.30 0.67 425
11/19/2016 Pub 250 0.64 0.09 425
11/19/2016 Pub 250 0.93 0.37 425
11/19/2016 Pub 250 0.69 0.42 425
df['Quantile'] = df.Sales.groupby(df.index)
.transform(lambda x: np.percentile(x.unique(), 50))
print (df)
Category Sales Ratio 1 Ratio 2 Quantile
11/19/2016 Bar 300 0.46 0.96 425
11/19/2016 Bar 300 0.56 0.78 425
11/19/2016 Bar 300 0.43 0.96 425
11/19/2016 Bar 300 0.47 0.94 425
11/19/2016 Casino 550 0.92 0.12 425
11/19/2016 Casino 550 0.43 0.74 425
11/19/2016 Casino 550 0.98 0.65 425
11/19/2016 Casino 550 0.76 0.67 425
11/19/2016 Casino 550 0.79 0.80 425
11/19/2016 Casino 550 0.90 0.91 425
11/19/2016 Casino 550 0.89 0.31 425
11/19/2016 Cafe 700 0.69 0.99 425
11/19/2016 Cafe 700 0.07 0.18 425
11/19/2016 Cafe 700 0.75 0.59 425
11/19/2016 Cafe 700 0.07 0.64 425
11/19/2016 Cafe 700 0.14 0.42 425
11/19/2016 Cafe 700 0.30 0.67 425
11/19/2016 Pub 250 0.64 0.09 425
11/19/2016 Pub 250 0.93 0.37 425
11/19/2016 Pub 250 0.69 0.42 425

Row based chart plot (Seaborn or Matplotlib)

Given that my data is a pandas dataframe and looks like this:
Ref +1 +2 +3 +4 +5 +6 +7
2013-05-28 1 -0.44 0.03 0.06 -0.31 0.13 0.56 0.81
2013-07-05 2 0.84 1.03 0.96 0.90 1.09 0.59 1.15
2013-08-21 3 0.09 0.25 0.06 0.09 -0.09 -0.16 0.56
2014-10-15 4 0.35 1.16 1.91 3.44 2.75 1.97 2.16
2015-02-09 5 0.09 -0.10 -0.38 -0.69 -0.25 -0.85 -0.47
How can I plot a chart of the 5 lines (1 for each ref), where the X axis are the columns (+1, +2...), and starts from 0? If is in seaborn, even better. But matplotlib solutions are also welcome.

Plotting a dataframe in pandas is generally all about reshaping the table so that the individual lines you want are in separate columns, and the x-values are in the index. Some of these reshape operations are a bit ugly, but you can do:
df = pd.read_clipboard()
plot_table = pd.melt(df.reset_index(), id_vars=['index', 'Ref'])
plot_table = plot_table.pivot(index='variable', columns='Ref', values='value')
# Add extra row to have all lines start from 0:
plot_table.loc['+0', :] = 0
plot_table = plot_table.sort_index()
plot_table
Ref 1 2 3 4 5
variable
+0 0.00 0.00 0.00 0.00 0.00
+1 -0.44 0.84 0.09 0.35 0.09
+2 0.03 1.03 0.25 1.16 -0.10
+3 0.06 0.96 0.06 1.91 -0.38
+4 -0.31 0.90 0.09 3.44 -0.69
+5 0.13 1.09 -0.09 2.75 -0.25
+6 0.56 0.59 -0.16 1.97 -0.85
+7 0.81 1.15 0.56 2.16 -0.47
Now that you have a table with the right shape, plotting is pretty automatic:
plot_table.plot()