We Keep Coding

ValueError: at least one array or dtype is required

Here is my df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1162 entries, 0 to 1161
Data columns (total 61 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Algorithms&DataStructures 428 non-null float64
1 C/C++Programming 688 non-null float64
2 Calculus1 835 non-null float64
3 Calculus2 752 non-null float64
4 Calculus3 366 non-null float64
5 ChemistryLaboratory 497 non-null float64
6 ChemistryforEngineers 823 non-null float64
7 ComputerArchitecture 433 non-null float64
And this is the function used to impute the NaN value:
from sklearn.neighbors import KNeighborsRegressor
# function that imputes a dataframe
def impute_knn(df):
''' inputs: pandas df containing feature matrix '''
''' outputs: dataframe with NaN imputed '''
# imputation with KNN unsupervised method
# separate dataframe into numerical/categorical
ldf = df.select_dtypes(include=[np.number]) # select numerical columns in df
ldf_putaside = df.select_dtypes(exclude=[np.number]) # select categorical columns in df
# define columns w/ and w/o missing data
cols_nan = ldf.columns[ldf.isna().any()].tolist() # columns w/ nan
cols_no_nan = ldf.columns.difference(cols_nan).values # columns w/o nan
for col in cols_nan:
imp_test = ldf[ldf[col].isna()] # indicies which have missing data will become our test set
imp_train = ldf.dropna() # all indicies which which have no missing data
model = KNeighborsRegressor(n_neighbors=5) # KNR Unsupervised Approach
knr = model.fit(imp_train[cols_no_nan], imp_train[col])
ldf.loc[df[col].isna(), col] = knr.predict(imp_test[cols_no_nan])
return pd.concat([ldf,ldf_putaside],axis=1)
I got it from: Bayesian Regression | House Price Prediction
However, when I apply it to my dataframe, it reports an error:
Full error:
ValueError Traceback (most recent call last)
<ipython-input-284-b13fac408835> in <module>
----> 1 df2 = impute_knn(df)
2 # looks like we have a full feature matrix
3 df2.info()
5 frames
/usr/local/lib/python3.8/dist-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator)
663
664 if all(isinstance(dtype, np.dtype) for dtype in dtypes_orig):
--> 665 dtype_orig = np.result_type(*dtypes_orig)
666
667 if dtype_numeric:
<__array_function__ internals> in result_type(*args, **kwargs)
ValueError: at least one array or dtype is required
I'd love to hear your comment! Thank you!

Seaborn Python Filter Columns [duplicate]

I have two queries:
I want to remove the empty bar from the bar graph (present in the first column).
I have to use this graph in a PowerPoint presentation. How can I increase the height of the bar graph such that it fixes the height of the slide? I have tried to increase the height but it is not increasing any further. Is it possible? If not what are other options that I can try?
plt.figure(figsize=(40,20))
g = sns.catplot(x = 'Subject', y = 'EE Score',data = df , hue = 'Session',col='Grade',sharey = True,sharex = True,
hue_order=["2017-18", "2018-19", "2019-20"], kind="bar");
#plt.legend(bbox_to_anchor=(1, 1), loc=2)
g.set(ylim=(0, 100))
g.set_axis_labels("Subject", "EE Score")
ax = g.facet_axis(0,0)
for p in ax.patches:
ax.text(p.get_x() + 0.015,
p.get_height() * 1.02,
'{0:.1f}'.format(p.get_height()),
color='black', rotation='horizontal', size=12)
ax = g.facet_axis(0,1)
for p in ax.patches:
ax.text(p.get_x() + 0.015,
p.get_height() * 1.02,
'{0:.1f}'.format(p.get_height()),
color='black', rotation='horizontal', size=12)
ax = g.facet_axis(0,2)
for p in ax.patches:
ax.text(p.get_x() + 0.015,
p.get_height() * 1.02,
'{0:.1f}'.format(p.get_height()),
color='black', rotation='horizontal', size=12)
ax = g.facet_axis(0,3)
for p in ax.patches:
ax.text(p.get_x() + 0.015,
p.get_height() * 1.02,
'{0:.1f}'.format(p.get_height()),
color='black', rotation='horizontal', size=12)
#g.set_ylabel('')
plt.savefig('2.png', bbox_inches = 'tight')

Like #JohanC, I initially thought it was not possible to remove an empty category from a catplot(). However, Michael's comment provides the solution: sharex=False.
This solution will not work if the column used for x= is a category dtype, which can be checked with pandas.DataFrame.info()
Tested in python 3.10, pandas 1.4.2, matplotlib 3.5.1, seaborn 0.11.2
seaborn is a high-level api for matplotlib
object dtype x-axis
import seaborn as sns
titanic = sns.load_dataset('titanic')
# remove one category
titanic.drop(titanic.loc[(titanic['class']=='First')&(titanic['who']=='child')].index, inplace=True)
g = sns.catplot(x="who", y="survived", col="class", data=titanic, kind="bar", ci=None, sharex=False, hue='embarked', estimator=sum)
categorical dtype x-axis
See that tips.day is categorical and sharex=False will not work
The column can be converted to object dtype with tips.day = tips.day.astype('str'), in which case, sharex=False will work, but the days of the week will not be ordered.
tips = sns.load_dataset('tips')
print(tips.info())
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244 entries, 0 to 243
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 total_bill 244 non-null float64
1 tip 244 non-null float64
2 sex 244 non-null category
3 smoker 244 non-null category
4 day 244 non-null category
5 time 244 non-null category
6 size 244 non-null int64
dtypes: category(4), float64(2), int64(1)
memory usage: 7.4 KB
g = sns.catplot(x="day", y="total_bill", col="time", kind="bar", data=tips, ci=None, sharex=False, hue='smoker')
With converting the column to a object dtype
Note the days are no longer ordered.
tips.day = tips.day.astype('str')
print(tips.info())
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 244 entries, 0 to 243
Data columns (total 7 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 total_bill 244 non-null float64
1 tip 244 non-null float64
2 sex 244 non-null category
3 smoker 244 non-null category
4 day 244 non-null object
5 time 244 non-null category
6 size 244 non-null int64
dtypes: category(3), float64(2), int64(1), object(1)
memory usage: 8.8+ KB
g = sns.catplot(x="day", y="total_bill", col="time", kind="bar", data=tips, ci=None, sharex=False, hue='smoker')

getting strange error while calculating z-score

i want to calculate z-score of my whole dataset. i have tried two types of code but unfortunately they both gave me the same error.
my 1 code is here:
zee=stats.zscore(df)
print(zee)
my 2 code is:
from scipy import stats
import numpy as np
z = np.abs(stats.zscore(df))
print(z)
am using jupyter
The error i have got:
-----
TypeError Traceback (most recent call last)
<ipython-input-23-ef429aebacfd> in <module>
1 from scipy import stats
2 import numpy as np
----> 3 z = np.abs(stats.zscore(df))
4 print(z)
~/.local/lib/python3.8/site-packages/scipy/stats/stats.py in zscore(a, axis, ddof, nan_policy)
2495 sstd = np.nanstd(a=a, axis=axis, ddof=ddof, keepdims=True)
2496 else:
-> 2497 mns = a.mean(axis=axis, keepdims=True)
2498 sstd = a.std(axis=axis, ddof=ddof, keepdims=True)
2499
~/.local/lib/python3.8/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims)
160 ret = umr_sum(arr, axis, dtype, out, keepdims)
161 if isinstance(ret, mu.ndarray):
--> 162 ret = um.true_divide(
163 ret, rcount, out=ret, casting='unsafe', subok=False)
164 if is_float16_result and out is None:
TypeError: unsupported operand type(s) for /: 'str' and 'int'
and here the info of my dataframe,if theres something wrong with my datafarme.
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 14 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 Region 100 non-null object
1 Country 100 non-null object
2 Item Type 100 non-null object
3 Sales Channel 100 non-null object
4 Order Priority 100 non-null object
5 Order Date 100 non-null object
6 Order ID 100 non-null int64
7 Ship Date 100 non-null object
8 Units Sold 100 non-null int64
9 Unit Price 100 non-null float64
10 Unit Cost 100 non-null float64
11 Total Revenue 100 non-null float64
12 Total Cost 100 non-null float64
13 Total Profit 100 non-null float64
dtypes: float64(5), int64(2), object(7)
memory usage: 11.1+ KB
thanks in advance.

Your df contains non float/int values, please try sending only int/float cols to your zscore func.
stats.zscore(df[['Unit Cost', 'Total Revenue', 'Total Cost', 'Total Profit']])

Plotting Pandas' pivot_table from long data

I have a xls file with data organized in long format. I have four columns: the variable name, the country name, the year and the value.
After importing the data in Python with pandas.read_excel, I want to plot the time series of one variable for different countries. To do so, I create a pivot table that transforms the data in wide format. When I try to plot with matplotlib, I get an error
ValueError: could not convert string to float: 'ZAF'
(where 'ZAF' is the label of one country)
What's the problem?
This is the code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data = pd.read_excel('raw_emissions_energy.xls','raw data', index_col = None, thousands='.',parse_cols="A,C,F,M")
data['Year'] = data['Year'].astype(str)
data['COU'] = data['COU'].astype(str)
# generate sub-datasets for specific VARs
data_CO2PROD = pd.pivot_table(data[(data['VAR']=='CO2_PBPROD')], index='COU', columns='Year')
plt.plot(data_CO2PROD)
The xls file with raw data looks like:
raw data Excel view
This is what I get from data_CO2PROD.info()
<class 'pandas.core.frame.DataFrame'>
Index: 105 entries, ARE to ZAF
Data columns (total 16 columns):
(Value, 1990) 104 non-null float64
(Value, 1995) 105 non-null float64
(Value, 2000) 105 non-null float64
(Value, 2001) 105 non-null float64
(Value, 2002) 105 non-null float64
(Value, 2003) 105 non-null float64
(Value, 2004) 105 non-null float64
(Value, 2005) 105 non-null float64
(Value, 2006) 105 non-null float64
(Value, 2007) 105 non-null float64
(Value, 2008) 105 non-null float64
(Value, 2009) 105 non-null float64
(Value, 2010) 105 non-null float64
(Value, 2011) 105 non-null float64
(Value, 2012) 105 non-null float64
(Value, 2013) 105 non-null float64
dtypes: float64(16)
memory usage: 13.9+ KB
None

Using data_CO2PROD.plot() instead of plt.plot(data_CO2PROD) allowed me to plot the data. http://pandas.pydata.org/pandas-docs/stable/visualization.html.
Simple code:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data= pd.DataFrame(np.random.randn(3,4), columns=['VAR','COU','Year','VAL'])
data['VAR'] = ['CC','CC','KK']
data['COU'] =['ZAF','NL','DK']
data['Year']=['1987','1987','2006']
data['VAL'] = [32,33,35]
data['Year'] = data['Year'].astype(str)
data['COU'] = data['COU'].astype(str)
# generate sub-datasets for specific VARs
data_CO2PROD = pd.pivot_table(data=data[(data['VAR']=='CC')], index='COU', columns='Year')
data_CO2PROD.plot()
plt.show()

I think you need add parameter values to pivot_table:
data_CO2PROD = pd.pivot_table(data=data[(data['VAR']=='CC')],
index='COU',
columns='Year',
values='Value')
data_CO2PROD.plot()
plt.show()

pandas dataframe conversion for linear regression

I read the CSV file and get a dataframe (name: data) that has a few columns, the first a few are in format numeric long(type:pandas.core.series.Series) and the last column(label) is a binary response variable string 'P(ass)'/'F(ail)'
import statsmodels.api as sm
label = data.ix[:, -1]
label[label == 'P'] = 1
label[label == 'F'] = 0
fea = data.ix[:, 0: -1]
logit = sm.Logit(label, fea)
result = logit.fit()
print result.summary()
Pandas throws me this error message: "ValueError: Pandas data cast to numpy dtype of object. Check input data with np.asarray(data)"
Numpy,Pandas etc modules are imported already. I tried to convert fea columns to float but still does not go through. Could someone tell me how to correct?
Thanks
update:
data.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 500 entries, 68135 to 3002
Data columns (total 8 columns):
TestQty 500 non-null int64
WaferSize 500 non-null int64
ChuckTemp 500 non-null int64
Notch 500 non-null int64
ORIGINALDIEX 500 non-null int64
ORIGINALDIEY 500 non-null int64
DUTNo 500 non-null int64
PassFail 500 non-null object
dtypes: int64(7), object(1)
memory usage: 35.2+ KB
data.sum()
TestQty 530
WaferSize 6000
ChuckTemp 41395
Notch 135000
ORIGINALDIEX 12810
ORIGINALDIEY 7885
DUTNo 271132
PassFail 20
dtype: float64

Shouldn't your features be this:
fea = data.ix[:, 0:-1]
From you data, you see that PassFail sums to 20 before you convert 'P' to 1 and 'F' to zero. I believe that is the source of your error.
To see what is in there, try:
data.PassFail.unique()
To verify that it totals to 500 (the number of rows in the DataFrame):
sum(label[label == 0]) + sum(label[label == 1)
Finally, try passing values to the function rather than Series and DataFrames:
logit = sm.Logit(label.values, fea.values)