We Keep Coding

How to make a barplot for the target variable with each predictive variable?

I have a pandas df, that looks something like this (after scaling):
Age blood_Press golucse Cholesterol
0 1.953859 -1.444088 -1.086684 -1.981315
1 0.357992 -0.123270 -0.585981 0.934929
2 0.997219 0.998712 2.005212 0.019169
3 2.589318 -0.528543 -1.123484 -1.299904
4 2.088141 0.792976 0.021526 -0.777959
and a binary target feature:
y
0 1.0
1 1.0
2 1.0
3 0.0
4 1.0
I want to make a bar chart for each predictive feature with the y target. So the y values would be on the x-axis, which is just 1 or 0, and on the y-axis would be the values for the predictive feature. For example, something that looks like this (ignore the features used here, just an example of what I need). So here instead of male and female I'd have 1 and 0...
the code for this plot is something like this:
myPlot = sns.catplot(data = data, x = 'the y feature' , y = 'the x feature', kind = 'bar')
myPlot.fig.suptitle('title title', size=15, y=1.);
myPlot.set_ylabels('Y label whatever', fontsize=15, x=1.02)
myPlot.fig.set_size_inches(9,8);
But I don't want to repeat it for every feature, I'm sure it's much simpler than that. But how?

Setup
print(df)
Age blood_Press golucse Cholesterol y
0 1.953859 -1.444088 -1.086684 -1.981315 1.0
1 0.357992 -0.123270 -0.585981 0.934929 1.0
2 0.997219 0.998712 2.005212 0.019169 1.0
3 2.589318 -0.528543 -1.123484 -1.299904 0.0
4 2.088141 0.792976 0.021526 -0.777959 1.0
Melt the dataframe to convert from wide to long format
m = df.melt(id_vars=['y'], var_name='feature')
print(m)
# y feature value
# 0 1.0 Age 1.953859
# 1 1.0 Age 0.357992
# 2 1.0 Age 0.997219
# 3 0.0 Age 2.589318
# 4 1.0 Age 2.088141
# 5 1.0 blood_Press -1.444088
# 6 1.0 blood_Press -0.123270
# 7 1.0 blood_Press 0.998712
# 8 0.0 blood_Press -0.528543
# 9 1.0 blood_Press 0.792976
# 10 1.0 golucse -1.086684
# 11 1.0 golucse -0.585981
# 12 1.0 golucse 2.005212
# 13 0.0 golucse -1.123484
# 14 1.0 golucse 0.021526
# 15 1.0 Cholesterol -1.981315
# 16 1.0 Cholesterol 0.934929
# 17 1.0 Cholesterol 0.019169
# 18 0.0 Cholesterol -1.299904
# 19 1.0 Cholesterol -0.777959
Then use the catplot method and pass the col parameter as feature column
sns.catplot(data=m, x='y', y='value', col='feature', kind='bar', col_wrap=2)

Seaborn custom axis sxale: matplotlib.scale.FuncScale

I'm trying to figure out how to get a custom scale for my axis. My x-axis goes from 0 to 1,000,000 in 100,000 step increments, but I want to scale each of these numbers by 1/100, so that they go from 0 to 1,000 in 100 step increments. matplotlib.scale.FuncScale, but I'm having trouble getting it to work.
Here's what the plot currently looks like:
My code looks like this:
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
dataPlot = pd.DataFrame({"plot1" : [1, 2, 3], "plot2" : [4, 5, 6], "plot3" : [7, 8, 9]})
ax = sns.lineplot(data = dataPlot, dashes = False, palette = ["blue", "red", "green"])
ax.set_xlim(1, numRows)
ax.set_xticks(range(0, numRows, 100000))
plt.ticklabel_format(style='plain')
plt.scale.FuncScale("xaxis", ((lambda x : x / 1000), (lambda y : y * 1000)))
When I run this code specifically, I get AttributeError: module 'matplotlib.pyplot' has no attribute 'scale', so I tried adding import matplotlib as mpl to the top of the code and then changing the last line to be mpl.scale.FuncScale("xaxis", ((lambda x : x / 1000), (lambda y : y * 1000))) and that actually ran without error, but but it didn't change anything.
How can I get this to properly scale the axis?

Based on the clarification from the question comments a straightforward solution scaling the x-axis data in the dataframe (x-data in the question case being the df index) and then plot.
Using example data since the code from the question wasn't running on its own.
x starting range is 0 to 100, and then scaled to 0 to 10, but that's equivalent to any other starting range and scaling.
1st the default df.plot: (just as reference)
import pandas as pd
import numpy as np
arr = np.arange(0, 101, 1) * 1.5
df = pd.DataFrame(arr, columns=['y_data'])
print(df)
y_data
0 0.0
1 1.5
2 3.0
3 4.5
4 6.0
.. ...
96 144.0
97 145.5
98 147.0
99 148.5
100 150.0
df.plot()
Note that per default df.plot uses the index as x-axis.
2nd scaling the x-data in the dataframe:
The interims dfs are only displayed to follow along.
Preparation
df.reset_index(inplace=True)
Getting the original index data as a column to further work with (see scaling below).
index y_data
0 0 0.0
1 1 1.5
2 2 3.0
3 3 4.5
4 4 6.0
.. ... ...
96 96 144.0
97 97 145.5
98 98 147.0
99 99 148.5
100 100 150.0
df = df.rename(columns = {'index':'x_data'}) # just to be more explicit
x_data y_data
0 0 0.0
1 1 1.5
2 2 3.0
3 3 4.5
4 4 6.0
.. ... ...
96 96 144.0
97 97 145.5
98 98 147.0
99 99 148.5
100 100 150.0
Scaling
df['x_data'] = df['x_data'].apply(lambda x: x/10)
x_data y_data
0 0.0 0.0
1 0.1 1.5
2 0.2 3.0
3 0.3 4.5
4 0.4 6.0
.. ... ...
96 9.6 144.0
97 9.7 145.5
98 9.8 147.0
99 9.9 148.5
100 10.0 150.0
3rd df.plot with specific columns:
df.plot(x='x_data', y = 'y_data')
By x= a specific column instead of the default = index is used as the x-axis.
Note that the y data hasn't changed but the x-axis is now scaled compared to the "1st the default df.plot" above.

ValueError: operands could not be broadcast together with shapes (1,55) (42,)

To Download Dataset click link
I am trying to find out disease type based on the symptoms, by using a machine learning model. All are going well but when I trying to predict the disease type based on given symptoms it gives me "ValueError: operands could not be broadcast together with shapes (1,55) (42,) " that error. to solve this i have seen many of the similar post but not able to solve it.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
import re
import string
import nltk
from nltk.corpus import stopwords
from sklearn.metrics import f1_score
from sklearn.model_selection import train_test_split
from nltk.stem.snowball import SnowballStemmer
from nlppreprocess import NLP
import math
import string
punct = string.punctuation
import spacy
import en_core_web_sm
nlp = en_core_web_sm.load()
#nlp = spacy.load("en_core_web_sm")
from spacy.lang.en.stop_words import STOP_WORDS
from sklearn.metrics import confusion_matrix,accuracy_score, classification_report, roc_curve, auc
from sklearn.naive_bayes import GaussianNB
gnb = GaussianNB()
input data
w = pd.read_csv("symptom_disease.csv")
w = w.fillna(int(0))
X = w.drop(["Disease"],axis=1)
m = w["Disease"]
data = [1,2,3,4,5,6,7,8,9,10]
y = pd.DataFrame(data,columns=["disease"])
gnb=gnb.fit(X,np.ravel(y))
X.head()
X.head()
output:
Passing much less urine Bleeding from any body part Feeling extremely lethargic/weak Excessive sleepiness/restlessness Altered mental status Seizure/fits Breathlessness Blood in sputum Chest pain Sound/noise in breathing ... diarrhoea sweats and chills difficulty breathing sweating and shivering rapid heartbeat sweating shivering loss of appetite coughing up blood vomiting
0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
1 0.0 0.0 0.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 1.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
import spacy
nlp = spacy.load("en_core_web_sm")
t = ['Passing much less urine','Bleeding from any body part','Feeling extremely lethargic/weak','Excessive sleepiness/restlessness','Altered mental status','Seizure/fits','Breathlessness','Blood in sputum','Chest pain','Sound/noise in breathing','Drooling of saliva','Difficulty in opening mouth','Eye irritation','Runny nose','Stuffy nose','watery eyes','Sneezing','itchy nose','itchy throat','fever','headache','intense pain','fatigue','dry cough','bloody stools','loose stools','nausea','shortness of breath','tight chest','cough','short of breath','muscle pains','diarrhoea','sweats and chills','difficulty breathing','sweating and shivering','rapid heartbeat','sweating','shivering','loss of appetite','coughing up blood','vomiting','Weakness','Stomach pain','constipation','Cough','Chills','Abdominal pain','Yellow skin color','skin color yellow','Dark-colored urine','clay-colored stool','yellow color urine','weight loss','itchy skin']
#t = ['Passing much less urine', 'Bleeding from any body part', 'Feeling extremely lethargic/weak', 'Excessive sleepiness/restlessness', 'Altered mental status', 'Seizure/fits', 'Breathlessness', 'Blood in sputum', 'Chest pain', 'Sound/noise in breathing', 'Drooling of saliva', 'Difficulty in opening mouth']
docs = nlp.pipe(t)
l1= []
for doc in docs:
clean_doc = " ".join([tok.lemma_.lower() for tok in doc if not tok.is_stop and not tok.is_punct])
l1.append(clean_doc)
l2=[]
for i in range(0,len(l1)):
l2.append(0)
print(l2)
import spacy
nlp = spacy.load("en_core_web_sm")
psymptoms = ["Blood in sputum","Chest pain","Sound/noise in breathing","Breathlessness"]
docs = nlp.pipe(psymptoms)
sym= []
for doc in docs:
clean_doc = " ".join([tok.lemma_.lower() for tok in doc if not tok.is_stop and not tok.is_punct])
sym.append(clean_doc)
for k in range(0,len(l1)):
for z in sym:
#print(z)
if(z==l1[k]):
l2[k]=1
inputtest = [l2]
predict = gnb.predict(inputtest)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-39-d99236746b75> in <module>
1 #print(inputtest)
----> 2 predict = gnb.predict(inputtest)
~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in predict(self, X)
76 check_is_fitted(self)
77 X = self._check_X(X)
---> 78 jll = self._joint_log_likelihood(X)
79 return self.classes_[np.argmax(jll, axis=1)]
80
~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in _joint_log_likelihood(self, X)
454 jointi = np.log(self.class_prior_[i])
455 n_ij = - 0.5 * np.sum(np.log(2. * np.pi * self.sigma_[i, :]))
--> 456 n_ij -= 0.5 * np.sum(((X - self.theta_[i, :]) ** 2) /
457 (self.sigma_[i, :]), 1)
458 joint_log_likelihood.append(jointi + n_ij)
ValueError: operands could not be broadcast together with shapes (1,55) (42,)
Error msg picture

Okay, finally I solved it.
Actually, there is a dimensional problem.
Problems come because I was given an input data dimension of the model is X=(10 rows × 42 columns) and y = (10 rows × 1 column).
and when use the model for prediction then I was given a test data dimension of = (1 rows × 55 columns). That's the problem of dimension. Now I changed my input data shape of X = (10 rows × 55 columns). So now it works fine and predicted well.

Python: Finding multiple linear trend lines in a scatter plot

I have the following pandas dataframe -
Atomic Number R C
0 2.0 49.0 0.040306
1 3.0 205.0 0.209556
2 4.0 140.0 0.107296
3 5.0 117.0 0.124688
4 6.0 92.0 0.100020
5 7.0 75.0 0.068493
6 8.0 66.0 0.082244
7 9.0 57.0 0.071332
8 10.0 51.0 0.045725
9 11.0 223.0 0.217770
10 12.0 172.0 0.130719
11 13.0 182.0 0.179953
12 14.0 148.0 0.147929
13 15.0 123.0 0.102669
14 16.0 110.0 0.120729
15 17.0 98.0 0.106872
16 18.0 88.0 0.061996
17 19.0 277.0 0.260485
18 20.0 223.0 0.164312
19 33.0 133.0 0.111359
20 36.0 103.0 0.069348
21 37.0 298.0 0.270709
22 38.0 245.0 0.177368
23 54.0 124.0 0.079491
The trend between r and C is generally a linear one. What I would like to do if possible is find an exhaustive list of all the possible combinations of 3 or more points and what their trends are with scipy.stats.linregress so that I can find groups of points that fit linearly the best.
Which would ideally look something like this for the data, (Source) but I am looking for all the other possible trends too.
So the question, how do I feed all the 16776915 possible combinations (sum_(i=3)^24 binomial(24, i)) of 3 or more points into lingress and is it even doable without a ton of code?

My following solution proposal is based on the RANSAC algorithm. It is method to fit a mathematical model (e.g. a line) to data with heavy of outliers.
RANSAC is one specific method from the field of robust regression.
My solution below first fits a line with RANSAC. Then you remove the data points close to this line from your data set (which is the same as keeping the outliers), fit RANSAC again, remove data, etc until only very few points are left.
Such approaches always have parameters which are data dependent (e.g. noise level or proximity of the lines). In the following solution and MIN_SAMPLES and residual_threshold are parameters which might require some adaption to the structure of your data:
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model
MIN_SAMPLES = 3
x = np.linspace(0, 2, 100)
xs, ys = [], []
# generate points for thee lines described by a and b,
# we also add some noise:
for a, b in [(1.0, 2), (0.5, 1), (1.2, -1)]:
xs.extend(x)
ys.extend(a * x + b + .1 * np.random.randn(len(x)))
xs = np.array(xs)
ys = np.array(ys)
plt.plot(xs, ys, "r.")
colors = "rgbky"
idx = 0
while len(xs) > MIN_SAMPLES:
# build design matrix for linear regressor
X = np.ones((len(xs), 2))
X[:, 1] = xs
ransac = linear_model.RANSACRegressor(
residual_threshold=.3, min_samples=MIN_SAMPLES
)
res = ransac.fit(X, ys)
# vector of boolean values, describes which points belong
# to the fitted line:
inlier_mask = ransac.inlier_mask_
# plot point cloud:
xinlier = xs[inlier_mask]
yinlier = ys[inlier_mask]
# circle through colors:
color = colors[idx % len(colors)]
idx += 1
plt.plot(xinlier, yinlier, color + "*")
# only keep the outliers:
xs = xs[~inlier_mask]
ys = ys[~inlier_mask]
plt.show()
In the following plot points shown as stars belong to the clusters detected by my code. You also see a few points depicted as circles which are the points remaining after the iterations. The few black stars form a cluster which you could get rid of by increasing MIN_SAMPLES and / or residual_threshold.

How to make axis tick labels visible on the other side of the plot in gridspec?

Plotting my favourite example dataframe,which looks like this:
x val1 val2 val3
0 0.0 10.0 NaN NaN
1 0.5 10.5 NaN NaN
2 1.0 11.0 NaN NaN
3 1.5 11.5 NaN 11.60
4 2.0 12.0 NaN 12.08
5 2.5 12.5 12.2 12.56
6 3.0 13.0 19.8 13.04
7 3.5 13.5 13.3 13.52
8 4.0 14.0 19.8 14.00
9 4.5 14.5 14.4 14.48
10 5.0 NaN 19.8 14.96
11 5.5 15.5 15.5 15.44
12 6.0 16.0 19.8 15.92
13 6.5 16.5 16.6 16.40
14 7.0 17.0 19.8 18.00
15 7.5 17.5 17.7 NaN
16 8.0 18.0 19.8 NaN
17 8.5 18.5 18.8 NaN
18 9.0 19.0 19.8 NaN
19 9.5 19.5 19.9 NaN
20 10.0 20.0 19.8 NaN
I have two subplots, for some other reasons it is best for me to use gridspec. The plotting code is as follows (it is quite comprehensive, so I would like to avoid major changes in the code that otherwise works perfectly and just doesn't do one unimportant detail):
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import gridspec
import matplotlib as mpl
df = pd.read_csv('H:/DocumentsRedir/pokus/dataframe.csv', delimiter=',')
# setting limits for x and y
ylimit=(0,10)
yticks1=np.arange(0,11,1)
xlimit1=(10,20)
xticks1 = np.arange(10,21,1)
# general plot formatting (axes colour, background etc.)
plt.style.use('ggplot')
plt.rc('axes',edgecolor='black')
plt.rc('axes', facecolor = 'white')
plt.rc('grid', color = 'grey')
plt.rc('grid', alpha = 0.3) # alpha is percentage of transparency
colours = ['g','b','r']
title1 = 'The plot'
# GRIDSPEC INTRO - rows, cols, distance of individual plots
fig = plt.figure(figsize=(6,4))
gs=gridspec.GridSpec(1,2, hspace=0.15, wspace=0.08,width_ratios=[1,1])
## SUBPLOT of GRIDSPEC with lines
# the first plot
axes1 = plt.subplot(gs[0,0])
for count, vals in enumerate(df.columns.values[1:]):
X = np.asarray(df[vals])
h = vals
p1 = plt.plot(X,df.index,color=colours[count],linestyle='-',linewidth=1.5,label=h)
# formatting
p1 = plt.ylim(ylimit)
p1 = plt.yticks(yticks1, yticks1, rotation=0)
p1 = axes1.yaxis.set_minor_locator(mpl.ticker.MultipleLocator(0.1))
p1 = plt.setp(axes1.get_yticklabels(),fontsize=8)
p1 = plt.gca().invert_yaxis()
p1 = plt.ylabel('x [unit]', fontsize=14)
p1 = plt.xlabel("Value [unit]", fontsize=14)
p1 = plt.tick_params('both', length=5, width=1, which='minor', direction = 'in')
p1 = axes1.xaxis.set_minor_locator(mpl.ticker.MultipleLocator(0.1))
p1 = plt.xlim(xlimit1)
p1 = plt.xticks(xticks1, xticks1, rotation=0)
p1 = plt.setp(axes1.get_xticklabels(),fontsize=8)
p1 = plt.legend(loc='best',fontsize = 8, ncol=2) #
# the second plot (something random)
axes2 = plt.subplot(gs[0,1])
for count, vals in enumerate(df.columns.values[1:]):
nonans = df[vals].dropna()
result=nonans-0.5
p2 = plt.plot(result,nonans.index,color=colours[count],linestyle='-',linewidth=1.5)
p2 = plt.ylim(ylimit)
p2 = plt.yticks(yticks1, yticks1, rotation=0)
p2 = axes2.yaxis.set_minor_locator(mpl.ticker.MultipleLocator(0.1))
p2 = plt.gca().invert_yaxis()
p2 = plt.xlim(xlimit1)
p2 = plt.xticks(xticks1, xticks1, rotation=0)
p2 = axes2.xaxis.set_minor_locator(mpl.ticker.MultipleLocator(0.1))
p2 = plt.setp(axes2.get_xticklabels(),fontsize=8)
p2 = plt.xlabel("Other value [unit]", fontsize=14)
p2 = plt.tick_params('x', length=5, width=1, which='minor', direction = 'in')
p2 = plt.setp(axes2.get_yticklabels(), visible=False)
fig.suptitle(title1, size=16)
plt.show()
However, is it possible to show the y tick labels of the second subplot on the right hand side? The current code produces this:
And I would like to know if there is an easy way to get this:

No, ok, found out it is precisely what I wanted.
I want the TICKS to be on BOTH sides, just the LABELS to be on the right. The solution above removes my ticks from the left side of the subplot, which doesn't look good. However, this answer seems to get the right solution :)
To sum up:
to get the ticks on both sides and labels on the right, this is what fixes it:
axes2.yaxis.tick_right(‌​)
axes2.yaxis.set_ticks_p‌​osition('both')
And if you need the same for x axis, it's axes2.xaxis.tick_top(‌​)

try something like
axes2.yaxis.tick_right()
Just look around Python Matplotlib Y-Axis ticks on Right Side of Plot.