hello I am trying to preform logistic regression using tensor-flow ( sorry if my code looks dumb) and I have written the cost function once in numpy and once in tensor-flow , I am getting different results for the same starting weights , could some one help me?
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_blobs
DataSize=1000
data, y = make_blobs(n_samples=1000, centers=2, n_features=2,random_state=1,center_box=(-5.0,5.0))
plt.scatter(data[:,0],data[:,1])
plt.show(block=False)
x=np.linspace(-1,5,1000)
b=np.ones([1,1])
W=np.ones([2,1])
asd=W*x.T+b
pred=np.dot(data,W)+b
plt.plot(x,asd[0])
plt.show(block=False)
result=((1))/(1+np.exp(-pred))
s=np.log(result)
J=-(y.T.dot(s)+(1-y).T.dot(1-s))/1000
print ("cost in numpy",J)
#
with tf.variable_scope("scopi",reuse=True):
X = tf.placeholder(tf.float32 )
Y = tf.placeholder(tf.float32 )
b = tf.Variable(tf.ones((1,1)),name="bias")
W = tf.Variable(tf.ones((1,2)),name="weights")
ypred=W*X+b
hx=tf.reduce_sum(tf.sigmoid(ypred),reduction_indices=1)
#cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
J=-tf.reduce_sum(tf.mul(tf.transpose(Y),hx)+tf.mul(tf.transpose(1-Y),(1-hx)))/1000
opti=tf.train.AdamOptimizer(0.1).minimize(J)
with tf.Session() as session:
session.run(tf.initialize_all_variables())
h = session.run(J, feed_dict={X: data, Y: y})
print ("cost in tensorflow", h)
# epoch = 100
# for i in range(epoch):
# for j in range(DataSize):
# session.run(opti, feed_dict={X: data[j], Y: y[j]})
#
#
#
#
#
# if i%10==0:
#
# a=session.run(J,feed_dict={X:data,Y:y})
#
# print ("cost ", a)
Cost sample of the cost functions:
('cost in numpy', array([ 2.37780175])) ('cost in tensorflow', 0.073667422)
You are initializing the weights to random values in this line:
session.run(tf.initialize_all_variables())
After that line you can set the values with something like this:
session.run(tf.assign(b,tf.ones((1,2))))
Related
import tensorflow as tf
import matplotlib.pyplot as plt
import pandas as pd
import pylab as pl
import numpy as np
import tensorflow as tf
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (20, 6)
df1 = pd.read_csv("TrainData.csv")
df2 = pd.read_csv("TestData.csv")
train_data_X = np.asanyarray(df1['ENGINE SIZE'])
train_data_Y = np.asanyarray(df1['CO2 EMISSIONS'])
test_data_X = np.asanyarray(df2['ENGINE SIZE'])
test_data_Y = np.asanyarray(df2['CO2 EMISSIONS'])
W = tf.Variable(20.0, name= 'Weight')
b = tf.Variable(30.0, name= 'Bias')
X = tf.placeholder(tf.float32, name= 'Input')
Y = tf.placeholder(tf.float32, name= 'Output')
Y = W*X + b
loss = tf.reduce_mean(tf.square(Y - train_data_Y))
optimizer = tf.train.GradientDescentOptimizer(0.05)
train = optimizer.minimize(loss)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
loss_values = []
train_data = []
for step in range(100):
_, loss_val, a_val, b_val = sess.run([train, loss, W, b], feed_dict={X:train_data_X, Y:train_data_Y})
loss_values.append(loss_val)
if step % 5 == 0:
print(step, loss_val, a_val, b_val)
train_data.append([a_val, b_val])
plt.plot(loss_values, 'ro')
plt.show()
I am trying to make a linear regression model to detect CO2 emission by giving size of engine as input. I am using the above code in tensorflow.
1) When I use this code Weight and Bias remains unchanged. What is the problem in code?
2) Also if I want engine size and milage both as inputs. what code changes should be made
Thanks in advance
There were few mistakes in the code which are mentioned below :
You were using placeholder Y = W*X + b, which in later section of the code was used to feed data (feed_dict={X:train_data_X, Y:train_data_Y}). You should have used another variable for prediction (not the placeholder which you were using to feed data) and then you should have been able to calculate loss function. However, required changes have been made. Check prediction= W*X + b in the below code
You were passing complete data in feed_dict at once (feed_dict={X:train_data_X, Y:train_data_Y}). However, you need to pass single data value at a time (feed_dict={X:x, Y:y})
Below code with the needful correction should be working fine.
import tensorflow as tf
import matplotlib.pyplot as plt
import pandas as pd
import pylab as pl
import numpy as np
import tensorflow as tf
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (20, 6)
df1 = pd.read_csv("TrainData.csv")
df2 = pd.read_csv("TestData.csv")
train_data_X = np.asanyarray(df1['ENGINE SIZE'])
train_data_Y = np.asanyarray(df1['CO2 EMISSIONS'])
test_data_X = np.asanyarray(df2['ENGINE SIZE'])
test_data_Y = np.asanyarray(df2['CO2 EMISSIONS'])
W = tf.Variable(20.0, name= 'Weight')
b = tf.Variable(30.0, name= 'Bias')
X = tf.placeholder(tf.float32, name= 'Input')
Y = tf.placeholder(tf.float32, name= 'Output')
prediction= W*X + b
loss = tf.reduce_mean(tf.square(prediction - Y))
optimizer = tf.train.GradientDescentOptimizer(0.05)
train = optimizer.minimize(loss)
loss_values = []
train_data = []
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
for step in range(100):
for (x,y) in zip(train_data_X,train_data_Y):
_, loss_val, a_val, b_val = sess.run([train, loss, W, b], feed_dict={X:x, Y:y})
loss_values.append(loss_val)
if step % 5 == 0:
print(step, loss_val, a_val, b_val)
train_data.append([a_val, b_val])
plt.plot(loss_values, 'ro')
plt.show()
Note : Because of incorrect choice of loss function, your loss keeps on increasing with every step.
I have mentioned a loss function below, which might work for your data. I am not sure how your data looks like, but you can give this a try if you want and let me know if this worked.
n_samples = train_data_X.shape[0]
loss = tf.reduce_sum(tf.pow(prediction - Y, 2)) / (2 * n_samples)
Response to your second query.
Assuming that your data has column name as MILEAGE, You can perform the below changes in train_data_X and test_data_X. Rest of the code will remain the same as above.
train_data_X = np.asanyarray(df1[['ENGINE SIZE','MILEAGE']])
train_data_Y = np.asanyarray(df1['CO2 EMISSIONS'])
test_data_X = np.asanyarray(df2[['ENGINE SIZE','MILEAGE']])
test_data_Y = np.asanyarray(df2['CO2 EMISSIONS'])
I am trying to implement linear regression using tensor-flow. Following is the code I am using.
import tensorflow as tf
import numpy as np
import pandas as pd
import os
rng = np.random
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
# reading data from a csv file
file1 = pd.read_csv('out.csv')
x_data=file1['^GSPC']
# converting datafram into array
x_data=x_data.values
y_data=file1['FB']
#converting dataframe into array
y_data=y_data.values
n_steps = 1000 #Total number of steps
n_iterations = [] #Nth iteration value
n_loss = [] #Loss at nth iteration
learned_weight = [] #weight at nth iteration
learned_bias = [] #bias value at nth iteration
# Try to find values for W and b that compute y_data = W * x_data + b
W = tf.Variable(rng.randn())
b = tf.Variable(rng.rand())
y = W * x_data + b
# Minimize the mean squared errors.
loss=tf.reduce_sum(tf.pow(y-y_data, 2))/(2*28)
optimizer = tf.train.GradientDescentOptimizer(0.01)
train = optimizer.minimize(loss)
with tf.Session() as sess:
# Before starting, initialize the variables. We will 'run' this first.
sess.run(tf.global_variables_initializer())
for step in range(n_steps):
sess.run(train)
n_iterations.append(step)
n_loss.append(loss.eval())
learned_weight.append(W.eval())
learned_bias.append(b.eval())
print("Final Weight: "+str(learned_weight[-1])+", Final Bias: "+str(learned_bias[-1]) + ", Final cost:"+str(n_loss[-1]))
The problem is every time I run the code I get different result (weights, bias and cost(loss)). I have studied from a few resources that weights, bias and cost should be approximately same in every run.
Secondly, the line i.e ( y=weights*x_data+bias) does not quite fit the training data.
Thirdly, I have to convert dataframe x_data and y_data to array by implementing the following
x_data=x_data.values
y_data=y_data.values
if I don’t do as shown above my code run the following error:
Traceback (most recent call last): File "python", line 33, in File "tensorflow/python/framework/fast_tensor_util.pyx", line 120, in tensorflow.python.framework.fast_tensor_util.AppendObjectArrayToTensorProto TypeError: Expected binary or unicode string, got tf.Tensor 'sub:0' shape=(28,) dtype=float32
Please help me understanding what I am doing wrong!
P.S: My questions may sound stupid because I am new to tensor flow and machine learning.
The code is implemented wrongly:
Use tf.Placeholders for data that will be passed into the model.
Use the feed_dict attribute of sess.run to pass data to the placeholder when executing the graph.
Here's an updated example:
Build the Graph
import numpy as np
import tensorflow as tf
import numpy as np
# dataset
X_data = np.random.randn(100,3)
y_data = 2*np.sum(X_data, 1)+0.01
# reshape y to be a column vector
y_data = np.reshape(y_data, [-1, 1])
# parameters
n_steps = 1000 #Total number of steps
batch_size = 20
input_length = X_data.shape[0] # => 100
display_cost = 500
# data placeholders
X = tf.placeholder(shape=[None, 3],dtype = tf.float32)
y = tf.placeholder(shape=[None, 1],dtype = tf.float32)
# build the model
W = tf.Variable(initial_value = tf.random_normal([3,1]))
b = tf.Variable(np.random.rand())
y_fitted = tf.add(tf.matmul(X, W), b)
# Minimize the mean squared errors
loss=tf.losses.mean_squared_error(labels=y, predictions=y_fitted)
optimizer = tf.train.GradientDescentOptimizer(0.01).minimize(loss)
Execute in Session
# execute in Session
with tf.Session() as sess:
# initialize all variables
tf.global_variables_initializer().run()
# Train the model
for steps in range(n_steps):
mini_batch = zip(range(0, input_length, batch_size),
range(batch_size, input_length+1, batch_size))
# train data in mini-batches
for (start, end) in mini_batch:
sess.run(optimizer, feed_dict = {X: X_data[start:end],
y: y_data[start:end]})
# print training performance
if (steps+1) % display_cost == 0:
print('Step: {}'.format((steps+1)))
# evaluate loss function
cost = sess.run(loss, feed_dict = {X: X_data,
y: y_data})
print('Cost: {}'.format(cost))
# report rmse for training and test data
print('\nFinal Weight: {}'.format(W.eval()))
print('\nFinal Bias: {}'.format(b.eval()))
Output for 2 runs
# Run 1
Step: 500
Cost: 3.1569701713918263e-11
Step: 1000
Cost: 3.1569701713918263e-11
Final Weight: [[2.0000048]
[2.0000024]
[1.9999973]]
Final Bias: 0.010000854730606079
# Run 2
Step: 500
Cost: 7.017615221566187e-12
Step: 1000
Cost: 7.017615221566187e-12
Final Weight: [[1.9999975]
[1.9999989]
[1.9999999]]
Final Bias: 0.0099998963996768
Indeed, the weight and bias are approximately the same for multiple calls to build a classifier using the same dataset. Also when doing numerical computations, Numpy ndarrays are mostly the preferred data format hence the conversion using .values.
I am new to tensorflow, and trying to write loss function(squared loss) using basic python operators, but it is not working. Can anyone tell me where I went wrong. Thanks in adavnce
n = x_data.shape[0]
L = (Y_pred-y)**2
loss = (1/n)*tf.reduce_sum(L)
I get loss=0.0 when I run the corresponding session
_ ,_m, _c, _l = session.run([optimizer,m,c,loss], feed_dict={x: x_data, y: y_data})
y is a placeholder
loss = tf.reduce_mean(tf.squared_difference(Y_pred,y))
this works just fine?
Complete Code:
import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
#downloading dataset
!wget -nv -O /resources/data/PierceCricketData.csv https://ibm.box.com/shared/static/reyjo1hk43m2x79nreywwfwcdd5yi8zu.csv
df = pd.read_csv("/resources/data/PierceCricketData.csv")
df.head()
%matplotlib inline
x_data, y_data = (df["Chirps"].values,df["Temp"].values)
plt.plot(x_data, y_data, 'ro')
# label the axis
plt.xlabel("# Chirps per 15 sec")
plt.ylabel("Temp in Farenhiet")
x = tf.placeholder(tf.float32, shape=x_data.shape)
y = tf.placeholder(tf.float32, shape=y_data.shape)
m = tf.Variable(3.0, name='m')
c = tf.Variable(2.0, name='c')
Y_pred = m*x+c
n = x_data.shape[0]
L = (Y_pred*nf-y*nf)**2
loss = (1/n)*tf.reduce_sum(L)
# loss = tf.reduce_mean(tf.squared_difference(Y_pred,y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01).minimize(loss)
session = tf.Session()
session.run(tf.global_variables_initializer())
convergenceTolerance = 0.0001
previous_m = np.inf
previous_c = np.inf
steps = {}
steps['m'] = []
steps['c'] = []
losses=[]
for k in range(100000):
_ ,_m, _c, _l = session.run([optimizer,m,c,loss], feed_dict={x: x_data, y: y_data})
steps['m'].append(_m)
steps['c'].append(_c)
losses.append(_l)
if (np.abs(previous_m - _m) <= convergenceTolerance) or (np.abs(previous_c - _c) <= convergenceTolerance):
print "Finished by Convergence Criterion"
print k
print _l
break
previous_m = _m,
previous_c = _c,
print(losses)
Output I get is [0.0, 0.0]
Why?
Here is the official TensorFlow implementation of mean_squared_error :
from tensorflow.python.framework import ops, math_ops
#tf_export("losses.mean_squared_error")
def mean_squared_error(labels, predictions, weights=1.0, scope=None,
loss_collection=ops.GraphKeys.LOSSES,
reduction=Reduction.SUM_BY_NONZERO_WEIGHTS):
if labels is None:
raise ValueError("labels must not be None.")
if predictions is None:
raise ValueError("predictions must not be None.")
with ops.name_scope(scope, "mean_squared_error",(predictions, labels, weights)) as scope:
predictions = math_ops.to_float(predictions)
labels = math_ops.to_float(labels)
predictions.get_shape().assert_is_compatible_with(labels.get_shape())
losses = math_ops.squared_difference(predictions, labels)
return compute_weighted_loss(losses, weights, scope, loss_collection, reduction=reduction)
As you can see in their source code you should make sure that the tensors have the same dtype. Hope that answers your question.
import numpy as np
import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
#reproducible random seed
seed = 1
np.random.seed(seed)
#Import and normalize the data
df = pd.read_csv('creditcard.csv')
#Exploring the data
# print df.head()
# print df.describe()
# print df.isnull().sum()
# count_class = pd.value_counts(df['Class'])
# count_class.plot(kind = 'bar')
# plt.title('Fraud class histogram')
# plt.xlabel('class')
# plt.ylabel('Frequency')
# plt.show()
# print('Clearly the data is totally unbalanced!')
#to normalize the amount column
# data['normAmount'] = StandardScaler().fit_transform(data['Amount'].reshape(-1, 1))
df['normAmount'] = StandardScaler().fit_transform(df['Amount'].values.reshape(-1, 1))
df = df.drop(['Time','V28','V27','V26','V25','V24','V23','V22','V20','V15','V13','V8','Amount'], axis =1)
X = df.iloc[:,df.columns!='Class']
Y = df.iloc[:,df.columns=='Class']
# number of records in the minority class
number_record_fraud = len(df[df.Class==1])
fraud_indices = np.array(df[df.Class==1].index)
#picking normal class
normal_indices = np.array(df[df.Class==0].index)
#select random x(number_record_fraud) numbers from normal_indices
random_normal_indices = np.random.choice(normal_indices,number_record_fraud,replace=False)
random_normal_indices = np.array(random_normal_indices)
#under sample data
under_sample_indices = np.concatenate([fraud_indices,random_normal_indices])
under_sample_data = df.iloc[under_sample_indices,:]
X_undersample = under_sample_data.iloc[:,under_sample_data.columns!='Class']
Y_undersample = under_sample_data.iloc[:,under_sample_data.columns=='Class']
# split data into train and test dataset
X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size = 0.3)
X_train_undersample,X_test_undersample,Y_train_undersample,Y_test_undersample = train_test_split(X_undersample,Y_undersample,test_size=0.3)
#parameters
learning_rate = 0.05
training_epoch = 10
batch_size = 43
display_step = 1
#tf graph input
x = tf.placeholder(tf.float32,[None,18])
y = tf.placeholder(tf.float32,[None,1])
#set model weights
w = tf.Variable(tf.zeros([18,1]))
b = tf.Variable(tf.zeros([1]))
#construct model
pred = tf.nn.softmax(tf.matmul(x,w) + b) #softmax activation
#minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred),reduction_indices=1))
#Gradient descent
optimizer = tf.train.AdamOptimizer(learning_rate).minimize(cost)
#initializing variables
init = tf.global_variables_initializer()
#launch the graph
with tf.Session() as sess:
sess.run(init)
#training cycle
for epoch in range(training_epoch):
total_batch = len(X_train_undersample)/batch_size
avg_cost = 0
#loop over all the batches
for batch in range(total_batch):
batch_xs = X_train.iloc[(batch)*batch_size:(batch+1) *batch_size]
batch_ys = Y_train.iloc[(batch)*batch_size:(batch+1) *batch_size]
# run optimizer and cost operation
_,c= sess.run([optimizer,cost],feed_dict={x:batch_xs,y:batch_ys})
avg_cost += c/total_batch
correct_prediction = tf.equal(tf.argmax(pred,1),tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
#disply log per epoch step
if (epoch+1) % display_step == 0:
train_accuracy, newCost = sess.run([accuracy, cost], feed_dict={x: X_test,y: Y_test})
print "test_set_accuracy:",accuracy.eval({x:X_test_undersample,y:Y_test_undersample})*100
print "whole_set_accuracy:",accuracy.eval({x:X,y:Y})*100
# print train_accuracy
# print "cost",newCost
print
print 'optimization finished.'
Things I've tried to figure out what's causing it:
Tried changing train dataset length.
Dropped some not needed fields.
Tried putting validation blocks.
Dataset :link
There can be multiple reasons of why it is overfitting , and as well there can be multiple ways to debug it and to fix it. Its hard to tell just from the code, because it also depends on the data, but here are some common reaons as well as fixes:
Too small dataset, adding more data its a common overfitting fix
Too complex model, if you have many features, or complex polonomial features, try to reducing complexity using feature selection
Add regularization: i dont see regularization in your code, try to add it.
I'm trying to fit an exponentially decaying model (y=Ax^b + C)to some data but have yet to get a value other than 0 for for b. I have two "working" sets of code right now, one steps through each X,Y pair, and the other attempts to use the entire [X,Y] array, but I'm not sure that I have implemented that correctly. For now I'd like for it to correctly fit a curve. The linear model works fine so I'm not sure where this is going south.
Data is here - PASTEBIN
#!/usr/bin/python
import numpy as np
import tensorflow as tf
import sys
import matplotlib.pyplot as plt
k=0
xdata= []
ydata = []
# Open the data and read it in, ignore the header.
with open('curvedata_full_formatted.csv') as f:
for line in f:
k+=1
if k==1:continue
items = line.split(',')
xdata.append(float(items[0]))
ydata.append(float(items[1]))
# Model linear regression y = A*x^B+C
# x - data to be fed into the model - 1 feature
x = tf.placeholder(tf.float32, [None, 1])
# A - training variable - 1 feature, 1 output
A = tf.Variable(tf.zeros([1,1]))
# B - training variable - 1 output
B = tf.Variable(tf.zeros([1,1]))
# C - training variable - 1 output
C = tf.Variable(tf.zeros([1]))
# x^B
xb = tf.exp(B)
# A*x^b
product = tf.mul(A,xb)
# Prediction
y = tf.add(product,C)
# Actual value ybar
y_ = tf.placeholder(tf.float32)
# Cost function sum((y_-y)**2)
cost = tf.reduce_mean(tf.square(y_-y))
# Training using Gradient Descent to minimize cost
train_step = tf.train.GradientDescentOptimizer(1*10**-9).minimize(cost)
sess = tf.Session()
init = tf.initialize_all_variables()
sess.run(init)
steps = 150
for i in range(steps):
# Read in data from log file and use as x,y
for (X,Y) in zip(xdata,ydata):
#xs = np.array([[xdata]])
#ys = np.array([[ydata]])
# Train
# Feed dict x placeholder xs, y_ placeholder ys
X = np.array([[X]])
Y = np.array([[Y]])
feed = { x: X, y_: Y }
sess.run(train_step, feed_dict=feed)
sys.stdout.write("\rIteration %i " %i +"cost %.15f" % sess.run(cost, feed_dict=feed))
sys.stdout.flush()
print ''
print 'A: %f'%sess.run(A)
print 'B: %f'%sess.run(B)
print 'C: %f'%sess.run(C)
As a test, try starting the optimizer with initial values close to the expected final parameters. This test will tell you whether or not the problem is in the selection of initial parameter values.