I'm trying to implement a sort of protected division using Tensorflow.where but somehow it seems to be skipping the condition set on the where statement.
The main idea is, when dividing x/y , if y == 0. then the result of the division of be x instead of throwing and error.
My code is as follows:
def Pdivide(x,y):
result = tf.where(y == 0., x, x/y)
return result
But somehow that condition is being skipped:
>>> a = tf.Variable([1.7, 0.2, 0., 1.1, 0.9, 0.3, 23., -1.])
>>> b = tf.Variable([0., 0., 0., 1., 1., 0., 1., 1.])
>>>Pdivide(a,b)
>>>(inf, inf, nan, 1.1, 0.9, inf, 23, -1)
Intended output:
>>>(1.7, 0.2, 0., 1.1, 0.9, 0.3, 23, -1)
PS: Using eager execution.
Ok so the answer is pretty simple apparently.
For some reason the tensor elements cannot me compared with simple == but using tf.equal(y, 0.) solves the problem and produces the correct output.
Related
I want to run a function in a for loop. Firstly, I have a list of arrays and each arry includes some constants. These constants go to function. Then, I make function and finally import arrays stored as a list into the created function. At the moment it is only using the constants stored in the last array of the list storing constants. I want to create the first function using the first array of constants and run that function for the first array of inps. I have checked this solution but I could not solve my issue.
constants=[np.array([2., 2., 2., 2.]),
np.array([20., 40., 30., 10.])]
inps=[np.array([[1., 1., 1.],[2., 2., 2.]]),
np.array([[3., 3., 3.],[4., 4., 4.]])]
This is my code:
def fun(row=i):
row[0] = a * row[0]
for i in constants:
i=i.tolist()
a = i[0]
return row[0], row[1], row[2]
out=[]
for j in inps:
j=[j]
new = np.asarray(list(map(fun, [x for point in j for x in point])))
out.append(new)
Then, I want to get:
out= [np.array([[2., 1., 1.],
[4., 2., 2.]]),
np.array([[60., 3. , 3. ],
[80., 4. , 4. ]])]
Simply, I want to multiply first value of the first array of constants to first column of first array of inps and replace it with the result. Then, I want to multiply the second of constants tothe second array of inps and so on.
But my code is creating only one function and performs the last function created by the cnstants coming from constants[lasti] for all the arrays of inps.
It is giving me the following result:
[array([[40., 1., 1.],
[80., 2., 2.]]),
array([[120., 3., 3.],
[160., 4., 4.]])]
In advance, I appreciate any help.
Not sure you need the function at all. This produces the output you are looking for:
import numpy as np
constants = [
np.array([2.0, 2.0, 2.0, 2.0]),
np.array([20.0, 40.0, 30.0, 10.0])]
inps = [
np.array([[1.0, 1.0, 1.0], [2.0, 2.0, 2.0]]),
np.array([[3.0, 3.0, 3.0], [4.0, 4.0, 4.0]]),
]
for index, inp in enumerate(inps):
inp[:,0] *= constants[index][0]
print(inps)
Output:
[array([[2., 1., 1.],
[4., 2., 2.]]),
array([[60., 3., 3.],
[80., 4., 4.]])]
I have compared many Quadratic Programming(QP) solvers like cvxopt, qpoases and osqp and found that osqp works faster and better for my application.
Now, I want to minimize an indefinite quadratic function with both equality and inequality constraints that may get violated depending on various factors. So I want to use l1 penalty method that penalizes the violating constraints.
for example,
I have modified an example, to violate the constraints.
import osqp
import scipy.sparse as sparse
import numpy as np
# Define problem data
P = sparse.csc_matrix([[4., 1.], [1., 2.]])
q = np.array([1., 1.])
A = sparse.csc_matrix([[1., 0.], [0., 1.], [1., 0.], [0., 1.]])
l = np.array([0., 0., 0.2, 1.1])
u = np.array([1., 1., 0.2, 1.1])
# Create an OSQP object
prob = osqp.OSQP()
# Setup workspace and change alpha parameter
prob.setup(P, q, A, l, u, alpha=1.0)
# Solve problem
res = prob.solve()
print res.x
Obviously, this is an infeasible problem, so we need to change the objective function to penalize the error.
So, I need help to formulate this problem that can be solved using osqp's python interface.
Or, please let me know if there is any other python interface available to solve this kind of constraint violation problems.
In general abs functions can be dangerous (they are non-differentiable). A standard way to deal with this is to add slacks. E.g.
g(x) <= 0
becomes
g(x) <= s
s >= 0
Now add a term mu*s to the objective.
For
h(x) = 0
one could do
h(x) = s1 - s2
s1, s2 >= 0
and add mu*(s1+s2) to the objective.
As usual: this is just one approach (there are other formulations).
I had the same problem and this question helped a lot. This is how I solved it in OSQP interface.
I redefined example to be:
# Define problem data
P = sparse.csc_matrix([[4., 1.], [1., 2.]])
q = np.array([1., 1.])
A = sparse.csc_matrix([[1., 0.], [0., 1.], [1., 1.]])
l = np.array([0., 0., 3])
u = np.array([1., 1., 3])
Here first and second variable are constrained to be at most 1. But their sum should equal 3. This makes this problem unfeasible.
Now let's transform inequality constraints as Erwin suggested by adding two slack variables.
# Redefine problem data with 2 slack variableы
# Added quadratic penalties to variables s1 and s2 with penalty coefficient == 1
P = sparse.csc_matrix([[4., 1., 0., 0.], [1., 2., 0., 0.], [0., 0., 1., 0.], [0., 0., 0., 1.]])
# Zero linear penalties for s1 and s2.
q = np.array([1., 1., 0., 0.])
# First constraint is x1 <= s1, second is s1 >= 0.
# Third constraint is x2 <= s2, fourth is s2 >= 0.
A = sparse.csc_matrix([[1., 0., -1., 0.], [0., 0., 1., 0.], [0., 1., 0., -1.], [0., 0., 0., 1.], [1., 1., 0., 0.]])
l = np.array([-np.inf, 0., -np.inf, 0., 3])
u = np.array([0., np.inf, 0., np.inf, 3])
When I run solver, problem has a solution and is softly penalised for exceeding upper bounds.
iter objective pri res dua res rho time
1 -4.9403e-03 3.00e+00 5.99e+02 1.00e-01 8.31e-04s
50 1.3500e+01 1.67e-07 7.91e-08 9.96e-01 8.71e-04s
status: solved
number of iterations: 50
optimal objective: 13.5000
run time: 8.93e-04s
optimal rho estimate: 1.45e+00
[1.00 2.00 1.00 2.00]
Hope this helps somebody.
I started working with numba today, mainly because I have a nested for-loop that can take quite a while with regular python code.
I have a macports version of python-2.7 with llvm-3.6 and the pip version of numba (everything is up-to-date)
Here is the code I'm using:
import pandas as pd
from numba import jit
from numpy import nan, full
#jit
def movingAverage(adj_close, maxMA):
ma = full([len(adj_close), maxMA], nan, dtype=float64)
ind = range( 1, len(adj_close)+1 )
for d in ind:
m = max( 0, d-maxMA-1)
adj = adj_close[d-1:m:-1] if (m or d==maxMA+1) else adj_close[d-1::-1]
cs = adj.cumsum()
for i in range( len(adj) ):
ma[d-1][i] = ( cs[i] / (i+1) )
print ma
return ma
I'm calculating a rolling mean for the input adj_close for up to maxMA days.
adj_close is a array of values, one value per day
I started by creating ma, a holder for the values that are going to be calculated. And work out the vaules for each day individually (note that the first day can only have an average involving 1 day, the second, 2 and so on up to the maxMA)
If I input something like adj_close = array(range(5), dtype=float64) and maxMA = 3 get the right answer as follows:
array([[ 0., nan, nan],
[ 1., 0.5, nan],
[ 2., 1.5, 1.],
[ 3., 2.5, 2.],
[ 4., 3.5, 3.]])
However, If I take out the print ma line, just before the return of my function, it returns only part of the answer:
array([[ nan, nan, nan],
[ nan, nan, nan],
[ nan, nan, nan],
[ 3., 2.5, 2.],
[ 4., 3.5, 3.]])
Why is that happening? Why does #jit needs the print between those loops to get the answer right? What can I do to get rid of the print statement (that greatly increases the runtime)?
Edit: I'm accepting #JoshAdel suggestion and opened a issue at Numba's github. I'm, therefore, accepting #MSeifert answer as the workaround solved the problem for me.
I think numba does something strange here but probably because of the mixture of python and nopython mode. If I use Python 3.5 the returns are identical with and without print.
For python 2.7 I think the problem is because the for-loop is either compiled in nopython mode (without print) or in python mode (with print). But then converted to python when it exits the loop. But that's just guessing. But I tried it with:
import pandas as pd
from numba import jit
from numpy import nan, full
import numpy as np
#jit
def movingAverage(adj_close, maxMA):
ma = full([len(adj_close), maxMA], nan, dtype=np.float64)
ind = range( 1, len(adj_close)+1 )
for d in ind:
m = max( 0, d-maxMA-1)
adj = adj_close[d-1:m:-1] if (m or d==maxMA+1) else adj_close[d-1::-1]
cs = adj.cumsum()
for i in range( len(adj) ):
ma[d-1][i] = ( cs[i] / (i+1) )
if d == ind[-1]:
return ma # notice that I return it after the last loop but before the loop terminates.
#return ma
and it does return:
array([[ 0., nan, nan],
[ 1., 0.5, nan],
[ 2., 1.5, 1.],
[ 3., 2.5, 2.],
[ 4., 3.5, 3.]])
This is however not a very effient way because of the recalculation of len(adj_close)+1. This could be stored somewhere.
I am very new to Python... and I am having a hard time plugging the contents of my 1d array into a nonlinear equation so I can ultimately plot the results. My code is below:
import numpy as np
import matplotlib.pyplot as plt
def readfiles(file_list):
""" read <TAB> delemited files as strings
ignoring '# Comment' lines """
data = []
for fname in file_list:
data.append(
np.genfromtxt(fname,
comments='#', # skip comment lines
delimiter='\t',
dtype ="|S", autostrip=True).T)
return data
data = readfiles(['CR1000_rawMeasurements_15m.txt'])
def column(matrix, i):
return [row[i] for row in matrix]
x = column(data,18)
for i in x:
thermTemp1_degC = 1/(1.401E-3 + 2.377E-4*np.log(i) + 9.730E-8*np.log(i)**3)-273.15
All I have been successfully able to do is extract the column I need from my data. When I run this script, I get 'TypeError: Not implemented for this type.' (my 1d array, x, is just a column of zeros right now.) How can I fix this?
There are a few points to address here.
Returning the Correct Column
The array you've given in the comments is a little strange, but you can retrieve the columns with numpy:
data = [[ 737055., 0.], [ 737055., 0.], [ 737055., 0.], [ 737055., 0.], [ 737055., 0.], [ 735773., 0.], [ 735773., 0.], [ 735773., 0.]]]
data
=> [[[737055.0, 0.0],
[737055.0, 0.0],
[737055.0, 0.0],
[737055.0, 0.0],
[737055.0, 0.0],
[735773.0, 0.0],
[735773.0, 0.0],
[735773.0, 0.0]]]
column_0 = np.array(data)[0][:, 0]
column_1 = np.array(data)[0][:, 1]
column_0
=> array([ 737055., 737055., 737055., 737055., 737055., 735773.,
735773., 735773.])
column_1
=> array([ 0., 0., 0., 0., 0., 0., 0., 0.])
Performing the Calculation
As x is a numpy array (if you use the above column code) you don't need to put this in a for loop:
thermTemp1_degC = 1/(1.401E-3 + 2.377E-4*np.log(i) + 9.730E-8*np.log(i)**3)-273.15
Here thermTemp1_degC is a numpy array the same size as x.
I don't understand why the following code behaves the way it does.
import numpy as np
nbr_arrays = 4
nbr_fields_per_array = 3
nbr_subfields_per_field = 2
# pre-allocate zeros list
zeros = np.zeros(nbr_subfields_per_field)
data = []
for array in range(nbr_arrays):
# pre-allocate the subarray
empty_array = []
for empty_array_index in range(nbr_fields_per_array):
empty_array.append(zeros)
# append pre subarray to data
data.append(empty_array)
# fill up data
for j in range(nbr_fields_per_array):
for k in range(nbr_subfields_per_field):
data[array][j][k] = j*k*array
The generated output data reads now:
[[array([ 0., 6.]), array([ 0., 6.]), array([ 0., 6.])],
[array([ 0., 6.]), array([ 0., 6.]), array([ 0., 6.])],
[array([ 0., 6.]), array([ 0., 6.]), array([ 0., 6.])],
[array([ 0., 6.]), array([ 0., 6.]), array([ 0., 6.])]]
Even zeros reads completely differently:
array([ 0., 6.])
If I look at the identify of the different lists, this is what I get:
id(data[0][0])
Out[72]: 45790208
id(data[1][0])
Out[66]: 45790208
id(data[2][0])
Out[67]: 45790208
id(data[3][0])
Out[68]: 45790208
id(zeros)
Out[69]: 45790208
why are all the references the same? and why does zero suddenly contain non-zero values?
I'd really appreciate it if somebody could explain me what exactly is happening here, and how I have to modify my code to see the expected behaviour (output).
EDIT:
not using zeros but using [[0]*nbr_subfields_per_field for x in range(nbr_fields_per_array)] instead gives me the expected result. but why? why doesn't the original code work?
Modified code that works:
data = []
for array in range(nbr_arrays):
empty_array = [[0]*nbr_subfields_per_field for x in range(nbr_fields_per_array)]
''' this is causing the weird behaviour
empty_array = []
for empty_array_index in range(nbr_fields_per_array):
empty_array.append(zeros)
'''
data.append(empty_array)
for j in range(nbr_fields_per_array):
for k in range(nbr_subfields_per_field):
data[array][j][k] = j*k*array
# pre-allocate zeros list
zeros = np.zeros(nbr_subfields_per_field)
This creates a single object.
for empty_array_index in range(nbr_fields_per_array):
empty_array.append(zeros)
This keeps appending the same object.
Stop pre-allocating.
Numpy can set up multidimensional arrays for you, if you want. Since you're going to initialize the whole array immediately after creating it, the empty method seems like the most appropriate:
data = np.empty((nbr_arrays, nbr_fields_per_array, nbr_subfields_per_field))