We Keep Coding

Neural Network increases loss instead of decreasing it

I used two online references to construct a neural network in python with four input nodes, a layer of 4 hidden nodes, and 6 output nodes. When I run the network, the loss increases rather than decreasing which I believe means its predictions are getting worse.
Sorry for the ocean of code, I have no idea where in the code the issue could be. Nothing that I did has been able to fix this. Is there something wrong with my code, or is my assumption about the loss function wrong?
import numpy as np
#defining inputs and real outputs
inputData = np.array([[10.0, 5.0, 15.0, 3.0],
[9.0, 6.0, 16.0, 4.0],
[8.0, 4.0, 17.0, 5.0],
[7.0, 3.0, 18.0, 6.0],
[6.0, 2.0, 19.0, 7.0]])
statsReal = np.array([[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
[0.0, 0.2, 0.4, 0.6, 0.8, 1.0]])
def sigmoid(x):
return 1/(1 + np.exp(-x))
def sigmoid_d_dx(x):
return sigmoid(x) * (1 - sigmoid(x))
def softmax(A):
expA = np.exp(A)
return expA / expA.sum(axis=1, keepdims=True)
#defining the hidden and output nodes, and the weights and biases for hidden and output layers
instances = inputData.shape[0]
attributes = inputData.shape[1]
hidden_nodes = 4
output_nodes = 6
wh = np.random.rand(attributes,hidden_nodes)
bh = np.random.randn(hidden_nodes)
wo = np.random.rand(hidden_nodes,output_nodes)
bo = np.random.randn(output_nodes)
learningRate = 10e-4
error_cost = []
for epoch in range(100):
#Feedforward Phase 1
zh = np.dot(inputData, wh) + bh
ah = sigmoid(zh)
#Feedforward Phase 2
zo = np.dot(ah, wo) + bo
ao = softmax(zo)
#Backpropogation Phase 1
dcost_dzo = ao - statsReal
dzo_dwo = ah
dcost_wo = np.dot(dzo_dwo.T, dcost_dzo)
dcost_bo = dcost_dzo
#Backpropogation Phase 2
dzo_dah = wo
dcost_dah = np.dot(dcost_dzo, dzo_dah.T)
dah_dzh = sigmoid_d_dx(zh)
dzh_dwh = inputData
dcost_wh = np.dot(dzh_dwh.T, dah_dzh * dcost_dah)
dcost_bh = dcost_dah*dah_dzh
#Weight Updates
wh -= learningRate * dcost_wh
bh -= learningRate * dcost_bh.sum(axis=0)
wo -= learningRate * dcost_wo
bo -= learningRate * dcost_bo.sum(axis=0)
loss = np.sum(-statsReal * np.log(ao))
print(loss)
error_cost.append(loss)
print(error_cost)```

Your network is learning when you train with reasonable data.
Try this data for example. I added one distinct case for every class and one hot encoded the targets. I scaled the inputs to [0.0, 1.0]
inputData = np.array([[1.0, 0.5, 0.0, 0.0],
[0.0, 1.0, 0.5, 0.0],
[1.0, 0.0, 0.0, 1.0],
[0.0, 1.0, 0.0, 0.5],
[0.0, 0.0, 0.0, 1.0],
[1.0, 1.0, 0.5, 0.0]])
statsReal = np.array([[1.0, 0.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 1.0, 0.0, 0.0, 0.0, 0.0],
[0.0, 0.0, 1.0, 0.0, 0.0, 0.0],
[0.0, 0.0, 0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 0.0, 0.0, 1.0, 0.0],
[0.0, 0.0, 0.0, 0.0, 0.0, 1.0]])
Increase the learning rate
learningRate = 10e-2
Train for more epochs and print a little bit less often.
for epoch in range(1000):
#....
if epoch % 100 == 99: print(loss)
Output of your loss function
6.116573523774877
2.6901680150532847
1.323221228926058
0.7688474199923144
0.5186915091033664
0.38432651801528794
0.3024486736712547
0.24799685736356275
0.20944414625474833
0.1808455098847857

How to pass argument of type char ** from Python to C API [duplicate]

As seen here How do I convert a Python list into a C array by using ctypes? this code will take a python array and transform it to a C array.
import ctypes
arr = (ctypes.c_int * len(pyarr))(*pyarr)
Which would the way of doing the same with a list of lists or a lists of lists of lists?
For example, for the following variable
list3d = [[[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]], [[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]], [[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]]]
I have tried the following with no luck:
([[ctypes.c_double * 4] *2]*3)(*list3d)
# *** TypeError: 'list' object is not callable
(ctypes.c_double * 4 *2 *3)(*list3d)
# *** TypeError: expected c_double_Array_4_Array_2 instance, got list
Thank you!
EDIT: Just to clarify, I am trying to get one object that contains the whole multidimensional array, not a list of objects. This object's reference will be an input to a C DLL that expects a 3D array.

It works with tuples if you don't mind doing a bit of conversion first:
from ctypes import *
list3d = [
[[0.0, 1.0, 2.0, 3.0], [4.0, 5.0, 6.0, 7.0]],
[[0.2, 1.2, 2.2, 3.2], [4.2, 5.2, 6.2, 7.2]],
[[0.4, 1.4, 2.4, 3.4], [4.4, 5.4, 6.4, 7.4]],
]
arr = (c_double * 4 * 2 * 3)(*(tuple(tuple(j) for j in i) for i in list3d))
Check that it's initialized correctly in row-major order:
>>> (c_double * 24).from_buffer(arr)[:]
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0,
0.2, 1.2, 2.2, 3.2, 4.2, 5.2, 6.2, 7.2,
0.4, 1.4, 2.4, 3.4, 4.4, 5.4, 6.4, 7.4]
Or you can create an empty array and initialize it using a loop. enumerate over the rows and columns of the list and assign the data to a slice:
arr = (c_double * 4 * 2 * 3)()
for i, row in enumerate(list3d):
for j, col in enumerate(row):
arr[i][j][:] = col

I made the change accordingly
a = [[[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]], [[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]], [[40.0, 1.2, 6.0, 0.3], [50.0, 4.2, 0, 0]]]
arr = (((ctypes.c_float * len(a[0][0])) * len(a[0])) * len(a))
arr_instance=arr()
for i in range(0,len(a)):
for j in range(0,len(a[0])):
for k in range(0,len(a[0][0])):
arr_instance[i][j][k]=a[i][j][k]
The arr_instance is what you want.

List denies to be appended

Below is a portion of python code which is meant to work with FreeCAD libraries. I can provide the full code if requested.
What's weird in this code is that appending to a list, mr_fus.References, does no effect on the size of the list. I also tried to append to a dummy list, temp, and its size returned as expected.
Here is the definition of References.
Although type() indicates that References is a list, to me, it looks like no ordinary list. I am curious if it is possible for a list to deny to add element to itself.
temp = [] # just for comparison
for i in range(1,len(App.ActiveDocument.Shape004.Shape.Faces)):
mr_fus.References.append((App.ActiveDocument.Shape004.Shape, App.ActiveDocument.Shape004.Shape.Faces[i]))
temp.append((App.ActiveDocument.Shape004.Shape, App.ActiveDocument.Shape004.Shape.Faces[i]))
print(type(mr_fus.References)) # <class 'list'>
print(len(mr_fus.References)) # 0 # why??
print(len(temp)) # 18
EDIT: Here is a reproducible example though not minimal.
import sys
import math
import numpy
sys.path.append('/usr/lib/freecad/lib/')
import FreeCAD
import Draft
import Part
import BOPTools.JoinFeatures
doc = FreeCAD.newDocument('newdoc')
ZERO = 1e-10
def U(c, xl):
u = c[5]
if c[6] != 0:
t = c[0] + c[1] * abs((xl + c[2]) / c[3]) ** c[4]
if abs(t) <= ZERO:
t = 0
u += c[6] * t ** (1./c[7])
return u;
class Fuselage:
def H(xl):
if xl < 0.4:
c = [1.0, -1.0, -0.4, 0.4, 1.8, 0.0, 0.25, 1.8]
elif xl < 0.8:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 0.25, 0.0, 0.0]
elif xl < 1.9:
c = [1.0, -1.0, -0.8, 1.1, 1.5, 0.05, 0.2, 0.6]
elif xl < 2.0:
c = [1.0, -1.0, -1.9, 0.1, 2.0, 0.0, 0.05, 2.0]
return U(c, xl);
def W(xl):
if xl < 0.4:
c = [1.0, -1.0, -0.4, 0.4, 2.0, 0.0, 0.25, 2.0]
elif xl < 0.8:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 0.25, 0.0, 0.0]
elif xl < 1.9:
c = [1.0, -1.0, -0.8, 1.1, 1.5, 0.05, 0.2, 0.6]
elif xl < 2.0:
c = [1.0, -1.0, -1.9, 0.1, 2.0, 0.0, 0.05, 2.0]
return U(c, xl);
def Z(xl):
if xl < 0.4:
c = [1.0, -1.0, -0.4, 0.4, 1.8, -0.08, 0.08, 1.8]
elif xl < 0.8:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
elif xl < 1.9:
c = [1.0, -1.0, -0.8, 1.1, 1.5, 0.04, -0.04, 0.6]
elif xl < 2.0:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 0.04, 0.0, 0.0]
return U(c, xl);
def N(xl):
if xl < 0.4:
c = [2.0, 3.0, 0.0, 0.4, 1.0, 0.0, 1.0, 1.0]
elif xl < 0.8:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 0.0, 0.0]
elif xl < 1.9:
c = [5.0, -3.0, -0.8, 1.1, 1.0, 0.0, 1.0, 1.0]
elif xl < 2.0:
c = [2.0, 0.0, 0.0, 0.0, 0.0, 0.04, 1.0, 1.0]
return U(c, xl);
class Pylon:
def H(xl):
if xl < 0.8:
c = [1.0, -1.0, -0.8, 0.4, 3.0, 0.0, 0.145, 3.0]
elif xl < 1.018:
c = [1.0, -1.0, -0.8, 0.218, 2.0, 0.0, 0.145, 2.0]
return U(c, xl);
def W(xl):
if xl < 0.8:
c = [1.0, -1.0, -0.8, 0.4, 3.0, 0.0, 0.166, 3.0]
elif xl < 1.018:
c = [1.0, -1.0, -0.8, 0.218, 2.0, 0.0, 0.166, 2.0]
return U(c, xl);
def Z(xl):
if xl < 0.4:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 0.125, 0.0, 0.0]
elif xl < 1.018:
c = [1.0, -1.0, -0.8, 1.1, 1.5, 0.065, 0.06, 0.6]
return U(c, xl);
def N(xl):
if xl < 0.4:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 0.0, 0.0]
elif xl < 1.018:
c = [0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 0.0, 0.0]
return U(c, xl);
def yl(xl, phi, part):
return r(part.H(xl), part.W(xl), part.N(xl), phi) * math.sin(phi)
def zl(xl, phi, part):
return r(part.H(xl), part.W(xl), part.N(xl), phi) * math.cos(phi) + part.Z(xl)
def r(H, W, N, phi):
a = abs(0.5 * H * math.sin(phi)) ** N + abs(0.5 * W * math.cos(phi)) ** N
b = (0.25 * H * W) ** N
return (b / a) ** (1./N)
xm = numpy.linspace(0.00001, 2, 10)
xp = numpy.linspace(0.40001, 1.018, 10)
p = numpy.linspace(0, 2*math.pi, 10)
def makepart(part, x):
polygons = []
for i in range(len(x)-1):
points = []
for j in range(len(p)-1):
points.append(FreeCAD.Vector(x[i], yl(x[i], p[j], part), zl(x[i], p[j], part)))
points.append(points[0])
polygons.append(Part.makePolygon(points))
loft = Part.makeLoft(polygons)
cap1 = Part.Face(polygons[0])
cap2 = Part.Face(polygons[-1])
shell = Part.Shell(loft.Faces+[cap1, cap2])
Part.show(shell)
return shell
fuselage = makepart(Fuselage, xm)
pylon = makepart(Pylon, xp)
# join-connect fuselage and pylon
# let's call the joined object 'heli'
heli = BOPTools.JoinFeatures.makeConnect(name = 'Connected')
heli.Objects = [App.ActiveDocument.Shape, App.ActiveDocument.Shape001]
heli.Proxy.execute(heli)
heli.purgeTouched()
# make heli a solid
s = Part.Solid(Part.Shell(heli.Shape.Faces))
Part.show(s)
# make a sphere
sphere = Part.makeSphere(2,FreeCAD.Vector(1,0,0))
Part.show(sphere)
# cut heli from sphere
cut = sphere.cut(s)
Part.show(cut)
# extract seem of sphere
import CompoundTools.CompoundFilter
f = CompoundTools.CompoundFilter.makeCompoundFilter(name = 'CompoundFilter')
f.Base = App.ActiveDocument.Shape004
f.FilterType = 'window-volume'
f.Proxy.execute(f)
f.purgeTouched()
# make a line from seem of sphere to heli
line = Draft.makeWire([heli.Shape.Vertex27.Point, App.ActiveDocument.CompoundFilter.Shape.Vertex1.Point])
# split heli and line
import BOPTools.SplitFeatures
split = BOPTools.SplitFeatures.makeBooleanFragments(name= 'BooleanFragments')
split.Objects = [App.ActiveDocument.Shape004, App.ActiveDocument.Line]
split.Mode = 'Standard'
split.Proxy.execute(split)
split.purgeTouched()
#export to step
#split.Shape.exportStep("robin.step")
import ObjectsFem
mesh = ObjectsFem.makeMeshGmsh(FreeCAD.ActiveDocument, 'FEMMeshGmsh')
mesh.CharacteristicLengthMin = 0.5
mesh.CharacteristicLengthMax = 0.5
mesh.ElementDimension = 3
FreeCAD.ActiveDocument.ActiveObject.Part = FreeCAD.ActiveDocument.Shape004
mr_fus = ObjectsFem.makeMeshRegion(FreeCAD.ActiveDocument, FreeCAD.ActiveDocument.FEMMeshGmsh, 0.5, 'fus')
mr_outer = ObjectsFem.makeMeshRegion(FreeCAD.ActiveDocument, FreeCAD.ActiveDocument.FEMMeshGmsh, 1.0, 'outer')
mr_fus.CharacteristicLength = 0.7
temp = []
for i in range(1,len(App.ActiveDocument.Shape004.Shape.Faces)):
mr_fus.References.append((App.ActiveDocument.Shape004.Shape, App.ActiveDocument.Shape004.Shape.Faces[i]))
temp.append((App.ActiveDocument.Shape004.Shape, App.ActiveDocument.Shape004.Shape.Faces[i]))
print(type(mr_fus.References))
print(len(mr_fus.References))
print(len(temp))
EDIT: About FreeCAD version:
OS: Ubuntu 19.04
Word size of OS: 64-bit
Word size of FreeCAD: 64-bit
Version: 0.18.4.
Build type: Release
Python version: 3.7.3
Qt version: 5.12.2
Coin version: 4.0.0a
OCC version: 7.3.0
Locale: English/United States (en_US)

Normalize with respect to row and column

I have an array of probabilities. I would like the columns to sum to 1 (representing probability) and the rows to sum to X (where X is an integer, say 9 for example).
I thought that I could normalize the columns, and then normalize the rows and times by X. But this didn't work, the resulting sums of the rows and columns were not perfectly 1.0 and X.
This is what I tried:
# B is 5 rows by 30 columns
# Normalizing columns to 1.0
col_sum = []
for col in B.T:
col_sum.append(sum(col))
for row in range(B.shape[0]):
for col in range(B.shape[1]):
if B[row][col] != 0.0 and B[row][col] != 1.0:
B[row][col] = (B[row][col] / col_sum[col])
# Normalizing rows to X (9.0)
row_sum = []
for row in B:
row_sum.append(sum(row))
for row in range(B.shape[0]):
for col in range(B.shape[1]):
if B[row][col] != 0.0 and B[row][col] != 1.0:
B[row][col] = (B[row][col] / row_sum[row]) * 9.0

I'm not sure if I understood correctly, but it seems like what you're trying to accomplish might mathematically not be feasible?
Imagine you have a 2x2 matrix where you want the rows to sum up to 1 and the columns to 10. Even if you made all the numbers in the columns 1 (their max possible value) you would still not be able to sum them up to 10 in their columns?

This can only work if your matrix's number of columns is X times the number of rows. For example, if X = 3 and you have 5 rows, then you must have 15 columns. So, you could make your 5x30 matrix work for X=6 but not X=9.
The reason for this is that, if each column sums up to 1.0, the total of all values in the matrix will be 1.0 times the number of columns. And since you want each row to sum up to X, then the total of all values must also be X times the number of rows.
So: Columns * 1.0 = X * Rows
If that constraint is met, you only have to adjust all values proportionally to X/sum(row) and both dimensions will work automatically unless the initial values are not properly balanced. If the matrix is not already balanced, adjusting the values would be similar to solving a sudoku (allegedly an NP problem) and the result would largely be unrelated to the initial values. The matrix is balanced when all rows, adjusted to have the same sum, result in all columns having the same sum.
[0.7, 2.1, 1.4, 0.7, 1.4, 1.4, 0.7, 1.4, 1.4, 2.1, 0.7, 2.1, 1.4, 2.1, 1.4] 21
[2.8, 1.4, 0.7, 2.1, 1.4, 2.1, 0.7, 1.4, 2.1, 1.4, 0.7, 0.7, 1.4, 0.7, 1.4] 21
[1.4, 1.4, 1.4, 1.4, 1.4, 1.4, 1.4, 1.4, 1.4, 0.7, 2.8, 0.7, 0.7, 1.4, 2.1] 21
[1.4, 1.4, 1.4, 1.4, 2.1, 1.4, 1.4, 1.4, 0.7, 0.7, 2.1, 1.4, 1.4, 1.4, 1.4] 21
[0.7, 0.7, 2.1, 1.4, 0.7, 0.7, 2.8, 1.4, 1.4, 2.1, 0.7, 2.1, 2.1, 1.4, 0.7] 21
apply x = x * 3 / 21 to all elements ...
[0.1, 0.3, 0.2, 0.1, 0.2, 0.2, 0.1, 0.2, 0.2, 0.3, 0.1, 0.3, 0.2, 0.3, 0.2] 3.0
[0.4, 0.2, 0.1, 0.3, 0.2, 0.3, 0.1, 0.2, 0.3, 0.2, 0.1, 0.1, 0.2, 0.1, 0.2] 3.0
[0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.1, 0.4, 0.1, 0.1, 0.2, 0.3] 3.0
[0.2, 0.2, 0.2, 0.2, 0.3, 0.2, 0.2, 0.2, 0.1, 0.1, 0.3, 0.2, 0.2, 0.2, 0.2] 3.0
[0.1, 0.1, 0.3, 0.2, 0.1, 0.1, 0.4, 0.2, 0.2, 0.3, 0.1, 0.3, 0.3, 0.2, 0.1] 3.0
[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

Iterating forward and backward in Python

I have a coding interface which has a counter component. It simply increments by 1 with every update. Consider it an infinite generator of {1,2,3,...} over time which I HAVE TO use.
I need to use this value and iterate from -1.5 to 1.5. So, the iteration should start from -1.5 and reach 1.5 and then from 1.5 back to -1.5.
How should I use this infinite iterator to generate an iteration in that range?

You can use cycle from itertools to repeat a sequence.
from itertools import cycle
# build the list with 0.1 increment
v = [(x-15)/10 for x in range(31)]
v = v + list(reversed(v))
cv = cycle(v)
for c in my_counter:
x = next(cv)
This will repeat the list v:
-1.5, -1.4, -1.3, -1.2, -1.1, -1.0, -0.9, -0.8, -0.7, -0.6, -0.5, -0.4,
-0.3, -0.2, -0.1, 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0,
1.1, 1.2, 1.3, 1.4, 1.5, 1.5, 1.4, 1.3, 1.2, 1.1, 1.0, 0.9, 0.8, 0.7,
0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0, -0.1, -0.2, -0.3, -0.4, -0.5, -0.6,
-0.7, -0.8, -0.9, -1.0, -1.1, -1.2, -1.3, -1.4, -1.5, -1.5, -1.4, -1.3,
-1.2, -1.1, -1.0, -0.9, -0.8, -0.7, -0.6, -0.5, -0.4, -0.3, -0.2, -0.1,
0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2, 1.3,
1.4, 1.5, 1.5, 1.4, 1.3, 1.2, 1.1, 1.0, 0.9 ...

Something like:
import itertools
infGenGiven = itertools.count() # This is similar your generator
def func(x):
if x%2==0:
return 1.5
else:
return -1.5
infGenCycle = itertools.imap(func, infGenGiven)
count=0
while count<10:
print infGenCycle.next()
count+=1
Output:
1.5
-1.5
1.5
-1.5
1.5
-1.5
1.5
-1.5
1.5
-1.5
Note that this starts 1.5 because the first value in infGenGiven is 0, although for your generator it is 1 and so the infGenCycle output will give you what you want.

Thank you all.
I guess the best approach is to use the trigonometric functions (sine or cosine) which oscillate between plus and minus one.
More details at: https://en.wikipedia.org/wiki/Trigonometric_functions
Cheers