I am constructing a time delayed recurrent model and I need to know about how and when TensorFlow computes its backwards step.
Consider the following model and pseudo-code:
unit_1 = LSTM(unit_size)
unit_2 = LSTM(unit_size)
unit_3 = LSTM(unit_size)
unit_4 = LSTM(unit_size)
ip_W = Variable([4 * unit_size, output_size])
ip_b = Variable([output_size])
prev_1 = tf.zeros([unit_size])
prev_2 = tf.zeros([unit_size])
prev_3 = tf.zeros([unit_size])
prev_4 = tf.zeros([unit_size])
for t, input in enumerate(input_data):
if t%1==0:
prev_1 = unit_1([input])
if t%2==0:
prev_2 = unit_2([input])
if t%3==0:
prev_3 = unit_3([input])
if t%4==0:
prev_4 = unit_4([input])
concat = tf.concat(0,[prev_1, prev_2, prev_3, prev_4])
output[t] = tf.matmul(concat, ip_W) + ip_B
Here is a gist to a usable version of this code, based on tensorflow/python/ops/rnn.py
My Question:
For time steps where cells are not called (i.e. at T=1, unit_0 is called, while all the rest are not) are their weights updated? I'm torn as to whether it would be a good idea to have them update at every state or not. The cells themselves haven't been exposed to any new data in the steps, so I afraid that back propagating may result in over-correction. Would appreciate other's insights into this.
Let me know if any clarification is necessary.
Related
I am trying to write a pyomo model in python3, but I am facing this error that I can't seem to figure out - 'list' object has no attribute 'is_expression_type'. Below is my pyomo model, any help would be appreciated.
R_avg_tolist = [[0.00043159478649482775,
0.00045388639592182584,
0.0006735271301199177,
0.00044026758948786,
0.0037176592984565836]]
Cov_list = [[5.884677519869241e-05,
5.756542207262417e-05,
6.017027849080026e-05,
6.180151597797322e-05,
-0.0005074353586106837],
[5.756542207262417e-05,
6.0380562653096757e-05,
6.613608499966434e-05,
6.737370769879904e-05,
-0.0005362752804115953],
[6.017027849080026e-05,
6.613608499966434e-05,
8.206495000024503e-05,
8.01694525889321e-05,
-0.0005958716888916681],
[6.180151597797322e-05,
6.737370769879904e-05,
8.01694525889321e-05,
0.00010129901491226823,
-0.000608829853150321],
[-0.0005074353586106837,
-0.0005362752804115953,
-0.0005958716888916681,
-0.000608829853150321,
0.007373689071617548]]
import pyomo.environ as pyo
# Optimization Problem
def create_model(rho,R_avg,Cov):
m = pyo.ConcreteModel()
init_x = {}
m.idx = pyo.Set(initialize=[0,1,2,3,4])
for i in m.idx:
init_x[i] = 0
m.x = pyo.Var(m.idx,initialize=init_x,bounds=(0,None))
def Obj_func(m):
b = []
mult_result = 0
for i in m.idx:
a = 0
for j in m.idx:
a+= m.x[j]*Cov[j][i]
b.append(a)
for i in m.idx:
mult_result += b[i]*m.x[i]
return mult_result
m.OBJ = pyo.Objective(rule=Obj_func)
def constraint1(m):
sum=0
for i in m.idx:
sum+=m.x[i]
return sum ==100
m.C1 = pyo.Constraint(rule=constraint1(m))
def constraint2(m):
sum=0
for i in m.idx:
sum += R_avg_tolist[i]*m.x[i]
return sum >=0.08
m.C2 = pyo.Constraint(rule=constraint2(m))
return m
When I run model using below code, I face the attribute error - 'list' object has no attribute 'is_expression_type'.
rho = 0.0008
model1 = create_model(rho,R_avg_tolist,Cov_list)
solver = SolverFactory('ipopt')
results = solver.solve(model1, tee = True)
Probably not what you want to hear, but your model has many syntax probs. It's obviously a course assignment... Do you have somebody (instructor/TA) to go over this with who can advise a bit?
You didn't include enough info about which line of code caused the issue, but there are several problem areas. I've posted many simple pyomo examples if you scan through some of them, you'll get some ideas, along with the documentation and whatever you have from your course notes....
A few pointers may help:
Do NOT overwrite keywords/functions by using them as variables. When you write:
sum = 0
sum ....
you are nuking the python function sum by making that name a variable and assigning it the value of 0. You should be using sum in several of your functions with verbiage like:
sum(m.X[i] for i in m.idx) # or similar
You seem to be confused on making valid pyomo expressions. That is the core job of pyomo ... to make expressions and fill the model. For example, in your constraint1, you can just make an expression (without a function) and add it to your model. You can do a 1-liner there because the constraint is not a "for each". You could:
m.C1 = pyo.Constraint(expr=sum(m.x[i] for x in m.idx) == 100)
In general, when you are starting:
Add 1 thing to your model, and then print the model:
model.pprint()
See if it looks right, if not, fix it. Then repeat!
i'm working on graphs and big dataset of complex network's. i run SIR algorithm on them with ndlib library.
but each iteration takes something like 1Sec and it make code takes 10-12 h to complete .
i was wondering is there any way to make it parallelised ?
the code is like down bellow
this line of the code is core :
sir = model.infected_SIR_MODEL(it, infectionList, False)
is there any simple method to make it run on multi thread or parallelised ?
count = 500
for i in numpy.arange(1, count, 1):
for it in model.get_nodes():
sir = model.infected_SIR_MODEL(it, infectionList, False)
each iteration :
for u in self.graph.nodes():
u_status = self.status[u]
eventp = np.random.random_sample()
neighbors = self.graph.neighbors(u)
if isinstance(self.graph, nx.DiGraph):
neighbors = self.graph.predecessors(u)
if u_status == 0:
infected_neighbors = len([v for v in neighbors if self.status[v] == 1])
if eventp < self.BetaList[u] * infected_neighbors:
actual_status[u] = 1
elif u_status == 1:
if eventp < self.params['model']['gamma']:
actual_status[u] = 2
So, if the iterations are independent, then I don't see the point of iteration over count=500. Either way the multiprocessing library might be of interest to you.
I've prepared 2 stub solutions (i.e. alter to your exact needs).
The first expects that every input is static (the changes in solutions as far as I understand the OP's question raise from the random state generation inside each iteration). With the second, you can update the input data between iterations of i. I've not tried the code as I don't have the model so it might not work directly.
import multiprocessing as mp
# if everything is independent (eg. "infectionList" is static and does not change during the iterations)
def worker(model, infectionList):
sirs = []
for it in model.get_nodes():
sir = model.infected_SIR_MODEL(it, infectionList, False)
sirs.append(sir)
return sirs
count = 500
infectionList = []
model = "YOUR MODEL INSTANCE"
data = [(model, infectionList) for _ in range(1, count+1)]
with mp.Pool() as pool:
results = pool.starmap(worker, data)
The second proposed solution if "infectionList" or something else gets updated in each iteration of "i":
def worker2(model, it, infectionList):
sir = model.infected_SIR_MODEL(it, infectionList, False)
return sir
with mp.Pool() as pool:
for i in range(1, count+1):
data = [(model, it, infectionList) for it in model.get_nodes()]
results = pool.starmap(worker2, data)
# process results, update something go to next iteration....
Edit: Updated the answer to separate proposals more clearly.
I am attempting to implement the algorithm from the TD-Gammon article by Gerald Tesauro. The core of the learning algorithm is described in the following paragraph:
I have decided to have a single hidden layer (if that was enough to play world-class backgammon in the early 1990's, then it's enough for me). I am pretty certain that everything except the train() function is correct (they are easier to test), but I have no idea whether I have implemented this final algorithm correctly.
import numpy as np
class TD_network:
"""
Neural network with a single hidden layer and a Temporal Displacement training algorithm
taken from G. Tesauro's 1995 TD-Gammon article.
"""
def __init__(self, num_input, num_hidden, num_output, hnorm, dhnorm, onorm, donorm):
self.w21 = 2*np.random.rand(num_hidden, num_input) - 1
self.w32 = 2*np.random.rand(num_output, num_hidden) - 1
self.b2 = 2*np.random.rand(num_hidden) - 1
self.b3 = 2*np.random.rand(num_output) - 1
self.hnorm = hnorm
self.dhnorm = dhnorm
self.onorm = onorm
self.donorm = donorm
def value(self, input):
"""Evaluates the NN output"""
assert(input.shape == self.w21[1,:].shape)
h = self.w21.dot(input) + self.b2
hn = self.hnorm(h)
o = self.w32.dot(hn) + self.b3
return(self.onorm(o))
def gradient(self, input):
"""
Calculates the gradient of the NN at the given input. Outputs a list of dictionaries
where each dict corresponds to the gradient of an output node, and each element in
a given dict gives the gradient for a subset of the weights.
"""
assert(input.shape == self.w21[1,:].shape)
J = []
h = self.w21.dot(input) + self.b2
hn = self.hnorm(h)
o = self.w32.dot(hn) + self.b3
for i in range(len(self.b3)):
db3 = np.zeros(self.b3.shape)
db3[i] = self.donorm(o[i])
dw32 = np.zeros(self.w32.shape)
dw32[i, :] = self.donorm(o[i])*hn
db2 = np.multiply(self.dhnorm(h), self.w32[i,:])*self.donorm(o[i])
dw21 = np.transpose(np.outer(input, db2))
J.append(dict(db3 = db3, dw32 = dw32, db2 = db2, dw21 = dw21))
return(J)
def train(self, input_states, end_result, a = 0.1, l = 0.7):
"""
Trains the network using a single series of input states representing a game from beginning
to end, and a final (supervised / desired) output for the end state
"""
outputs = [self(input_state) for input_state in input_states]
outputs.append(end_result)
for t in range(len(input_states)):
delta = dict(
db3 = np.zeros(self.b3.shape),
dw32 = np.zeros(self.w32.shape),
db2 = np.zeros(self.b2.shape),
dw21 = np.zeros(self.w21.shape))
grad = self.gradient(input_states[t])
for i in range(len(self.b3)):
for key in delta.keys():
td_sum = sum([l**(t-k)*grad[i][key] for k in range(t + 1)])
delta[key] += a*(outputs[t + 1][i] - outputs[t][i])*td_sum
self.w21 += delta["dw21"]
self.w32 += delta["dw32"]
self.b2 += delta["db2"]
self.b3 += delta["db3"]
The way I use this is I play through a whole game (or rather, the neural net plays against itself), and then I send the states of that game, from start to finish, into train(), along with the final result. It then takes this game log, and applies the above formula to alter weights using the first game state, then the first and second game states, and so on until the final time, when it uses the entire list of game states. Then I repeat that many times and hope that the network learns.
To be clear, I am not after feedback on my code writing. This was never meant to be more than a quick and dirty implementation to see that I have all the nuts and bolts in the right spots.
However, I have no idea whether it is correct, as I have thus far been unable to make it capable of playing tic-tac-toe at any reasonable level. There could be many reasons for that. Maybe I'm not giving it enough hidden nodes (I have used 10 to 12). Maybe it needs more games to train (I have used 200 000). Maybe it would do better with different normalisation functions (I've tried sigmoid and ReLU, leaky and non-leaky, in different variations). Maybe the learning parameters are not tuned right. Maybe tic-tac-toe and its deterministic gameplay means it "locks in" on certain paths in the game tree. Or maybe the training implementation is just wrong. Which is why I'm here.
Have I misunderstood Tesauro's algorithm?
I can't say that I entirely understand your implementation, but this line jumps out to me:
td_sum = sum([l**(t-k)*grad[i][key] for k in range(t + 1)])
Comparing with the formula you reference:
I see at least two differences:
Your implementation sums over t+1 elements compared to t elements in the formula
The gradient should be indexed with the same k as used in l**(t-k), but in your implementation it is indexed with i and key, without any reference to k
Perhaps if you fix these discrepancies your solution will behave more as expected.
I'm trying to use the Python library Pygmo2 (https://esa.github.io/pagmo2/index.html) to parallelize an optimization problem.
To my understanding, parallelization can be achieved with an archipelago of islands (in this case, mp_island).
As a minimal working example, one of the tutorials from the official site can serve: https://esa.github.io/pagmo2/docs/python/tutorials/using_archipelago.html
I extracted the code:
class toy_problem:
def __init__(self, dim):
self.dim = dim
def fitness(self, x):
return [sum(x), 1 - sum(x*x), - sum(x)]
def gradient(self, x):
return pg.estimate_gradient(lambda x: self.fitness(x), x)
def get_nec(self):
return 1
def get_nic(self):
return 1
def get_bounds(self):
return ([-1] * self.dim, [1] * self.dim)
def get_name(self):
return "A toy problem"
def get_extra_info(self):
return "\tDimensions: " + str(self.dim)
import pygmo as pg
a_cstrs_sa = pg.algorithm(pg.cstrs_self_adaptive(iters=1000))
p_toy = pg.problem(toy_problem(50))
p_toy.c_tol = [1e-4, 1e-4]
archi = pg.archipelago(n=32,algo=a_cstrs_sa, prob=p_toy, pop_size=70)
print(archi)
archi.evolve()
print(archi)
Looking at the documentation of the old version of the library (http://esa.github.io/pygmo/documentation/migration.html), migration between islands seems to be an essential feature of the island parallelization model.
Also, to my understanding, optimization algorithms like evolutionary algorithms could not work without it.
However, in the documentation of Pygmo2, I can nowhere find how to perform migration.
Is it happening automatically in an archipelago?
Does it depend on the selected algorithm?
Is it not yet implemented in Pygmo2?
Is the documentation on this yet missing or did I just not find it?
Can somebody enlighten me?
pagmo2 is now implementing migration since v2.11, the PR has benn completed and merged into master. Almost all capabilities present in pagmo1.x are restored. We will still add more topologies in the future, but they can already be implemented manually. Refer to docs here: https://esa.github.io/pagmo2/docs/cpp/cpp_docs.html
Tutorial and example are missing and will be added in the near future (help is welcome)
The migration framework has not been fully ported from pagmo1 to pagmo2 yet. There is a long-standing PR open here:
https://github.com/esa/pagmo2/pull/102
We will complete the implementation of the migration framework in the next few months, hopefully by the beginning of the summer.
IMHO, the PyGMO2/pagmo documentation is confirming the migration feature to be present.
The archipelago class is the main parallelization engine of pygmo. It essentially is a container of island able to initiate evolution (optimization tasks) in each island asynchronously while keeping track of the results and of the information exchange (migration) between the tasks ...
With an exception of thread_island-s ( where some automated inference may take place and enforce 'em for thread-safe UDI-s ), all other island types - { mp_island | ipyparallel_island }-s do create a GIL-independent form of a parallelism, yet the computing is performed via an async-operated .evolve() method
In original PyGMO, the archipelago class was auto .__init__()-ed with attribute topology = unconnected(), unless specified explicitly, as documented in PyGMO, having a tuple of call-interfaces for archipelago.__init__() method ( showing just the matching one ):
__init__( <PyGMO.algorithm> algo,
<PyGMO.problem> prob,
<int> n_isl,
<int> n_ind [, topology = unconnected(),
distribution_type = point_to_point,
migration_direction = destination
]
)
But, adding that, one may redefine the default, so as to meet one's PyGMO evolutionary process preferences:
topo = topology.erdos_renyi( nodes = 100,
p = 0.03
) # Erdos-Renyi ( random ) topology
or
set a Clustered Barabási-Albert, with ageing vertices graph topology:
topo = topology.clustered_ba( m0 = 3,
m = 3,
p = 0.5,
a = 1000,
nodes = 0
) # clustered Barabasi-Albert,
# # with Ageing vertices topology
or:
topo = topology.watts_strogatz( nodes = 100,
p = 0.1
) # Watts-Strogatz ( circle
# + links ) topology
and finally, set it by assignment into the class-instance attribute:
archi = pg.archipelago( n = 32,
algo = a_cstrs_sa,
prob = p_toy,
pop_size = 70
) # constructs an archipelago
archi.topology = topo # sets the topology to the
# # above selected, pre-defined <topo>
I'm trying to make an Elman Network (aka Simple Recurent Network) with Pybrain, I think the code should look something like this:
n = RecurentNetwork()
n.addInputModule(LinearLayer(5, name = 'in'))
n.addModule(TanhLayer(10, name = 'hidden'))
n.addModule(LinearLayer(10, name = 'context'))
n.addOutputModule(LinearLayer(5, name = 'out'))
n.addConnection(FullConnection(n['in'], n['hidden'], name = 'in_to_hidden'))
n.addConnection(FullConnection(n['hidden'], n['out'], name = 'hidden_to_out'))
n.addConnection(IdentityConnection(n['hidden'], n['context'], name = 'hidden_to_context'))
n.addConnection(IdentityConnection(n['context'], n['hidden'], name = 'context_to_hidden'))
My problem is that I don't know how to get the context nodes (at time t) to keep the values of the hidden nodes of the last iteration (at time t-1) in order to give them to the hidden nodes in this iteration (at time t) and how to fix the weights in hidden_to_context to be 1. How it is right now I get an error saying there is a "loop" in the net (and indeed there is one). Any help would be much appreciated. Thank you very much.
Cheers,
Bruno
I would look at this section:
http://pybrain.org/docs/tutorial/netmodcon.html#using-recurrent-networks
In particular,
The RecurrentNetwork class has one additional method, .addRecurrentConnection(), which looks back in time one timestep.