I am working on a Dial a Ride Problem (DARP). I have a lage amount of nodes and edges (338 nodes and 826 edges). I've imported the node/edge data from OSMnx and am trying to solve the model with Gurobi Optimizer in Python.
To be able to use the OSMnx data with Gurobi, I created a matrix = len(nodes) x len(nodes) matrix and therein printed the length of the edge if two nodes were connected, and a large number otherwise. In the optimization, a x[i,j] = len(nodes) x len(nodes) binary decision variable is used to decide if an edge is traversed or not.
The problem I am encountering is a large computing time for just one request (+1 hour). I think this is because the model also has to consider all the other indices from this large matrix, even though they can be ignored completely since they represent that two nodes are unconnected.
My question therefore is if someone can help me find some preprocessing techniques or something else that might reduce my computational time. For example, tell the model that it can ignore indices from this matrix if the value is too high or maybe a more efficient node/edge storage file that Gurobi can use more efficiently.
Thanks in advance.
If your graph is sparse, the optimization model should be sparse, too. Specifically, you should only create a variable x[i,j] if the edge (i,j) exists in the graph. For an example of how to do this, see the netflow.py sample in the examples/python subdirectory of Gurobi.
I want to concatenate arrays of different dimensions to feed them to my neural network that will have as first layer the AdaptiveAveragePooling1d. I have a dataset that is composed of several signals (1D arrays), each one with a different length. For example:
array1 = np.random.randn(1200,1)
array2 = np.random.randn(950,1)
array3 = np.random.randn(1000,1)
I want to obtain a tensor in which I concatenate these three signals to obtain a 2D tensor.
However if I try to do
tensor = torch.Tensor([array1, array2, array3])
It gives me this error:
ValueError: expected sequence of length 1200 at dim 2 (got 950)
Is there a way to obtain such thing?
EDIT
More information about the dataset:
Each signal window represents a heart beat on the ECG registration, taken from several patients, sampled with a sampling frequency of 1000Hz
The beats can have different lengths, because it depends on the heart rate of the patient itself
For each beat I need to predict the length of the QRS interval (the target of the network) that I have, expressed in milliseconds
I have already thought of interpolating the shortest samples to the the length of the longest ones, but then I would also have to change the length of the QRS interval in the labels, is that right?
I have read of this AdaptiveAveragePooling1d layer, that would allow me to input the network with samples of different sizes. But my problem is how do I input the network a dataset in which each sample has a different length? How do I group them without using a filling method with NaNs or zeros?
I hope I explained myself.
This disobeys the definition of a tensor and is impossible. If a tensor is of shape (NxMx1), all of the N matrices must be of size (Mx1).
There are still ways to get all your arrays to the same length. Look at where your data is coming from and what its structure is and figure out which of the following solutions would work. Some of these may change the signal's derivative in a way you don't like
Cropping arrays to the same size (ie cutting start/end off) or zero padding the shorter ones to the length of the longer one (I really dislike this one and it would only work for very specific applications)
'Stretching' the arrays to the same size by using interpolation
Shortening the arrays to the same size by subsampling
For some applications, maybe even passing the coefficients of a
fourier series from the signals
EDIT
For heart rate, which should be a roughly periodic signal, I'd definitely crop the signal which should work quite well. Passing FFT(equally cropped signals) or Fourier coefficients may also yield interesting results, but from my experience with neural spike data, training on the FFT of a signal like this doesn't perform any better when you have enough data to train off.
Also if you're using a fully connected network, a using 1D convolutions is a good alternative to try.
I am using odeint in scipy to integrate a function. The function is basically the velocities in x,y,z directions which I need to integrate to find the corresponding x,y,z position coordinates. Odeint in python asks for a list of timesteps and return values for those number of timesteps only. But, I require all values of x,y,z coordinate positions calculated at each internally defined timestep and not just at the timesteps I send as a parameter. So, is there a way to just send the min and max timestep and get all the values calculated at each internally defined timestep between the given min and max timestep?
The reason I need this is that when I plot the x,y,z coordinates returned, I am getting sharp turns and not smooth paths. So, in order to plot a smooth path, I will require all the coordinates calculated at each internally defined timestep.
If I am not wrong, the ODE45 function in Matlab returns all values calculated at every automatically defined internal timestep. Is there a way to get this to work in python?
You get this functionality with scipy.integrate.solve_ivp or by fashioning your own time loop using the stepper classes, the old ode class or the new RK45, Radau, LSODA etc.
Note that in Matlab the option parameter 'Refine' is set to the default value 4, so that for every internal step there are 3 interpolated points added so that the output looks curved despite large time steps due to the step size control. This trick is not present in the python routines, you would have to enrich the output yourself.
Has anyone tried to predict a specific pattern in time series data?
Example: In a specific time, there is a huge upward spike in certain variables in a time series...
How would I build a model to predict that spike when next time it occurs?
Please do respond if anyone working in this area.
I tried with converting that particular series of data in a NumPy array and trying to feed in the model.But Its not allowing me.
Here is the data looks like
This data is generated in a controlled manner so that we can have these spikes near to near.. In actual case this could b random, and our main objective is to catch this pattern and make a count.
Das, you could try implementing LSTM based Neural Network Models.
See:
https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/
It is still preferred that the data contains a trend. If the upward spike happens around the same time of the recurring time interval, it is more likely that you get a better prediction result.
In the image you shared, there seems to be trend in the data. Hence LSTM models can pretty efficiently extract the pattern and output a prediction.
Statistical modelling of the data can also provide better results.
See: https://orangematter.solarwinds.com/2019/12/15/holt-winters-forecasting-simplified/
Das, if outputting the total number of peaks is solely the requirement, then I think heavy neural network models are bit of an overkill. However, neural network models also can pretty well do the job, but require lot of data input for training and fine tuning the weights and biases to give a really good result.
How about you try implementing a thresholding based technique, where you increment a counter every time the data value crosses the preset threshold? In such an approach you should ensure to group very nearby peaks together so that the count is just one for that case. Here you could set a threshold on the x axis too.
ie:- For instance with respect to the given plot, let the y-threshold be 4. Then you will get a count 5 if you consider the y axis threshold (y value 4) alone. This is because for x value at 15:48.2, there are two peaks that cross y value 4. So suppose you set a threshold in the x axis too, then these nearby peaks shall be grouped together within the preset limit and the final count will be 4 (which is the requirement).
Can Tensorflow handle inputs of varying shapes (sizes)?
The project
I'm developing a image/shape recognizer which captures an array of {x:#,y:#} positions.
For example, a circle might look like this
[{"x":0.38,"y":0.32},{"x":0.33,"y":0.35},{"x":0.31,"y":0.4},{"x":0.31,"y":0.46},{"x":0.34,"y":0.51},{"x":0.39,"y":0.52},{"x":0.44,"y":0.51},{"x":0.47,"y":0.47},{"x":0.49,"y":0.42},{"x":0.47,"y":0.37},{"x":0.42,"y":0.34},{"x":0.37,"y":0.33}]
and a square like this
[{"x":0.15,"y":0.19},{"x":0.15,"y":0.25},{"x":0.15,"y":0.31},{"x":0.15,"y":0.37},{"x":0.14,"y":0.42},{"x":0.14,"y":0.48},{"x":0.14,"y":0.53},{"x":0.14,"y":0.59},{"x":0.14,"y":0.64},{"x":0.2,"y":0.64},{"x":0.26,"y":0.64},{"x":0.31,"y":0.65},{"x":0.37,"y":0.65},{"x":0.43,"y":0.65},{"x":0.49,"y":0.65},{"x":0.54,"y":0.65},{"x":0.6,"y":0.65},{"x":0.65,"y":0.65},{"x":0.67,"y":0.6},{"x":0.68,"y":0.55},{"x":0.68,"y":0.5},{"x":0.68,"y":0.44},{"x":0.68,"y":0.38},{"x":0.68,"y":0.32},{"x":0.67,"y":0.27},{"x":0.67,"y":0.22},{"x":0.66,"y":0.17},{"x":0.61,"y":0.15},{"x":0.56,"y":0.13},{"x":0.51,"y":0.13},{"x":0.45,"y":0.13},{"x":0.39,"y":0.13},{"x":0.33,"y":0.13},{"x":0.27,"y":0.13},{"x":0.22,"y":0.14},{"x":0.17,"y":0.15}]
Because the length of these shapes can vary I was wondering how Tensorflow would handle it...as I understand it, the input "shape" needs to always be the same length, right?
Yes, the shape should be the same. But, in your case, you can make sure that for a batch, all the arrays have the same number of elements by adding dummy elements to those which fall short in length.
Just make sure that for a batch, your shape is same.