To solve problem 83 of project euler I tried to use the A* algorithm. The algorithm works fine for the given problem and I get the correct result. But when I visualized the algorithm I realized that it seems as if the algorithm checked way to many possible nodes. Is it because I didn't implement the algorithm properly or am I missing something else? I tried using two different heuristic functions which you can see in the code below, but the output didn't change much.
Are any tips to make the code efficient?
import heapq
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from matplotlib import animation
import numpy as np
class PriorityQueue:
def __init__(self):
self.elements = []
def empty(self):
return not self.elements
def put(self, item, priority):
heapq.heappush(self.elements, (priority, item))
def get(self):
return heapq.heappop(self.elements)[1]
class A_star:
def __init__(self, data, start, end):
self.data = data
self.start = start
self.end = end
self.a = len(self.data)
self.b = len(self.data[0])
def h_matrix(self):
elements = sorted([self.data[i][j] for j in range(self.b) for i in range(self.a)])
n = self.a + self.b - 1
minimum = elements[:n]
h = []
for i in range(self.a):
h_i = []
for j in range(self.b):
h_i.append(sum(minimum[:(n-i-j-1)]))
h.append(h_i)
return h
def astar(self):
h = self.h_matrix()
open_list = PriorityQueue()
open_list.put(self.start, 0)
came_from = {}
cost_so_far = {}
came_from[self.start] = None
cost_so_far[self.start] = self.data[0][0]
checked = []
while not open_list.empty():
current = open_list.get()
checked.append(current)
if current == self.end:
break
neighbors = [(current[0]+x, current[1]+y) for x, y in {(-1,0), (0,-1), (1,0), (0,1)}
if 0 <= current[0]+x < self.a and 0 <= current[1]+y < self.b]
for next in neighbors:
new_cost = cost_so_far[current] + self.data[next[0]][next[1]]
if next not in cost_so_far or new_cost < cost_so_far[next]:
cost_so_far[next] = new_cost
priority = new_cost + h[next[0]][next[1]]
open_list.put(next, priority)
came_from[next] = current
return came_from, checked, cost_so_far[self.end]
def reconstruct_path(self):
paths = self.astar()[0]
best_path = [self.end]
while best_path[0] is not None:
new = paths[best_path[0]]
best_path.insert(0, new)
return best_path[1:]
def minimum(self):
return self.astar()[2]
if __name__ == "__main__":
liste = [[131, 673, 234, 103, 18], [201, 96, 342, 965, 150], [630, 803, 746, 422, 111], [537, 699, 497, 121, 956], [805, 732, 524, 37, 331]]
path = A_star(liste, (0,0), (4,4))
print(path.astar())
#print(path.reconstruct_path())
path.plot_path(speed=200)
Here you can see my visualization for the 80x80 matrix given in the problem. Blue are all the points in checked and red is the optimal path. From my understanding it shouldn't be the case that every point in the matrix is in checked i.e. blue.
https://i.stack.imgur.com/LKkdh.png
My initial guess would be that my heuristic function is not good enough. If I choose h=0, which would mean Dijkstra Algorithm the length of my checked list is 6400. Contrary if I use my custom h the length is 6455. But how can I improve the heuristic function for an arbitrary matrix?
Let me start at the end of your post:
Marking cells as checked
if I use my custom h the length is 6455.
You should not have a size of checked that exceeds the number of cells. So let me first suggest an improvement for that: instead of using a list, use set, and skip anything popped from the priority queue that is already in the set. The relevant code will then look like this:
checked = set()
while not open_list.empty():
current = open_list.get()
if current in checked: # don't repeat the same search
continue
checked.add(current)
And if in the end you need the list version of checked, just return it that way:
return came_from, list(checked), cost_so_far[self.end]
Now to the main question:
Improving heuristic function
From my understanding it shouldn't be the case that every point in the matrix is in checked i.e. blue. My initial guess would be that my heuristic function is not good enough.
That is the right explanation. Combine that with the fact that the given matrix has paths which have a total cost which come quite close, so there is a competition field that involves much of the matrix.
how can I improve the heuristic function for an arbitrary matrix?
One idea is to consider the following. A path must include at least one element from each "forward" diagonal (/). So if we work with the minimum value on each such diagonal and create running sums (backwards -- starting from the target), we'll have a workable value for h.
Here is the code for that idea:
def h_matrix(self):
min_diagonals = [float("inf")] * (self.a + self.b - 1)
# For each diagonal get the minimum cost on that diagonal
for i, row in enumerate(self.data):
for j, cost in enumerate(row):
min_diagonals[i+j] = min(min_diagonals[i+j], cost)
# Create a running sum in backward direction
for i in range(len(min_diagonals) - 2, -1, -1):
min_diagonals[i] += min_diagonals[i + 1]
min_diagonals.append(0) # Add an entry for allowing the next logic to work
# These sums are a lower bound for the remaining cost towards the target
h = [
[min_diagonals[i + j + 1] for j in range(self.b)]
for i in range(self.a)
]
return h
With these improvements, we get these counts:
len(cost_so_far) == 6374
len(checked) == 6339
This represents still a large portion of the matrix, but at least a few cells were left out.
Related
this is my code to find the height of a tree of up to 10^5 nodes. May I know why I get the following error?
Warning, long feedback: only the beginning and the end of the feedback message is shown, and the middle was replaced by " ... ". Failed case #18/24: time limit exceeded
Input:
100000
Your output:
stderr:
(Time used: 6.01/3.00, memory used: 24014848/2147483648.)
Is there a way to speed up this algo?
This is the exact problem description:
Problem Description
Task. You are given a description of a rooted tree. Your task is to compute and output its height. Recall
that the height of a (rooted) tree is the maximum depth of a node, or the maximum distance from a
leaf to the root. You are given an arbitrary tree, not necessarily a binary tree.
Input Format. The first line contains the number of nodes π. The second line contains π integer numbers
from β1 to π β 1 β parents of nodes. If the π-th one of them (0 β€ π β€ π β 1) is β1, node π is the root,
otherwise itβs 0-based index of the parent of π-th node. It is guaranteed that there is exactly one root.
It is guaranteed that the input represents a tree.
Constraints. 1 β€ π β€ 105
Output Format. Output the height of the tree.
# python3
import sys, threading
from collections import deque, defaultdict
sys.setrecursionlimit(10**7) # max depth of recursion
threading.stack_size(2**27) # new thread will get stack of such size
class TreeHeight:
def read(self):
self.n = int(sys.stdin.readline())
self.parent = list(map(int, sys.stdin.readline().split()))
def compute_height(self):
height = 0
nodes = [[] for _ in range(self.n)]
for child_index in range(self.n):
if self.parent[child_index] == -1:
# child_index = child value
root = child_index
nodes[0].append(root)
# do not add to index
else:
parent_index = None
counter = -1
updating_child_index = child_index
while parent_index != -1:
parent_index = self.parent[updating_child_index]
updating_child_index = parent_index
counter += 1
nodes[counter].append(child_index)
# nodes[self.parent[child_index]].append(child_index)
nodes2 = list(filter(lambda x: x, nodes))
height = len(nodes2)
return(height)
def main():
tree = TreeHeight()
tree.read()
print(tree.compute_height())
threading.Thread(target=main).start()
First, why are you using threading? Threading isn't good. It is a source of potentially hard to find race conditions and confusing complexity. Plus in Python, thanks to the GIL, you often don't get any performance win.
That said, your algorithm essentially looks like this:
for each node:
travel all the way to the root
record its depth
If the tree is entirely unbalanced and has 100,000 nodes, then for each of 100,000 nodes you have to visit an average of 50,000 other nodes for roughly 5,000,000,000 operations. This takes a while.
What you need to do is stop constantly traversing the tree back to the root to find the depths. Something like this should work.
import sys
class TreeHeight:
def read(self):
self.n = int(sys.stdin.readline())
self.parent = list(map(int, sys.stdin.readline().split()))
def compute_height(self):
height = [None for _ in self.parent]
todo = list(range(self.n))
while 0 < len(todo):
node = todo.pop()
if self.parent[node] == -1:
height[node] = 1
elif height[node] is None:
if height[self.parent[node]] is None:
# We will try again after doing our parent
todo.append(node)
todo.append(self.parent[node])
else:
height[node] = height[self.parent[node]] + 1
return max(height)
if __name__ == "__main__":
tree = TreeHeight()
tree.read()
print(tree.compute_height())
(Note, I switched to a standard indent, and then made that indent 4. See this classic study for evidence that an indent in the 2-4 range is better for comprehension than an indent of 8. And, of course, the pep8 standard for Python specifies 4 spaces.)
Here is the same code showing how to handle accidental loops, and hardcode a specific test case.
import sys
class TreeHeight:
def read(self):
self.n = int(sys.stdin.readline())
self.parent = list(map(int, sys.stdin.readline().split()))
def compute_height(self):
height = [None for _ in self.parent]
todo = list(range(self.n))
in_redo = set()
while 0 < len(todo):
node = todo.pop()
if self.parent[node] == -1:
height[node] = 1
elif height[node] is None:
if height[self.parent[node]] is None:
if node in in_redo:
# This must be a cycle, lie about its height.
height[node] = -100000
else:
in_redo.add(node)
# We will try again after doing our parent
todo.append(node)
todo.append(self.parent[node])
else:
height[node] = height[self.parent[node]] + 1
return max(height)
if __name__ == "__main__":
tree = TreeHeight()
# tree.read()
tree.n = 5
tree.parent = [-1, 0, 4, 0, 3]
print(tree.compute_height())
I'm trying to find some Python library or method that will create a formula that describes an an arbitrary list of integers like: [1,1,2,4,3,1,2,3,4,1,...]
For example:
from some_awesome_library import magic_method
seq = [1,1,2,4,3,1,2,3,4,1,4,3,2,2,4,3]
my_func(sequence):
equation = magic_method(seq)
return equation
print(my_func(seq))
The order of the sequence matters, but it has certain rules. For example, all integers will be between 1 and 4, and there will be an equal number of each integer within the sequence.
I've looked into numpy.polyfit and scipy.optimize.leastsq. I suspect that scipy is what I need, but it'd be great to have confirmation of that approach and any suggestions for the types of mathematical functions I should look into using (I'm not much of a math's person - just studied up to college calculus). Maybe a some sort of modulo function? Maybe a sine wave?
Thanks in advance for any help or suggestions you have.
EDIT: Thanks for your comments below. I'm curious about Sudoku puzzles, specifically N=2 puzzles. I suspect that if you take the entire solution space and line them up in a certain order that patterns will emerge that might be useful for solving Sudoku faster. I've got a class that represents the solution space called SolutionManager and returns "slices" of the solution space that look like the list of integers shown above. Below is an image of one such example for an N=2 puzzle solution space (generated with Jupyter Notebook):
I think I can see patterns in the data, but I'm trying to figure out how to develop formulas that represent these patterns. I also suspect that reordering the solutions will make for simpler equations describing those patterns.
To prove that I'm trying to write a genetic algorithm that will reorder the solutions described in the SolutionManager according to how simple the equations describing them are. I'm stuck on writing the fitness function, which should rate the SolutionManager instance by how simple it's described equations are.
Code for the SolutionManager is below:
class SolutionManager:
"""Object that contains all possible Solutions for an n=2 Sudoku array"""
def __init__(self, solutions_file):
input_file = open(solutions_file, 'r')
grids = [r.strip() for r in input_file.readlines()]
list_of_solutions = []
i = 0
for grid in grids:
list_of_cubes = []
i += 1
for r in range(4):
for c in range(4):
pos = r * 4 + c
digit = int(grid[pos])
cube = Cube(i, c, r, digit)
list_of_cubes.append(cube)
list_of_solutions.append(Solution(list_of_cubes))
self.solutions = list_of_solutions
assert isinstance(self.solutions, list)
assert all(isinstance(x, Solution) for x in self.solutions)
"""Get a vertical slice of the Solution Space"""
def get_vertical_slice(self, slice_index):
assert slice_index <= 4
assert slice_index >= 0
slice = []
for sol in self.solutions:
slice.append(sol.get_column(slice_index))
return slice
"""Get a horizontal slice of the Solution Space"""
def get_horizontal_slice(self, slice_index):
assert slice_index <= 4
assert slice_index >= 0
slice = []
for sol in self.solutions:
slice.append(sol.get_row(slice_index))
return slice
"""Sorts the solutions by a vertical axis using an algorithm"""
def sort_solutions_by_vertical_axis(self, axis_index):
pass
class Solution:
def __init__(self, cubes):
assert (len(cubes) == 16)
self.solution_cubes = cubes
def get_column(self, c):
return list(_cube for _cube in self.solution_cubes if _cube.row == c)
def get_row(self, r):
return list(_cube for _cube in self.solution_cubes if _cube.column == r)
def get_column_value(self, c, v):
single_cube = list(_cube for _cube in self.solution_cubes if _cube.row == r and _cube.value == v)
assert (len(single_cube) == 1)
return single_cube[0]
def get_row_value(self, r, v):
single_cube = list(_cube for _cube in self.solution_cubes if _cube.column == c and _cube.value == v)
assert (len(single_cube) == 1)
return single_cube[0]
def get_position(self, r, c):
single_cube = list(_cube for _cube in self.solution_cubes if _cube.column == c and _cube.row == r)
assert (len(single_cube) == 1)
return single_cube[0]
class Cube:
def __init__(self, d, c, r, v):
self.depth = d
self.column = c
self.row = r
self.value = v
def __str__(self):
return str(self.value)
EDIT: This was a while ago and I've since got it working, if you'd like to see the code it's included at github.com/LewisGaul/minegaulerQt.
I'm trying to write a program to calculate probabilities for the game minesweeper, and have had some difficulty working out how best to structure it. While it may seem quite simple at first with the example below, I would like to know the best way to allow for more complex configurations. Note I am not looking for help with how to calculate probabilities - I know the method, I just need to implement it!
To make it clear what I'm trying to calculate, I will work through a simple example which can be done by hand. Consider a minesweeper configuration
# # # #
# 1 2 #
# # # #
where # represents an unclicked cell. The 1 tells us there is exactly 1 mine in the leftmost 7 unclicked cells, the 2 tells us there are exactly 2 in the rightmost 7. To calculate the probability of each individual cell containing a mine, we need to determine all the different cases (only 2 in this simple case):
1 mine in leftmost 3 cells, 2 mines in rightmost 3 cells (total of 3 mines, 3x3=9 combinations).
1 mine in center 4 cells, 1 mine in rightmost 3 cells (total of 2 mines, 4x3=12 combinations).
Given the probability of a mine being in a random cell is about 0.2, it is (in a random selection of cells) about 4 times more likely there is a total of 2 mines rather than a total of 3, so the total number of mines in a configuration matters, as well as the number of combinations of each configuration. So in this case the probability of case 1 is 9/(9+4x12)=0.158, and the probability of there being a mine in a given leftmost cell is therefore about 0.158/3=0.05, as those cells are effectively equivalent (they share exactly the same revealed neighbours).
I have created a GUI with Tkinter which allows me to easily enter configurations such as the one in the example, which stores the grid as a numpy array. I then made a NumberGroup class which isolates each of the clicked/numbered cells, storing the number and a set of the coordinates of its unclicked neighbours. These can be subtracted to get equivalence groups... Although this would not be as straightforward if there were three or more numbers instead of just two. But I am unsure how to go from here to getting the different configurations. I toyed with making a Configuration class, but am not hugely familiar with how different classes should work together. See working code below (numpy required).
Note: I am aware I could have attempted to use a brute force approach, but if possible I would like to avoid that, keeping the equivalent groups separate (in the above example there are 3 equivalence groups, the leftmost 3, the middle 4, the rightmost 3). I would like to hear your thoughts on this.
import numpy as np
grid = np.array(
[[0, 0, 0, 0],
[0, 2, 1, 0],
[0, 0, 0, 0]]
)
dims = (3, 4) #Dimensions of the grid
class NumberGroup(object):
def __init__(self, mines, coords, dims=None):
"""Takes a number of mines, and a set of coordinates."""
if dims:
self.dims = dims
self.mines = mines
self.coords = coords
def __repr__(self):
return "<Group of {} cells with {} mines>".format(
len(self.coords), self.mines)
def __str__(self):
if hasattr(self, 'dims'):
dims = self.dims
else:
dims = (max([c[0] for c in self.coords]) + 1,
max([c[1] for c in self.coords]) + 1)
grid = np.zeros(dims, int)
for coord in self.coords:
grid[coord] = 1
return str(grid).replace('0', '.').replace('1', '#')
def __sub__(self, other):
if type(other) is NumberGroup:
return self.coords - other.coords
elif type(other) is set:
return self.coords - other.coords
else:
raise TypeError("Can only subtract a group or a set from another.")
def get_neighbours(coord, dims):
x, y = coord
row = [u for u in range(x-1, x+2) if u in range(dims[0])]
col = [v for v in range(y-1, y+2) if v in range(dims[1])]
return {(u, v) for u in row for v in col}
groups = []
all_coords = [(i, j) for i in range(dims[0])
for j in range(dims[1])]
for coord, nr in [(c, grid[c]) for c in all_coords if grid[c] > 0]:
empty_neighbours = {c for c in get_neighbours(coord, dims)
if grid[c] == 0}
if nr > len(empty_neighbours):
print "Error: number {} in cell {} is too high.".format(nr, coord)
break
groups.append(NumberGroup(nr, empty_neighbours, dims))
print groups
for g in groups:
print g
print groups[0] - groups[1]
UPDATE:
I have added a couple of other classes and restructured a bit (see below for working code), and it is now capable of creating and displaying the equivalence groups, which is a step in the right direction. However I still need to work out how to iterate through all the possible mine-configurations, by assigning a number of mines to each group in a way that creates a valid configuration. Any help is appreciated.
For example,
# # # #
# 2 1 #
# # # #
There are three equivalence groups G1: the left 3, G2: the middle 4, G3: the right 3. I want the code to loop through, assigning groups with mines in the following way:
G1=2 (max the first group) => G2=0 => G3=1 (this is all configs with G1=2)
G1=1 (decrease by one) => G2=1 => G3=0 (this is all with G1=1)
G1=0 => G2=2 INVALID
So we arrive at both configurations. This needs to work for more complicated setups!
import numpy as np
def get_neighbours(coord, dims):
x, y = coord
row = [u for u in range(x-1, x+2) if u in range(dims[0])]
col = [v for v in range(y-1, y+2) if v in range(dims[1])]
return {(u, v) for u in row for v in col}
class NrConfig(object):
def __init__(self, grid):
self.grid = grid
self.dims = grid.shape # Dimensions of grid
self.all_coords = [(i, j) for i in range(self.dims[0])
for j in range(self.dims[1])]
self.numbers = dict()
self.groups = []
self.configs = []
self.get_numbers()
self.get_groups()
self.get_configs()
def __str__(self):
return str(self.grid).replace('0', '.')
def get_numbers(self):
for coord, nr in [(c, self.grid[c]) for c in self.all_coords
if self.grid[c] > 0]:
empty_neighbours = {c for c in get_neighbours(
coord, self.dims) if self.grid[c] == 0}
if nr > len(empty_neighbours):
print "Error: number {} in cell {} is too high.".format(
nr, coord)
return
self.numbers[coord] = Number(nr, coord, empty_neighbours,
self.dims)
def get_groups(self):
coord_neighbours = dict()
for coord in [c for c in self.all_coords if self.grid[c] == 0]:
# Must be a set so that order doesn't matter!
coord_neighbours[coord] = {self.numbers[c] for c in
get_neighbours(coord, self.dims) if c in self.numbers}
while coord_neighbours:
coord, neighbours = coord_neighbours.popitem()
equiv_coords = [coord] + [c for c, ns in coord_neighbours.items()
if ns == neighbours]
for c in equiv_coords:
if c in coord_neighbours:
del(coord_neighbours[c])
self.groups.append(EquivGroup(equiv_coords, neighbours, self.dims))
def get_configs(self):
pass # WHAT GOES HERE?!
class Number(object):
"""Contains information about the group of cells around a number."""
def __init__(self, nr, coord, neighbours, dims):
"""Takes a number of mines, and a set of coordinates."""
self.nr = nr
self.coord = coord
# A list of the available neighbouring cells' coords.
self.neighbours = neighbours
self.dims = dims
def __repr__(self):
return "<Number {} with {} empty neighbours>".format(
int(self), len(self.neighbours))
def __str__(self):
grid = np.zeros(self.dims, int)
grid[self.coord] = int(self)
for coord in self.neighbours:
grid[coord] = 9
return str(grid).replace('0', '.').replace('9', '#')
def __int__(self):
return self.nr
class EquivGroup(object):
"""A group of cells which are effectively equivalent."""
def __init__(self, coords, nrs, dims):
self.coords = coords
# A list of the neighbouring Number objects.
self.nr_neighbours = nrs
self.dims = dims
if self.nr_neighbours:
self.max_mines = min(len(self.coords),
max(map(int, self.nr_neighbours)))
else:
self.max_mines = len(coords)
def __repr__(self):
return "<Equivalence group containing {} cells>".format(
len(self.coords))
def __str__(self):
grid = np.zeros(self.dims, int)
for coord in self.coords:
grid[coord] = 9
for number in self.nr_neighbours:
grid[number.coord] = int(number)
return str(grid).replace('0', '.').replace('9', '#')
grid = np.array(
[[0, 0, 0, 0],
[0, 2, 1, 0],
[0, 0, 0, 0]]
)
config = NrConfig(grid)
print config
print "Number groups:"
for n in config.numbers.values():
print n
print "Equivalence groups:"
for g in config.groups:
print g
If you don't want to brute-force it, you could model the process as a decision tree. Suppose we start with your example:
####
#21#
####
If we want to start placing mines in a valid configuration, we at this point essentially have eight choices. Since it doesn't really matter which square we pick within an equivalence group, we can narrow that down to three choices. The tree branches. Let's go down one branch:
*###
#11#
####
I placed a mine in G1, indicated by the asterisk. Also, I've updated the numbers (just one number in this case) associated with this equivalence group, to indicate that these numbered squares can now border one fewer mines.
This hasn't reduced our freedom of choice for the following step, we can still place a mine in any of the equivalence groups. Let's place another one in G1:
*XX#
*01#
XXX#
Another asterisk marks the new mine, and the numbered square has again been lowered by one. It has now reached zero, meaning it cannot border any more mines. That means that for our next choice of mine placement, all the equivalence groups dependent upon this numbered square are ruled out. Xs mark squares where we can now not place any mine. We can only make one choice now:
*XX*
*00X
XXXX
Here the branch ends and you've found a valid configuration. By running along all the branches in this tree in this manner, you should find all of them. Here we found your first configuration. Of course, there's more than one way to get there. If we had started by placing a mine in G3, we would have been forced to place the other two in G1. That branch leads to the same configuration, so you should check for duplicates. I don't see a way to avoid this redundancy right now.
The second configuration is found by either starting with G2, or placing one mine in G1 and then the second in G2. In either case you again end up at a branch end:
**XX
X00X
XXXX
Invalid configurations like your example with zero mines in G1 do not pop up. There are no valid choices along the tree that lead you there. Here is the whole tree of valid choices.
Choice 1: 1 | 2 | 3
Choice 2: 1 2 3 | 1 | 1
Choice 3: 3 1 | |1
Valid configurations are the branch ends at which no further choice is possible, i.e.
113
12
131
21
311
which obviously fall into two equivalent classes if we disregard the order of the numbers.
from wikipedia https://en.wikipedia.org/wiki/Stepwise_regression
Forward selection, which involves starting with no variables in the
model, testing the addition of each variable using a chosen model
comparison criterion, adding the variable (if any) that improves the
model the most, and repeating this process until none improves the
model.
I think that the implementation of this algorithm is very interesting because it can be seen as a combinatorial version of the hill climbing algorithm where the neighbours function is equivalent to add a variable to the current model.
I have not enough experience to write this algorithm in an optimized way. This is my current implementation:
class FSR():
def __init__(self, n_components):
self.n_components = n_components
def cost(self, index):
lr = LinearRegression().fit(self.x[:, index], self.y)
hat_y = lr.predict(self.x[:, index])
e = np.linalg.norm(hat_y - self.y)
return e
def next_step_fsr(self, comp, cand):
""" given the current components and candidates the function
return the new components, the new candidates and the new EV"""
if comp == []:
er = np.inf
else:
er = self.cost(comp)
for i in range(len(cand)):
e = cand.popleft()
comp.append(e)
new_er = self.cost(comp)
if new_er < er:
new_comp = comp.copy()
new_cand = deque(i for i in cand)
er = new_er
comp.pop()
cand.append(e)
return new_comp, new_cand, new_er
def fsr(self):
n, p = self.x.shape
er = []
comp = []
cand = deque(range(p))
for i in range(self.n_components):
comp, cand, new_er = self.next_step_fsr(comp, cand)
er.append(new_er)
return comp, er
def fit(self, x, y):
self.x = x
self.y = y
self.comp_, self.er_ = self.fsr()
I would like to know how can I improve the speed of this code.
x = np.random.normal(0,1, (100,20))
y = x[:,1] + x[:,2] + np.random.normal(0,.1, 100)
fsr = FSR(n_components=2)
fsr.fit(x,y)
print('selected component = ', fsr.comp_)
I want the final code too look not too different from the posted one. This is because I would like to extend the problem to different combinatorial problems with different cost function as well.
I think that the function that should be changed is next_step_fsr where given the current selected variables try which one is the best variable to include in the model. In particular I am interested in situation where x has a lot of columns (like 10000). I think that the current bottle neck is the line new_cand = deque(i for i in cand) where the list of candidates is copied.
I've written some python code to calculate a certain quantity from a cosmological simulation. It does this by checking whether a particle in contained within a box of size 8,000^3, starting at the origin and advancing the box when all particles contained within it are found. As I am counting ~2 million particles altogether, and the total size of the simulation volume is 150,000^3, this is taking a long time.
I'll post my code below, does anybody have any suggestions on how to improve it?
Thanks in advance.
from __future__ import division
import numpy as np
def check_range(pos, i, j, k):
a = 0
if i <= pos[2] < i+8000:
if j <= pos[3] < j+8000:
if k <= pos[4] < k+8000:
a = 1
return a
def sigma8(data):
N = []
to_do = data
print 'Counting number of particles per cell...'
for k in range(0,150001,8000):
for j in range(0,150001,8000):
for i in range(0,150001,8000):
temp = []
n = []
for count in range(len(to_do)):
n.append(check_range(to_do[count],i,j,k))
to_do[count][1] = n[count]
if to_do[count][1] == 0:
temp.append(to_do[count])
#Only particles that have not been found are
# searched for again
to_do = temp
N.append(sum(n))
print 'Next row'
print 'Next slice, %i still to find' % len(to_do)
print 'Calculating sigma8...'
if not sum(N) == len(data):
return 'Error!\nN measured = {0}, total N = {1}'.format(sum(N), len(data))
else:
return 'sigma8 = %.4f, variance = %.4f, mean = %.4f' % (np.sqrt(sum((N-np.mean(N))**2)/len(N))/np.mean(N), np.var(N),np.mean(N))
I'll try to post some code, but my general idea is the following: create a Particle class that knows about the box that it lives in, which is calculated in the __init__. Each box should have a unique name, which might be the coordinate of the bottom left corner (or whatever you use to locate your boxes).
Get a new instance of the Particle class for each particle, then use a Counter (from the collections module).
Particle class looks something like:
# static consts - outside so that every instance of Particle doesn't take them along
# for the ride...
MAX_X = 150,000
X_STEP = 8000
# etc.
class Particle(object):
def __init__(self, data):
self.x = data[xvalue]
self.y = data[yvalue]
self.z = data[zvalue]
self.compute_box_label()
def compute_box_label(self):
import math
x_label = math.floor(self.x / X_STEP)
y_label = math.floor(self.y / Y_STEP)
z_label = math.floor(self.z / Z_STEP)
self.box_label = str(x_label) + '-' + str(y_label) + '-' + str(z_label)
Anyway, I imagine your sigma8 function might look like:
def sigma8(data):
import collections as col
particles = [Particle(x) for x in data]
boxes = col.Counter([x.box_label for x in particles])
counts = boxes.most_common()
#some other stuff
counts will be a list of tuples which map a box label to the number of particles in that box. (Here we're treating particles as indistinguishable.)
Using list comprehensions is much faster than using loops---I think the reason is that you're basically relying more on the underlying C, but I'm not the person to ask. Counter is (supposedly) highly-optimized as well.
Note: None of this code has been tested, so you shouldn't try the cut-and-paste-and-hope-it-works method here.