I was trying to implement a cost function for a Programming assignment in Andrew Ng Deep Learning course which requires my own, original work. I am also not allowed to reproduce the assignment code without permission, but am doing so anyway in this question.
The expected result for the cost = 6.000064773192205, But with this code, my result for cost = 4.50006477319. Does anyone have any idea what I did wrong in this code?
removed code
There is an error in your sigmoid function. You are supposed to calculate negative of np.dot(np.transpose(w), X) + b).
Here is the one I have used
A = 1 / (1 + np.exp(-(np.dot(np.transpose(w), X) + b)))
np.sum(np.multiply(Y, np.log(A)) + np.multiply((1-Y), np.log(1-A))) /m
Just in case you find it useful (and as I'm doing the same source), you could also have called the sigmoid() function you defined in the previous step from within propagate() by using this instead:
A = sigmoid(np.dot(w.T,X) + b)
It's not necessary as evidenced by your work, but it's a bit cleaner.
Related
I'm working on a dynamic programming task of finding a minimum cost path along a directed graph (all possible paths have the same number of weighted nodes).
The approach for solving the problem is a recursive function along with a dynamic programming.
Since this dynamic programming task is encountered in many unrelated problems during the code, the concept of threading could be helpful.
The problem is, that in python, 'threading' won't help much.
what are efficient ways of handling such a task in python?
Here's the code:
def rec_fun(pos, path_size, weights, directions):
cost = weights[d][i, j]
if path_size == 0:
key = str(i) + ',' + str(j) + ',' + str(d)
dict.update({key: pix_cost})
return cost
else:
key = str(i) + ',' + str(j) + ',' + str(d)
if key in dict:
return dict[key]
else:
val = cost + min(rec_fun(pos + direction[0], path_size - 1, weights, direction),
rec_fun(pos + direction[1], path_size - 1, weights, direction),
rec_fun(pos + direction[2], path_size - 1, weights, direction))
dict.update({key: val})
return val
So first off, dynamic programming is just a simple paradigm for solving a certain type of problem. There is not really something specific you can do to optimize dynamic programming in general, in turn that means general python optimizations apply.
So from the code you posted the most striking thing I see is the use of recursions, this is relatively inefficient in python, so start off by moving to for (ideal) or while loops.
Following is a non-exhaustive list of possible ways to make your code run faster in python (with increasing effort):
Try Numba to speed your functions up significantly. It requires very little work (often the #jit decorator is enough) to optimize your code to almost cython level in some cases.
Make use of Numpy to vectorize your code where possible.
Use Processes multiprocessing.Process instead of Threads, as you probably already figured, python threads don't work the same way as in other programming languages due to the global interpreter lock. I think here it's important to note that sharing memory between processes is not really a good practice and you should avoid that if anyhow possible.
Use Python's C-Extension Cython to write all performance critical parts of your code.
With Python 3, you can easily achieve dynamic programming by caching the results of recursive calls using lru_cache from functools.
You can wrap your function as such:
#functools.lru_cache(max_size=None)
def rec_fun(pos, path_size, weights, directions):
# your code...
Note: Since this caches all calls so it could store more results than you necessarily need compared to implementing DP with your own array or list.
I've read the official documenation and read the comments here https://github.com/Theano/theano/blob/ddfd7d239a1e656cee850cdbc548da63f349c37d/theano/compile/function.py#L74-L324, and one man told me that it tells Theano to compile the symbolic computation graph into an actual program that you can run.
However, I still cannot figure out how does it know, for example in this code:
self.update_fun = theano.function(
inputs=[self.input_mat, self.output_mat],
outputs=self.cost,
updates=updates,
allow_input_downcast=True)
how to compute all that, if it has no body? I mean all those things are computed in some code above these pasted lines, but... is theano.function actually looking to source code to find out how to compute those things? I'm just guessing and would really like to know how it works.
Maybe the problem I have in the explanation that "it tells Theano to compile the symbolic computation graph into an actual program" is that I have no clue what symbolic computation graph is, so that would be another question very related to the previous.
Explanation would be appreciated.
I'm no expert but here's my take at explaining it:
Yes the 'body' is defined in the code above. But theano doesn't 'interpret' that code directly like the python interpreter would. The code in question is just creating theano objects that will allow theano to compile the desired function. Let's take a simple example: how you would create a function f(x) = 2x + x**3.
You first create a symbolic input variable x. Then you define the 'body' of the function by building the symbolic expression of f(x):
y = 2 * x + x**3 # defines a new symbolic variable which depends on x
This y object is equivalent to a graph representing the formula. Something like Plus(Times(2,x), Power(x,3)).
You finally call theano.function with input=x and output=y. Then theano does its magic and compiles the actual function f(x) = y = 2 * x + x**3 from the information (the graph) 'contained' in y.
Does it make things clearer?
I don't know how to phrase this, but my current structure uses recursion. So I call upon the same function when the looping within the function is finished, and use the results of the loop as an argument to continue the program with new data to work through.
So this function of mine spits out a number, among other things. I've assigned this resulting number from the function to a variable.
My gap in knowledge is however coming up short on the thought process/logic to a solution. Although it sounds simple. I've thought about maybe a way to over ride the original information with a global variable of the same name, but that doesn't seem to over ride it since at the top of the function is the variable declaration. I've also thought about using arguments with the *args method. But that isn't working either.
My function, has a loop in it. That requires variable X for example. On the loops first run, X already has a base number to work with. This X is used in the loop as part of the equation, after all the loops run through, I get a result and assign that to variable Y.
I need to essentially add Y to X, essentially making X = X + Y and then recall the function.
Trying this line of thinking, and declaring an updated variable doesn't work with this method because when I recall the function, it updates it to the base number again.
I've also explored if statements checking if the number is X and declaring the updated variable in the else statement otherwise. But that too is not providing a fruitful outcome.
What line of thinking, or method should I be applying to solve this problem?
I can explain more if necessary, me being a new to Python I've come up short on the search for the answer before posting, maybe because I don't know the terminology for this type of method to find such answers.
I appreciate your time in reading this, thanks!
def loop1Yo(f):
matrixDict = {}
timer = 0
x = f[0]
for element in f[1:]:
depTime = 1479215120 + xyzDuration ###(This part wouldnt work but I think explains what I am trying to do.)
directionResult = (depTime) + (element)
sortedMatrixList = sorted(matrixDict.items(), key=operator.itemgetter(1))
xyz = sortedMatrixList
xyzDuration = sortedMatrixList[0][1]
f = [x for x,_ in xyz]
print (f)
loop1Yo(f)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 months ago.
Improve this question
Functional programming is one of the programming paradigms in Python. It treats computation as the evaluation of mathematical functions and avoids state and mutable data. I am trying to understand how Python incorporates functional programming.
Consider the following factorial program (factorial.py):
def factorial(n, total):
if n == 0:
return total
else:
return factorial(n-1, total*n)
num = raw_input("Enter a natural number: ")
print factorial(int(num), 1)
That code avoids mutable data, because we are not changing the value of any variable. We are only recursively calling the factorial function with a new value.
If the example given above for functional programming is correct, then what does avoiding state mean?
Does functional programming only mean that I must use only functions whenever I have computations (as given in the above example)?
If the given example is wrong, what is a simple example with an explanation?
The example is correct for functional programming. And a good example of what not to do in Python because it is inefficient and doesn't scale. Python doesn't have any tail-call optimisation, so recursive calls should not be used solely to avoid imperative loops. If you really start programming in this style in Python, your programs will end with runtime errors eventually.
You are describing pure functional programming which is not something Python could be used for.
Python supports functional programming to some degree in the sense that functions are first class values. That means functions can be passed to other functions and returned as results from functions. And the standard library contains functions also found in most functional programming languages standard libraries, like map(), filter(), reduce(), and the stuff in the functools and itertools modules.
Borrowing from http://ua.pycon.org/static/talks/kachayev/#/8 where he makes a comparison between the way one thinks of imperative and functional programs. The example is borrowed.
Imperative:
expr, res = "28+32+++32++39", 0
for t in expr.split("+"):
if t != "":
res += int(t)
print res
Functional:
from operator import add
expr = "28+32+++32++39"
print reduce(add, map(int, filter(bool, expr.split("+"))))
If the given example is wrong, then kindly provide another simple example with an explanation.
It’s not, and it shows a realistic problem to start with, which you’ll see if you call factorial with a huge number. The maximum recursion depth will be reached. Python doesn’t have tail call optimization.
It the example given above for functional programming is correct, then what does avoiding state mean.
It means (in Python) that once a variable had been assigned, you should not reassign a new value, or change the value that you assigned to that variable.
Secondly, does functional programming only mean that, I must use only functions whenever I have computations (as given in the above example)
Functional programming is quite broad. Python is a multi-paradigm language that supports some functional programming concepts.
Functional programming means that all computations should be seen as mathematical functions.
I wrote a post about it that explains all the above in greater detail: Use Functional Programming In Python
Basics
Functional programming is a paradigm where we use only expressions and no statements. Statements do something like an if, whereas expressions evaluate mathematical stuff. We try to avoid mutability (values getting changed), since we only want pure functions. Pure functions don't have side effects, meaning that a function, given the same input, always produces the same output. We want functions to be pure, since they are easier to debug. By doing this, we are describing what a function is, as opposed to giving steps about how to do something and we write a lot less code. Functional programming is often called declarative, while other approaches are called imperative.
Utilities
Know about built-in functions, since they are so useful. For starters: abs, round, pow, max & min, sum, len, sorted, reversed, zip, and range.
Lambdas
Anonymous functions or lambdas are pure functions that take input and produce a value. There isn't any return statement, since we are returning what we are evaluating. You can also give them a name by just declaring a variable.
(lambda x, y: x + y)(1, 1)
add = lambda x, y: x + y
add(1, 1)
Ternary operator
Since we are working with expressions, instead of using if statements for logic, we use the ternary operator.
(expression) if (condition) else (expression2)
Map, filter & reduce
We also need a way of looping. This comes with list comprehension.
In this example we add one to each array element.
inc = lambda x: [i + 1 for i in x]
We can also operate on elements that meet a condition.
evens = lambda x: [i for i in x if (i % 2) == 0]
Some people say that that is the correct Pythonic way of doing things. But people who are more serious use a different approach: map, filter, and reduce. These are the building blocks of functional programming. While map and filter are built-in functions, reduce used to be, but now is in the functools module. To use it, import it:
from functools import reduce
Map is a function that takes a function and calls it on the elements of an array. The result of a map function is unfortunately unreadable, so you need to turn it into a tuple or a collection.
inc = lambda x: tuple(map(lambda y: y + 1, x))
Filter is a function that calls a function on an array, keeps the elements that output True and removes the ones that are False. Like map, it's not readable.
evens = lambda x: tuple(filter(lambda y: (y % 2) == 0, x))
Reduce takes a function that has two parameters. One is the result of the last time it was called & one is the new value. It keeps doing this until it reduces down to a value.
from functools import reduce
m_sum = lambda x: reduce(lambda y, z: y + z, x)
Let and do
A lot of functional programming languages have do. Do is a function that takes n arguments, evaluates all of them and returns the value of the last one. Python doesn't have do, but I created one myself.
do = lambda *args: tuple(map(lambda y: y, args))[-1]
Here's an example using do that prints something and exits.
print_err = lambda x: do(print(x), exit())
We use do to get imperative advantages.
Let allows some expression to be equal to something in a expression. In Python, most people do that as follows.
def something(x):
y = x + 10
return y * 3
Python 3.8 adds the expression assignment operators :=. So that code can now be written as a lambda.
something = lambda x: do(y := x + 10, y * 3)
Working with impure systems
The console, the file system, the web, etc. are immutable and impure and we need a way to work with them. In some languages like Haskell you have monads, which are wrappers for functions. Clojure allows the use of impure functions. In Python, a multi-paradigm language, we don't need anything, since it's not just functional. Print is an impure function.
Recursion
Your example uses recursion. Recursion uses the call stack, which is a small chunk of memory that is used for calling functions. It can get full and crash. A lot of functional programming languages use something known as lazy evaluation to help with the problem of recursion, but Python doesn't have it. So avoid recursion as much as possible.
Answers
Avoiding state is being immutable, meaning not changing the value of something.
In functional programming, everything is an expression, which is a function. We don't have that in Python though, since it's multi-paradigm.
In a more functional way, your code would be written as follows. Also it's not functional due to using an if statement instead of a ternary expression.
from functools import reduce
factorial = lambda x: reduce(lambda y, z: y*z, range(1,x+1))
This produces a range of 1 to x and multiplies all the values using range.
I was looking at an article on Peter Norvig's website, where he's trying to answer the following question (this is not my question, btw)
"Can I do the equivalent of (test ? result : alternative) in Python?"
here's one of the options listed by him,
def if_(test, result, alternative=None):
"If test is true, 'do' result, else alternative. 'Do' means call if callable."
if test:
if callable(result): result = result()
return result
else:
if callable(alternative): alternative = alternative()
return alternative
And here's a usage example.
>>> fact = lambda n: if_(n <= 1, 1, lambda: n * fact(n-1))
>>> fact(6)
720
I understand how this works (I think), but I was just playing with the code, and decided to see what happens when I change the third argument in the definition of 'fact' above to
n * fact(n-1), that is, change it to a non-callable expression. On running it, the interpreter goes into a never ending loop. I have a pretty good idea of why that is happening, that is, the if_ function is returning back the same expression that it is receiving. But what is the type of that expression? What exactly is going on here? I am not looking for a detailed explanation , but just for some pointers to python's evaluation model which might help my understanding.
Thanks!
The reason the loop never terminates when you change fact to n * fact(n-1) is that n * fact(n-1) has to evaluate first (as the third argument to if). Evaluating it leads to another call to fact, ad infinitum (since there is no longer any base case to stop it).
Previously, you were passing a function object (lambda), which would not be evaluated until the body of if, and its result would be checked via test.
This is known (I believe) as eager evaluation, where function arguments are evaluated before they are passed to the function. In a lazy-evaluation scheme, the arguments would not be evaluated until they were used in the function body.