Related
I've been learning and experimenting with decorators. I understand what they do: they allow you to write modular code by allowing you to add functionality to existing functions without changing them.
I found a great thread which really helped me learn how to do it by explaining all the ins and outs here: How to make a chain of function decorators?
But what is missing from this thread (and from other resources I've looked at) is WHY do we need the decorator syntax? Why do I need nested functions to create a decorator? Why can't I just take an existing function, write another one with some extra functionality, and feed the first into the 2nd to make it do something else?
Is it just because this is the convention? What am I missing? I assume my inexperience is showing here.
I will try to keep it simple and explain it with examples.
One of the things that I often do is measure the time taken by API's that I build and then publish them on AWS.
Since it is a very common use case I have created a decorator for it.
def log_latency():
def actual_decorator(f):
#wraps(f)
def wrapped_f(*args, **kwargs):
t0 = time()
r = f(*args, **kwargs)
t1 = time()
async_daemon_execute(public_metric, t1 - t0, log_key)
return r
return wrapped_f
return actual_decorator
Now if there is any method that I want to measure the latency, I just put annotate it with the required decorator.
#log_latency()
def batch_job_api(param):
pass
Suppose you want to write a secure API which only works if you send a header with a particular value then you can use a decorator for it.
def secure(f):
#wraps(f)
def wrapper(*args, **kw):
try:
token = request.headers.get("My_Secret_Token")
if not token or token != "My_Secret_Text":
raise AccessDenied("Required headers missing")
return f(*args, **kw)
return wrapper
Now just write
#secure
def my_secure_api():
pass
I have also been using the above syntax for API specific exceptions, and if a method needs a database interaction instead of acquiring a connection I use #session decorator which tells that this method will use a database connection and you don't need to handle one yourself.
I could have obviously avoided it, by writing a function that checks for the header or prints time taken by API on AWS, but that just looks a bit ugly and nonintuitive.
There is no convention for this, at least I am not aware of them but it definitely makes the code more readable and easier to manage.
Most of the IDE's also have different color syntax for annotation which makes it easier to understand and organize code.
So, I would it may just be because of inexperience; if you start using them you will automatically start knowing where to use them.
From PEP 318 -- Decorators for Functions and Methods (with my own added emphasis):
Motivation
The current method of applying a transformation to a
function or method places the actual transformation after the function
body. For large functions this separates a key component of the
function's behavior from the definition of the rest of the function's
external interface. For example:
def foo(self):
perform method operation
foo = classmethod(foo)
This becomes less readable with longer methods. It also seems less than pythonic to name
the function three times for what is conceptually a single
declaration. A solution to this problem is to move the transformation
of the method closer to the method's own declaration. The intent of
the new syntax is to replace
def foo(cls):
pass
foo = synchronized(lock)(foo)
foo = classmethod(foo)
with an alternative that places the decoration in the function's declaration:
#classmethod
#synchronized(lock)
def foo(cls):
pass
I came across a neatly-written article which explains decorators in Python, why we should leverage them, how to use them and more.
Some of the benefits of using decorators in Python include:
Allows us to extend functions without actually modifying them.
Helps us stick with the Open-Closed Principle.
It helps reduce code duplication -- we can factorize out repeated code into a decorator.
Decorators help keep a project organized and easy to maintain.
Reference: https://tomisin.dev/blog/understanding-decorators-in-python
A great example of why decorators are useful is the numba package.
It provides a decorator to speed up Python functions:
#jit
def my_slow_function():
# ...
The #jit decorator does some very amazing and complex operations - it compiles the whole function (well, parts of it) to machine code. If Python did not have decorators you would have to write the function in machine code yourself... or more seriously, the syntax would look like this:
def my_slow_function():
# ...
my_slow_function = jit(my_slow_function)
The purpose of the decorator syntax is to make the second example nicer. It's just syntactic sugar so you don't have to type the function name three times.
A pure function is a function similar to a Mathematical function, where there is no interaction with the "Real world" nor side-effects. From a more practical point of view, it means that a pure function can not:
Print or otherwise show a message
Be random
Depend on system time
Change global variables
And others
All this limitations make it easier to reason about pure functions than non-pure ones. The majority of the functions should then be pure so that the program can have less bugs.
In languages with a huge type-system like Haskell the reader can know right from the start if a function is or is not pure, making the successive reading easier.
In Python this information may be emulated by a #pure decorator put on top of the function. I would also like that decorator to actually do some validation work. My problem lies in the implementation of such a decorator.
Right now I simply look the source code of the function for buzzwords such as global or random or print and complains if it finds one of them.
import inspect
def pure(function):
source = inspect.getsource(function)
for non_pure_indicator in ('random', 'time', 'input', 'print', 'global'):
if non_pure_indicator in source:
raise ValueError("The function {} is not pure as it uses `{}`".format(
function.__name__, non_pure_indicator))
return function
However it feels like a weird hack, that may or may not work depending on your luck, could you please help me in writing a better decorator?
I kind of see where you are coming from but I don't think this can work. Let's take a simple example:
def add(a,b):
return a + b
So this probably looks "pure" to you. But in Python the + here is an arbitrary function which can do anything, just depending on the bindings in force when it is called. So that a + b can have arbitrary side effects.
But it's even worse than that. Even if this is just doing standard integer + then there's more 'impure' stuff going on.
The + is creating a new object. Now if you are sure that only the caller has a reference to that new object then there is a sense in which you can think of this as a pure function. But you can't be sure that, during the creation process of that object, no reference to it leaked out.
For example:
class RegisteredNumber(int):
numbers = []
def __new__(cls,*args,**kwargs):
self = int.__new__(cls,*args,**kwargs)
self.numbers.append(self)
return self
def __add__(self,other):
return RegisteredNumber(super().__add__(other))
c = RegisteredNumber(1) + 2
print(RegisteredNumber.numbers)
This will show that the supposedly pure add function has actually changed the state of the RegisteredNumber class. This is not a stupidly contrived example: in my production code base we have classes which track each created instance, for example, to allow access via key.
The notion of purity just doesn't make much sense in Python.
(not an answer, but too long for a comment)
So if a function can return different values for the same set of arguments, it is not pure?
Remember that functions in Python are objects, so you want to check the purity of an object...
Take this example:
def foo(x):
ret, foo.x = x*x+foo.x, foo.x+1
return ret
foo.x=0
calling foo(3) repeatedly gives:
>>> foo(3)
9
>>> foo(3)
10
>>> foo(3)
11
...
Moreover, reading globals does not require to use the global statement, or the global() builtin inside your function. Global variables might change somewhere else, affecting the purity of your function.
All the above situation might be difficult to detect at runtime.
In Python, to reduce clutter, is there a way to "Declare" members/methods of a class as "self" or a workaround.
class range:
#range_start # Declare as self
#range_end # Declare as self
def __init__(self,my_range_start,my_range_end):
range_start = my_range_start
range_end = my_range_end
def print_range(self):
print ("%x %x" % (range_start,range_end))
There isn't any straightforward way to do this. Also, if you did, you'd violate some rather deep set python conventions and your code would become much harder to read for other Python programmers. Too high a cost, in my opinion, to reduce visual clutter.
You can not achieve what you ask for in the way that you want, the idea of implicit self has been discussed and discarded by the python community.
However, concerning your code example, you can reduce the amount of code:
class range(tuple):
def print_range(self):
print("%x %x" % self)
Still, that code is bad, because adding a print_range() function is the wrong approach (why is it called print_range() and not print(), btw?). Instead, provide implementations of __str__() and __repr__() to format the object.
Some more notes:
There is a namedtuple() function that creates a type similar to a tuple, which you should perhaps take a look at.
Calling the object self is a convention, you can name it anything you like. Others won't easily be able to read your code though.
Concerning style, check out PEP 8. There is a bunch of conventions that you should adhere to, unless you have a good reason not to.
There is a useful Ruby idiom that uses tap which allows you to create an object, do some operations on it and return it (I use a list here only as an example, my real code is more involved):
def foo
[].tap do |a|
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a << b
end
end
>> foo
=> [1]
With Rails there's a similar method called returning, so you can write:
def foo
returning([]) do |a|
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a << b
end
end
which speaks for itself. No matter how much processing you do on the object, it's still clear that it's the return value of the function.
In Python I have to write:
def foo():
a = []
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a.append(b)
return a
and I wonder if there is a way to port this Ruby idiom into Python. My first thought was to use with statement, but return with is not valid syntax.
Short answer: Ruby encourages method chaining, Python doesn't.
I guess the right question is: What is Ruby's tap useful for?
Now I don't know a lot about Ruby, but by googling I got the impression that tap is conceptually useful as method chaining.
In Ruby, the style: SomeObject.doThis().doThat().andAnotherThing() is quite idiomatic. It underlies the concept of fluent interfaces, for example. Ruby's tap is a special case of this where instead of having SomeObject.doThis() you define doThis on the fly.
Why I am explaining all this? Because it tells us why tap doesn't have good support in Python. With due caveats, Python doesn't do call chaining.
For example, Python list methods generally return None rather than returning the mutated list. Functions like map and filter are not list methods. On the other hand, many Ruby array methods do return the modified array.
Other than certain cases like some ORMs, Python code doesn't use fluent interfaces.
In the end it is the difference between idiomatic Ruby and idiomatic Python. If you are going from one language to the other you need to adjust.
You can implement it in Python as follows:
def tap(x, f):
f(x)
return x
Usage:
>>> tap([], lambda x: x.append(1))
[1]
However it won't be so much use in Python 2.x as it is in Ruby because lambda functions in Python are quite restrictive. For example you can't inline a call to print because it is a keyword, so you can't use it for inline debugging code. You can do this in Python 3.x although it isn't as clean as the Ruby syntax.
>>> tap(2, lambda x: print(x)) + 3
2
5
If you want this bad enough, you can create a context manager
class Tap(object):
def __enter__(self, obj):
return obj
def __exit__(*args):
pass
which you can use like:
def foo():
with Tap([]) as a:
a.append(1)
return a
There's no getting around the return statement and with really doesn't do anything here. But you do have Tap right at the start which clues you into what the function is about I suppose. It is better than using lambdas because you aren't limited to expressions and can have pretty much whatever you want in the with statement.
Overall, I would say that if you want tap that bad, then stick with ruby and if you need to program in python, use python to write python and not ruby. When I get around to learning ruby, I intend to write ruby ;)
I had an idea to achieve this using function decorators, but due to the distinction in python between expressions and statements, this ended up still requiring the return to be at the end.
The ruby syntax is rarely used in my experience, and is far less readable than the explicit python approach. If python had implicit returns or a way to wrap multiple statements up into a single expression then this would be doable - but it has neither of those things by design.
Here's my - somewhat pointless - decorator approach, for reference:
class Tapper(object):
def __init__(self, initial):
self.initial = initial
def __call__(self, func):
func(self.initial)
return self.initial
def tap(initial):
return Tapper(initial)
if __name__ == "__main__":
def tapping_example():
#tap([])
def tapping(t):
t.append(1)
t.append(2)
return tapping
print repr(tapping_example())
I partly agree with others in that it doesn't make much sense to implement this in Python. However, IMHO, Mark Byers's way is the way, but why lambdas(and all that comes with them)? can't you write a separate function to be called when needed?
Another way to do basically the same could be
map(afunction(), avariable)
but this beautiful feature is not a built-in in Python 3, I hear.
Hardly any Ruby programmers use tap in this way. In fact, all top Ruby programmers i know say tap has no use except in debugging.
Why not just do this in your code?
[].push(1)
and remember Array supports a fluent interface, so you can even do this:
[].push(1).push(2)
This question already has answers here:
The Zen of Python [closed]
(22 answers)
Python: Am I missing something? [closed]
(16 answers)
Closed 8 years ago.
I would be interested in knowing what the StackOverflow community thinks are the important language features (idioms) of Python. Features that would define a programmer as Pythonic.
Python (pythonic) idiom - "code expression" that is natural or characteristic to the language Python.
Plus, Which idioms should all Python programmers learn early on?
Thanks in advance
Related:
Code Like a Pythonista: Idiomatic Python
Python: Am I missing something?
Python is a language that can be described as:
"rules you can fit in the
palm of your hand with a huge bag of
hooks".
Nearly everything in python follows the same simple standards. Everything is accessible, changeable, and tweakable. There are very few language level elements.
Take for example, the len(data) builtin function. len(data) works by simply checking for a data.__len__() method, and then calls it and returns the value. That way, len() can work on any object that implements a __len__() method.
Start by learning about the types and basic syntax:
Dynamic Strongly Typed Languages
bool, int, float, string, list, tuple, dict, set
statements, indenting, "everything is an object"
basic function definitions
Then move on to learning about how python works:
imports and modules (really simple)
the python path (sys.path)
the dir() function
__builtins__
Once you have an understanding of how to fit pieces together, go back and cover some of the more advanced language features:
iterators
overrides like __len__ (there are tons of these)
list comprehensions and generators
classes and objects (again, really simple once you know a couple rules)
python inheritance rules
And once you have a comfort level with these items (with a focus on what makes them pythonic), look at more specific items:
Threading in python (note the Global Interpreter Lock)
context managers
database access
file IO
sockets
etc...
And never forget The Zen of Python (by Tim Peters)
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
This page covers all the major python idioms: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
An important idiom in Python is docstrings.
Every object has a __doc__ attribute that can be used to get help on that object. You can set the __doc__ attribute on modules, classes, methods, and functions like this:
# this is m.py
""" module docstring """
class c:
"""class docstring"""
def m(self):
"""method docstring"""
pass
def f(a):
"""function f docstring"""
return
Now, when you type help(m), help(m.f) etc. it will print the docstring as a help message.
Because it's just part of normal object introspection this can be used by documention generating systems like epydoc or used for testing purposes by unittest.
It can also be put to more unconventional (i.e. non-idiomatic) uses such as grammars in Dparser.
Where it gets even more interesting to me is that, even though doc is a read-only attribute on most objects, you can use them anywhere like this:
x = 5
""" pseudo docstring for x """
and documentation tools like epydoc can pick them up and format them properly (as opposed to a normal comment which stays inside the code formatting.
Decorators get my vote. Where else can you write something like:
def trace(num_args=0):
def wrapper(func):
def new_f(*a,**k):
print_args = ''
if num_args > 0:
print_args = str.join(',', [str(x) for x in a[0:num_args]])
print('entering %s(%s)' %(f.__name__,print_args))
rc = f(*a,**k)
if rc is not None:
print('exiting %s(%s)=%s' %(f.__name__,str(rc)))
else:
print('exiting %s(%s)' %(f.__name__))
return rc
return new_f
return wrapper
#trace(1)
def factorial(n):
if n < 2:
return 1
return n * factorial(n-1)
factorial(5)
and get output like:
entering factorial(5)
entering factorial(4)
entering factorial(3)
entering factorial(2)
entering factorial(1)
entering factorial(0)
exiting factorial(0)=1
exiting factorial(1)=1
exiting factorial(2)=2
exiting factorial(3)=6
exiting factorial(4)=24
exiting factorial(5)=120
Everything connected to list usage.
Comprehensions, generators, etc.
Personally, I really like Python syntax defining code blocks by using indentation, and not by the words "BEGIN" and "END" (as in Microsoft's Basic and Visual Basic - I don't like these) or by using left- and right-braces (as in C, C++, Java, Perl - I like these).
This really surprised me because, although indentation has always been very important to me, I didn't make to much "noise" about it - I lived with it, and it is considered a skill to be able to read other peoples, "spaghetti" code. Furthermore, I never heard another programmer suggest making indentation a part of a language. Until Python! I only wish I had realized this idea first.
To me, it is as if Python's syntax forces you to write good, readable code.
Okay, I'll get off my soap-box. ;-)
From a more advanced viewpoint, understanding how dictionaries are used internally by Python. Classes, functions, modules, references are all just properties on a dictionary. Once this is understood it's easy to understand how to monkey patch and use the powerful __gettattr__, __setattr__, and __call__ methods.
Here's one that can help. What's the difference between:
[ foo(x) for x in range(0, 5) ][0]
and
( foo(x) for x in range(0, 5) ).next()
answer:
in the second example, foo is called only once. This may be important if foo has a side effect, or if the iterable being used to construct the list is large.
Two things that struck me as especially Pythonic were dynamic typing and the various flavors of lists used in Python, particularly tuples.
Python's list obsession could be said to be LISP-y, but it's got its own unique flavor. A line like:
return HandEvaluator.StraightFlush, (PokerCard.longFaces[index + 4],
PokerCard.longSuits[flushSuit]), []
or even
return False, False, False
just looks like Python and nothing else. (Technically, you'd see the latter in Lua as well, but Lua is pretty Pythonic in general.)
Using string substitutions:
name = "Joe"
age = 12
print "My name is %s, I am %s" % (name, age)
When I'm not programming in python, that simple use is what I miss most.
Another thing you cannot start early enough is probably testing. Here especially doctests are a great way of testing your code by explaining it at the same time.
doctests are simple text file containing an interactive interpreter session plus text like this:
Let's instantiate our class::
>>> a=Something(text="yes")
>>> a.text
yes
Now call this method and check the results::
>>> a.canify()
>>> a.text
yes, I can
If e.g. a.text returns something different the test will fail.
doctests can be inside docstrings or standalone textfiles and are executed by using the doctests module. Of course the more known unit tests are also available.
I think that tutorials online and books only talk about doing things, not doing things in the best way. Along with the python syntax i think that speed in some cases is important.
Python provides a way to benchmark functions, actually two!!
One way is to use the profile module, like so:
import profile
def foo(x, y, z):
return x**y % z # Just an example.
profile.run('foo(5, 6, 3)')
Another way to do this is to use the timeit module, like this:
import timeit
def foo(x, y, z):
return x**y % z # Can also be 'pow(x, y, z)' which is way faster.
timeit.timeit('foo(5, 6, 3)', 'from __main__ import *', number = 100)
# timeit.timeit(testcode, setupcode, number = number_of_iterations)