We Keep Coding

What does Python's # operator do? [duplicate]

What does the # symbol do in Python?

An # symbol at the beginning of a line is used for class and function decorators:
PEP 318: Decorators
Python Decorators
The most common Python decorators are:
#property
#classmethod
#staticmethod
An # in the middle of a line is probably matrix multiplication:
# as a binary operator.

Example
class Pizza(object):
def __init__(self):
self.toppings = []
def __call__(self, topping):
# When using '#instance_of_pizza' before a function definition
# the function gets passed onto 'topping'.
self.toppings.append(topping())
def __repr__(self):
return str(self.toppings)
pizza = Pizza()
#pizza
def cheese():
return 'cheese'
#pizza
def sauce():
return 'sauce'
print pizza
# ['cheese', 'sauce']
This shows that the function/method/class you're defining after a decorator is just basically passed on as an argument to the function/method immediately after the # sign.
First sighting
The microframework Flask introduces decorators from the very beginning in the following format:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
This in turn translates to:
rule = "/"
view_func = hello
# They go as arguments here in 'flask/app.py'
def add_url_rule(self, rule, endpoint=None, view_func=None, **options):
pass
Realizing this finally allowed me to feel at peace with Flask.

In Python 3.5 you can overload # as an operator. It is named as __matmul__, because it is designed to do matrix multiplication, but it can be anything you want. See PEP465 for details.
This is a simple implementation of matrix multiplication.
class Mat(list):
def __matmul__(self, B):
A = self
return Mat([[sum(A[i][k]*B[k][j] for k in range(len(B)))
for j in range(len(B[0])) ] for i in range(len(A))])
A = Mat([[1,3],[7,5]])
B = Mat([[6,8],[4,2]])
print(A # B)
This code yields:
[[18, 14], [62, 66]]

This code snippet:
def decorator(func):
return func
#decorator
def some_func():
pass
Is equivalent to this code:
def decorator(func):
return func
def some_func():
pass
some_func = decorator(some_func)
In the definition of a decorator you can add some modified things that wouldn't be returned by a function normally.

What does the “at” (#) symbol do in Python?
In short, it is used in decorator syntax and for matrix multiplication.
In the context of decorators, this syntax:
#decorator
def decorated_function():
"""this function is decorated"""
is equivalent to this:
def decorated_function():
"""this function is decorated"""
decorated_function = decorator(decorated_function)
In the context of matrix multiplication, a # b invokes a.__matmul__(b) - making this syntax:
a # b
equivalent to
dot(a, b)
and
a #= b
equivalent to
a = dot(a, b)
where dot is, for example, the numpy matrix multiplication function and a and b are matrices.
How could you discover this on your own?
I also do not know what to search for as searching Python docs or Google does not return relevant results when the # symbol is included.
If you want to have a rather complete view of what a particular piece of python syntax does, look directly at the grammar file. For the Python 3 branch:
~$ grep -C 1 "#" cpython/Grammar/Grammar
decorator: '#' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
--
testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
augassign: ('+=' | '-=' | '*=' | '#=' | '/=' | '%=' | '&=' | '|=' | '^=' |
'<<=' | '>>=' | '**=' | '//=')
--
arith_expr: term (('+'|'-') term)*
term: factor (('*'|'#'|'/'|'%'|'//') factor)*
factor: ('+'|'-'|'~') factor | power
We can see here that # is used in three contexts:
decorators
an operator between factors
an augmented assignment operator
Decorator Syntax:
A google search for "decorator python docs" gives as one of the top results, the "Compound Statements" section of the "Python Language Reference." Scrolling down to the section on function definitions, which we can find by searching for the word, "decorator", we see that... there's a lot to read. But the word, "decorator" is a link to the glossary, which tells us:
decorator
A function returning another function, usually applied as a function transformation using the #wrapper syntax. Common
examples for decorators are classmethod() and staticmethod().
The decorator syntax is merely syntactic sugar, the following two
function definitions are semantically equivalent:
def f(...):
...
f = staticmethod(f)
#staticmethod
def f(...):
...
The same concept exists for classes, but is less commonly used there.
See the documentation for function definitions and class definitions
for more about decorators.
So, we see that
#foo
def bar():
pass
is semantically the same as:
def bar():
pass
bar = foo(bar)
They are not exactly the same because Python evaluates the foo expression (which could be a dotted lookup and a function call) before bar with the decorator (#) syntax, but evaluates the foo expression after bar in the other case.
(If this difference makes a difference in the meaning of your code, you should reconsider what you're doing with your life, because that would be pathological.)
Stacked Decorators
If we go back to the function definition syntax documentation, we see:
#f1(arg)
#f2
def func(): pass
is roughly equivalent to
def func(): pass
func = f1(arg)(f2(func))
This is a demonstration that we can call a function that's a decorator first, as well as stack decorators. Functions, in Python, are first class objects - which means you can pass a function as an argument to another function, and return functions. Decorators do both of these things.
If we stack decorators, the function, as defined, gets passed first to the decorator immediately above it, then the next, and so on.
That about sums up the usage for # in the context of decorators.
The Operator, #
In the lexical analysis section of the language reference, we have a section on operators, which includes #, which makes it also an operator:
The following tokens are operators:
+ - * ** / // % #
<< >> & | ^ ~
< > <= >= == !=
and in the next page, the Data Model, we have the section Emulating Numeric Types,
object.__add__(self, other)
object.__sub__(self, other)
object.__mul__(self, other)
object.__matmul__(self, other)
object.__truediv__(self, other)
object.__floordiv__(self, other)
[...]
These methods are called to implement the binary arithmetic operations (+, -, *, #, /, //, [...]
And we see that __matmul__ corresponds to #. If we search the documentation for "matmul" we get a link to What's new in Python 3.5 with "matmul" under a heading "PEP 465 - A dedicated infix operator for matrix multiplication".
it can be implemented by defining __matmul__(), __rmatmul__(), and
__imatmul__() for regular, reflected, and in-place matrix multiplication.
(So now we learn that #= is the in-place version). It further explains:
Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of # allows
writing cleaner code:
S = (H # beta - r).T # inv(H # V # H.T) # (H # beta - r)
instead of:
S = dot((dot(H, beta) - r).T,
dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
While this operator can be overloaded to do almost anything, in numpy, for example, we would use this syntax to calculate the inner and outer product of arrays and matrices:
>>> from numpy import array, matrix
>>> array([[1,2,3]]).T # array([[1,2,3]])
array([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> array([[1,2,3]]) # array([[1,2,3]]).T
array([[14]])
>>> matrix([1,2,3]).T # matrix([1,2,3])
matrix([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> matrix([1,2,3]) # matrix([1,2,3]).T
matrix([[14]])
Inplace matrix multiplication: #=
While researching the prior usage, we learn that there is also the inplace matrix multiplication. If we attempt to use it, we may find it is not yet implemented for numpy:
>>> m = matrix([1,2,3])
>>> m #= m.T
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: In-place matrix multiplication is not (yet) supported. Use 'a = a # b' instead of 'a #= b'.
When it is implemented, I would expect the result to look like this:
>>> m = matrix([1,2,3])
>>> m #= m.T
>>> m
matrix([[14]])

What does the “at” (#) symbol do in Python?
# symbol is a syntactic sugar python provides to utilize decorator,
to paraphrase the question, It's exactly about what does decorator do in Python?
Put it simple decorator allow you to modify a given function's definition without touch its innermost (it's closure).
It's the most case when you import wonderful package from third party. You can visualize it, you can use it, but you cannot touch its innermost and its heart.
Here is a quick example,
suppose I define a read_a_book function on Ipython
In [9]: def read_a_book():
...: return "I am reading the book: "
...:
In [10]: read_a_book()
Out[10]: 'I am reading the book: '
You see, I forgot to add a name to it.
How to solve such a problem? Of course, I could re-define the function as:
def read_a_book():
return "I am reading the book: 'Python Cookbook'"
Nevertheless, what if I'm not allowed to manipulate the original function, or if there are thousands of such function to be handled.
Solve the problem by thinking different and define a new_function
def add_a_book(func):
def wrapper():
return func() + "Python Cookbook"
return wrapper
Then employ it.
In [14]: read_a_book = add_a_book(read_a_book)
In [15]: read_a_book()
Out[15]: 'I am reading the book: Python Cookbook'
Tada, you see, I amended read_a_book without touching it inner closure. Nothing stops me equipped with decorator.
What's about #
#add_a_book
def read_a_book():
return "I am reading the book: "
In [17]: read_a_book()
Out[17]: 'I am reading the book: Python Cookbook'
#add_a_book is a fancy and handy way to say read_a_book = add_a_book(read_a_book), it's a syntactic sugar, there's nothing more fancier about it.

If you are referring to some code in a python notebook which is using Numpy library, then # operator means Matrix Multiplication. For example:
import numpy as np
def forward(xi, W1, b1, W2, b2):
z1 = W1 # xi + b1
a1 = sigma(z1)
z2 = W2 # a1 + b2
return z2, a1

Decorators were added in Python to make function and method wrapping (a function that receives a function and returns an enhanced one) easier to read and understand. The original use case was to be able to define the methods as class methods or static methods on the head of their definition. Without the decorator syntax, it would require a rather sparse and repetitive definition:
class WithoutDecorators:
def some_static_method():
print("this is static method")
some_static_method = staticmethod(some_static_method)
def some_class_method(cls):
print("this is class method")
some_class_method = classmethod(some_class_method)
If the decorator syntax is used for the same purpose, the code is shorter and easier to understand:
class WithDecorators:
#staticmethod
def some_static_method():
print("this is static method")
#classmethod
def some_class_method(cls):
print("this is class method")
General syntax and possible implementations
The decorator is generally a named object ( lambda expressions are not allowed) that accepts a single argument when called (it will be the decorated function) and returns another callable object. "Callable" is used here instead of "function" with premeditation. While decorators are often discussed in the scope of methods and functions, they are not limited to them. In fact, anything that is callable (any object that implements the _call__ method is considered callable), can be used as a decorator and often objects returned by them are not simple functions but more instances of more complex classes implementing their own __call_ method.
The decorator syntax is simply only a syntactic sugar. Consider the following decorator usage:
#some_decorator
def decorated_function():
pass
This can always be replaced by an explicit decorator call and function reassignment:
def decorated_function():
pass
decorated_function = some_decorator(decorated_function)
However, the latter is less readable and also very hard to understand if multiple decorators are used on a single function.
Decorators can be used in multiple different ways as shown below:
As a function
There are many ways to write custom decorators, but the simplest way is to write a function that returns a subfunction that wraps the original function call.
The generic patterns is as follows:
def mydecorator(function):
def wrapped(*args, **kwargs):
# do some stuff before the original
# function gets called
result = function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
# return wrapper as a decorated function
return wrapped
As a class
While decorators almost always can be implemented using functions, there are some situations when using user-defined classes is a better option. This is often true when the decorator needs complex parametrization or it depends on a specific state.
The generic pattern for a nonparametrized decorator as a class is as follows:
class DecoratorAsClass:
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
# do some stuff before the original
# function gets called
result = self.function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
Parametrizing decorators
In real code, there is often a need to use decorators that can be parametrized. When the function is used as a decorator, then the solution is simple—a second level of wrapping has to be used. Here is a simple example of the decorator that repeats the execution of a decorated function the specified number of times every time it is called:
def repeat(number=3):
"""Cause decorated function to be repeated a number of times.
Last value of original function call is returned as a result
:param number: number of repetitions, 3 if not specified
"""
def actual_decorator(function):
def wrapper(*args, **kwargs):
result = None
for _ in range(number):
result = function(*args, **kwargs)
return result
return wrapper
return actual_decorator
The decorator defined this way can accept parameters:
>>> #repeat(2)
... def foo():
... print("foo")
...
>>> foo()
foo
foo
Note that even if the parametrized decorator has default values for its arguments, the parentheses after its name is required. The correct way to use the preceding decorator with default arguments is as follows:
>>> #repeat()
... def bar():
... print("bar")
...
>>> bar()
bar
bar
bar
Finally lets see decorators with Properties.
Properties
The properties provide a built-in descriptor type that knows how to link an attribute to a set of methods. A property takes four optional arguments: fget , fset , fdel , and doc . The last one can be provided to define a docstring that is linked to the attribute as if it were a method. Here is an example of a Rectangle class that can be controlled either by direct access to attributes that store two corner points or by using the width , and height properties:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
def _width_get(self):
return self.x2 - self.x1
def _width_set(self, value):
self.x2 = self.x1 + value
def _height_get(self):
return self.y2 - self.y1
def _height_set(self, value):
self.y2 = self.y1 + value
width = property(
_width_get, _width_set,
doc="rectangle width measured from left"
)
height = property(
_height_get, _height_set,
doc="rectangle height measured from top"
)
def __repr__(self):
return "{}({}, {}, {}, {})".format(
self.__class__.__name__,
self.x1, self.y1, self.x2, self.y2
)
The best syntax for creating properties is using property as a decorator. This will reduce the number of method signatures inside of the class
and make code more readable and maintainable. With decorators the above class becomes:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
#property
def width(self):
"""rectangle height measured from top"""
return self.x2 - self.x1
#width.setter
def width(self, value):
self.x2 = self.x1 + value
#property
def height(self):
"""rectangle height measured from top"""
return self.y2 - self.y1
#height.setter
def height(self, value):
self.y2 = self.y1 + value

Starting with Python 3.5, the '#' is used as a dedicated infix symbol for MATRIX MULTIPLICATION (PEP 0465 -- see https://www.python.org/dev/peps/pep-0465/)

# can be a math operator or a DECORATOR but what you mean is a decorator.
This code:
def func(f):
return f
func(lambda :"HelloWorld")()
using decorators can be written like:
def func(f):
return f
#func
def name():
return "Hello World"
name()
Decorators can have arguments.
You can see this GeeksforGeeks post: https://www.geeksforgeeks.org/decorators-in-python/

It indicates that you are using a decorator. Here is Bruce Eckel's example from 2008.

Python decorator is like a wrapper of a function or a class. It’s still too conceptual.
def function_decorator(func):
def wrapped_func():
# Do something before the function is executed
func()
# Do something after the function has been executed
return wrapped_func
The above code is a definition of a decorator that decorates a function.
function_decorator is the name of the decorator.
wrapped_func is the name of the inner function, which is actually only used in this decorator definition. func is the function that is being decorated.
In the inner function wrapped_func, we can do whatever before and after the func is called. After the decorator is defined, we simply use it as follows.
#function_decorator
def func():
pass
Then, whenever we call the function func, the behaviours we’ve defined in the decorator will also be executed.
EXAMPLE :
from functools import wraps
def mydecorator(f):
#wraps(f)
def wrapped(*args, **kwargs):
print "Before decorated function"
r = f(*args, **kwargs)
print "After decorated function"
return r
return wrapped
#mydecorator
def myfunc(myarg):
print "my function", myarg
return "return value"
r = myfunc('asdf')
print r
Output :
Before decorated function
my function asdf
After decorated function
return value

To say what others have in a different way: yes, it is a decorator.
In Python, it's like:
Creating a function (follows under the # call)
Calling another function to operate on your created function. This returns a new function. The function that you call is the argument of the #.
Replacing the function defined with the new function returned.
This can be used for all kinds of useful things, made possible because functions are objects and just necessary just instructions.

# symbol is also used to access variables inside a plydata / pandas dataframe query, pandas.DataFrame.query.
Example:
df = pandas.DataFrame({'foo': [1,2,15,17]})
y = 10
df >> query('foo > #y') # plydata
df.query('foo > #y') # pandas

Non-strict by-name arguments in Python?

Question
Is there any way to declare function arguments as non-strict (passed by-name)?
If this is not possible directly: are there any helper functions or decorators that help me achieve something similar?
Concrete example
Here is a littly toy-example to experiment with.
Suppose that I want to build a tiny parser-combinator library that can cope with the following classic grammar for arithmetic expressions with parentheses (numbers replaced by a single literal value 1 for simplicity):
num = "1"
factor = num
| "(" + expr + ")"
term = factor + "*" + term
| factor
expr = term + "+" + expr
| term
Suppose that I define a parser combinator as an object that has a method parse that can take list of tokens, current position, and either throw a parse error, or return a result and a new position. I can nicely define a ParserCombinator base class that provides + (concatenation) and | (alternative). Then I can define parser combinators that accept constant strings, and implement + and |:
# Two kinds of errors that can be thrown by a parser combinator
class UnexpectedEndOfInput(Exception): pass
class ParseError(Exception): pass
# Base class that provides methods for `+` and `|` syntax
class ParserCombinator:
def __add__(self, next):
return AddCombinator(self, next)
def __or__(self, other):
return OrCombinator(self, other)
# Literally taken string constants
class Lit(ParserCombinator):
def __init__(self, string):
self.string = string
def parse(self, tokens, pos):
if pos < len(tokens):
t = tokens[pos]
if t == self.string:
return t, (pos + 1)
else:
raise ParseError
else:
raise UnexpectedEndOfInput
def lit(str):
return Lit(str)
# Concatenation
class AddCombinator(ParserCombinator):
def __init__(self, first, second):
self.first = first
self.second = second
def parse(self, tokens, pos):
x, p1 = self.first.parse(tokens, pos)
y, p2 = self.second.parse(tokens, p1)
return (x, y), p2
# Alternative
class OrCombinator(ParserCombinator):
def __init__(self, first, second):
self.first = first
self.second = second
def parse(self, tokens, pos):
try:
return self.first.parse(tokens, pos)
except:
return self.second.parse(tokens, pos)
So far, everything is fine. However, because the non-terminal symbols of the grammar are defined in a mutually recursive fashion, and I cannot eagerly unfold the tree of all possible parser combinations, I have to work with factories of parser combinators, and wrap them into something like this:
# Wrapper that prevents immediate stack overflow
class LazyParserCombinator(ParserCombinator):
def __init__(self, parserFactory):
self.parserFactory = parserFactory
def parse(self, tokens, pos):
return self.parserFactory().parse(tokens, pos)
def p(parserFactory):
return LazyParserCombinator(parserFactory)
This indeed allows me to write down the grammar in a way that is very close to the EBNF:
num = p(lambda: lit("1"))
factor = p(lambda: num | (lit("(") + expr + lit(")")))
term = p(lambda: (factor + lit("*") + term) | factor)
expr = p(lambda: (term + lit("+") + expr) | term)
And it actually works:
tokens = [str(x) for x in "1+(1+1)*(1+1+1)+1*(1+1)"]
print(expr.parse(tokens, 0))
However, the p(lambda: ...) in every line is a bit annoying. Is there some idiomatic way to get rid of it? It would be nice if one could somehow pass the whole RHS of a rule "by-name", without triggering the eager evaluation of the infinite mutual recursion.
What I've tried
I've checked what's available in the core language: it seems that only if, and and or can "short-circuit", please correct me if I'm wrong.
I've tried looking at how other non-toy-example libraries do this.
For example,
funcparserlib
uses explicit forward declarations to avoid mutual recursion
(look at the forward_decl and value.define
part in github README.md example code).
The parsec.py uses some special #generate decorators
and seems to do something like monadic parsing using coroutines.
That's all very nice, but my goal is to understand what options
I have with regards to the basic evaluation strategies available
in Python.
I've also found something like the lazy_object_proxy.Proxy, but it didn't seem to help to instantiate such objects in more concise way.
So, is there a nicer way to pass arguments by-name and avoid the blowup of mutually recursively defined values?

It's a nice idea, but it's not something that Python's syntax allows for: Python expressions are always evaluated strictly (with the exception of if blocks and and and or short-circuiting expressions).
In particular, the problem is that in an expression like:
num = p(lit("1"))
The p function argument is always received with a new name binding to the same object. The object resulting from evaluating lit("1") is not named anything (until a name is created by the formal parameter to p), so there is no name there to bind to. Conversely, there must be an object there, or otherwise p wouldn't be able to receive a value at all.
What you could do is add a new object to use instead of a lambda to defer evaluation of a name. For example, something like:
class DeferredNamespace(object):
def __init__(self, namespace):
self.__namespace = namespace
def __getattr__(self, name):
return DeferredLookup(self.__namespace, name)
class DeferredLookup(object):
def __init__(self, namespace, name):
self.__namespace = namespace
self.__name = name
def __getattr__(self, name):
return getattr(getattr(self.__namespace, self.__name), name)
d = DeferredNamespace(locals())
num = p(d.lit("1"))
In this case, d.lit actually doesn't return lit, it returns a DeferredLookup object that will use getattr(locals(), 'lit') to resolve its members when they are actually used. Note that this captures locals() eagerly, which you might not want; you can adapt that to use a lambda, or better yet just create all your entities in some other namespace anyway.
You still get the wart of the d. in the syntax, which may or may not be a deal-breaker, depending on your goals with this API.

Special solution for functions that must accept exactly one by-name argument
If you want to define a function f that has to take one single argument by-name, consider making f into a #decorator. Instead of an argument littered with lambdas, the decorator can then directly receive the function definition.
The lambdas in the question appear because we need a way to make the execution of the right hand sides lazy. However, if we change the definitions of non-terminal symbols to be defs rather than local variables, the RHS is also not executed immediately. Then what we have to do is to convert these defs into ParserCombinators somehow. For this, we can use decorators.
We can define a decorator that wraps a function into a LazyParserCombinator as follows:
def rule(f):
return LazyParserCombinator(f)
and then apply it to the functions that hold the definitions of each grammar rule:
#rule
def num(): return lit("1")
#rule
def factor(): return num | (lit("(") + expr + lit(")"))
#rule
def term(): return factor + lit("*") + term | factor
#rule
def expr(): return (term + lit("+") + expr) | term
The syntactic overhead within the right hand sides of the rules is minimal (no overhead for referencing other rules, no p(...)-wrappers or ruleName()-parentheses needed), and there is no counter-intuitive boilerplate with lambdas.
Explanation:
Given a higher order function h, we can use it to decorate other function f as follows:
#h
def f():
<body>
What this does is essentially:
def f():
<body>
f = h(f)
and h is not constrained to returning functions, it can also return other objects, like ParserCombinators above.

How do I decorate a partially applied function in Python 3?

I created a decorator factory that is parameterized by a custom logging function like so:
def _log_error(logger):
def decorator(f):
#wraps(f)
def wrapper(*args, **kwargs):
try:
return f(*args, **kwargs)
except Exception as e:
logger(e)
return None
return wrapper
return decorator
Which I now want to use to decorate a partially-applied function foo:
foo = partial(bar, someparam)
I've tried all of the following:
#_log_error(logger)
foo = partial(bar, someparam)
log_error = _log_error(logger)
#log_error
foo = partial(...)
foo = partial(...)
#log_error
foo
#log_error
(foo = partial(...))
AFAICT both log_error = _log_error(logger) / #log_error and #_log_error(logger) seem totally valid ways of producing the decorator and it works fine on normally declared functions. But when trying to use on the partially applied function I get syntax errors at the start of foo =, and googling while yielding excellent resources on working with decorators and functools.partial in general have not given me anything on this specific case.

Decorators don't work on assignments. But since using a decorator is the same thing as calling the decorator, you can do
foo = _log_error(logger)(partial(bar, someparam))

Either way works
Here's another way you can do it using Either – This answer gets its inspiration from Brian Lonsdorf's egghead series: Professor Frisby Introduces Composable Functional JavaScript
We'll take some of what we learned there and write some super sweet functional python codes
class Map (dict):
def __init__(self, **xw):
super(Map, self).__init__(**xw)
self.__dict__ = self
def Left (x):
return Map(
fold = lambda f, g: f(x),
bimap = lambda f, g: Left(f(x))
)
def Right (x):
return Map(
fold = lambda f, g: g(x),
bimap = lambda f, g: Right(g(x))
)
Note: This is a very incomplete implementation of Left and Right but it's enough to get this specific job done. To take advantage of the full power of this super-powered data type, you'll want a complete implementation.
Generics promote code reuse
We'll setup a few more generic functions
def identity (x):
return x
def try_catch (f):
try:
return Right(f())
except Exception as e:
return Left(e)
def partial (f, *xs, **xw):
def aux (*ys, **yw):
return f(*xs, *ys, **xw, **yw)
return aux
Now we have enough to define log_error – the syntax is a little wonky for writing curried functions in Python, but everything works as expected.
In plain English: we try applying f and get back a value. If the value is an error (Left), call logger, otherwise return the value (identity)
def log_error (logger):
def wrapper (f):
def aux (*xs, **xw):
return try_catch (lambda: f(*xs, **xw)).bimap(logger, identity)
return aux
return wrapper
Putting it all together
Now let's try it with a little function
def foo (x,y,z):
return (x + y) * z
What you wanted to do was wrap a partially applied function in your using your custom logger
foo_logger = log_error(lambda err: print("ERROR:" + str(err))) (partial(foo,'a'))
foo_logger('b',3).fold(print, print)
# ('a' + 'b') * 3
# 'ab' * 3
# => ababab
foo_logger(1,3).fold(print, print)
# ('a' + 1) * 3
# ERROR: Can't convert 'int' object to str implicitly
# => None
Understanding the results
As you can see, when there is no error present (Right), evaluation just keeps on moving and the computed value is passed to print.
When an error occurs (Left), the logger picks it up and logs the error message to the console. Because the logging function has no return value, None is passed along to print

Applying functools.wraps to nested wrappers

I have a base decorator that takes arguments but that also is built upon by other decorators. I can't seem to figure where to put the functools.wraps in order to preserve the full signature of the decorated function.
import inspect
from functools import wraps
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#wraps(func)
def wrap(f):
#wraps(f)
def wrapped_f(*args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
return wrap
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2, a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
The result of this code should be and is:
saysay hello argumentsarguments
However, running help on say_hello gives me:
say_hello(*args, **kwargs)
doc string for say_hello
Everything is preserved except the parameter names.
It seems like I just need to add another #wraps() somewhere, but where?

I experimented with this:
>>> from functools import wraps
>>> def x(): print(1)
...
>>> #wraps(x)
... def xyz(a,b,c): return x
>>> xyz.__name__
'x'
>>> help(xyz)
Help on function x in module __main__:
x(a, b, c)
AFAIK, this has nothing to do with wraps itself, but an issue related to help. Indeed, because help inspects your objects to provide the information, including __doc__ and other attributes, this is why you get this behavior, although your wrapped function has different argument list. Though, wraps doesn't update that automatically (the argument list) what it really updates is this tuple and the __dict__ which is technically the objects namespace:
WRAPPER_ASSIGNMENTS = ('__module__', '__name__', '__qualname__', '__doc__',
'__annotations__')
WRAPPER_UPDATES = ('__dict__',)
If you aren't sure about how wraps work, probably it'll help if your read the the source code from the standard library: functools.py.
It seems like I just need to add another #wraps() somewhere, but where?
No, you don't need to add another wraps in your code, help as I stated above works that way by inspecting your objects. The function's arguments are associated with code objects (__code__) because your function's arguments are stored/represented in that object, wraps has no way to update the argument of the wrapper to be like the wrapped function (continuing with the above example):
>>> xyz.__code__.co_varnames
>>> xyz.__code__.co_varnames = x.__code__.co_varnames
AttributeError: readonly attribute
If help displayed that function xyz has this argument list () instead of (a, b, c) then this is clearly wrong! And the same applies for wraps, to change the argument list of the wrapper to the wrapped, would be cumbersome! So this should not be a concern at all.
>>> #wraps(x, ("__code__",))
... def xyz(a,b,c): pass
...
>>> help(xyz)
Help on function xyz in module __main__:
xyz()
But xyz() returns x():
>>> xyz()
1
For other references take a look at this question or the Python Documentation
What does functools.wraps do?

direprobs was correct in that no amount of functools wraps would get me there. bravosierra99 pointed me to somewhat related examples. However, I couldn't find a single example of signature preservation on nested decorators in which the outer decorator takes arguments.
The comments on Bruce Eckel's post on decorators with arguments gave me the biggest hints in achieving my desired result.
The key was in removing the middle function from within my _process_arguments function and placing its parameter in the next, nested function. It kind of makes sense to me now...but it works:
import inspect
from decorator import decorator
# Base decorator
def _process_arguments(func, *indices):
""" Apply the pre-processing function to each selected parameter """
#decorator
def wrapped_f(f, *args):
params = inspect.getargspec(f)[0]
args_out = list()
for ind, arg in enumerate(args):
if ind in indices:
args_out.append(func(arg))
else:
args_out.append(arg)
return f(*args_out)
return wrapped_f
# Function that will be used to process each parameter
def double(x):
return x * 2
# Decorator called by end user
def double_selected(*args):
return _process_arguments(double, *args)
# End-user's function
#double_selected(2, 0)
def say_hello(a1, a2,a3):
""" doc string for say_hello """
print('{} {} {}'.format(a1, a2, a3))
say_hello('say', 'hello', 'arguments')
print(help(say_hello))
And the result:
saysay hello argumentsarguments
Help on function say_hello in module __main__:
say_hello(a1, a2, a3)
doc string for say_hello

Named keywords in decorators?

I've been playing around in depth with attempting to write my own version of a memoizing decorator before I go looking at other people's code. It's more of an exercise in fun, honestly. However, in the course of playing around I've found I can't do something I want with decorators.
def addValue( func, val ):
def add( x ):
return func( x ) + val
return add
#addValue( val=4 )
def computeSomething( x ):
#function gets defined
If I want to do that I have to do this:
def addTwo( func ):
return addValue( func, 2 )
#addTwo
def computeSomething( x ):
#function gets defined
Why can't I use keyword arguments with decorators in this manner? What am I doing wrong and can you show me how I should be doing it?

You need to define a function that returns a decorator:
def addValue(val):
def decorator(func):
def add(x):
return func(x) + val
return add
return decorator
When you write #addTwo, the value of addTwo is directly used as a decorator. However, when you write #addValue(4), first addValue(4) is evaluated by calling the addValue function. Then the result is used as a decorator.

You want to partially apply the function addValue - give the val argument, but not func. There are generally two ways to do this:
The first one is called currying and used in interjay's answer: instead of a function with two arguments, f(a,b) -> res, you write a function of the first arg that returns another function that takes the 2nd arg g(a) -> (h(b) -> res)
The other way is a functools.partial object. It uses inspection on the function to figure out what arguments a function needs to run (func and val in your case ). You can add extra arguments when creating a partial and once you call the partial, it uses all the extra arguments given.
from functools import partial
#partial(addValue, val=2 ) # you can call this addTwo
def computeSomething( x ):
return x
Partials are usually a much simpler solution for this partial application problem, especially with more than one argument.

Decorators with any kinds of arguments -- named/keyword ones, unnamed/positional ones, or some of each -- essentially, ones you call on the #name line rather than just mention there -- need a double level of nesting (while the decorators you just mention have a single level of nesting). That goes even for argument-less ones if you want to call them in the # line -- here's the simplest, do-nothing, double-nested decorator:
def double():
def middling():
def inner(f):
return f
return inner
return middling
You'd use this as
#double()
def whatever ...
note the parentheses (empty in this case since there are no arguments needed nor wanted): they mean you're calling double, which returns middling, which decorates whatever.
Once you've seen the difference between "calling" and "just mentioning", adding (e.g. optional) named args is not hard:
def doublet(foo=23):
def middling():
def inner(f):
return f
return inner
return middling
usable either as:
#doublet()
def whatever ...
or as:
#doublet(foo=45)
def whatever ...
or equivalently as:
#doublet(45)
def whatever ...