We Keep Coding

How to remove function implementation from a python file?

I am designing python assignments for a class. I define functions, write docstrings and then I implement them. Afterward, I'd like to remove all my implementations of the functions and replace only the code (not the doc-strings, function names, and arguments) with a raise NotImplementedError.
Is there any tool (e.g. IDE) which removes all the code for me automatically, so that I don't have to replace the implemented function by myself? I was thinking about writing a small script, but I thought I might ask here before I do this ...
If anyone has written something similar or knows of a quick way how to this, I would appreciate this a lot.
Here's a minimal example of what I'd like to achieve:
test.py
def add(a,b):
"""
Adds two numbers
"""
return a+b
def multiply(a,b):
"""
Multiplies two numbers
"""
return a*b
should become in an automated fashion (and of course for much larger files):
test.py
def add(a,b):
"""
Adds two numbers
"""
raise NotImplementedEror
def multiply(a,b):
"""
Multiplies two numbers
"""
raise NotImplementedEror

I don't know of a tool to do specifically this, but Python provides great AST manipulation tools within its own standard library via the ast module. You'll need a third party module to "unparse" the result after transformation back into regular Python code, and after a quick search I found this one seems to do the trick, although there do seem to be many others.
Here's a bit of sample code to get you in the right direction. Obviously, you'll need to tweak this to get the exact behavior you want, especially if you want to provide classes instead of just top-level functions (as I've written nothing to handle that use case). But with a bit of Python knowledge, it should be possible to automate.
Also, this is Python 3 (which, as of the start of 2020, is the only supported Python version). If you're still on Python 2, it may require some modifications.
import ast
import astunparse
# Read our file using the built-in Python AST module.
with open('filename.py') as f:
data = ast.parse(f.read(), 'filename.py')
# Loop through all declarations.
for decl in data.body:
# Only modify functions
if isinstance(decl, ast.FunctionDef):
# The docstring is the first statement of the body. So we don't
# want to change it. Instead, replace the rest of the body with
# our pre-built "raise" call. Note that I figured out what "raise"
# looked like in AST form by running
#
# ast.dump(ast.parse("raise NotImplementedError()"))
#
decl.body[1:] = [ast.Raise(ast.Call(ast.Name('NotImplementedError'), [], []), None)]
# Use astunparse to pretty print the result as Python code.
print(astunparse.unparse(data))
It's definitely possible to automate, if you're willing to take the time to do it. If you're planning to do this for several assignments, or even over several semesters, you may consider making a script for it. But if you're just doing it once, it may be more worth your time to just do it by hand.

Ruby's tap idiom in Python

There is a useful Ruby idiom that uses tap which allows you to create an object, do some operations on it and return it (I use a list here only as an example, my real code is more involved):
def foo
[].tap do |a|
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a << b
end
end
>> foo
=> [1]
With Rails there's a similar method called returning, so you can write:
def foo
returning([]) do |a|
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a << b
end
end
which speaks for itself. No matter how much processing you do on the object, it's still clear that it's the return value of the function.
In Python I have to write:
def foo():
a = []
b = 1 + 2
# ... and some more processing, maybe some logging, etc.
a.append(b)
return a
and I wonder if there is a way to port this Ruby idiom into Python. My first thought was to use with statement, but return with is not valid syntax.

Short answer: Ruby encourages method chaining, Python doesn't.
I guess the right question is: What is Ruby's tap useful for?
Now I don't know a lot about Ruby, but by googling I got the impression that tap is conceptually useful as method chaining.
In Ruby, the style: SomeObject.doThis().doThat().andAnotherThing() is quite idiomatic. It underlies the concept of fluent interfaces, for example. Ruby's tap is a special case of this where instead of having SomeObject.doThis() you define doThis on the fly.
Why I am explaining all this? Because it tells us why tap doesn't have good support in Python. With due caveats, Python doesn't do call chaining.
For example, Python list methods generally return None rather than returning the mutated list. Functions like map and filter are not list methods. On the other hand, many Ruby array methods do return the modified array.
Other than certain cases like some ORMs, Python code doesn't use fluent interfaces.
In the end it is the difference between idiomatic Ruby and idiomatic Python. If you are going from one language to the other you need to adjust.

You can implement it in Python as follows:
def tap(x, f):
f(x)
return x
Usage:
>>> tap([], lambda x: x.append(1))
[1]
However it won't be so much use in Python 2.x as it is in Ruby because lambda functions in Python are quite restrictive. For example you can't inline a call to print because it is a keyword, so you can't use it for inline debugging code. You can do this in Python 3.x although it isn't as clean as the Ruby syntax.
>>> tap(2, lambda x: print(x)) + 3
2
5

If you want this bad enough, you can create a context manager
class Tap(object):
def __enter__(self, obj):
return obj
def __exit__(*args):
pass
which you can use like:
def foo():
with Tap([]) as a:
a.append(1)
return a
There's no getting around the return statement and with really doesn't do anything here. But you do have Tap right at the start which clues you into what the function is about I suppose. It is better than using lambdas because you aren't limited to expressions and can have pretty much whatever you want in the with statement.
Overall, I would say that if you want tap that bad, then stick with ruby and if you need to program in python, use python to write python and not ruby. When I get around to learning ruby, I intend to write ruby ;)

I had an idea to achieve this using function decorators, but due to the distinction in python between expressions and statements, this ended up still requiring the return to be at the end.
The ruby syntax is rarely used in my experience, and is far less readable than the explicit python approach. If python had implicit returns or a way to wrap multiple statements up into a single expression then this would be doable - but it has neither of those things by design.
Here's my - somewhat pointless - decorator approach, for reference:
class Tapper(object):
def __init__(self, initial):
self.initial = initial
def __call__(self, func):
func(self.initial)
return self.initial
def tap(initial):
return Tapper(initial)
if __name__ == "__main__":
def tapping_example():
#tap([])
def tapping(t):
t.append(1)
t.append(2)
return tapping
print repr(tapping_example())

I partly agree with others in that it doesn't make much sense to implement this in Python. However, IMHO, Mark Byers's way is the way, but why lambdas(and all that comes with them)? can't you write a separate function to be called when needed?
Another way to do basically the same could be
map(afunction(), avariable)
but this beautiful feature is not a built-in in Python 3, I hear.

Hardly any Ruby programmers use tap in this way. In fact, all top Ruby programmers i know say tap has no use except in debugging.
Why not just do this in your code?
[].push(1)
and remember Array supports a fluent interface, so you can even do this:
[].push(1).push(2)

Should I make my python code less fool-proof to improve readability?

I try to make my code fool-proof, but I've noticed that it takes a lot of time to type things out and it takes more time to read the code.
Instead of:
class TextServer(object):
def __init__(self, text_values):
self.text_values = text_values
# <more code>
# <more methods>
I tend to write this:
class TextServer(object):
def __init__(self, text_values):
for text_value in text_values:
assert isinstance(text_value, basestring), u'All text_values should be str or unicode.'
assert 2 <= len(text_value), u'All text_values should be at least two characters long.'
self.__text_values = frozenset(text_values) # <They shouldn't change.>
# <more code>
#property
def text_values(self):
# <'text_values' shouldn't be replaced.>
return self.__text_values
# <more methods>
Is my python coding style too paranoid? Or is there a way to improve readability while keeping it fool-proof?
Note 1: I've added the comments between < and > just for clarification.
Note 2: The main fool I try to prevent to abuse my code is my future self.

Here's some good advice on Python idioms from this page:
Catch errors rather than avoiding them to avoid cluttering your code with special cases. This idiom is called EAFP ('easier to ask forgiveness than permission'), as opposed to LBYL ('look before you leap'). This often makes the code more readable. For example:
Worse:
#check whether int conversion will raise an error
if not isinstance(s, str) or not s.isdigit:
return None
elif len(s) > 10: #too many digits for int conversion
return None
else:
return int(str)
Better:
try:
return int(str)
except (TypeError, ValueError, OverflowError): #int conversion failed
return None
(Note that in this case, the second version is much better, since it correctly handles leading + and -, and also values between 2 and 10 billion (for 32-bit machines). Don't clutter your code by anticipating all the possible failures: just try it and use appropriate exception handling.)

"Is my python coding style too paranoid? Or is there a way to improve readability while keeping it fool-proof?"
Who's the fool you're protecting yourself from?
You? Are you worried that you didn't remember the API you wrote?
A peer? Are you worried that someone in the next cubicle will actively make an effort to pass the wrong things through an API? You can talk to them to resolve this problem. It saves a lot of code if you provide documentation.
A total sociopath who will download your code, refuse to read the API documentation, and then call all the methods with improper arguments? What possible help can you provide to them?
The "fool-proof" coding isn't really very helpful, since all of these scenarios are more easily addressed another way.
If you're fool-proofing against yourself, perhaps that's not really sensible.
If you're fool-proofing for a co-worker or peer, you should -- perhaps -- talk to them and make sure they understand the API docs.
If you're fool-proofing against some hypothetical sociopathic programmer who's out to subvert the API, there's nothing you can do. It's Python. They have the source. Why would they go through the effort to misuse the API when they can just edit the source to break things?

It's unusual in Python to use a private instance attribute, and then expose it through a property as you have. Just use self.text_values.

Your code is too paranoid (especially when you want to protect only against yourself).
In Python circles, LBYL is generally (but not always) frowned upon. But there's also the (often unstated) assumption that one has (good) unit tests.
Me personally? I think readability is of paramount importance. I mean, if you yourself think it's hard to read, what will others think? And less readable code is also more likely to catch bugs. Not to mention making working on it harder/more time consuming (you have to dig to find what the code actually does in all that LBYLing)

If you try to make your code totally foolproof, someone will invent a better fool. Seriously, a good rule of thumb is to guard against likely errors, but don't clutter your code trying to think of every conceivable way a caller could break you.

instead of spending time in assert and private variable i prefer spend time in documentation and test cases. i prefer read documentation and when i have to read code i prefer read tests. this is more true the more code grow up. in the same time tests give you a fool-proof code and useful use cases.

I base the need for error checking code on the consequences of the errors that it's checking for. If crap data gets into my system, how long will it be before I discover it, how hard will it be to determine that the problem was crap data, and how difficult will it be to fix? For cases like the one you've posted, the answers are generally "not long," "not hard," and "not difficult."
But data that's going to be persisted somewhere and then used as the input to a complicated algorithm in six weeks? I'll check the hell out of that.

I don't think this is specific to python. I'm a firm believer in Design by Contract: Ideally all functions should have clear pre- and post-conditions; unfortunately most languages (Eiffel being the canonical exception) don't provide particularly convenient ways to achieve this which contributes to the apparent conflict between clarity and correctness.
As a practical matter, one approach is to write a 'checkValues' method so as to avoid cluttering __init__. You can even compress it to:
def __init__(self, text_values):
self.text_values = checkValues( text_values )
def checkValues(text_values):
for text_value in text_values:
assert isinstance(text_value, basestring), u'All text_values should be str or unicode.'
assert 2 <= len(text_value), u'All text_values should be at least two characters long.'
return( frozenset( text_values ) )
Another approach would be using a folding text editor that can hide/show the pre-conditions with the aid of some commenting conventions; this would also be useful for auto-generating documentation.

Take all the energy you're putting into argument checking and channel it instead into writing clear, concise doc-strings.
Unless you're writing code for nuclear reactors; in which case I would appreciate you doing both.

Another way to think is: you only need to catch errors that you can correct. If you're checking the input just for aborting with an AssertionError, you're better off just allowing the code to raise the appropriate exception later so you can debug correctly.
This line in particular is pretty bad since it stops duck-typing:
assert isinstance(text_value, basestring), u'All text_values should be str or unicode.'

Python 3 and static typing

I didn't really pay as much attention to Python 3's development as I would have liked, and only just noticed some interesting new syntax changes. Specifically from this SO answer function parameter annotation:
def digits(x:'nonnegative number') -> "yields number's digits":
# ...
Not knowing anything about this, I thought it could maybe be used for implementing static typing in Python!
After some searching, there seemed to be a lot discussion regarding (entirely optional) static typing in Python, such as that mentioned in PEP 3107, and "Adding Optional Static Typing to Python" (and part 2)
..but, I'm not clear how far this has progressed. Are there any implementations of static typing, using the parameter-annotation? Did any of the parameterised-type ideas make it into Python 3?

Thanks for reading my code!
Indeed, it's not hard to create a generic annotation enforcer in Python. Here's my take:
'''Very simple enforcer of type annotations.
This toy super-decorator can decorate all functions in a given module that have
annotations so that the type of input and output is enforced; an AssertionError is
raised on mismatch.
This module also has a test function func() which should fail and logging facility
log which defaults to print.
Since this is a test module, I cut corners by only checking *keyword* arguments.
'''
import sys
log = print
def func(x:'int' = 0) -> 'str':
'''An example function that fails type checking.'''
return x
# For simplicity, I only do keyword args.
def check_type(*args):
param, value, assert_type = args
log('Checking {0} = {1} of {2}.'.format(*args))
if not isinstance(value, assert_type):
raise AssertionError(
'Check failed - parameter {0} = {1} not {2}.'
.format(*args))
return value
def decorate_func(func):
def newf(*args, **kwargs):
for k, v in kwargs.items():
check_type(k, v, ann[k])
return check_type('<return_value>', func(*args, **kwargs), ann['return'])
ann = {k: eval(v) for k, v in func.__annotations__.items()}
newf.__doc__ = func.__doc__
newf.__type_checked = True
return newf
def decorate_module(module = '__main__'):
'''Enforces type from annotation for all functions in module.'''
d = sys.modules[module].__dict__
for k, f in d.items():
if getattr(f, '__annotations__', {}) and not getattr(f, '__type_checked', False):
log('Decorated {0!r}.'.format(f.__name__))
d[k] = decorate_func(f)
if __name__ == '__main__':
decorate_module()
# This will raise AssertionError.
func(x = 5)
Given this simplicity, it's strange at the first sight that this thing is not mainstream. However, I believe there are good reasons why it's not as useful as it might seem. Generally, type checking helps because if you add integer and dictionary, chances are you made some obvious mistake (and if you meant something reasonable, it's still better to be explicit than implicit).
But in real life you often mix quantities of the same computer type as seen by compiler but clearly different human type, for example the following snippet contains an obvious mistake:
height = 1.75 # Bob's height in meters.
length = len(sys.modules) # Number of modules imported by program.
area = height * length # What's that supposed to mean???
Any human should immediately see a mistake in the above line provided it knows the 'human type' of variables height and length even though it looks to computer as perfectly legal multiplication of int and float.
There's more that can be said about possible solutions to this problem, but enforcing 'computer types' is apparently a half-solution, so, at least in my opinion, it's worse than no solution at all. It's the same reason why Systems Hungarian is a terrible idea while Apps Hungarian is a great one. There's more at the very informative post of Joel Spolsky.
Now if somebody was to implement some kind of Pythonic third-party library that would automatically assign to real-world data its human type and then took care to transform that type like width * height -> area and enforce that check with function annotations, I think that would be a type checking people could really use!

As mentioned in that PEP, static type checking is one of the possible applications that function annotations can be used for, but they're leaving it up to third-party libraries to decide how to do it. That is, there isn't going to be an official implementation in core python.
As far as third-party implementations are concerned, there are some snippets (such as http://code.activestate.com/recipes/572161/), which seem to do the job pretty well.
EDIT:
As a note, I want to mention that checking behavior is preferable to checking type, therefore I think static typechecking is not so great an idea. My answer above is aimed at answering the question, not because I would do typechecking myself in such a way.

"Static typing" in Python can only be implemented so that the type checking is done in run-time, which means it slows down the application. Therefore you don't want that as a generality. Instead you want some of your methods to check it's inputs. This can be easily done with plain asserts, or with decorators if you (mistakenly) think you need it a lot.
There is also an alternative to static type checking, and that is to use an aspect oriented component architecture like The Zope Component Architecture. Instead of checking the type, you adapt it. So instead of:
assert isinstance(theobject, myclass)
you do this:
theobject = IMyClass(theobject)
If theobject already implements IMyClass nothing happens. If it doesn't, an adapter that wraps whatever theobject is to IMyClass will be looked up, and used instead of theobject. If no adapter is found, you get an error.
This combined the dynamicism of Python with the desire to have a specific type in a specific way.

This is not an answer to question directly, but I found out a Python fork that adds static typing: mypy-lang.org, of course one can't rely on it as it's still small endeavor, but interesting.

Sure, static typing seems a bit "unpythonic" and I don't use it all the time. But there are cases (e.g. nested classes, as in domain specific language parsing) where it can really speed up your development.
Then I prefer using beartype explained in this post*. It comes with a git repo, tests and an explanation what it can and what it can't do ... and I like the name ;)
* Please don't pay attention to Cecil's rant about why Python doesn't come with batteries included in this case.

What are the important language features (idioms) of Python to learn early on [duplicate]

This question already has answers here:
The Zen of Python [closed]
(22 answers)
Python: Am I missing something? [closed]
(16 answers)
Closed 8 years ago.
I would be interested in knowing what the StackOverflow community thinks are the important language features (idioms) of Python. Features that would define a programmer as Pythonic.
Python (pythonic) idiom - "code expression" that is natural or characteristic to the language Python.
Plus, Which idioms should all Python programmers learn early on?
Thanks in advance
Related:
Code Like a Pythonista: Idiomatic Python
Python: Am I missing something?

Python is a language that can be described as:
"rules you can fit in the
palm of your hand with a huge bag of
hooks".
Nearly everything in python follows the same simple standards. Everything is accessible, changeable, and tweakable. There are very few language level elements.
Take for example, the len(data) builtin function. len(data) works by simply checking for a data.__len__() method, and then calls it and returns the value. That way, len() can work on any object that implements a __len__() method.
Start by learning about the types and basic syntax:
Dynamic Strongly Typed Languages
bool, int, float, string, list, tuple, dict, set
statements, indenting, "everything is an object"
basic function definitions
Then move on to learning about how python works:
imports and modules (really simple)
the python path (sys.path)
the dir() function
__builtins__
Once you have an understanding of how to fit pieces together, go back and cover some of the more advanced language features:
iterators
overrides like __len__ (there are tons of these)
list comprehensions and generators
classes and objects (again, really simple once you know a couple rules)
python inheritance rules
And once you have a comfort level with these items (with a focus on what makes them pythonic), look at more specific items:
Threading in python (note the Global Interpreter Lock)
context managers
database access
file IO
sockets
etc...
And never forget The Zen of Python (by Tim Peters)
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

This page covers all the major python idioms: http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html

An important idiom in Python is docstrings.
Every object has a __doc__ attribute that can be used to get help on that object. You can set the __doc__ attribute on modules, classes, methods, and functions like this:
# this is m.py
""" module docstring """
class c:
"""class docstring"""
def m(self):
"""method docstring"""
pass
def f(a):
"""function f docstring"""
return
Now, when you type help(m), help(m.f) etc. it will print the docstring as a help message.
Because it's just part of normal object introspection this can be used by documention generating systems like epydoc or used for testing purposes by unittest.
It can also be put to more unconventional (i.e. non-idiomatic) uses such as grammars in Dparser.
Where it gets even more interesting to me is that, even though doc is a read-only attribute on most objects, you can use them anywhere like this:
x = 5
""" pseudo docstring for x """
and documentation tools like epydoc can pick them up and format them properly (as opposed to a normal comment which stays inside the code formatting.

Decorators get my vote. Where else can you write something like:
def trace(num_args=0):
def wrapper(func):
def new_f(*a,**k):
print_args = ''
if num_args > 0:
print_args = str.join(',', [str(x) for x in a[0:num_args]])
print('entering %s(%s)' %(f.__name__,print_args))
rc = f(*a,**k)
if rc is not None:
print('exiting %s(%s)=%s' %(f.__name__,str(rc)))
else:
print('exiting %s(%s)' %(f.__name__))
return rc
return new_f
return wrapper
#trace(1)
def factorial(n):
if n < 2:
return 1
return n * factorial(n-1)
factorial(5)
and get output like:
entering factorial(5)
entering factorial(4)
entering factorial(3)
entering factorial(2)
entering factorial(1)
entering factorial(0)
exiting factorial(0)=1
exiting factorial(1)=1
exiting factorial(2)=2
exiting factorial(3)=6
exiting factorial(4)=24
exiting factorial(5)=120

Everything connected to list usage.
Comprehensions, generators, etc.

Personally, I really like Python syntax defining code blocks by using indentation, and not by the words "BEGIN" and "END" (as in Microsoft's Basic and Visual Basic - I don't like these) or by using left- and right-braces (as in C, C++, Java, Perl - I like these).
This really surprised me because, although indentation has always been very important to me, I didn't make to much "noise" about it - I lived with it, and it is considered a skill to be able to read other peoples, "spaghetti" code. Furthermore, I never heard another programmer suggest making indentation a part of a language. Until Python! I only wish I had realized this idea first.
To me, it is as if Python's syntax forces you to write good, readable code.
Okay, I'll get off my soap-box. ;-)

From a more advanced viewpoint, understanding how dictionaries are used internally by Python. Classes, functions, modules, references are all just properties on a dictionary. Once this is understood it's easy to understand how to monkey patch and use the powerful __gettattr__, __setattr__, and __call__ methods.

Here's one that can help. What's the difference between:
[ foo(x) for x in range(0, 5) ][0]
and
( foo(x) for x in range(0, 5) ).next()
answer:
in the second example, foo is called only once. This may be important if foo has a side effect, or if the iterable being used to construct the list is large.

Two things that struck me as especially Pythonic were dynamic typing and the various flavors of lists used in Python, particularly tuples.
Python's list obsession could be said to be LISP-y, but it's got its own unique flavor. A line like:
return HandEvaluator.StraightFlush, (PokerCard.longFaces[index + 4],
PokerCard.longSuits[flushSuit]), []
or even
return False, False, False
just looks like Python and nothing else. (Technically, you'd see the latter in Lua as well, but Lua is pretty Pythonic in general.)

Using string substitutions:
name = "Joe"
age = 12
print "My name is %s, I am %s" % (name, age)
When I'm not programming in python, that simple use is what I miss most.

Another thing you cannot start early enough is probably testing. Here especially doctests are a great way of testing your code by explaining it at the same time.
doctests are simple text file containing an interactive interpreter session plus text like this:
Let's instantiate our class::
>>> a=Something(text="yes")
>>> a.text
yes
Now call this method and check the results::
>>> a.canify()
>>> a.text
yes, I can
If e.g. a.text returns something different the test will fail.
doctests can be inside docstrings or standalone textfiles and are executed by using the doctests module. Of course the more known unit tests are also available.

I think that tutorials online and books only talk about doing things, not doing things in the best way. Along with the python syntax i think that speed in some cases is important.
Python provides a way to benchmark functions, actually two!!
One way is to use the profile module, like so:
import profile
def foo(x, y, z):
return x**y % z # Just an example.
profile.run('foo(5, 6, 3)')
Another way to do this is to use the timeit module, like this:
import timeit
def foo(x, y, z):
return x**y % z # Can also be 'pow(x, y, z)' which is way faster.
timeit.timeit('foo(5, 6, 3)', 'from __main__ import *', number = 100)
# timeit.timeit(testcode, setupcode, number = number_of_iterations)