I am designing python assignments for a class. I define functions, write docstrings and then I implement them. Afterward, I'd like to remove all my implementations of the functions and replace only the code (not the doc-strings, function names, and arguments) with a raise NotImplementedError.
Is there any tool (e.g. IDE) which removes all the code for me automatically, so that I don't have to replace the implemented function by myself? I was thinking about writing a small script, but I thought I might ask here before I do this ...
If anyone has written something similar or knows of a quick way how to this, I would appreciate this a lot.
Here's a minimal example of what I'd like to achieve:
test.py
def add(a,b):
"""
Adds two numbers
"""
return a+b
def multiply(a,b):
"""
Multiplies two numbers
"""
return a*b
should become in an automated fashion (and of course for much larger files):
test.py
def add(a,b):
"""
Adds two numbers
"""
raise NotImplementedEror
def multiply(a,b):
"""
Multiplies two numbers
"""
raise NotImplementedEror
I don't know of a tool to do specifically this, but Python provides great AST manipulation tools within its own standard library via the ast module. You'll need a third party module to "unparse" the result after transformation back into regular Python code, and after a quick search I found this one seems to do the trick, although there do seem to be many others.
Here's a bit of sample code to get you in the right direction. Obviously, you'll need to tweak this to get the exact behavior you want, especially if you want to provide classes instead of just top-level functions (as I've written nothing to handle that use case). But with a bit of Python knowledge, it should be possible to automate.
Also, this is Python 3 (which, as of the start of 2020, is the only supported Python version). If you're still on Python 2, it may require some modifications.
import ast
import astunparse
# Read our file using the built-in Python AST module.
with open('filename.py') as f:
data = ast.parse(f.read(), 'filename.py')
# Loop through all declarations.
for decl in data.body:
# Only modify functions
if isinstance(decl, ast.FunctionDef):
# The docstring is the first statement of the body. So we don't
# want to change it. Instead, replace the rest of the body with
# our pre-built "raise" call. Note that I figured out what "raise"
# looked like in AST form by running
#
# ast.dump(ast.parse("raise NotImplementedError()"))
#
decl.body[1:] = [ast.Raise(ast.Call(ast.Name('NotImplementedError'), [], []), None)]
# Use astunparse to pretty print the result as Python code.
print(astunparse.unparse(data))
It's definitely possible to automate, if you're willing to take the time to do it. If you're planning to do this for several assignments, or even over several semesters, you may consider making a script for it. But if you're just doing it once, it may be more worth your time to just do it by hand.
Related
I've only recently learned about decorators, and despite reading nearly every search result I can find about this question, I cannot figure this out. All I want to do is define some function "calc(x,y)", and wrap its result with a series of external functions, without changing anything inside of my function, nor its calls in the script, such as:
#tan
#sqrt
def calc(x,y):
return (x+y)
### calc(x,y) = tan(sqrt(calc(x,y))
### Goal is to have every call of calc in the script automatically nest like that.
After reading about decorators for almost 10 hours yesterday, I got the strong impression this is what they were used for. I do understand that there are various ways to modify how the functions are passed to one another, but I can't find any obvious guide on how to achieve this. I read that maybe functools wraps can be used for this purpose, but I cannot figure that out either.
Most of the desire here is to be able to quickly and easily test how different functions modify the results of others, without having to tediously wrap functions between parenthesis... That is, to avoid having to mess with parenthesis at all, having my modifier test functions defined on their own lines.
A decorator is simply a function that takes a function and returns another function.
def tan(f):
import math
def g(x,y):
return math.tan(f(x,y))
return g
How are we supposed to write files in Python while staying functionally pure? Normally I would do something like this
from typing import Iterable
from io import IOBase
def transform_input(input_lines: Iterable[str]) -> Iterable[str]: ...
def print_pack(input_lines: Iterable[str], output: IOBase) -> None:
for line in input_lines:
print(line, file=output)
def main(*args, **kwargs):
# Somehow we get a bunch iterables with strings and a list of output streams
packs_of_input = ... # Iterable[Iterable[str]]
output_streams = ... # Iterable[IOBase]
packs_to_print = map(transform_input, packs_of_input)
for pack, output_stream in zip(packs_to_print, output_streams):
print_pack(pack, output_stream)
We can replace the for-loop with something like this
list(map(lambda pack_stream: print_pack(*pack_stream), zip(packs_to_print, output_streams))
but it would only make it look like the printing is done functionally. The problem is that print_pack is not a pure function, that is all its effort results in a side-effect and it returns nothing.
How are we supposed to write files and remain functionally-pure (or almost pure)?
Essentially, in Python, you need to have an impure function somewhere, so there's no way to have 100% pure functions in this application. In the end you need to do some IO, and IO is impure.
However, what you can do is try to represent a particular layer of abstraction within your application as pure functions, and isolate the part that does the actual side-effects in another module. You can do this pretty easily in an ad-hoc way -- for example, by accumulating the contents of the file you want to write as a pure immutable data structure in your main code. Then your side-effecting code can be reduced in size, since all it needs to do is dump a string to a file.
We can look to Haskell for a more rigorous way to purely represent the full power of side-effecting operations with pure functions and data structures -- using the Monad abstraction. Essentially, a Monad is something that you can bind callbacks to, to create a chain of effectful computations all based on pure functions. For the IO monad, the Haskell runtime takes care of actually performing the side-effects once you return an IO value from the main function -- so all the code you write is technically pure functions, and the runtime takes care of the IO.
The Effect library (disclaimer: I wrote it) basically implements a certain flavor of Monad (or something very close to a monad) in Python. This lets you represent arbitrary IO (and other side-effects) as pure objects and functions, and put the actual performance of those effects off to the side. So your application code can be 100% pure, as long as you have a kind of library of relatively simple side-effecting functions.
So, for example, to implement a function that writes a list of lines to a file with Effects, you'd do something like this:
#do
def write_lines_to_file(lines, filename):
file_handle = yield open_file(filename)
for line in lines:
yield write_data(file_handle, line)
# alternatively:
# from effect.fold import sequence; from functools import partial
# yield sequence(map(partial(write_data, file_handle), lines))
yield close_file(file_handle)
The Effect library provides this special do decorator that lets you use an imperative-looking syntax to describe a pure effectful operation. The above is function is equivalent to this one:
def write_lines_to_file(lines, filename):
file_handle_eff = open_file(filename).on(
lambda file_handle:
sequence(map(partial(write_data, file_handle), lines)).on(
lambda _: close_file(file_handle)))
These both assume that three functions exist: open_file, write_data, and close_file. These functions are assumed to return Effect objects that represent the intent to perform those actions. In the end, an Effect is essentially an intent (some transparent description of an action that is requested), and one or more callbacks to run when the result of that action has been completed. The interesting distinction is that write_lines_to_file does not actually write lines to a file; it simply returns some representation of the intent to write some lines to a file.
To actually perform this effect, you need to use the sync_perform function, like sync_perform(dispatcher, write_lines_to_file(lines, filename)). This is an impure function that actually runs the performers for all the effects that your pure representation of an effectful computation use.
I could go into the details of how open_file, write_data, and close_file would need to be implemented, and the details of what the "dispatcher" argument are, but really the documentation at https://effect.readthedocs.org/ is probably the right thing to reference at this point.
I also gave a talk at Strange Loop about Effect and its implementation, which you can watch on YouTube: https://www.youtube.com/watch?v=D37dc9EoFus
It's worth noting that Effect is a pretty heavy-handed way to keep code purely functional. You can get a long way toward maintainable code by taking a "functional core/imperative shell" approach and trying your best to write most of your code as pure functions and minimizing the effectful code. But if you're interested in a more rigorous approach, I think Effect is good. My team uses it in production and it has helped out a lot, especially with its testing API.
I'm looking for the best recipie to allow inline definition of functions, or multi-line lambda, in python.
For example, I'd like to do the following:
def callfunc(func):
func("Hello")
>>> callfunc(define('x', '''
... print x, "World!"
... '''))
Hello World!
I've found an example for the define function in this answer:
def define(arglist, body):
g = {}
exec("def anonfunc({0}):\n{1}".format(
arglist,
"\n".join(" {0}".format(line) for line in body.splitlines())), g)
return g["anonfunc"]
This is one possible solution, but it is not ideal. Desireable features would be:
be smarter about indentation,
hide the innards better (e.g. don't have anonfunc in the function's scope)
provide access to variables in the surrounding scope / captures
better error handling
and some things I haven't thought of. I had a really nice implementation once that did most of the above, but I lost in unfortunately. I'm wondering if someone else has made something similar.
Disclaimer:
I'm well aware this is controversial among Python users, and regarded as a hack or unpythonic. I'm also aware of the discussions regaring multi-line-lambdas on the python-dev mailing list, and that a similar feature was omitted on purpose. However, from the same discussions I've learned that there is also interest in such a function by many others.
I'm not asking whether this is a good idea or not, but instead: Given that one has decided to implement this, (either out of fun and curiosity, madness, genuinely thinking this is a nice idea, or being held at gunpoint) how to make anonymous define work as close as possible to def using python's (2.7 or 3.x) current facilities?
Examples:
A bit more as to why, this can be really handy for callbacks in GUIs:
# gtk example:
self.ntimes = 0
button.connect('clicked', define('*a', '''
self.ntimes += 1
label.set_text("Button has been clicked %d times" % self.ntimes)
''')
The benefit over defining a function with def is that your code is in a more logical order. This is simplified code taken from a Twisted application:
# twisted example:
def sayHello(self):
d = self.callRemote(HelloCommand)
def handle_response(response):
# do something, this happens after (x)!
pass
d.addCallback(handle_response) # (x)
Note how it seems out of order. I usually break stuff like this up, to keep the code order == execution order:
def sayHello_d(self):
d = self.callRemote(HelloCommand)
d.addCallback(self._sayHello_2)
return d
def _sayHello_2(self, response):
# handle response
pass
This is better wrt. ordering but more verbose. Now, with the anonymous functions trick:
d = self.callRemote(HelloCommand)
d.addCallback(define('response', '''
print "callback"
print "got response from", response["name"]
'''))
If you come from a javascript or ruby background, python's abilities to deal with anonymous functions may indeed seem limited, but this is for a reason. Python designers decided that clarity of code is more important than conciseness. If you don't like that, you probably don't like python at all. There's nothing wrong about that, there are many other choices - why not to try a language that tastes better to you?
Putting chunks of code into strings and interpreting them on the fly is definitely a wrong way to "extend" a language, just because none of tools you're working with - from syntax highlighters to the python interpreter itself - would be able to deal with "stringified" code in a sensible way.
To answer the question as asked: what you're doing there is essentially an attempt to construct some better-than-python programming language and compile it to python on the fly. The idea is not new in the world of scripting languages and can be productive or not (CoffeeScript is an example of a successful implementation), but your very approach is wrong. format() not the tool you're looking for when working with code. If you're writing a compiler, do it properly: use a parser (e.g. pyparsing) to read your code in an AST, walk through the AST to generate python code (or even bytecode), catch syntax errors as you go and take measures to provide better runtime feedback (e.g. error context, line numbers etc). Finally, make sure your compiler works across different python versions and implementations.
Or just use ruby.
I have two functions like the following:
def fitnesscompare(x, y):
if x.fitness>y.fitness:
return 1
elif x.fitness==y.fitness:
return 0
else: #x.fitness<y.fitness
return -1
that are used with 'sort' to sort on different attributes of class instances.
These are used from within other functions and methods in the program.
Can I make them visible everywhere rather than having to pass them to each object in which they are used?
Thanks
The best approach (to get the visibility you ask about) is to put this def statement in a module (say fit.py), import fit from any other module that needs access to items defined in this one, and use fit.fitnesscompare in any of those modules as needed.
What you ask, and what you really need, may actually be different...:
as I explained in another post earlier today, custom comparison functions are not the best way to customize sorting in Python (which is why in Python 3 they're not even allowed any more): rather, a custom key-extraction function will serve you much better (future-proof, more general, faster). I.e., instead of calling, say
somelist.sort(cmp=fit.fitnesscompare)
call
somelist.sort(key=fit.fitnessextract)
where
def fitnessextract(x):
return x.fitness
or, for really blazing speed,
import operator
somelist.sort(key=operator.attrgetter('fitness'))
Defining a function with def makes that function available within whatever scope you've defined it in. At module level, using def will make that function available to any other function inside that module.
Can you perhaps post an example of what is not working for you? The code you've posted appears to be unrelated to your actual problem.
This seems like it should be easy, but I can't find the answer anywhere - nor able to derive one myself. How do you turn an unquoted python function/lambda into an AST?
Here is what I'd like to be able to do.
import ast
class Walker(ast.NodeVisitor):
pass
# ...
# note, this doesnt work as ast.parse wants a string
tree = ast.parse(lambda x,y: x+y)
Walker().visit(tree)
In general, you can't. For example, 2 + 2 is an expression -- but if you pass it to any function or method, the argument being passed is just the number 4, no way to recover what expression it was computed from. Function source code can sometimes be recovered (though not for a lambda), but "an unquoted Python expression" gets evaluated so what you get is just the object that's the expression's value.
What problem are you trying to solve? There may be other, viable approaches.
Edit: tx to the OP for clarifying. There's no way to do it for lambda or some other corner cases, but as I mention function source code can sometimes be recovered...:
import ast
import inspect
def f():
return 23
tree = ast.parse(inspect.getsource(f))
print ast.dump(tree)
inspect.getsource raises IOError if it can't get the source code for whatever object you're passing it. I suggest you wrap the parsing and getsource call into an auxiliary function that can accept a string (and just parses it) OR a function (and tries getsource on it, possibly giving better errors in the IOError case).
If you only get access to the function/lambda you only have the compiled python bytecode. The exact Python AST can't be reconstructed from the bytecode because there is information loss in the compilation process. But you can analyze the bytecode and create AST's for that. There is one such analyzer in GeniuSQL. I also have a small proof of concept that analyzes bytecode and creates SQLAlchemy clauseelements from this.
The process I used for analyzing is the following:
Split the code into a list of opcodes with potential arguments.
Find the basic blocks in the code by going through the opcodes and for every jump create a basic block boundary after the jump and before the jump target
Create a control flow graph from the basic blocks.
Go through all the basic blocks with abstract interpretation tracking stack and variable assignments in SSA form.
To create the output expression just get the calculated SSA return value.
I have pasted my proof of concept and example code using it. This is non-clean quickly hacked together code, but you're free to build on it if you like. Leave a note if you decide to make something useful from it.
The Meta library allows you to recover the source in many cases, with some exceptions such as comprehensions and lambdas.
import meta, ast
source = '''
a = 1
b = 2
c = (a ** b)
'''
mod = ast.parse(source, '<nofile>', 'exec')
code = compile(mod, '<nofile>', 'exec')
mod2 = meta.decompile(code)
source2 = meta.dump_python_source(mod2)
assert source == source2
You can't generate AST from compiled bytecode. You need the source code.
Your lambda expression is a function, which has lots of information, but I don't think it still has source code associated with. I'm not sure you can get what you want.