Related
Can you add new statements (like print, raise, with) to Python's syntax?
Say, to allow..
mystatement "Something"
Or,
new_if True:
print "example"
Not so much if you should, but rather if it's possible (short of modifying the python interpreters code)
You may find this useful - Python internals: adding a new statement to Python, quoted here:
This article is an attempt to better understand how the front-end of Python works. Just reading documentation and source code may be a bit boring, so I'm taking a hands-on approach here: I'm going to add an until statement to Python.
All the coding for this article was done against the cutting-edge Py3k branch in the Python Mercurial repository mirror.
The until statement
Some languages, like Ruby, have an until statement, which is the complement to while (until num == 0 is equivalent to while num != 0). In Ruby, I can write:
num = 3
until num == 0 do
puts num
num -= 1
end
And it will print:
3
2
1
So, I want to add a similar capability to Python. That is, being able to write:
num = 3
until num == 0:
print(num)
num -= 1
A language-advocacy digression
This article doesn't attempt to suggest the addition of an until statement to Python. Although I think such a statement would make some code clearer, and this article displays how easy it is to add, I completely respect Python's philosophy of minimalism. All I'm trying to do here, really, is gain some insight into the inner workings of Python.
Modifying the grammar
Python uses a custom parser generator named pgen. This is a LL(1) parser that converts Python source code into a parse tree. The input to the parser generator is the file Grammar/Grammar[1]. This is a simple text file that specifies the grammar of Python.
[1]: From here on, references to files in the Python source are given relatively to the root of the source tree, which is the directory where you run configure and make to build Python.
Two modifications have to be made to the grammar file. The first is to add a definition for the until statement. I found where the while statement was defined (while_stmt), and added until_stmt below [2]:
compound_stmt: if_stmt | while_stmt | until_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
until_stmt: 'until' test ':' suite
[2]: This demonstrates a common technique I use when modifying source code I’m not familiar with: work by similarity. This principle won’t solve all your problems, but it can definitely ease the process. Since everything that has to be done for while also has to be done for until, it serves as a pretty good guideline.
Note that I've decided to exclude the else clause from my definition of until, just to make it a little bit different (and because frankly I dislike the else clause of loops and don't think it fits well with the Zen of Python).
The second change is to modify the rule for compound_stmt to include until_stmt, as you can see in the snippet above. It's right after while_stmt, again.
When you run make after modifying Grammar/Grammar, notice that the pgen program is run to re-generate Include/graminit.h and Python/graminit.c, and then several files get re-compiled.
Modifying the AST generation code
After the Python parser has created a parse tree, this tree is converted into an AST, since ASTs are much simpler to work with in subsequent stages of the compilation process.
So, we're going to visit Parser/Python.asdl which defines the structure of Python's ASTs and add an AST node for our new until statement, again right below the while:
| While(expr test, stmt* body, stmt* orelse)
| Until(expr test, stmt* body)
If you now run make, notice that before compiling a bunch of files, Parser/asdl_c.py is run to generate C code from the AST definition file. This (like Grammar/Grammar) is another example of the Python source-code using a mini-language (in other words, a DSL) to simplify programming. Also note that since Parser/asdl_c.py is a Python script, this is a kind of bootstrapping - to build Python from scratch, Python already has to be available.
While Parser/asdl_c.py generated the code to manage our newly defined AST node (into the files Include/Python-ast.h and Python/Python-ast.c), we still have to write the code that converts a relevant parse-tree node into it by hand. This is done in the file Python/ast.c. There, a function named ast_for_stmt converts parse tree nodes for statements into AST nodes. Again, guided by our old friend while, we jump right into the big switch for handling compound statements and add a clause for until_stmt:
case while_stmt:
return ast_for_while_stmt(c, ch);
case until_stmt:
return ast_for_until_stmt(c, ch);
Now we should implement ast_for_until_stmt. Here it is:
static stmt_ty
ast_for_until_stmt(struct compiling *c, const node *n)
{
/* until_stmt: 'until' test ':' suite */
REQ(n, until_stmt);
if (NCH(n) == 4) {
expr_ty expression;
asdl_seq *suite_seq;
expression = ast_for_expr(c, CHILD(n, 1));
if (!expression)
return NULL;
suite_seq = ast_for_suite(c, CHILD(n, 3));
if (!suite_seq)
return NULL;
return Until(expression, suite_seq, LINENO(n), n->n_col_offset, c->c_arena);
}
PyErr_Format(PyExc_SystemError,
"wrong number of tokens for 'until' statement: %d",
NCH(n));
return NULL;
}
Again, this was coded while closely looking at the equivalent ast_for_while_stmt, with the difference that for until I've decided not to support the else clause. As expected, the AST is created recursively, using other AST creating functions like ast_for_expr for the condition expression and ast_for_suite for the body of the until statement. Finally, a new node named Until is returned.
Note that we access the parse-tree node n using some macros like NCH and CHILD. These are worth understanding - their code is in Include/node.h.
Digression: AST composition
I chose to create a new type of AST for the until statement, but actually this isn't necessary. I could've saved some work and implemented the new functionality using composition of existing AST nodes, since:
until condition:
# do stuff
Is functionally equivalent to:
while not condition:
# do stuff
Instead of creating the Until node in ast_for_until_stmt, I could have created a Not node with an While node as a child. Since the AST compiler already knows how to handle these nodes, the next steps of the process could be skipped.
Compiling ASTs into bytecode
The next step is compiling the AST into Python bytecode. The compilation has an intermediate result which is a CFG (Control Flow Graph), but since the same code handles it I will ignore this detail for now and leave it for another article.
The code we will look at next is Python/compile.c. Following the lead of while, we find the function compiler_visit_stmt, which is responsible for compiling statements into bytecode. We add a clause for Until:
case While_kind:
return compiler_while(c, s);
case Until_kind:
return compiler_until(c, s);
If you wonder what Until_kind is, it's a constant (actually a value of the _stmt_kind enumeration) automatically generated from the AST definition file into Include/Python-ast.h. Anyway, we call compiler_until which, of course, still doesn't exist. I'll get to it an a moment.
If you're curious like me, you'll notice that compiler_visit_stmt is peculiar. No amount of grep-ping the source tree reveals where it is called. When this is the case, only one option remains - C macro-fu. Indeed, a short investigation leads us to the VISIT macro defined in Python/compile.c:
#define VISIT(C, TYPE, V) {\
if (!compiler_visit_ ## TYPE((C), (V))) \
return 0; \
It's used to invoke compiler_visit_stmt in compiler_body. Back to our business, however...
As promised, here's compiler_until:
static int
compiler_until(struct compiler *c, stmt_ty s)
{
basicblock *loop, *end, *anchor = NULL;
int constant = expr_constant(s->v.Until.test);
if (constant == 1) {
return 1;
}
loop = compiler_new_block(c);
end = compiler_new_block(c);
if (constant == -1) {
anchor = compiler_new_block(c);
if (anchor == NULL)
return 0;
}
if (loop == NULL || end == NULL)
return 0;
ADDOP_JREL(c, SETUP_LOOP, end);
compiler_use_next_block(c, loop);
if (!compiler_push_fblock(c, LOOP, loop))
return 0;
if (constant == -1) {
VISIT(c, expr, s->v.Until.test);
ADDOP_JABS(c, POP_JUMP_IF_TRUE, anchor);
}
VISIT_SEQ(c, stmt, s->v.Until.body);
ADDOP_JABS(c, JUMP_ABSOLUTE, loop);
if (constant == -1) {
compiler_use_next_block(c, anchor);
ADDOP(c, POP_BLOCK);
}
compiler_pop_fblock(c, LOOP, loop);
compiler_use_next_block(c, end);
return 1;
}
I have a confession to make: this code wasn't written based on a deep understanding of Python bytecode. Like the rest of the article, it was done in imitation of the kin compiler_while function. By reading it carefully, however, keeping in mind that the Python VM is stack-based, and glancing into the documentation of the dis module, which has a list of Python bytecodes with descriptions, it's possible to understand what's going on.
That's it, we're done... Aren't we?
After making all the changes and running make, we can run the newly compiled Python and try our new until statement:
>>> until num == 0:
... print(num)
... num -= 1
...
3
2
1
Voila, it works! Let's see the bytecode created for the new statement by using the dis module as follows:
import dis
def myfoo(num):
until num == 0:
print(num)
num -= 1
dis.dis(myfoo)
Here's the result:
4 0 SETUP_LOOP 36 (to 39)
>> 3 LOAD_FAST 0 (num)
6 LOAD_CONST 1 (0)
9 COMPARE_OP 2 (==)
12 POP_JUMP_IF_TRUE 38
5 15 LOAD_NAME 0 (print)
18 LOAD_FAST 0 (num)
21 CALL_FUNCTION 1
24 POP_TOP
6 25 LOAD_FAST 0 (num)
28 LOAD_CONST 2 (1)
31 INPLACE_SUBTRACT
32 STORE_FAST 0 (num)
35 JUMP_ABSOLUTE 3
>> 38 POP_BLOCK
>> 39 LOAD_CONST 0 (None)
42 RETURN_VALUE
The most interesting operation is number 12: if the condition is true, we jump to after the loop. This is correct semantics for until. If the jump isn't executed, the loop body keeps running until it jumps back to the condition at operation 35.
Feeling good about my change, I then tried running the function (executing myfoo(3)) instead of showing its bytecode. The result was less than encouraging:
Traceback (most recent call last):
File "zy.py", line 9, in
myfoo(3)
File "zy.py", line 5, in myfoo
print(num)
SystemError: no locals when loading 'print'
Whoa... this can't be good. So what went wrong?
The case of the missing symbol table
One of the steps the Python compiler performs when compiling the AST is create a symbol table for the code it compiles. The call to PySymtable_Build in PyAST_Compile calls into the symbol table module (Python/symtable.c), which walks the AST in a manner similar to the code generation functions. Having a symbol table for each scope helps the compiler figure out some key information, such as which variables are global and which are local to a scope.
To fix the problem, we have to modify the symtable_visit_stmt function in Python/symtable.c, adding code for handling until statements, after the similar code for while statements [3]:
case While_kind:
VISIT(st, expr, s->v.While.test);
VISIT_SEQ(st, stmt, s->v.While.body);
if (s->v.While.orelse)
VISIT_SEQ(st, stmt, s->v.While.orelse);
break;
case Until_kind:
VISIT(st, expr, s->v.Until.test);
VISIT_SEQ(st, stmt, s->v.Until.body);
break;
[3]: By the way, without this code there’s a compiler warning for Python/symtable.c. The compiler notices that the Until_kind enumeration value isn’t handled in the switch statement of symtable_visit_stmt and complains. It’s always important to check for compiler warnings!
And now we really are done. Compiling the source after this change makes the execution of myfoo(3) work as expected.
Conclusion
In this article I've demonstrated how to add a new statement to Python. Albeit requiring quite a bit of tinkering in the code of the Python compiler, the change wasn't difficult to implement, because I used a similar and existing statement as a guideline.
The Python compiler is a sophisticated chunk of software, and I don't claim being an expert in it. However, I am really interested in the internals of Python, and particularly its front-end. Therefore, I found this exercise a very useful companion to theoretical study of the compiler's principles and source code. It will serve as a base for future articles that will get deeper into the compiler.
References
I used a few excellent references for the construction of this article. Here they are, in no particular order:
PEP 339: Design of the CPython compiler - probably the most important and comprehensive piece of official documentation for the Python compiler. Being very short, it painfully displays the scarcity of good documentation of the internals of Python.
"Python Compiler Internals" - an article by Thomas Lee
"Python: Design and Implementation" - a presentation by Guido van Rossum
Python (2.5) Virtual Machine, A guided tour - a presentation by Peter Tröger
original source
One way to do things like this is to preprocess the source and modify it, translating your added statement to python. There are various problems this approach will bring, and I wouldn't recommend it for general usage, but for experimentation with language, or specific-purpose metaprogramming, it can occassionally be useful.
For instance, lets say we want to introduce a "myprint" statement, that instead of printing to the screen instead logs to a specific file. ie:
myprint "This gets logged to file"
would be equivalent to
print >>open('/tmp/logfile.txt','a'), "This gets logged to file"
There are various options as to how to do the replacing, from regex substitution to generating an AST, to writing your own parser depending on how close your syntax matches existing python. A good intermediate approach is to use the tokenizer module. This should allow you to add new keywords, control structures etc while interpreting the source similarly to the python interpreter, thus avoiding the breakage crude regex solutions would cause. For the above "myprint", you could write the following transformation code:
import tokenize
LOGFILE = '/tmp/log.txt'
def translate(readline):
for type, name,_,_,_ in tokenize.generate_tokens(readline):
if type ==tokenize.NAME and name =='myprint':
yield tokenize.NAME, 'print'
yield tokenize.OP, '>>'
yield tokenize.NAME, "open"
yield tokenize.OP, "("
yield tokenize.STRING, repr(LOGFILE)
yield tokenize.OP, ","
yield tokenize.STRING, "'a'"
yield tokenize.OP, ")"
yield tokenize.OP, ","
else:
yield type,name
(This does make myprint effectively a keyword, so use as a variable elsewhere will likely cause problems)
The problem then is how to use it so that your code is usable from python. One way would just be to write your own import function, and use it to load code written in your custom language. ie:
import new
def myimport(filename):
mod = new.module(filename)
f=open(filename)
data = tokenize.untokenize(translate(f.readline))
exec data in mod.__dict__
return mod
This requires you handle your customised code differently from normal python modules however. ie "some_mod = myimport("some_mod.py")" rather than "import some_mod"
Another fairly neat (albeit hacky) solution is to create a custom encoding (See PEP 263) as this recipe demonstrates. You could implement this as:
import codecs, cStringIO, encodings
from encodings import utf_8
class StreamReader(utf_8.StreamReader):
def __init__(self, *args, **kwargs):
codecs.StreamReader.__init__(self, *args, **kwargs)
data = tokenize.untokenize(translate(self.stream.readline))
self.stream = cStringIO.StringIO(data)
def search_function(s):
if s!='mylang': return None
utf8=encodings.search_function('utf8') # Assume utf8 encoding
return codecs.CodecInfo(
name='mylang',
encode = utf8.encode,
decode = utf8.decode,
incrementalencoder=utf8.incrementalencoder,
incrementaldecoder=utf8.incrementaldecoder,
streamreader=StreamReader,
streamwriter=utf8.streamwriter)
codecs.register(search_function)
Now after this code gets run (eg. you could place it in your .pythonrc or site.py) any code starting with the comment "# coding: mylang" will automatically be translated through the above preprocessing step. eg.
# coding: mylang
myprint "this gets logged to file"
for i in range(10):
myprint "so does this : ", i, "times"
myprint ("works fine" "with arbitrary" + " syntax"
"and line continuations")
Caveats:
There are problems to the preprocessor approach, as you'll probably be familiar with if you've worked with the C preprocessor. The main one is debugging. All python sees is the preprocessed file which means that text printed in the stack trace etc will refer to that. If you've performed significant translation, this may be very different from your source text. The example above doesn't change line numbers etc, so won't be too different, but the more you change it, the harder it will be to figure out.
Yes, to some extent it is possible. There is a module out there that uses sys.settrace() to implement goto and comefrom "keywords":
from goto import goto, label
for i in range(1, 10):
for j in range(1, 20):
print i, j
if j == 3:
goto .end # breaking out from nested loop
label .end
print "Finished"
Short of changing and recompiling the source code (which is possible with open source), changing the base language is not really possible.
Even if you do recompile the source, it wouldn't be python, just your hacked-up changed version which you need to be very careful not to introduce bugs into.
However, I'm not sure why you'd want to. Python's object-oriented features makes it quite simple to achieve similar results with the language as it stands.
General answer: you need to preprocess your source files.
More specific answer: install EasyExtend, and go through following steps
i) Create a new langlet ( extension language )
import EasyExtend
EasyExtend.new_langlet("mystmts", prompt = "my> ", source_ext = "mypy")
Without additional specification a bunch of files shall be created under EasyExtend/langlets/mystmts/ .
ii) Open mystmts/parsedef/Grammar.ext and add following lines
small_stmt: (expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | exec_stmt | assert_stmt | my_stmt )
my_stmt: 'mystatement' expr
This is sufficient to define the syntax of your new statement. The small_stmt non-terminal is part of the Python grammar and it's the place where the new statement is hooked in. The parser will now recognize the new statement i.e. a source file containing it will be parsed. The compiler will reject it though because it still has to be transformed into valid Python.
iii) Now one has to add semantics of the statement. For this one has to edit
msytmts/langlet.py and add a my_stmt node visitor.
def call_my_stmt(expression):
"defines behaviour for my_stmt"
print "my stmt called with", expression
class LangletTransformer(Transformer):
#transform
def my_stmt(self, node):
_expr = find_node(node, symbol.expr)
return any_stmt(CST_CallFunc("call_my_stmt", [_expr]))
__publish__ = ["call_my_stmt"]
iv) cd to langlets/mystmts and type
python run_mystmts.py
Now a session shall be started and the newly defined statement can be used:
__________________________________________________________________________________
mystmts
On Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)]
__________________________________________________________________________________
my> mystatement 40+2
my stmt called with 42
Quite a few steps to come to a trivial statement, right? There isn't an API yet that lets one define simple things without having to care about grammars. But EE is very reliable modulo some bugs. So it's just a matter of time that an API emerges that lets programmers define convenient stuff like infix operators or small statements using just convenient OO programming. For more complex things like embedding whole languages in Python by means of building a langlet there is no way of going around a full grammar approach.
Here's a very simple but crappy way to add new statements, in interpretive mode only. I'm using it for little 1-letter commands for editing gene annotations using only sys.displayhook, but just so I could answer this question I added sys.excepthook for the syntax errors as well. The latter is really ugly, fetching the raw code from the readline buffer. The benefit is, it's trivially easy to add new statements this way.
jcomeau#intrepid:~/$ cat demo.py; ./demo.py
#!/usr/bin/python -i
'load everything needed under "package", such as package.common.normalize()'
import os, sys, readline, traceback
if __name__ == '__main__':
class t:
#staticmethod
def localfunction(*args):
print 'this is a test'
if args:
print 'ignoring %s' % repr(args)
def displayhook(whatever):
if hasattr(whatever, 'localfunction'):
return whatever.localfunction()
else:
print whatever
def excepthook(exctype, value, tb):
if exctype is SyntaxError:
index = readline.get_current_history_length()
item = readline.get_history_item(index)
command = item.split()
print 'command:', command
if len(command[0]) == 1:
try:
eval(command[0]).localfunction(*command[1:])
except:
traceback.print_exception(exctype, value, tb)
else:
traceback.print_exception(exctype, value, tb)
sys.displayhook = displayhook
sys.excepthook = excepthook
>>> t
this is a test
>>> t t
command: ['t', 't']
this is a test
ignoring ('t',)
>>> ^D
I've found a guide on adding new statements:
https://troeger.eu/files/teaching/pythonvm08lab.pdf
Basically, to add new statements, you must edit Python/ast.c (among other things) and recompile the python binary.
While it's possible, don't. You can achieve almost everything via functions and classes (which wont require people to recompile python just to run your script..)
It's possible to do this using EasyExtend:
EasyExtend (EE) is a preprocessor
generator and metaprogramming
framework written in pure Python and
integrated with CPython. The main
purpose of EasyExtend is the creation
of extension languages i.e. adding
custom syntax and semantics to Python.
It's not exactly adding new statements to the language syntax, but macros are a powerful tool: https://github.com/lihaoyi/macropy
Some things can be done with decorators. Let's e.g. assume, Python had no with statement. We could then implement a similar behaviour like this:
# ====== Implementation of "mywith" decorator ======
def mywith(stream):
def decorator(function):
try: function(stream)
finally: stream.close()
return decorator
# ====== Using the decorator ======
#mywith(open("test.py","r"))
def _(infile):
for l in infile.readlines():
print(">>", l.rstrip())
It is a pretty unclean solution however as done here. Especially the behaviour where the decorator calls the function and sets _ to None is unexpected. For clarification: This decorator is equivalent to writing
def _(infile): ...
_ = mywith(open(...))(_) # mywith returns None.
and decorators are normally expected to modify, not to execute, functions.
I used such a method before in a script where I had to temporarily set the working directory for several functions.
OUTDATED:
The Logix project is now deprecated and no longer developed, per the Logix website.
There is a language based on python called Logix with which you CAN do such things. It hasn't been under development for a while, but the features that you asked for do work with the latest version.
Not without modifying the interpreter. I know a lot of languages in the past several years have been described as "extensible", but not in the way you're describing. You extend Python by adding functions and classes.
Ten years ago you couldn't, and I doubt that's changed. However, it wasn't that hard to modify the syntax back then if you were prepared to recompile python, and I doubt that's changed, either.
I've just started getting into adding docstrings to my classes/methods and I'm having difficulty in formatting them such that they are easily readable when printed. Some of the lines within the docstrings are long enough to wrap around on my IDE, and when I print the docstring in the console there are large gaps of whitespace in these breaks. Additionally, I would like to maintain a consistent indent scheme throughout the docstring, but these linebreaks violate it by forcing lines to print with not indent.
Are there certain best practices to docstring writing that I'm ignoring? Are there ways to print large strings such that formatting is respected?
Hope this makes sense, thanks.
Normally you use the help utility for viewing docstrings, it deals with inconsistency in whitespace:
>>> def test():
""" first line
HELLO THERE#
ME TOO
DOC STRING HERE
"""
return 1
>>> help(test)
Help on function test in module __main__:
test()
first line
HELLO THERE#
ME TOO
DOC STRING HERE
>>> def test2():
"""
DOC string
text here"""
return 5
>>> help(test2)
Help on function test2 in module __main__:
test2()
DOC string
text here
So while you can refer to PEP 8 for usual conventions, you can also just decide what format you like and just try to be consistent within your application.
I am trying to design the package and module system for a programming language (Heron) which can be both compiled and interpreted, and from what I have seen I really like the Python approach. Python has a rich choice of modules, which seems to contribute largely to its success.
What I don`t know is what happens in Python if a module is included in two different compiled packages: are there separate copies made of the data or is it shared?
Related to this are a bunch of side-questions:
Am I right in assuming that packages can be compiled in Python?
What are there pros and cons to the two approaches (copying or sharing of module data)?
Are there widely known problems with the Python module system, from the point of view of the Python community? For example is there a PEP under consideration for enhancing modules/packages?
Are there certain aspects of the Python module/package system which wouldn`t work well for a compiled language?
Well, you asked a lot of questions. Here are some hints to get a bit further:
a. Python code is lexed and compiled into Python specific instructions, but not compiled to machine executable code. The ".pyc" file is automatically created whenever you run python code that does not match the existing .pyc timestamp. This feature can be turned off. You might play with the dis module to see these instructions.
b. When a module is imported, it is executed (top to bottom) in its own namespace and that namespace cached globally. When you import from another module, the module is not executed again. Remember that def is just a statement. You may want to put a print('compiling this module') statement in your code to trace it.
It depends.
There were recent enhancements, mostly around specifying which module needed to be loaded. Modules can have relative paths so that a huge project might have multiple modules with the a same name.
Python itself won't work for a compiled language. Google for "unladen swallow blog" to see the tribulations of trying to speed up a language where "a = sum(b)" can change meanings between executions. Outside of corner cases, the module system forms a nice bridge between source code and a compiled library system. The approach works well, and Python's easy wrapping of C code (swig, etc.) helps.
Modules are the only truly global objects in Python, with all other global data based around the module system (which uses sys.modules as a registry). Packages are simply modules with special semantics for importing submodules. "Compiling" a .py file into a .pyc or .pyo isn't compilation as understood for most languages: it only checks the syntax and creates a code object which, when executed in the interpreter, creates the module object.
example.py:
print "Creating %s module." % __name__
def show_def(f):
print "Creating function %s.%s." % (__name__, f.__name__)
return f
#show_def
def a():
print "called: %s.a" % __name__
Interactive session:
>>> import example
# first sys.modules['example'] is checked
# since it doesn't exist, example.py is found and "compiled" to example.pyc
# (since example.pyc doesn't exist, same would happen if it was outdated, etc.)
Creating example module. # module code is executed
Creating function example.a. # def statement executed
>>> example.a()
called: example.a
>>> import example
# sys.modules['example'] found, local variable example assigned to that object
# no 'Creating ..' output
>>> d = {"__name__": "fake"}
>>> exec open("example.py") in d
# the first import in this session is very similar to this
# in that it creates a module object (which has a __dict__), initializes a few
# variables in it (__builtins__, __name__, and others---packages' __init__
# modules have their own as well---look at some_module.__dict__.keys() or
# dir(some_module))
# and executes the code from example.py in this dict (or the code object stored
# in example.pyc, etc.)
Creating fake module. # module code is executed
Creating function fake.a. # def statement executed
>>> d.keys()
['__builtins__', '__name__', 'a', 'show_def']
>>> d['a']()
called: fake.a
Your questions:
They are compiled, in a sense, but not as you would expect if you're familiar with how C compilers work.
If the data is immutable, copying is feasible, and should be indistinguishable from sharing except for object identity (is operator and id() in Python).
Imports may or may not execute code (they always assign a local variable to an object, but that poses no problems) and may or may not modify sys.modules. You must be careful to not import in threads, and generally it is best to do all imports at the top of every module: this leads to a cascading graph so all the imports are done at once and then __main__ continues and does the Real Work™.
I don't know of any current PEP, but there's already a lot of complex machinery in place, too. For example packages can have a __path__ attribute (really a list of paths) so submodules don't have to be in the same directory, and these paths can even be computed at runtime! (Example mungepath package below.) You can have your own import hooks, use import statements inside functions, directly call __import__, and I wouldn't be surprised to find 2-3 other unique ways to work with packages and modules.
A subset of the import system would work in a traditionally-compiled language, as long as it was similar to something like C's #include. You could run the "first level" of execution (creating the module objects) in the compiler, and compile those results. There are significant drawbacks to this, however, and amounts to separate execution contexts for module-level code and functions executed at runtime (and some functions would have to run in both contexts!). (Remember in Python that every statement is executed at runtime, even def and class statements.)
I believe this is the main reason traditionally-compiled languages restrict "top-level" code to class, function, and object declarations, eliminating this second context. Even then, you have initialization problems for global objects in C/C++ (and others), unless managed carefully.
mungepath/__init__.py:
print __path__
__path__.append(".") # CWD, would be different in non-example code
print __path__
from . import example # this is example.py from above, and is NOT in mungepath/
# note that this is a degenerate case, in that we now have two names for the
# 'same' module: example and mungepath.example, but they're really different
# modules with different functions (use 'is' or 'id()' to verify)
Interactive session:
>>> import example
Creating example module.
Creating function example.a.
>>> example.__dict__.keys()
['a', '__builtins__', '__file__', 'show_def', '__package__',
'__name__', '__doc__']
>>> import mungepath
['mungepath']
['mungepath', '.']
Creating mungepath.example module.
Creating function mungepath.example.a.
>>> mungepath.example.a()
called: mungepath.example.a
>>> example is mungepath.example
False
>>> example.a is mungepath.example.a
False
Global data is scoped at the interpreter level.
"packages" can be compiled as a package is just a collection of modules which themselves can be compiled.
I am not sure I understand given the established scoping of data.
How to do conditional compilation in Python ?
Is it using DEF ?
Python isn't compiled in the same sense as C or C++ or even Java, python files are compiled "on the fly", you can think of it as being similar to a interpreted language like Basic or Perl.1
You can do something equivalent to conditional compile by just using an if statement. For example:
if FLAG:
def f():
print "Flag is set"
else:
def f():
print "Flag is not set"
You can do the same for the creation classes, setting of variables and pretty much everything.
The closest way to mimic IFDEF would be to use the hasattr function. E.g.:
if hasattr(aModule, 'FLAG'):
# do stuff if FLAG is defined in the current module.
You could also use a try/except clause to catch name errors, but the idiomatic way would be to set a variable to None at the top of your script.
Python code is byte compiled into an intermediate form like Java, however there generally isn't a separate compilation step. The "raw" source files that end in .py are executable.
There is actually a way to get conditional compilation, but it's very limited.
if __debug__:
doSomething()
The __debug__ flag is a special case. When calling python with the -O or -OO options, __debug__ will be false, and the compiler will ignore that statement. This is used primarily with asserts, which is why assertions go away if you 'really compile' your scripts with optimization.
So if your goal is to add debugging code, but prevent it from slowing down or otherwise affecting a 'release' build, this does what you want. But you cannot assign a value to __debug__, so that's about all you can use it for.
Use pypreprocessor
Which can also be found on PYPI (Python Package Index) and can be fetched using pip.
The basic example of usage is:
from pypreprocessor import pypreprocessor
pypreprocessor.parse()
#define debug
#ifdef debug
print('The source is in debug mode')
#else
print('The source is not in debug mode')
#endif
You can also output the postprocessed code to a file by specifying...
pypreprocessor.output = 'output_file_name.py'
anywhere between the pypreprocessor import and the call to parse().
The module is essentially the python implementation of C preprocessor conditional compilation.
SideNote: This is compatible with both python2x and python 3k
Disclaimer: I'm the author of pypreprocessor
Update:
I forgot to mention before. Unlike the if/else or if _debug: approaches described in other answers, this is a true preprocessor. The bytecode produced will not contain the code that is conditionally excluded.
Python compiles a module automatically when you import it, so the only way to avoid compiling it is to not import it. You can write something like:
if some_condition:
import some_module
But that would only work for complete modules. In C and C++ you typically use a preprocessor for conditional compilation. There is nothing stopping you from using a preprocessor on your Python code, so you could write something like:
#ifdef SOME_CONDITION
def some_function():
pass
#endif
Run that through a C preprocessor and you'd have real conditional compilation and some_function will only be defined if SOME_CONDITION is defined.
BUT (and this is important): Conditional compilation is probably not what you want. Remember that when you import a module, Python simply executes the code in it. The def and class statements in the module are actually executed when you import the module. So the typical way of implementing what other languages would use conditional compilation for is just a normal if statement, like:
if some_condition:
def some_function():
pass
This will only define some_function if some_condition is true.
It's stuff like this that makes dynamic languages so powerful while remaining conceptually simple.
Doesn't make much sense in a dynamic environment. If you are looking for conditional definition of functions, you can use if:
if happy:
def makemehappy():
return "I'm good"
You could use the method discussed here: Determine if variable is defined in Python as a substitute for #ifdef
Can you add new statements (like print, raise, with) to Python's syntax?
Say, to allow..
mystatement "Something"
Or,
new_if True:
print "example"
Not so much if you should, but rather if it's possible (short of modifying the python interpreters code)
You may find this useful - Python internals: adding a new statement to Python, quoted here:
This article is an attempt to better understand how the front-end of Python works. Just reading documentation and source code may be a bit boring, so I'm taking a hands-on approach here: I'm going to add an until statement to Python.
All the coding for this article was done against the cutting-edge Py3k branch in the Python Mercurial repository mirror.
The until statement
Some languages, like Ruby, have an until statement, which is the complement to while (until num == 0 is equivalent to while num != 0). In Ruby, I can write:
num = 3
until num == 0 do
puts num
num -= 1
end
And it will print:
3
2
1
So, I want to add a similar capability to Python. That is, being able to write:
num = 3
until num == 0:
print(num)
num -= 1
A language-advocacy digression
This article doesn't attempt to suggest the addition of an until statement to Python. Although I think such a statement would make some code clearer, and this article displays how easy it is to add, I completely respect Python's philosophy of minimalism. All I'm trying to do here, really, is gain some insight into the inner workings of Python.
Modifying the grammar
Python uses a custom parser generator named pgen. This is a LL(1) parser that converts Python source code into a parse tree. The input to the parser generator is the file Grammar/Grammar[1]. This is a simple text file that specifies the grammar of Python.
[1]: From here on, references to files in the Python source are given relatively to the root of the source tree, which is the directory where you run configure and make to build Python.
Two modifications have to be made to the grammar file. The first is to add a definition for the until statement. I found where the while statement was defined (while_stmt), and added until_stmt below [2]:
compound_stmt: if_stmt | while_stmt | until_stmt | for_stmt | try_stmt | with_stmt | funcdef | classdef | decorated
if_stmt: 'if' test ':' suite ('elif' test ':' suite)* ['else' ':' suite]
while_stmt: 'while' test ':' suite ['else' ':' suite]
until_stmt: 'until' test ':' suite
[2]: This demonstrates a common technique I use when modifying source code I’m not familiar with: work by similarity. This principle won’t solve all your problems, but it can definitely ease the process. Since everything that has to be done for while also has to be done for until, it serves as a pretty good guideline.
Note that I've decided to exclude the else clause from my definition of until, just to make it a little bit different (and because frankly I dislike the else clause of loops and don't think it fits well with the Zen of Python).
The second change is to modify the rule for compound_stmt to include until_stmt, as you can see in the snippet above. It's right after while_stmt, again.
When you run make after modifying Grammar/Grammar, notice that the pgen program is run to re-generate Include/graminit.h and Python/graminit.c, and then several files get re-compiled.
Modifying the AST generation code
After the Python parser has created a parse tree, this tree is converted into an AST, since ASTs are much simpler to work with in subsequent stages of the compilation process.
So, we're going to visit Parser/Python.asdl which defines the structure of Python's ASTs and add an AST node for our new until statement, again right below the while:
| While(expr test, stmt* body, stmt* orelse)
| Until(expr test, stmt* body)
If you now run make, notice that before compiling a bunch of files, Parser/asdl_c.py is run to generate C code from the AST definition file. This (like Grammar/Grammar) is another example of the Python source-code using a mini-language (in other words, a DSL) to simplify programming. Also note that since Parser/asdl_c.py is a Python script, this is a kind of bootstrapping - to build Python from scratch, Python already has to be available.
While Parser/asdl_c.py generated the code to manage our newly defined AST node (into the files Include/Python-ast.h and Python/Python-ast.c), we still have to write the code that converts a relevant parse-tree node into it by hand. This is done in the file Python/ast.c. There, a function named ast_for_stmt converts parse tree nodes for statements into AST nodes. Again, guided by our old friend while, we jump right into the big switch for handling compound statements and add a clause for until_stmt:
case while_stmt:
return ast_for_while_stmt(c, ch);
case until_stmt:
return ast_for_until_stmt(c, ch);
Now we should implement ast_for_until_stmt. Here it is:
static stmt_ty
ast_for_until_stmt(struct compiling *c, const node *n)
{
/* until_stmt: 'until' test ':' suite */
REQ(n, until_stmt);
if (NCH(n) == 4) {
expr_ty expression;
asdl_seq *suite_seq;
expression = ast_for_expr(c, CHILD(n, 1));
if (!expression)
return NULL;
suite_seq = ast_for_suite(c, CHILD(n, 3));
if (!suite_seq)
return NULL;
return Until(expression, suite_seq, LINENO(n), n->n_col_offset, c->c_arena);
}
PyErr_Format(PyExc_SystemError,
"wrong number of tokens for 'until' statement: %d",
NCH(n));
return NULL;
}
Again, this was coded while closely looking at the equivalent ast_for_while_stmt, with the difference that for until I've decided not to support the else clause. As expected, the AST is created recursively, using other AST creating functions like ast_for_expr for the condition expression and ast_for_suite for the body of the until statement. Finally, a new node named Until is returned.
Note that we access the parse-tree node n using some macros like NCH and CHILD. These are worth understanding - their code is in Include/node.h.
Digression: AST composition
I chose to create a new type of AST for the until statement, but actually this isn't necessary. I could've saved some work and implemented the new functionality using composition of existing AST nodes, since:
until condition:
# do stuff
Is functionally equivalent to:
while not condition:
# do stuff
Instead of creating the Until node in ast_for_until_stmt, I could have created a Not node with an While node as a child. Since the AST compiler already knows how to handle these nodes, the next steps of the process could be skipped.
Compiling ASTs into bytecode
The next step is compiling the AST into Python bytecode. The compilation has an intermediate result which is a CFG (Control Flow Graph), but since the same code handles it I will ignore this detail for now and leave it for another article.
The code we will look at next is Python/compile.c. Following the lead of while, we find the function compiler_visit_stmt, which is responsible for compiling statements into bytecode. We add a clause for Until:
case While_kind:
return compiler_while(c, s);
case Until_kind:
return compiler_until(c, s);
If you wonder what Until_kind is, it's a constant (actually a value of the _stmt_kind enumeration) automatically generated from the AST definition file into Include/Python-ast.h. Anyway, we call compiler_until which, of course, still doesn't exist. I'll get to it an a moment.
If you're curious like me, you'll notice that compiler_visit_stmt is peculiar. No amount of grep-ping the source tree reveals where it is called. When this is the case, only one option remains - C macro-fu. Indeed, a short investigation leads us to the VISIT macro defined in Python/compile.c:
#define VISIT(C, TYPE, V) {\
if (!compiler_visit_ ## TYPE((C), (V))) \
return 0; \
It's used to invoke compiler_visit_stmt in compiler_body. Back to our business, however...
As promised, here's compiler_until:
static int
compiler_until(struct compiler *c, stmt_ty s)
{
basicblock *loop, *end, *anchor = NULL;
int constant = expr_constant(s->v.Until.test);
if (constant == 1) {
return 1;
}
loop = compiler_new_block(c);
end = compiler_new_block(c);
if (constant == -1) {
anchor = compiler_new_block(c);
if (anchor == NULL)
return 0;
}
if (loop == NULL || end == NULL)
return 0;
ADDOP_JREL(c, SETUP_LOOP, end);
compiler_use_next_block(c, loop);
if (!compiler_push_fblock(c, LOOP, loop))
return 0;
if (constant == -1) {
VISIT(c, expr, s->v.Until.test);
ADDOP_JABS(c, POP_JUMP_IF_TRUE, anchor);
}
VISIT_SEQ(c, stmt, s->v.Until.body);
ADDOP_JABS(c, JUMP_ABSOLUTE, loop);
if (constant == -1) {
compiler_use_next_block(c, anchor);
ADDOP(c, POP_BLOCK);
}
compiler_pop_fblock(c, LOOP, loop);
compiler_use_next_block(c, end);
return 1;
}
I have a confession to make: this code wasn't written based on a deep understanding of Python bytecode. Like the rest of the article, it was done in imitation of the kin compiler_while function. By reading it carefully, however, keeping in mind that the Python VM is stack-based, and glancing into the documentation of the dis module, which has a list of Python bytecodes with descriptions, it's possible to understand what's going on.
That's it, we're done... Aren't we?
After making all the changes and running make, we can run the newly compiled Python and try our new until statement:
>>> until num == 0:
... print(num)
... num -= 1
...
3
2
1
Voila, it works! Let's see the bytecode created for the new statement by using the dis module as follows:
import dis
def myfoo(num):
until num == 0:
print(num)
num -= 1
dis.dis(myfoo)
Here's the result:
4 0 SETUP_LOOP 36 (to 39)
>> 3 LOAD_FAST 0 (num)
6 LOAD_CONST 1 (0)
9 COMPARE_OP 2 (==)
12 POP_JUMP_IF_TRUE 38
5 15 LOAD_NAME 0 (print)
18 LOAD_FAST 0 (num)
21 CALL_FUNCTION 1
24 POP_TOP
6 25 LOAD_FAST 0 (num)
28 LOAD_CONST 2 (1)
31 INPLACE_SUBTRACT
32 STORE_FAST 0 (num)
35 JUMP_ABSOLUTE 3
>> 38 POP_BLOCK
>> 39 LOAD_CONST 0 (None)
42 RETURN_VALUE
The most interesting operation is number 12: if the condition is true, we jump to after the loop. This is correct semantics for until. If the jump isn't executed, the loop body keeps running until it jumps back to the condition at operation 35.
Feeling good about my change, I then tried running the function (executing myfoo(3)) instead of showing its bytecode. The result was less than encouraging:
Traceback (most recent call last):
File "zy.py", line 9, in
myfoo(3)
File "zy.py", line 5, in myfoo
print(num)
SystemError: no locals when loading 'print'
Whoa... this can't be good. So what went wrong?
The case of the missing symbol table
One of the steps the Python compiler performs when compiling the AST is create a symbol table for the code it compiles. The call to PySymtable_Build in PyAST_Compile calls into the symbol table module (Python/symtable.c), which walks the AST in a manner similar to the code generation functions. Having a symbol table for each scope helps the compiler figure out some key information, such as which variables are global and which are local to a scope.
To fix the problem, we have to modify the symtable_visit_stmt function in Python/symtable.c, adding code for handling until statements, after the similar code for while statements [3]:
case While_kind:
VISIT(st, expr, s->v.While.test);
VISIT_SEQ(st, stmt, s->v.While.body);
if (s->v.While.orelse)
VISIT_SEQ(st, stmt, s->v.While.orelse);
break;
case Until_kind:
VISIT(st, expr, s->v.Until.test);
VISIT_SEQ(st, stmt, s->v.Until.body);
break;
[3]: By the way, without this code there’s a compiler warning for Python/symtable.c. The compiler notices that the Until_kind enumeration value isn’t handled in the switch statement of symtable_visit_stmt and complains. It’s always important to check for compiler warnings!
And now we really are done. Compiling the source after this change makes the execution of myfoo(3) work as expected.
Conclusion
In this article I've demonstrated how to add a new statement to Python. Albeit requiring quite a bit of tinkering in the code of the Python compiler, the change wasn't difficult to implement, because I used a similar and existing statement as a guideline.
The Python compiler is a sophisticated chunk of software, and I don't claim being an expert in it. However, I am really interested in the internals of Python, and particularly its front-end. Therefore, I found this exercise a very useful companion to theoretical study of the compiler's principles and source code. It will serve as a base for future articles that will get deeper into the compiler.
References
I used a few excellent references for the construction of this article. Here they are, in no particular order:
PEP 339: Design of the CPython compiler - probably the most important and comprehensive piece of official documentation for the Python compiler. Being very short, it painfully displays the scarcity of good documentation of the internals of Python.
"Python Compiler Internals" - an article by Thomas Lee
"Python: Design and Implementation" - a presentation by Guido van Rossum
Python (2.5) Virtual Machine, A guided tour - a presentation by Peter Tröger
original source
One way to do things like this is to preprocess the source and modify it, translating your added statement to python. There are various problems this approach will bring, and I wouldn't recommend it for general usage, but for experimentation with language, or specific-purpose metaprogramming, it can occassionally be useful.
For instance, lets say we want to introduce a "myprint" statement, that instead of printing to the screen instead logs to a specific file. ie:
myprint "This gets logged to file"
would be equivalent to
print >>open('/tmp/logfile.txt','a'), "This gets logged to file"
There are various options as to how to do the replacing, from regex substitution to generating an AST, to writing your own parser depending on how close your syntax matches existing python. A good intermediate approach is to use the tokenizer module. This should allow you to add new keywords, control structures etc while interpreting the source similarly to the python interpreter, thus avoiding the breakage crude regex solutions would cause. For the above "myprint", you could write the following transformation code:
import tokenize
LOGFILE = '/tmp/log.txt'
def translate(readline):
for type, name,_,_,_ in tokenize.generate_tokens(readline):
if type ==tokenize.NAME and name =='myprint':
yield tokenize.NAME, 'print'
yield tokenize.OP, '>>'
yield tokenize.NAME, "open"
yield tokenize.OP, "("
yield tokenize.STRING, repr(LOGFILE)
yield tokenize.OP, ","
yield tokenize.STRING, "'a'"
yield tokenize.OP, ")"
yield tokenize.OP, ","
else:
yield type,name
(This does make myprint effectively a keyword, so use as a variable elsewhere will likely cause problems)
The problem then is how to use it so that your code is usable from python. One way would just be to write your own import function, and use it to load code written in your custom language. ie:
import new
def myimport(filename):
mod = new.module(filename)
f=open(filename)
data = tokenize.untokenize(translate(f.readline))
exec data in mod.__dict__
return mod
This requires you handle your customised code differently from normal python modules however. ie "some_mod = myimport("some_mod.py")" rather than "import some_mod"
Another fairly neat (albeit hacky) solution is to create a custom encoding (See PEP 263) as this recipe demonstrates. You could implement this as:
import codecs, cStringIO, encodings
from encodings import utf_8
class StreamReader(utf_8.StreamReader):
def __init__(self, *args, **kwargs):
codecs.StreamReader.__init__(self, *args, **kwargs)
data = tokenize.untokenize(translate(self.stream.readline))
self.stream = cStringIO.StringIO(data)
def search_function(s):
if s!='mylang': return None
utf8=encodings.search_function('utf8') # Assume utf8 encoding
return codecs.CodecInfo(
name='mylang',
encode = utf8.encode,
decode = utf8.decode,
incrementalencoder=utf8.incrementalencoder,
incrementaldecoder=utf8.incrementaldecoder,
streamreader=StreamReader,
streamwriter=utf8.streamwriter)
codecs.register(search_function)
Now after this code gets run (eg. you could place it in your .pythonrc or site.py) any code starting with the comment "# coding: mylang" will automatically be translated through the above preprocessing step. eg.
# coding: mylang
myprint "this gets logged to file"
for i in range(10):
myprint "so does this : ", i, "times"
myprint ("works fine" "with arbitrary" + " syntax"
"and line continuations")
Caveats:
There are problems to the preprocessor approach, as you'll probably be familiar with if you've worked with the C preprocessor. The main one is debugging. All python sees is the preprocessed file which means that text printed in the stack trace etc will refer to that. If you've performed significant translation, this may be very different from your source text. The example above doesn't change line numbers etc, so won't be too different, but the more you change it, the harder it will be to figure out.
Yes, to some extent it is possible. There is a module out there that uses sys.settrace() to implement goto and comefrom "keywords":
from goto import goto, label
for i in range(1, 10):
for j in range(1, 20):
print i, j
if j == 3:
goto .end # breaking out from nested loop
label .end
print "Finished"
Short of changing and recompiling the source code (which is possible with open source), changing the base language is not really possible.
Even if you do recompile the source, it wouldn't be python, just your hacked-up changed version which you need to be very careful not to introduce bugs into.
However, I'm not sure why you'd want to. Python's object-oriented features makes it quite simple to achieve similar results with the language as it stands.
General answer: you need to preprocess your source files.
More specific answer: install EasyExtend, and go through following steps
i) Create a new langlet ( extension language )
import EasyExtend
EasyExtend.new_langlet("mystmts", prompt = "my> ", source_ext = "mypy")
Without additional specification a bunch of files shall be created under EasyExtend/langlets/mystmts/ .
ii) Open mystmts/parsedef/Grammar.ext and add following lines
small_stmt: (expr_stmt | print_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | exec_stmt | assert_stmt | my_stmt )
my_stmt: 'mystatement' expr
This is sufficient to define the syntax of your new statement. The small_stmt non-terminal is part of the Python grammar and it's the place where the new statement is hooked in. The parser will now recognize the new statement i.e. a source file containing it will be parsed. The compiler will reject it though because it still has to be transformed into valid Python.
iii) Now one has to add semantics of the statement. For this one has to edit
msytmts/langlet.py and add a my_stmt node visitor.
def call_my_stmt(expression):
"defines behaviour for my_stmt"
print "my stmt called with", expression
class LangletTransformer(Transformer):
#transform
def my_stmt(self, node):
_expr = find_node(node, symbol.expr)
return any_stmt(CST_CallFunc("call_my_stmt", [_expr]))
__publish__ = ["call_my_stmt"]
iv) cd to langlets/mystmts and type
python run_mystmts.py
Now a session shall be started and the newly defined statement can be used:
__________________________________________________________________________________
mystmts
On Python 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)]
__________________________________________________________________________________
my> mystatement 40+2
my stmt called with 42
Quite a few steps to come to a trivial statement, right? There isn't an API yet that lets one define simple things without having to care about grammars. But EE is very reliable modulo some bugs. So it's just a matter of time that an API emerges that lets programmers define convenient stuff like infix operators or small statements using just convenient OO programming. For more complex things like embedding whole languages in Python by means of building a langlet there is no way of going around a full grammar approach.
Here's a very simple but crappy way to add new statements, in interpretive mode only. I'm using it for little 1-letter commands for editing gene annotations using only sys.displayhook, but just so I could answer this question I added sys.excepthook for the syntax errors as well. The latter is really ugly, fetching the raw code from the readline buffer. The benefit is, it's trivially easy to add new statements this way.
jcomeau#intrepid:~/$ cat demo.py; ./demo.py
#!/usr/bin/python -i
'load everything needed under "package", such as package.common.normalize()'
import os, sys, readline, traceback
if __name__ == '__main__':
class t:
#staticmethod
def localfunction(*args):
print 'this is a test'
if args:
print 'ignoring %s' % repr(args)
def displayhook(whatever):
if hasattr(whatever, 'localfunction'):
return whatever.localfunction()
else:
print whatever
def excepthook(exctype, value, tb):
if exctype is SyntaxError:
index = readline.get_current_history_length()
item = readline.get_history_item(index)
command = item.split()
print 'command:', command
if len(command[0]) == 1:
try:
eval(command[0]).localfunction(*command[1:])
except:
traceback.print_exception(exctype, value, tb)
else:
traceback.print_exception(exctype, value, tb)
sys.displayhook = displayhook
sys.excepthook = excepthook
>>> t
this is a test
>>> t t
command: ['t', 't']
this is a test
ignoring ('t',)
>>> ^D
I've found a guide on adding new statements:
https://troeger.eu/files/teaching/pythonvm08lab.pdf
Basically, to add new statements, you must edit Python/ast.c (among other things) and recompile the python binary.
While it's possible, don't. You can achieve almost everything via functions and classes (which wont require people to recompile python just to run your script..)
It's possible to do this using EasyExtend:
EasyExtend (EE) is a preprocessor
generator and metaprogramming
framework written in pure Python and
integrated with CPython. The main
purpose of EasyExtend is the creation
of extension languages i.e. adding
custom syntax and semantics to Python.
It's not exactly adding new statements to the language syntax, but macros are a powerful tool: https://github.com/lihaoyi/macropy
Some things can be done with decorators. Let's e.g. assume, Python had no with statement. We could then implement a similar behaviour like this:
# ====== Implementation of "mywith" decorator ======
def mywith(stream):
def decorator(function):
try: function(stream)
finally: stream.close()
return decorator
# ====== Using the decorator ======
#mywith(open("test.py","r"))
def _(infile):
for l in infile.readlines():
print(">>", l.rstrip())
It is a pretty unclean solution however as done here. Especially the behaviour where the decorator calls the function and sets _ to None is unexpected. For clarification: This decorator is equivalent to writing
def _(infile): ...
_ = mywith(open(...))(_) # mywith returns None.
and decorators are normally expected to modify, not to execute, functions.
I used such a method before in a script where I had to temporarily set the working directory for several functions.
OUTDATED:
The Logix project is now deprecated and no longer developed, per the Logix website.
There is a language based on python called Logix with which you CAN do such things. It hasn't been under development for a while, but the features that you asked for do work with the latest version.
Not without modifying the interpreter. I know a lot of languages in the past several years have been described as "extensible", but not in the way you're describing. You extend Python by adding functions and classes.
Ten years ago you couldn't, and I doubt that's changed. However, it wasn't that hard to modify the syntax back then if you were prepared to recompile python, and I doubt that's changed, either.