I would like to find a way where the execution of Python scripts automatically write the result of the expressions in the top level, as is done in interactive mode.
For instance, if I have this script.py:
abs(3)
for x in [1,2,3]:
print abs(x)
abs(-4)
print abs(5)
and execute python script.py, I will get
1
2
3
5
but I would rather have
3
1
2
3
4
5
which is what one would get executing it interactively (modulo prompts).
More or less, I would like to achieve the contrary of Disable automatic printing in Python interactive session . It seems that the code module could help me, but I got no no success with it.
Well, I'm not seriously proposing using something like this, but you could (ab)use ast processing:
# -*- coding: utf-8 -*-
import ast
import argparse
_parser = argparse.ArgumentParser()
_parser.add_argument('file')
class ExpressionPrinter(ast.NodeTransformer):
visit_ClassDef = visit_FunctionDef = lambda self, node: node
def visit_Expr(self, node):
node = ast.copy_location(
ast.Expr(
ast.Call(ast.Name('print', ast.Load()),
[node.value], [], None, None)
),
node
)
ast.fix_missing_locations(node)
return node
def main(args):
with open(args.file) as source:
tree = ast.parse(source.read(), args.file, mode='exec')
new_tree = ExpressionPrinter().visit(tree)
exec(compile(new_tree, args.file, mode='exec'))
if __name__ == '__main__':
main(_parser.parse_args())
Output for your example script.py:
% python2 printer.py test2.py
3
1
2
3
4
5
Related
After running some compuations nicely in linear fashion with a moderator script (cf. below) calling an inner one performing the computation, I struggle
to bring it to execution when trying it with multiprocessing. It seems that each CPU core is running through this list set (testRegister) and launches a computation even if an other core already performed this task earlier (in the same session). How can I prevent this chaotic behaviour? It is my first time attempting calling multiple processors by Python.
Correction: The initial post did not show that the test is a string consisting calling "the inner script" with varying parameters m1 and m2 beside fixed arguments arg1 and arg2 belonging solely to this "inner script".
#!/usr/bin/env python3
import os
import subprocess as sub
import sys
import multiprocessing
fileRegister = []
testRegister = []
def fileCollector():
for file in os.listdir("."):
if file.endswith(".xyz"):
fileRegister.append(file)
fileRegister.sort()
return fileRegister
def testSetup():
data = fileRegister
while len(data) > 1:
for entry in fileRegister[1:]:
m0 = str(fileRegister[0])
m1 = str(entry)
test = str("python foo.py ") + str(m1) + str(" ") + str(m2) +\
str(" --arg1 --arg2") # formulate test condition
testRegister.append(test)
testRegister.sort()
del data[0]
return testRegister
def shortAnalysator():
for entry in testRegister:
print(str(entry))
sub.call(entry, shell=True)
del testRegister[0]
def polyAnalysator():
# apparently each CPU core works as if the register were not shared
# reference: https://docs.python.org/3.7/library/multiprocessing.html
if __name__ == '__main__':
jobs = []
for i in range(3): # safety marging to not consume all CPU
p = multiprocessing.Process(target=shortAnalysator)
jobs.append(p)
p.start()
fileCollector()
testSetup()
shortAnalysator() # proceeding expectably on one CPU (slow)
# polyAnalysator() # causing irritation
sys.exit()```
Your polyAnalysator is running the shortAnalysator three times. Try changing your polyAnalysator as follows, and add the f method. This uses the multiprocessing Pool:
from multiprocessing import Pool
def f(test):
sub.call(test, shell=True)
def polyAnalysator():
# apparently each CPU core works as if the register were not shared
# reference: https://docs.python.org/3.7/library/multiprocessing.html
with Pool(3) as p:
p.map(f, testRegister)
Please explain the output?
# maindir/config/config.py
foo = 0
# maindir/worker/worker.py
import maindir.config.config
def print_foo():
print maindir.config.config.foo
# maindir/main.py
import config.config
import worker.worker
if __name__ == '__main__':
print config.config.foo
config.config.foo = 5
print config.config.foo
worker.worker.print_foo()
OUTPUT:
$ env PYTHONPATH=. python maindir/main.py
0
5
0 # was expecting 5 ...
On the other hand if we switch to import config.config in worker.py, then there is no surprise:
# maindir/worker/worker.py
import config.config
def print_foo():
print config.config.foo
OUTPUT:
$ env PYTHONPATH=./maindir python maindir/main.py
0
5
5
I have been struggling with this for a few days now and still dont have a good solution. Instead of providing code this time which with this problem has lately been leading to unhelpful tangents, let me just give you an idea of exactly what I am trying to accomplish and perhaps this will streamline the solution.
All I am trying to do run a python program while inputting a few variables to control what the program does. Allow me to give a specific example.
Example Syntax Structure
program_name function_to_run variable_1 variable_2 variable_n
Generic Syntax Example
parrot add "Mr Fluffy" "Red" "15oz"
Another Example
datamine search "Chris"
So to expand on these examples. The first program "parrot" has an add function. When the program is run and the add function is used from the command line, the program expects three variables (Name, color, weight). In the second example, the program named "datamine" has a function named "search" that expects a single string (the search term). The idea is, the program (datamine) for example will have several functions that could be used. Perhaps "add", "search", "delete" are all examples and each will have different expected variables. Using datamine help would list out each function and the required and or optional components.
Using argparse, I have not been able to figure out a working implementation of this yet. From past experience, I think the solution will involved using custom actions. Can anyone please help with some example code? I am using Python 3 by the way.
Thanks for the help!
Use subparsers. The docs give a good example of how to use set_defaults to specify the function that should be called for each subparser:
One particularly effective way of handling sub-commands is to combine the use of the add_subparsers() method with calls to set_defaults() so that each subparser knows which Python function it should execute.
In your examples, parrot and datamine would be separate parsers in separate modules, and add and search would be subparsers under them respectively. For example, the datamine module would look something like this:
#!/usr/bin/env python
# datamine
def add(a, b):
print(a + b)
def search(query, search_all=True):
run_my_search_app(query, search_all=search_all)
if __name__ == '__main__':
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "add" command
parser_add = subparsers.add_parser('add')
parser_add.add_argument('-a', type=int, default=1)
parser_add.add_argument('-b', type=int, default=2)
parser_add.set_defaults(func=add)
# create the parser for the "search" command
parser_search = subparsers.add_parser('search')
parser_search.add_argument('query')
parser_search.add_argument('--search-all', action='store_true')
parser_search.set_defaults(func=search)
args = parser.parse_args()
args = vars(args)
func = args.pop("func")
func(**args)
If this file is executable in your shell as datamine, you can do:
datamine add -a 11 -b 5
datamine search foo --search-all
Without optional flags you don't need anything fancy - just look at sys.argv directly:
import sys
def my_add(*args):
print( ','.join(args))
def my_search(*args):
print(args)
fn_map = {"add": my_add, "search": my_search}
if sys.argv[1:]:
fn = fn_map[sys.argv[1]]
rest = sys.argv[2:]
fn(*rest)
sample runs
1951:~/mypy$ python stack43990444.py
1951:~/mypy$ python stack43990444.py add "Mr Fluffy" "Red" "15oz"
Mr Fluffy,Red,15oz
1951:~/mypy$ python stack43990444.py search "Chris"
('Chris',)
Fully functional extrapolation of code from your parrot example using subparsers. Data set (created by this code) and usage examples at the bottom. Beware, example set does not consist strictly of parrots
#!/usr/bin/env python3
import argparse
import json
def add_parrot(name, weight, kind, **kwargs):
print("Adding {} of type {} and size {}".format(name, kind, weight))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots.append({'name': name, 'weight': weight, 'type': kind})
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def delete_parrot(name, **kwargs):
print("Uh oh! What happened to {}?".format(name))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots[:] = [p for p in parrots if p.get('name') != name]
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def show_parrots(name=None, weight=0, kind=None, **kwargs):
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
for p in parrots:
if (name or weight or kind):
if name in p['name'] or weight == p['weight'] or kind == p['type']:
print("{}\t{}\t{}".format(
p['name'], p['weight'], p['type']))
else:
print("{}\t{}\t{}".format(p['name'], p['weight'], p['type']))
parser = argparse.ArgumentParser(description="Manage Parrots")
subparsers = parser.add_subparsers()
add_parser = subparsers.add_parser('insert', aliases=['add', 'a'])
add_parser.add_argument('name')
add_parser.add_argument('weight', type=int)
add_parser.add_argument('kind')
add_parser.set_defaults(func=add_parrot)
del_parser = subparsers.add_parser("delete", aliases=['del', 'd'])
del_parser.add_argument('name')
del_parser.set_defaults(func=delete_parrot)
ls_parser = subparsers.add_parser('list', aliases=['show', 'ls'])
ls_parser.add_argument('--name')
ls_parser.add_argument('--size', type=int)
ls_parser.add_argument('--type', dest='kind')
ls_parser.set_defaults(func=show_parrots)
args = parser.parse_args()
args.func(**vars(args))
Dataset and usage examples:
➜ ~ cat parrots.json
[{"name": "tweety", "weight": 4, "type": "yellow"}, {"name": "donald", "weight": 18, "type": "white"}, {"name": "daffy", "weight": 12, "type": "black"}]
➜ ~ ./parrot.py ls
tweety 4 yellow
donald 18 white
daffy 12 black
➜ ~ ./parrot.py ls --name tweety
tweety 4 yellow
➜ ~ ./parrot.py delete tweety
Uh oh! What happened to tweety?
➜ ~ ./parrot.py ls --name tweety
➜ ~
I am following an example which shows limitation of python bindings, from site http://eli.thegreenplace.net/2011/07/03/parsing-c-in-python-with-clang/
It uses "libclang visitation API directly".
import sys
import clang.cindex
def callexpr_visitor(node, parent, userdata):
if node.kind == clang.cindex.CursorKind.CALL_EXPR:
print 'Found %s [line=%s, col=%s]' % (
node.spelling, node.location.line, node.location.column)
return 2 # means continue visiting recursively
index = clang.cindex.Index.create()
tu = index.parse(sys.argv[1])
clang.cindex.Cursor_visit(
tu.cursor,
clang.cindex.Cursor_visit_callback(callexpr_visitor),
None)
The output shows all functions called along with their line numbers.
Found foo [line=8, col=5]
Found foo [line=10, col=9]
Found bar [line=15, col=5]
Found foo [line=16, col=9]
Found bar [line=17, col=9]
When I run the same code, I only get output
Found bar [line=15, col=5]
The version I use is llvm3.1 with windows (with the changes suggested in the link).
I feel, returning 2 is not calling the callback function again.
I have even tried using 'get_children' on node and traversing without callback, I get the same result.
import sys
import clang.cindex
#def callexpr_visitor(node, parent, userdata):
def callexpr_visitor(node):
if node.kind == clang.cindex.CursorKind.CALL_EXPR:
print 'Found %s [line=%s, col=%s]' % (
clang.cindex.Cursor_displayname(node), node.location.line, node.location.column)
for c in node.get_children():
callexpr_visitor(c)
#return 2 # means continue visiting recursively
index = clang.cindex.Index.create()
tu = index.parse(sys.argv[1])
#clang.cindex.Cursor_visit(
# tu.cursor,
# clang.cindex.Cursor_visit_callback(callexpr_visitor),
# None)
callexpr_visitor(tu.cursor)
I could not get the reason for this behavior after much search and trials.
Can anyone explain this please ?
Regards.
I think I found the reason for this.
If I change the return type of function 'foo', from 'bool' to 'int', I get the expected result.
This may be as 'bool' is not a keyword in 'c'.
That was simple.
In order to work around a bug in Apple's lldb (rdar://13702081) I very frequently need to type two commands in sequence, like this:
(lldb) p todo.matA
(vMAT_Array *) $2 = 0x000000010400b5a0
(lldb) po $2.dump
$3 = 0x0000000100503ce0 <vMAT_Int8Array: 0x10400b5a0; size: [9 1]> =
1
1
1
1
1
1
1
1
1
Is it possible to write a new lldb command using the Python library (or something) that could combine those steps for me? Ideally to something like:
(lldb) pmat todo
$3 = 0x0000000100503ce0 <vMAT_Int8Array: 0x10400b5a0; size: [9 1]> =
1
1
1
1
1
1
1
1
1
Solution
Thanks to Jason Molenda here is output from a working lldb command script:
(lldb) pmat Z
$0 = 0x0000000100112920 <vMAT_DoubleArray: 0x101880c20; size: [9 3]> =
7 9 0.848715
3 5 0.993378
0 1 1.11738
4 12 1.2013
11 13 1.20193
6 10 1.29206
14 15 1.53283
8 16 1.53602
2 17 1.68116
I did have to tweak the script provided in the answer below very slightly, using Jason's suggestions for working around the lldb bug with overly-complex expressions. Here is my final script:
# import this into lldb with a command like
# command script import pmat.py
import lldb
import shlex
import optparse
def pmat(debugger, command, result, dict):
# Use the Shell Lexer to properly parse up command options just like a
# shell would
command_args = shlex.split(command)
parser = create_pmat_options()
try:
(options, args) = parser.parse_args(command_args)
except:
return
target = debugger.GetSelectedTarget()
if target:
process = target.GetProcess()
if process:
frame = process.GetSelectedThread().GetSelectedFrame()
if frame:
var = frame.FindVariable(args[0])
if var:
array = var.GetChildMemberWithName("matA")
if array:
id = array.GetValueAsUnsigned (lldb.LLDB_INVALID_ADDRESS)
if id != lldb.LLDB_INVALID_ADDRESS:
debugger.HandleCommand ('po [0x%x dump]' % id)
def create_pmat_options():
usage = "usage: %prog"
description='''Print a dump of a vMAT_Array instance.'''
parser = optparse.OptionParser(description=description, prog='pmat',usage=usage)
return parser
#
# code that runs when this script is imported into LLDB
#
def __lldb_init_module (debugger, dict):
# This initializer is being run from LLDB in the embedded command interpreter
# Make the options so we can generate the help text for the new LLDB
# command line command prior to registering it with LLDB below
# add pmat
parser = create_pmat_options()
pmat.__doc__ = parser.format_help()
# Add any commands contained in this module to LLDB
debugger.HandleCommand('command script add -f %s.pmat pmat' % __name__)
You can do this either with a regex command or by creating your own python command and loading it in to lldb. In this specific instance the regex command won't help you because you'll hit the same crasher you're hitting. But just for fun, I'll show both solutions.
First, python. This python code gets the currently selected frame on the currently selected thread. It looks for a variable whose name is provided on the command argument. It finds a child of that variable called matA and it runs GetObjectDescription() on that SBValue object.
# import this into lldb with a command like
# command script import pmat.py
import lldb
import shlex
import optparse
def pmat(debugger, command, result, dict):
# Use the Shell Lexer to properly parse up command options just like a
# shell would
command_args = shlex.split(command)
parser = create_pmat_options()
try:
(options, args) = parser.parse_args(command_args)
except:
return
target = debugger.GetSelectedTarget()
if target:
process = target.GetProcess()
if process:
frame = process.GetSelectedThread().GetSelectedFrame()
if frame:
var = frame.FindVariable(args[0])
if var:
child = var.GetChildMemberWithName("matA")
if child:
print child.GetObjectDescription()
def create_pmat_options():
usage = "usage: %prog"
description='''Call po on the child called "matA"'''
parser = optparse.OptionParser(description=description, prog='pmat',usage=usage)
return parser
#
# code that runs when this script is imported into LLDB
#
def __lldb_init_module (debugger, dict):
# This initializer is being run from LLDB in the embedded command interpreter
# Make the options so we can generate the help text for the new LLDB
# command line command prior to registering it with LLDB below
# add pmat
parser = create_pmat_options()
pmat.__doc__ = parser.format_help()
# Add any commands contained in this module to LLDB
debugger.HandleCommand('command script add -f %s.pmat pmat' % __name__)
In use,
(lldb) br s -p break
Breakpoint 2: where = a.out`main + 31 at a.m:8, address = 0x0000000100000eaf
(lldb) r
Process 18223 launched: '/private/tmp/a.out' (x86_64)
Process 18223 stopped
* thread #1: tid = 0x1f03, 0x0000000100000eaf a.out`main + 31 at a.m:8, stop reason = breakpoint 2.1
#0: 0x0000000100000eaf a.out`main + 31 at a.m:8
5 #autoreleasepool {
6 struct var myobj;
7 myobj.matA = #"hello there";
-> 8 printf ("%s\n", [(id)myobj.matA UTF8String]); // break here
9 }
10 }
(lldb) p myobj
(var) $0 = {
(void *) matA = 0x0000000100001070
}
(lldb) comm scri imp ~/lldb/pmat.py
(lldb) pmat myobj
hello there
(lldb)
You can put the command script import line in your ~/.lldbinit file if you want to use this.
It's easy to use the Python APIs once you have a general idea of how the debugger is structured. I knew that I would find the variable based on the frame, so I looked at the help for the SBFrame object with
(lldb) script help (lldb.SBFrame)
The method FindVariable returns an SBValue so then I looked at the lldb.SBValue help page, etc. There's a lot of boilerplate in my example python above - you're really looking at 4 lines of python that do all the work.
If this is still triggering the code path that is crashing your lldb process, you can do the last little bit of the script in two parts - get the address of the object and run po on that raw address. e.g.
child = var.GetChildMemberWithName("matA")
if child:
id = child.GetValueAsUnsigned (lldb.LLDB_INVALID_ADDRESS)
if id != lldb.LLDB_INVALID_ADDRESS:
debugger.HandleCommand ('po 0x%x' % id)
Second, using a command regex:
(lldb) br s -p break
Breakpoint 1: where = a.out`main + 31 at a.m:8, address = 0x0000000100000eaf
(lldb) r
Process 18277 launched: '/private/tmp/a.out' (x86_64)
Process 18277 stopped
* thread #1: tid = 0x1f03, 0x0000000100000eaf a.out`main + 31 at a.m:8, stop reason = breakpoint 1.1
#0: 0x0000000100000eaf a.out`main + 31 at a.m:8
5 #autoreleasepool {
6 struct var myobj;
7 myobj.matA = #"hello there";
-> 8 printf ("%s\n", [(id)myobj.matA UTF8String]); // break here
9 }
10 }
(lldb) command regex pmat 's/(.*)/po %1.matA/'
(lldb) pmat myobj
$0 = 0x0000000100001070 hello there
(lldb)
You can't use the simpler command alias in this instance - you have to use a regex alias - because you're calling a command which takes raw input. Specifically, po is really an alias to expression and you need to use regex command aliases to substitute values into those.