We Keep Coding

How to parse a Windows file path in textX?

I need to parse through a file path in Windows, make sure I have provided a csv file. I have tested the regex in an online regex generator and made sure it matches the text I provide it.
Program.tx:
Program:
'begin'
commands*=Command
'end'
;
Command:
Test | Configuration
;
Test:
'test'
;
Configuration:
'configuration' location=/[a-zA-Z:a-zA-Z\\]+(\.csv$)/
;
test.dsl:
begin
configuration C:\Users\me\Desktop\test.csv
end
program.py:
from textx import metamodel_from_file
from Input import Input
class Robot(object):
def __init__(self):
self.input_location = None
def setInput(self, location):
self.input = Input(location)
def interpret(self, model):
for c in model.commands:
if c.__class__.__name__ == "Configuration":
self.setInput(c.location)
robot_mm = metamodel_from_file('Program.tx')
robot_model = robot_mm.model_from_file('test.dsl')
robot = Robot()
robot.interpret(robot_model)
Once I use Robot.interpret(), I cannot parse through the provided filepath
textx.exceptions.TextXSyntaxError: None:2:19: error: Expected '[a-zA-Z:a-zA-Z\\]+(\.csv$)' at position c:\Users\me\Desktop\test.dsl:(2, 19) => 'on *C:\Users\me\Des'.

After spending a day on the problem, turns out textX doesn't like the anchor character - '$'.

Python3 argparse

I have been struggling with this for a few days now and still dont have a good solution. Instead of providing code this time which with this problem has lately been leading to unhelpful tangents, let me just give you an idea of exactly what I am trying to accomplish and perhaps this will streamline the solution.
All I am trying to do run a python program while inputting a few variables to control what the program does. Allow me to give a specific example.
Example Syntax Structure
program_name function_to_run variable_1 variable_2 variable_n
Generic Syntax Example
parrot add "Mr Fluffy" "Red" "15oz"
Another Example
datamine search "Chris"
So to expand on these examples. The first program "parrot" has an add function. When the program is run and the add function is used from the command line, the program expects three variables (Name, color, weight). In the second example, the program named "datamine" has a function named "search" that expects a single string (the search term). The idea is, the program (datamine) for example will have several functions that could be used. Perhaps "add", "search", "delete" are all examples and each will have different expected variables. Using datamine help would list out each function and the required and or optional components.
Using argparse, I have not been able to figure out a working implementation of this yet. From past experience, I think the solution will involved using custom actions. Can anyone please help with some example code? I am using Python 3 by the way.
Thanks for the help!

Use subparsers. The docs give a good example of how to use set_defaults to specify the function that should be called for each subparser:
One particularly effective way of handling sub-commands is to combine the use of the add_subparsers() method with calls to set_defaults() so that each subparser knows which Python function it should execute.
In your examples, parrot and datamine would be separate parsers in separate modules, and add and search would be subparsers under them respectively. For example, the datamine module would look something like this:
#!/usr/bin/env python
# datamine
def add(a, b):
print(a + b)
def search(query, search_all=True):
run_my_search_app(query, search_all=search_all)
if __name__ == '__main__':
# create the top-level parser
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers()
# create the parser for the "add" command
parser_add = subparsers.add_parser('add')
parser_add.add_argument('-a', type=int, default=1)
parser_add.add_argument('-b', type=int, default=2)
parser_add.set_defaults(func=add)
# create the parser for the "search" command
parser_search = subparsers.add_parser('search')
parser_search.add_argument('query')
parser_search.add_argument('--search-all', action='store_true')
parser_search.set_defaults(func=search)
args = parser.parse_args()
args = vars(args)
func = args.pop("func")
func(**args)
If this file is executable in your shell as datamine, you can do:
datamine add -a 11 -b 5
datamine search foo --search-all

Without optional flags you don't need anything fancy - just look at sys.argv directly:
import sys
def my_add(*args):
print( ','.join(args))
def my_search(*args):
print(args)
fn_map = {"add": my_add, "search": my_search}
if sys.argv[1:]:
fn = fn_map[sys.argv[1]]
rest = sys.argv[2:]
fn(*rest)
sample runs
1951:~/mypy$ python stack43990444.py
1951:~/mypy$ python stack43990444.py add "Mr Fluffy" "Red" "15oz"
Mr Fluffy,Red,15oz
1951:~/mypy$ python stack43990444.py search "Chris"
('Chris',)

Fully functional extrapolation of code from your parrot example using subparsers. Data set (created by this code) and usage examples at the bottom. Beware, example set does not consist strictly of parrots
#!/usr/bin/env python3
import argparse
import json
def add_parrot(name, weight, kind, **kwargs):
print("Adding {} of type {} and size {}".format(name, kind, weight))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots.append({'name': name, 'weight': weight, 'type': kind})
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def delete_parrot(name, **kwargs):
print("Uh oh! What happened to {}?".format(name))
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
parrots[:] = [p for p in parrots if p.get('name') != name]
with open('parrots.json', 'w') as parrotdb:
json.dump(parrots, parrotdb)
def show_parrots(name=None, weight=0, kind=None, **kwargs):
with open('parrots.json', 'r') as parrotdb:
parrots = json.load(parrotdb)
for p in parrots:
if (name or weight or kind):
if name in p['name'] or weight == p['weight'] or kind == p['type']:
print("{}\t{}\t{}".format(
p['name'], p['weight'], p['type']))
else:
print("{}\t{}\t{}".format(p['name'], p['weight'], p['type']))
parser = argparse.ArgumentParser(description="Manage Parrots")
subparsers = parser.add_subparsers()
add_parser = subparsers.add_parser('insert', aliases=['add', 'a'])
add_parser.add_argument('name')
add_parser.add_argument('weight', type=int)
add_parser.add_argument('kind')
add_parser.set_defaults(func=add_parrot)
del_parser = subparsers.add_parser("delete", aliases=['del', 'd'])
del_parser.add_argument('name')
del_parser.set_defaults(func=delete_parrot)
ls_parser = subparsers.add_parser('list', aliases=['show', 'ls'])
ls_parser.add_argument('--name')
ls_parser.add_argument('--size', type=int)
ls_parser.add_argument('--type', dest='kind')
ls_parser.set_defaults(func=show_parrots)
args = parser.parse_args()
args.func(**vars(args))
Dataset and usage examples:
➜ ~ cat parrots.json
[{"name": "tweety", "weight": 4, "type": "yellow"}, {"name": "donald", "weight": 18, "type": "white"}, {"name": "daffy", "weight": 12, "type": "black"}]
➜ ~ ./parrot.py ls
tweety 4 yellow
donald 18 white
daffy 12 black
➜ ~ ./parrot.py ls --name tweety
tweety 4 yellow
➜ ~ ./parrot.py delete tweety
Uh oh! What happened to tweety?
➜ ~ ./parrot.py ls --name tweety
➜ ~

How can I use OptionParser to print to the command line?

I'd like to use option parser to print the result of a computation to the command line. So far, I have
parser = OptionParser()
parser.add_option('-s','--some', help = "Print some of the Suspects")
parser.add_option('-a','--all',help = "Print all of the Suspects")
(opts,args) = parser.parse_args()
If the user passes -s, I would like the first 25 rows of a dataframe to be printed (I know how to do this). If -a is passed, I would like the entire dataframe to be printed. What do I have left to do?

from optparse import OptionParser
parser = OptionParser()
parser.add_option('-s','--some', help = "Print some of the Suspects")
parser.add_option('-a','--all',help = "Print all of the Suspects")
(opts,args) = parser.parse_args()
if opts.some:
print "some results"
if opts.all:
print "all results"

Find function calls with python using Regular expressions

I am working with a language where the modules are defined as
<module_name> <inst_name>(.<port_name> (<net_name>)….);
or
module1 inst1 ( .input a,
.output b;
port b=a;);
I want to find all such modules, while ignoring function calls .
I'm having difficulty with regex. I am looking for this
text1 text2 ( .text3; text4 );
note that all the spaces except the ones between text 1 and text2 are optional and might be new lines instead of spaces.text 3 and text4 can be multi lines but all are in the form of
text3 - >
.blah1 (blah2),
.blah3 (blah4)
text4->
blah1 blah2=xyz;
blah3 blah4=qwe;
I am trying to do
re.split(r"^[a-zA-Z]*\s[a-zA-Z]*\s?\n?\([a-zA-Z]*\s?\n?;[a-zA-Z]*\);", data)
Doesn't work though.It just grabs everything. How do i fix it? Thanks !!
I do need to grab everything individually, eventually (module/instances/port/nets). I think I can split it once regex is working.

I think you need to write a parser that understands enough of the language to at least canonicalize it before you try extracting information. You could write a simple parser by hand, or you could use a parsing framework such as PLY or others of that ilk.
To give you a more concrete idea about what I'm suggesting, consider
the following code, which defines a parse_data function that, given
the contents of a file, will yield a series of tokens recognized in
that file:
import re
tokens = {
'lparen': '\(',
'rparen': '\)',
'comma': ',',
'semicolon': ';',
'whitespace': '\s+',
'equals': '=',
'identifier': '[.\d\w]+',
}
tokens = dict((k, re.compile(v)) for k,v in tokens.items())
def parse_data(data):
while data:
for tn, tv in tokens.items():
mo = tv.match(data)
if mo:
matched = data[mo.start():mo.end()]
data = data[mo.end():]
yield tn, matched
Using this, you could write something that would put your sample input
into canonical form:
with open('inputfile') as fd:
data = fd.read()
last_token = (None, None)
for tn, tv in parse(data):
if tn == 'whitespace' and last_token[0] != 'semicolon':
print ' ',
elif tn == 'whitespace':
pass
elif tn == 'semicolon' and last_token[0] == 'rparen':
print tv
else:
print tv,
last_token = (tn, tv)
Given input like this:
module1 inst1 ( .input a,
.output b;
port b=a;);
module2 inst2 ( .input a, .output b; port b=a;);
module3 inst3 ( .input a, .output b;
port b=a;);
The above code would yield:
module1 inst1 ( .input a , .output b ; port b = a ; ) ;
module2 inst2 ( .input a , .output b ; port b = a ; ) ;
module3 inst3 ( .input a , .output b ; port b = a ; ) ;
Which, because it is in standard form, would be much more amendable to
extracting information via simple pattern matching.
Note that while this code relies on reading the entire source file
into memory first, you could fairly easily write code that you parse a
file in fragments if you were concerned about memory utilization.

Grep reliably all C #defines

I need to analyse some C files and print out all the #define found.
It's not that hard with a regexp (for example)
def with_regexp(fname):
print("{0}:".format(fname))
for line in open(fname):
match = macro_regexp.match(line)
if match is not None:
print(match.groups())
But for example it doesn't handle multiline defines for example.
There is a nice way to do it in C for example with
gcc -E -dM file.c
the problem is that it returns all the #defines, not just the one from the given file, and I don't find any option to only use the given file..
Any hint?
Thanks
EDIT:
This is a first solution to filter out the unwanted defines, simply checking that the name of the define is actually part of the original file, not perfect but seems to work nicely..
def with_gcc(fname):
cmd = "gcc -dM -E {0}".format(fname)
proc = Popen(cmd, shell=True, stdout=PIPE)
out, err = proc.communicate()
source = open(fname).read()
res = set()
for define in out.splitlines():
name = define.split(' ')[1]
if re.search(name, source):
res.add(define)
return res

Sounds like a job for a shell one-liner!
What I want to do is remove the all #includes from the C file (so we don't get junk from other files), pass that off to gcc -E -dM, then remove all the built in #defines - those start with _, and apparently linux and unix.
If you have #defines that start with an underscore this won't work exactly as promised.
It goes like this:
sed -e '/#include/d' foo.c | gcc -E -dM - | sed -e '/#define \(linux\|unix\|_\)/d'
You could probably do it in a few lines of Python too.

In PowerShell you could do something like the following:
function Get-Defines {
param([string] $Path)
"$Path`:"
switch -regex -file $Path {
'\\$' {
if ($multiline) { $_ }
}
'^\s*#define(.*)$' {
$multiline = $_.EndsWith('\');
$_
}
default {
if ($multiline) { $_ }
$multiline = $false
}
}
}
Using the following sample file
#define foo "bar"
blah
#define FOO \
do { \
do_stuff_here \
do_more_stuff \
} while (0)
blah
blah
#define X
it prints
\x.c:
#define foo "bar"
#define FOO \
do { \
do_stuff_here \
do_more_stuff \
} while (0)
#define X
Not ideal, at least how idiomatic PowerShell functions should work, but should work well enough for your needs.

Doing this in pure python I'd use a small state machine:
def getdefines(fname):
""" return a list of all define statements in the file """
lines = open(fname).read().split("\n") #read in the file as a list of lines
result = [] #the result list
current = []#a temp list that holds all lines belonging to a define
lineContinuation = False #was the last line break escaped with a '\'?
for line in lines:
#is the current line the start or continuation of a define statement?
isdefine = line.startswith("#define") or lineContinuation
if isdefine:
current.append(line) #append to current result
lineContinuation = line.endswith("\\") #is the line break escaped?
if not lineContinuation:
#we reached the define statements end - append it to result list
result.append('\n'.join(current))
current = [] #empty the temp list
return result