We Keep Coding

Entity Draw with Understand Python API

I am trying to get each entity to draw a certain type of graph using the Understand Python API. All the inputs are good and the database is opened but the the single item is not drawn. There is no error and no output file. The code is listed below. The Python code is called from a C# application which calls upython.exe with the associated arguments.
The Python file receives the Scitools directory and opens the understand database. The entities are also an argument which is loaded into a temp file. The outputPath references the directory where the SVG file will be placed. I can't seem to figure out why the item.draw method isn't working.
import os
import sys
import argparse
import re
import json
#parse arguments
parser = argparse.ArgumentParser(description = 'Creates butterfly or call by graphs from and Understand DB')
parser.add_argument('PathToSci', type = str, help = "Path to Sci Understand libraries")
parser.add_argument('PathToDB', type = str, help = "Path to the Database you want to create graphs from")
parser.add_argument('PathToOutput', type = str, help='Path to where the graphs should be outputted')
parser.add_argument('TypeOfGraph', type = str,
help="The type of graph you want to generate. Same names as in Understand GUI. IE 'Butterfly' 'Called By' 'Control Flow' ")
parser.add_argument("entities", help='Path to json list file of Entity long names you wish to create graphs for')
args, unknown = parser.parse_known_args()
# they may have entered a path with a space broken into multiple strings
if len(unknown) > 0:
print("Unkown argument entered.\n Note: Individual arguments must be passed as a single string.")
quit()
pathToSci = args.PathToSci
pathToDB = args.PathToDB
graphType = args.TypeOfGraph
entities = json.load(open(args.entities,))
pathToOutput = args.PathToOutput
pathToSci = os.path.join(pathToSci, "Python")
sys.path.append(pathToSci)
import understand
db = understand.open(pathToDB)
count = 0
for name in entities:
count += 1
print("Completed: " + str(count) + "/" + str(len(entities)))
#if it is an empty name don't make a graph
if len(name) == 0:
break
pattern = re.compile((name + '$').replace("\\", "/"))
print("pattern: " + str(pattern))
sys.stdout.flush()
ent = db.lookup(pattern)
print("ent: " + str(ent))
sys.stdout.flush()
print("Type: " + str(type(ent[0])))
sys.stdout.flush()
for item in ent:
try:
filename = os.path.join(pathToOutput, item.longname() + ".svg")
print("Graph Type: " + graphType)
sys.stdout.flush()
print("filename: " + filename)
sys.stdout.flush()
print("Item Kind: " + str(ent[0].kind()))
sys.stdout.flush()
item.draw(graphType, filename)
except understand.UnderstandError:
print("error creating graph")
sys.stdout.flush()
except Exception as e:
print("Could not create graph for " + item.kind().longname() + ": " + item.longname())
sys.stdout.flush()
print(e)
sys.stdout.flush()
db.close()
The output is below:
Completed: 1/1
pattern: re.compile('CSC03.SIN_COS$')
ent: [#lCSC03.SIN_COS#kacsc03.sin_cos(long_float,csc03.sctype)long_float#f./../../../IOSSP/Source_Files/OGP/OGP_71/csc03/csc03.ada]
Type: <class 'understand.Ent'>
Graph Type: Butterfly
filename: C:\Users\M73720\Documents\DFS\DFS-OGP-25-Aug-2022-11-24\SVGs\Entities\CSC03.SIN_COS.svg
Item Kind: Function

It turns out that it was a problem in the Understand API. The latest build corrected the problem. This was found by talking with SciTools Support group.

python - latexmk: error: no such option: -d

I am getting an error when I want to run a python script:
The error is following one:
The code is given below:
#!/usr/bin/python
import subprocess
code_dir = "code"
title = "Stanford ACM-ICPC Team Notebook"
def get_sections():
sections = []
section_name = None
with open('contents.txt', 'r') as f:
for line in f:
if '#' in line: line = line[:line.find('#')]
line = line.strip()
if len(line) == 0: continue
if line[0] == '[':
section_name = line[1:-1]
subsections = []
if section_name is not None:
sections.append((section_name, subsections))
else:
tmp = line.split('\t', 1)
if len(tmp) == 1:
raise ValueError('Subsection parse error: %s' % line)
filename = tmp[0]
subsection_name = tmp[1]
if subsection_name is None:
raise ValueError('Subsection given without section')
subsections.append((filename, subsection_name))
return sections
def get_style(filename):
ext = filename.lower().split('.')[-1]
if ext in ['c', 'cc', 'cpp']:
return 'cpp'
elif ext in ['java']:
return 'java'
elif ext in ['py']:
return 'py'
else:
return 'txt'
# TODO: check if this is everything we need
def texify(s):
#s = s.replace('\'', '\\\'')
#s = s.replace('\"', '\\\"')
return s
def get_tex(sections):
tex = ''
for (section_name, subsections) in sections:
tex += '\\section{%s}\n' % texify(section_name)
for (filename, subsection_name) in subsections:
tex += '\\subsection{%s}\n' % texify(subsection_name)
tex += '\\raggedbottom\\lstinputlisting[style=%s]{%s/%s}\n' % (get_style(filename), code_dir, filename)
tex += '\\hrulefill\n'
tex += '\n'
return tex
if __name__ == "__main__":
sections = get_sections()
tex = get_tex(sections)
with open('contents.tex', 'w') as f:
f.write(tex)
latexmk_options = ["latexmk", "-pdf", "notebook.tex"]
subprocess.call(latexmk_options)
I have already tried to install latexmk, But didn't succeed.
Can you help me about the detailed instruction of installation latexmk. I have already googled much. And for copyright thats not even my code. Its a code from stanford acm to make their own. Now I want to use to make my own.

Make sure that latexmk is accessible from your command line. You can check this by typing latexmk -version from your command line. If it is not accessible from command line then you need to add the latexmk path to environment variable.
If latexmk is not installed follow this link to properly install the latexmk.
I think following these steps might fix your problem.

Python script that performs line matching over stale files generates inconsistent output

I created a python script to parse mail (exim) logfiles and execute pattern matching in order to get a top 100 list for most send domains on my smtp servers.
However, everytime I execute the script I get a different count.
These are stale logfiles, and I cannot find a functional flaw in my code.
Example output:
1:
70353 gmail.com
68337 hotmail.com
53657 yahoo.com
2:
70020 gmail.com
67741 hotmail.com
54397 yahoo.com
3:
70191 gmail.com
67917 hotmail.com
54438 yahoo.com
Code:
#!/usr/bin/env python
import os
import datetime
import re
from collections import defaultdict
class DomainCounter(object):
def __init__(self):
self.base_path = '/opt/mail_log'
self.tmp = []
self.date = datetime.date.today() - datetime.timedelta(days=14)
self.file_out = '/var/tmp/parsed_exim_files-' + str(self.date.strftime('%Y%m%d')) + '.decompressed'
def parse_log_files(self):
sub_dir = os.listdir(self.base_path)
for directory in sub_dir:
if re.search('smtp\d+', directory):
fileInput = self.base_path + '/' + directory + '/maillog-' + str(self.date.strftime('%Y%m%d')) + '.bz2'
if not os.path.isfile(self.file_out):
os.popen('touch ' + self.file_out)
proccessFiles = os.popen('/bin/bunzip2 -cd ' + fileInput + ' > ' + self.file_out)
accessFileHandle = open(self.file_out, 'r')
readFileHandle = accessFileHandle.readlines()
print "Proccessing %s." % fileInput
for line in readFileHandle:
if '<=' in line and ' for ' in line and '<>' not in line:
distinctLine = line.split(' for ')
recipientAddresses = distinctLine[1].strip()
recipientAddressList = recipientAddresses.strip().split(' ')
if len(recipientAddressList) > 1:
for emailaddress in recipientAddressList:
# Since syslog messages are transmitted over UDP some messages are dropped and needs to be filtered out.
if '#' in emailaddress:
(login, domein) = emailaddress.split("#")
self.tmp.append(domein)
continue
else:
try:
(login, domein) = recipientAddressList[0].split("#")
self.tmp.append(domein)
except Exception as e:
print e, '<<No valid email address found, skipping line>>'
accessFileHandle.close()
os.unlink(self.file_out)
return self.tmp
if __name__ == '__main__':
domainCounter = DomainCounter()
result = domainCounter.parse_log_files()
domainCounts = defaultdict(int)
top = 100
for domain in result:
domainCounts[domain] += 1
sortedDict = dict(sorted(domainCounts.items(), key=lambda x: x[1], reverse=True)[:int(top)])
for w in sorted(sortedDict, key=sortedDict.get, reverse=True):
print '%-3s %s' % (sortedDict[w], w)

proccessFiles = os.popen('/bin/bunzip2 -cd ' + fileInput + ' > ' + self.file_out)
This line is non-blocking. Therefore it will start the command, but the few following lines are already reading the file. This is basically a concurrency issue. Try to wait for the command to complete before reading the file.
Also see:
Python popen command. Wait until the command is finished since os.popen is deprecated since python-2.6 (depending on which version you are using).
Sidenote - The same happens to the line below. The file may, or may not, exist after executing the following line:
os.popen('touch ' + self.file_out)

Python function to extract multiple segments of a file path

I would like to write a Python function that is capable of taking a file path, like:
/abs/path/to/my/file/file.txt
And returning three string variables:
/abs - the root directory, plus the "top-most" directory in the path
file - the "bottom-most" directory in the path; the parent of file.txt
path/to/my - everything in between the top- and bottom-most directories in the path
So something with the following pseudo-code:
def extract_path_segments(file):
absPath = get_abs_path(file)
top = substring(absPath, 0, str_post(absPath, "/", FIRST))
bottom = substring(absPath, 0, str_post(absPath, "/", LAST))
middle = str_diff(absPath, top, bottom)
return (top, middle, bottom)
Thanks in advance for any help here!

You are looking for os.sep, together with various os.path module functions. Simply split the path by that character, then re-assemble the parts you want to use. Something like:
import os
def extract_path_segments(path, sep=os.sep):
path, filename = os.path.split(os.path.abspath(path))
bottom, rest = path[1:].split(sep, 1)
bottom = sep + bottom
middle, top = os.path.split(rest)
return (bottom, middle, top)
This does not deal very well with Windows paths, where both \ and / are legal path separators. In that case you also have a drive letter, so you'd have to special-case that as well anyway.
Output:
>>> extract_path_segments('/abs/path/to/my/file/file.txt')
('/abs', 'path/to/my', 'file')

use os.path.split:
import os.path
def split_path(path):
"""
Returns a 2-tuple of the form `root, list_of_path_parts`
"""
head,tail = os.path.split(path)
out = []
while tail:
out.append(tail)
head,tail = os.path.split(head)
return head,list(reversed(out))
def get_parts(path):
root,path_parts = split_path(path)
head = os.path.join(root,path_parts[0])
path_to = os.path.join(*path_parts[1:-2])
parentdir = path_parts[-2]
return head,path_to,parentdir
head,path_to,parentdir = get_parts('/foo/path/to/bar/baz')
print (head) #foo
print (path_to) #path/to
print (parentdir) #bar

Using os.path.split() and os.path.join() as we are supposed to
>>> import os
>>> pth = "/abs/path/to/my/file/file.txt"
>>> parts = []
>>> while True:
... pth, last = os.path.split(pth)
... if not last:
... break
... parts.append(last)
...
>>> pth + parts[-1]
'/abs'
>>> parts[1]
'file'
>>> os.path.join(*parts[-2:1:-1])
'path/to/my'
As a function
import os
def extract_path_segments(pth):
parts = []
while True:
pth, last = os.path.split(pth)
if not last:
break
parts.append(last)
return pth + parts[-1], parts[1], os.path.join(*parts[-2:1:-1])

>>> p = '/abs/path/to/my/file/file.txt'
>>> r = p.split('/')
>>> r[1],'/'.join(r[2:-2]),r[-2]
('abs', 'path/to/my', 'file')

Tab completion in Python's raw_input()

i know i can do this to get the effect of tab completion in python sure.
import readline
COMMANDS = ['extra', 'extension', 'stuff', 'errors',
'email', 'foobar', 'foo']
def complete(text, state):
for cmd in COMMANDS:
if cmd.startswith(text):
if not state:
return cmd
else:
state -= 1
readline.parse_and_bind("tab: complete")
readline.set_completer(complete)
raw_input('Enter section name: ')
I am now interested in doing tab completion with directories. (/home/user/doc >tab)
How would i go about doing such a task?

Here is a quick example of how to perform incremental completion of file system paths. I've modified your example, organizing it into a class where methods named complete_[name] indicate top-level commands.
I've switched the completion function to use the internal readline buffer to determine the state of the overall completion, which makes the state logic a bit simpler. The path completion is in the _complete_path(path) method, and I've hooked up the extra command to perform path completions on its arguments.
I'm sure the code could be further simplified but it should provide you a decent starting point:
import os
import re
import readline
COMMANDS = ['extra', 'extension', 'stuff', 'errors',
'email', 'foobar', 'foo']
RE_SPACE = re.compile('.*\s+$', re.M)
class Completer(object):
def _listdir(self, root):
"List directory 'root' appending the path separator to subdirs."
res = []
for name in os.listdir(root):
path = os.path.join(root, name)
if os.path.isdir(path):
name += os.sep
res.append(name)
return res
def _complete_path(self, path=None):
"Perform completion of filesystem path."
if not path:
return self._listdir('.')
dirname, rest = os.path.split(path)
tmp = dirname if dirname else '.'
res = [os.path.join(dirname, p)
for p in self._listdir(tmp) if p.startswith(rest)]
# more than one match, or single match which does not exist (typo)
if len(res) > 1 or not os.path.exists(path):
return res
# resolved to a single directory, so return list of files below it
if os.path.isdir(path):
return [os.path.join(path, p) for p in self._listdir(path)]
# exact file match terminates this completion
return [path + ' ']
def complete_extra(self, args):
"Completions for the 'extra' command."
if not args:
return self._complete_path('.')
# treat the last arg as a path and complete it
return self._complete_path(args[-1])
def complete(self, text, state):
"Generic readline completion entry point."
buffer = readline.get_line_buffer()
line = readline.get_line_buffer().split()
# show all commands
if not line:
return [c + ' ' for c in COMMANDS][state]
# account for last argument ending in a space
if RE_SPACE.match(buffer):
line.append('')
# resolve command to the implementation function
cmd = line[0].strip()
if cmd in COMMANDS:
impl = getattr(self, 'complete_%s' % cmd)
args = line[1:]
if args:
return (impl(args) + [None])[state]
return [cmd + ' '][state]
results = [c + ' ' for c in COMMANDS if c.startswith(cmd)] + [None]
return results[state]
comp = Completer()
# we want to treat '/' as part of a word, so override the delimiters
readline.set_completer_delims(' \t\n;')
readline.parse_and_bind("tab: complete")
readline.set_completer(comp.complete)
raw_input('Enter section name: ')
Usage:
% python complete.py
Enter section name: ext<tab>
extension extra
Enter section name: extra foo<tab>
foo.py foo.txt foo/
Enter section name: extra foo/<tab>
foo/bar.txt foo/baz.txt
Enter section name: extra foo/bar.txt
Update It will complete paths from the root if the user types /:
% python complete.py
Enter section name: extra /Use<tab>
/Users/.localized /Users/Shared/ /Users/user1 /Users/user2
Enter section name: extra /Users/use<tab>
/Users/user1 /Users/user2

This is enough to enable built in directory tab completion with raw_input():
import readline
readline.parse_and_bind("tab: complete")

This version is for python3, uses pathlib, and a minimalistic version that tab completes files/dirs. It is based on some of the above answers, but only works for files/dirs.
#!/usr/bin/python
import pathlib
import readline
def complete_path(text, state):
incomplete_path = pathlib.Path(text)
if incomplete_path.is_dir():
completions = [p.as_posix() for p in incomplete_path.iterdir()]
elif incomplete_path.exists():
completions = [incomplete_path]
else:
exists_parts = pathlib.Path('.')
for part in incomplete_path.parts:
test_next_part = exists_parts / part
if test_next_part.exists():
exists_parts = test_next_part
completions = []
for p in exists_parts.iterdir():
p_str = p.as_posix()
if p_str.startswith(text):
completions.append(p_str)
return completions[state]
# we want to treat '/' as part of a word, so override the delimiters
readline.set_completer_delims(' \t\n;')
readline.parse_and_bind("tab: complete")
readline.set_completer(complete_path)
print(input('tab complete a filename: '))

For path completion
import os
import sys
import readline
import glob
def path_completer(text, state):
"""
This is the tab completer for systems paths.
Only tested on *nix systems
"""
line = readline.get_line_buffer().split()
if '~' in text:
text = os.path.expanduser('~')
return [x for x in glob.glob(text+'*')][state]
if __name__=="__main__":
readline.set_completer_delims('\t')
readline.parse_and_bind("tab: complete")
readline.set_completer(path_completer)
ans = input("What file do you want? ")
print(ans)
Note that I've refined the code found at https://gist.github.com/iamatypeofwalrus/5637895