We Keep Coding

Reading a dynamically updated log file via readline

I'm trying to read a log file, written line by line, via readline.
I'm surprised to observe the following behaviour (code executed in the interpreter, but same happens when variations are executed from a file):
f = open('myfile.log')
line = readline()
while line:
print(line)
line = f.readline()
# --> This displays all lines the file contains so far, as expected
# At this point, I open the log file with a text editor (Vim),
# add a line, save and close the editor.
line = f.readline()
print(line)
# --> I'm expecting to see the new line, but this does not print anything!
Is this behaviour standard? Am I missing something?
Note: I know there are better way to deal with an updated file for instance with generators as pointed here: Reading from a frequently updated file. I'm just interested in understanding the issue with this precise use case.

For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.
On the contrary, your scripts reads from the original file, so it does not see any change on it.
To further test, instead of Vim, you can try to directly write on the file using:
echo "Hello World" >> myfile.log
You should see the new line from python.

for following your file, you can use this code:
f = open('myfile.log')
while True:
line = readline()
if not line:
print(line)

update /etc/sysctl.conf with Python's ConfigParser

I am able to use Python's ConfigParser library to read /etc/sysctl.conf by adding a [dummy] section and overriding ConfigParser's read() method as follows:
class SysctlConfigParser(ConfigParser.ConfigParser):
def read(self, fn):
text = open(fn).read()
contents = StringIO.StringIO("[dummy]\n" + text)
self.readfp(contents, fn)
Now the tricky part is to write back configuration updates that my python program made, because if I would now call ConfigParser.write() directly then it would add back this [dummy] section as well:
[dummy]
net.netfilter.nf_conntrack_max = 313
net.netfilter.nf_conntrack_expect_max = 640
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 5
Here are my questions:
Is there an elegant way to make ConfigParser not to add this [dummy] section? It seems odd if I would have to open this file again just to remove the first line that contains this dummy section.
Maybe ConfigParser is not the right tool to edit sysctl.conf? If so are there any other Python libraries that would allow to update sysctl.conf in a convenient way from Python?

ConfigParser is designed for parsing INI-style configuration files. /etc/sysconf.conf is not this sort of file.
You could use the Augeas bindings for Python if you want a parser that works out-of-the-box:
import augeas
aug = augeas.Augeas()
aug.set('/files/etc/sysctl.conf/net.ipv4.ip_forwarding', '1')
aug.set('/files/etc/sysctl.conf/fs.inotify.max_user_watches', '8192')
aug.save()
The format of the file is pretty trivial (just a collection of <name> = <value> lines with optional comments).

How to write OS X Finder Comments from python?

I'm working on a python script that creates numerous images files based on a variety of inputs in OS X Yosemite. I am trying to write the inputs used to create each file as 'Finder comments' as each file is created so that IF the the output is visually interesting I can look at the specific input values that generated the file. I've verified that this can be done easily with apple script.
tell application "Finder" to set comment of (POSIX file "/Users/mgarito/Desktop/Random_Pixel_Color/2015-01-03_14.04.21.png" as alias) to {Val1, Val2, Val3} as Unicode text
Afterward, upon selecting the file and showing its info (cmd+i) the Finder comments clearly display the expected text 'Val1, Val2, Val2'.
This is further confirmed by running mdls [File/Path/Name] before and after the applescript is used which clearly shows the expected text has been properly added.
The problem is I can't figure out how to incorporate this into my python script to save myself.
Im under the impression the solution should* be something to the effect of:
VarList = [Var1, Var2, Var3]
Fiele = [File/Path/Name]
file.os.system.add(kMDItemFinderComment, VarList)
As a side note I've also look at xattr -w [Attribute_Name] [Attribute_Value] [File/Path/Name] but found that though this will store the attribute, it is not stored in the desired location. Instead it ends up in an affiliated pList which is not what I'm after.

Here is my way to do that.
First you need to install applescript package using pip install applescript command.
Here is a function to add comments to a file:
def set_comment(file_path, comment_text):
import applescript
applescript.tell.app("Finder", f'set comment of (POSIX file "{file_path}" as alias) to "{comment_text}" as Unicode text')
and then I'm just using it like this:
set_comment('/Users/UserAccountName/Pictures/IMG_6860.MOV', 'my comment')

After more digging, I was able to locate a python applescript bundle: https://pypi.python.org/pypi/py-applescript
This got me to a workable answer, though I'd still prefer to do this natively in python if anyone has a better option?
import applescript
NewFile = '[File/Path/Name]' <br>
Comment = "Almost there.."
AddComment = applescript.AppleScript('''
on run {arg1, arg2}
tell application "Finder" to set comment of (POSIX file arg1 as alias) to arg2 as Unicode text
return
end run
''')
print(AddComment.run(NewFile, Comment))
print("Done")

This is the function to get comment of a file.
def get_comment(file_path):
import applescript
return applescript.tell.app("Finder", f'get comment of (POSIX file "{file_path}" as alias)').out
print(get_comment('Your Path'))

Another approach is to use appscript, a high-level Apple event bridge that is sadly no longer officially supported but still works (and saw an updated release in Jan. 2021). Here is an example of reading and setting the comment on a file:
import appscript
import mactypes
# Get a handle on the Finder.
finder = appscript.app('Finder')
# Tell Finder to select the file.
file = finder.items[mactypes.Alias("/path/to/a/file")]
# Print the current comment
comment = file.comment()
print("Current comment: " + comment)
# Set a new comment.
file.comment.set("New comment")
# Print the current comment again to verify.
comment = file.comment()
print("Current comment: " + comment)
Despite that the author of appscript recommends against using it in new projects, I used it recently to create a command-line utility called Urial for the specialized purpose of writing and updating URIs in Finder comments. Perhaps its code can serve as an an additional example of using appscript to manipulate Finder comments.

Automating PDF generation

What would be a solid tool to use for generating PDF reports? Particularly, we are interested in creating interactive PDFs that have video, like the example found here.
Right now we are using Python and reportlab to generate PDFs, but have not explored the library completely (mostly because the license pricing is a little prohibitive)
We have been looking at the Adobe's SDK and iText libraries but it's hard to say what the capabilities of either are.
The ability to generate a document from a template PDF would be a plus.
Any pointers or comments will be appreciated.
Thanks,

Recently, I needed to create PDF reports for a Django application; a ReportLab license was available, but I ended up choosing LaTeX. The benefit of this approach is that we could use Django templates to generate the LaTeX source, and not get over encumbered writing lots of code for the many reports we needed to create. Plus, we could take advantage of the relatively much more concise LaTeX syntax (which does have it's many quirks and is not suitable for every purpose).
This snippet provides a general overview of the approach. I found it necessary to make some changes, which I have provided at the end of this question. The main addition is detection for Rerun LaTeX messages, which indicates an additional pass is required. Usage is as simple as:
def my_view(request):
pdf_stream = process_latex(
'latex_template.tex',
context=RequestContext(request, {'context_obj': context_obj})
)
return HttpResponse(pdf_stream, content_type='application/pdf')
It is possible to embed videos in LaTeX generated PDFs, however I do not have any experience with it. Here is a top Google result.
This solution does require spawning a new process (pdflatex), so if you want a pure Python solution keep looking.
import os
from subprocess import Popen, PIPE
from tempfile import NamedTemporaryFile
from django.template import loader, Context
class LaTeXException(Exception):
pass
def process_latex(template, context={}, type='pdf', outfile=None):
"""
Processes a template as a LaTeX source file.
Output is either being returned or stored in outfile.
At the moment only pdf output is supported.
"""
t = loader.get_template(template)
c = Context(context)
r = t.render(c)
tex = NamedTemporaryFile()
tex.write(r)
tex.flush()
base = tex.name
names = dict((x, '%s.%s' % (base, x)) for x in (
'log', 'aux', 'pdf', 'dvi', 'png'))
output = names[type]
stdout = None
if type == 'pdf' or type == 'dvi':
stdout = pdflatex(base, type)
elif type == 'png':
stdout = pdflatex(base, 'dvi')
out, err = Popen(
['dvipng', '-bg', '-transparent', names['dvi'], '-o', names['png']],
cwd=os.path.dirname(base), stdout=PIPE, stderr=PIPE
).communicate()
os.remove(names['log'])
os.remove(names['aux'])
# pdflatex appears to ALWAYS return 1, never returning 0 on success, at
# least on the version installed from the Ubuntu apt repository.
# so instead of relying on the return code to determine if it failed,
# check if it successfully created the pdf on disk.
if not os.path.exists(output):
details = '*** pdflatex output: ***\n%s\n*** LaTeX source: ***\n%s' % (
stdout, r)
raise LaTeXException(details)
if not outfile:
o = file(output).read()
os.remove(output)
return o
else:
os.rename(output, outfile)
def pdflatex(file, type='pdf'):
out, err = Popen(
['pdflatex', '-interaction=nonstopmode', '-output-format', type, file],
cwd=os.path.dirname(file), stdout=PIPE, stderr=PIPE
).communicate()
# If the output tells us to rerun, do it by recursing over ourself.
if 'Rerun LaTeX.' in out:
return pdflatex(file, type)
else:
return out

I suggest to use https://github.com/mreiferson/py-wkhtmltox to render HTML to PDF.
And use any tool you choose to render reports as HTML. I like http://www.makotemplates.org/

Highlighting python stack traces

I'm working on quite complex project and time after time I have to narrow down problems looking at stack traces. They happen to be very long and involve “my” code, standard library code and 3rd party libraries code at same time. Most of time the real problem is in “my” code and locating it instantly in a stack trace is a bit hard for eyes. Under “my” code I mean the code that is under current working directory.
So I realized that I want something that will colorize stack traces and highlight lines that are mine. Compare original to highlighted.
I could write a python script that I could use this way:
nosetests | colorize_stack_trace.py
But I believe there is a quicker and more elegant way to do this using Linux toolset. Any ideas?
UPD:
Using supercat suggested by Dennis Williamson, the intermediate result is following bash function:
pyst() {
rc=/tmp/spcrc;
echo '#################### ### # # # ########################################' > $rc;
echo ' blk 0 r ^(.*)$' >> $rc;
echo ' mag b 0 r ^\s*File "'`pwd`'/(.*)"' >> $rc;
spc -c $rc;
}
Now I can do:
nosetests 2>&1 | pyst
Not too elegant, but works at some degree. There are two problems left:
I can't see any output before nosetests completes. I.e. I don't see the progress.
I have to write 2>&1 over and over again.
UPD 2:
Asking this question I had mainly nosetests in mind. And I just found great solution: rednose nose plugin. It highlights paths that are local plus do many more handy readability things.
Returning to original question: problems that I noted with supercat don't relate to it completely but its a problem of Unix shell streaming, flushing, piping, redirecting. So as as an answer to the question asked I accept an answer that suggests supercat.

Actually, there is a great Python syntax highlighting library called Pygments, which is also able to highlight tracebacks.
So, all you have to do is:
$ easy_install pygments # downloads and installs pygments
$ cat traceback.txt | pygmentize -l pytb
"pytb" is the shortcut for the PythonTracebackLexer. There is also a special lexer for Python 3 Tracebacks included, which is called "py3tb".
You can format the output in various formats (including html, latex, svg, several image formats and so on). But there is also a terminal formatter available (and if you are wondering... of course there are different color themes available!).
You can use -f html to select another formatter (in that case, the HTML formatter).

Take a look at Supercat (spc). It does both ANSI and HTML highlighting and can be configured for your particular output. It comes with some configuration files for source code files in C and Python, for example and log files, Changelogs, diffs and others.
Based on Dave Kirby's suggestion for vim, this does something similar:
less -p regex file_name
Or
some_command | less -p regex

There is a nice module just for that purpose:
colored-traceback # github
colored-traceback # pypi
You just have to download and install it via pip:
pip install colored-traceback
Import it into a top-level file of your project, for exmaple like this:
if DEBUG:
import colored_traceback
colored_traceback.add_hook()
And you get a traceback like that for every underling file (colors vary):
Traceback (most recent call last):
File "./workflowy.py", line 525, in <module>
main()
File "./workflowy.py", line 37, in main
projects = cli.load_json(args, input_is_pipe)
File "./workflowy.py", line 153, in load_json
return json.load(sys.stdin)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Maybe you could use the cgitb module (short official doc here) as a starting point (it creates nice HTML tracebacks). It should be relatively simple to make the modifications you need (e.g. add a color tag according to the file path). But of course this is only viable if you are willing to use the browser to view the traces.

Here's one using ipython:
import sys
from IPython.core.ultratb import ColorTB
print(''.join(ColorTB().structured_traceback(*sys.exc_info())))

As a starting point for colorizing (and otherwise formatting) text, you'll probably want to look at the curses library. Also see this how-to, which looks useful.
As for overriding Python's built-in error handling for all programs...I've never tried it, but I would think that would involve some fairly low-level changes. You could always wrap your code in a huge try/except block, but I assume you don't want to do that. I favor the more Unixy approach of writing a little script that does one thing, and does it well: have it take an input and, if it's a stack trace, colorize it. Otherwise, pass the text through unchanged. Using a pipe, like you suggested, may be the best way. (In this case, to pipe stderr you'd want to do something like this, which merges stderr with stdout before piping: cmd1 2>&1 | cmd2 )

Also you can open traceback file with vim, using :cfile command. Then you can open highlighted ant interactive list of files in traceback using :copen command and jump between these files using set of related vim commands.

Load the text into vim:
nosetests | vim -
Set vim to highlight all lines that match a search
:set hlsearch
Search for lines with "your" path
/.*/path/to/my/code.*
Voila - all the lines with your path will be highlighted.
If you want to highlight the next line as well then you can do that too:
/.*/path/to/my/code.*\n.*