I'm working on quite complex project and time after time I have to narrow down problems looking at stack traces. They happen to be very long and involve “my” code, standard library code and 3rd party libraries code at same time. Most of time the real problem is in “my” code and locating it instantly in a stack trace is a bit hard for eyes. Under “my” code I mean the code that is under current working directory.
So I realized that I want something that will colorize stack traces and highlight lines that are mine. Compare original to highlighted.
I could write a python script that I could use this way:
nosetests | colorize_stack_trace.py
But I believe there is a quicker and more elegant way to do this using Linux toolset. Any ideas?
UPD:
Using supercat suggested by Dennis Williamson, the intermediate result is following bash function:
pyst() {
rc=/tmp/spcrc;
echo '#################### ### # # # ########################################' > $rc;
echo ' blk 0 r ^(.*)$' >> $rc;
echo ' mag b 0 r ^\s*File "'`pwd`'/(.*)"' >> $rc;
spc -c $rc;
}
Now I can do:
nosetests 2>&1 | pyst
Not too elegant, but works at some degree. There are two problems left:
I can't see any output before nosetests completes. I.e. I don't see the progress.
I have to write 2>&1 over and over again.
UPD 2:
Asking this question I had mainly nosetests in mind. And I just found great solution: rednose nose plugin. It highlights paths that are local plus do many more handy readability things.
Returning to original question: problems that I noted with supercat don't relate to it completely but its a problem of Unix shell streaming, flushing, piping, redirecting. So as as an answer to the question asked I accept an answer that suggests supercat.
Actually, there is a great Python syntax highlighting library called Pygments, which is also able to highlight tracebacks.
So, all you have to do is:
$ easy_install pygments # downloads and installs pygments
$ cat traceback.txt | pygmentize -l pytb
"pytb" is the shortcut for the PythonTracebackLexer. There is also a special lexer for Python 3 Tracebacks included, which is called "py3tb".
You can format the output in various formats (including html, latex, svg, several image formats and so on). But there is also a terminal formatter available (and if you are wondering... of course there are different color themes available!).
You can use -f html to select another formatter (in that case, the HTML formatter).
Take a look at Supercat (spc). It does both ANSI and HTML highlighting and can be configured for your particular output. It comes with some configuration files for source code files in C and Python, for example and log files, Changelogs, diffs and others.
Based on Dave Kirby's suggestion for vim, this does something similar:
less -p regex file_name
Or
some_command | less -p regex
There is a nice module just for that purpose:
colored-traceback # github
colored-traceback # pypi
You just have to download and install it via pip:
pip install colored-traceback
Import it into a top-level file of your project, for exmaple like this:
if DEBUG:
import colored_traceback
colored_traceback.add_hook()
And you get a traceback like that for every underling file (colors vary):
Traceback (most recent call last):
File "./workflowy.py", line 525, in <module>
main()
File "./workflowy.py", line 37, in main
projects = cli.load_json(args, input_is_pipe)
File "./workflowy.py", line 153, in load_json
return json.load(sys.stdin)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Maybe you could use the cgitb module (short official doc here) as a starting point (it creates nice HTML tracebacks). It should be relatively simple to make the modifications you need (e.g. add a color tag according to the file path). But of course this is only viable if you are willing to use the browser to view the traces.
Here's one using ipython:
import sys
from IPython.core.ultratb import ColorTB
print(''.join(ColorTB().structured_traceback(*sys.exc_info())))
As a starting point for colorizing (and otherwise formatting) text, you'll probably want to look at the curses library. Also see this how-to, which looks useful.
As for overriding Python's built-in error handling for all programs...I've never tried it, but I would think that would involve some fairly low-level changes. You could always wrap your code in a huge try/except block, but I assume you don't want to do that. I favor the more Unixy approach of writing a little script that does one thing, and does it well: have it take an input and, if it's a stack trace, colorize it. Otherwise, pass the text through unchanged. Using a pipe, like you suggested, may be the best way. (In this case, to pipe stderr you'd want to do something like this, which merges stderr with stdout before piping: cmd1 2>&1 | cmd2 )
Also you can open traceback file with vim, using :cfile command. Then you can open highlighted ant interactive list of files in traceback using :copen command and jump between these files using set of related vim commands.
Load the text into vim:
nosetests | vim -
Set vim to highlight all lines that match a search
:set hlsearch
Search for lines with "your" path
/.*/path/to/my/code.*
Voila - all the lines with your path will be highlighted.
If you want to highlight the next line as well then you can do that too:
/.*/path/to/my/code.*\n.*
Related
(Background: On an NTFS partition, files and/or folders can be set to "compressed", like it's a file attribute. They'll show up in blue in Windows Explorer, and will take up less disk space than they normally would. They can be accessed by any program normally, compression/decompression is handled transparently by the OS - this is not a .zip file. In Windows, setting a file to compressed can be done from a command line with the "Compact" command.)
Let's say I've created a file called "testfile.txt", put some data in it, and closed it. Now, I want to set it to be NTFS compressed. Yes, I could shell out and run Compact, but is there a way to do it directly in Python code instead?
In the end, I ended up cheating a bit and simply shelling out to the command line Compact utility. Here is the function I ended up writing. Errors are ignored, and it returns the output text from the Compact command, if any.
def ntfscompress(filename):
import subprocess
_compactcommand = 'Compact.exe /C /I /A "{}"'.format(filename)
try:
_result = subprocess.run(_compactcommand, timeout=86400,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,text=True)
return(_result.stdout)
except:
return('')
I am trying to write a small program in bash and part of it needs to be able to get some values from a txt file where the different files are separated by a line, and then either add each line to a variable or add each line to one array.
So far I have tried this:
FILE=$"transfer_config.csv"
while read line
do
MYARRAY[$index]="$line"
index=$(($index+1))
done < $FILE
echo ${MYARRAY[0]}
This just produces a blank line though, and not what was on the first line of the config file.
I am not returned with any errors which is why I am not too sure why this is happening.
The bash script is called though a python script using os.system("$HOME/bin/mcserver_config/server_transfer/down/createRemoteFolder"), but if I simply call it after the python program has made the file which the bash script reads, it works.
I am almost 100% sure it is not an issue with the directories, because pwd at the top of the bash script shows it in the correct directory, and the python program is also creating the data file in the correct place.
Any help is much appreciated.
EDIT:
I also tried the subprocess.call("path_to_script", shell=True) to see if it would make a difference, I know it is unlikely but it didn't.
I suspect that when calling the bash script from python, having just created the file, you are not really finished with that file: you should either explicitly close the file or use a with construct.
Otherwise, the written data is still in any buffer (from the file object, or in the OS, or wherever). Only closing (or at least flushing) the file makes sure the data is indeed in the file.
BTW, instead of os.system, you should use the subprocess module...
Is there a way to retrieve the path to the interpreter a UNIX shell would use for a given script? (preferably in a Python API or as shell command)?
To be used like this:
$ get_bang ./myscript.py
/usr/bin/python3
Of course I could extract it manually using RE but I'm sure in real world that's more complicated than just handling the first line and I don't want to re-invent the wheel..
The reason I need this is I want to call the script from inside another script and I want to add parameters to the interpreter.
Actually, it isn't more complicated than reading (the first word) of the first line.
Try putting the shebang on the second line (or even just putting a space before the #) and see what happens.
Also see http://www.in-ulm.de/~mascheck/various/shebang/ and http://homepages.cwi.nl/~aeb/std/hashexclam-1.html for more than you've ever wanted to know about the shebang feature.
Many ways - for example:
sed -n '1s/^#!//p' filename
prints for example
/bin/sh
or (if multiword)
/usr/bin/env perl
or nothing, if here isn't shebang
I need to enter the contents of a text (.txt) file as input for a Python (.py) file. Assuming the name of the text file is TextFile and the name of the Python file PythonFile, then the code should be as follows:
python PythonFile.py < TextFile.txt
Yet, when I try to do this in IDLE and type in
import PythonFile < TextFile,
IDLE gives me an invalid syntax message, pointing to the < sign. I tried all sorts of variations on this theme (i.e.,using or not using the file name extensions), but still got the same invalid-syntax message. How is the syntax different for input redirection in IDLE?
If it works in the command line, then why do you want to do this in IDLE? There are ways to achieve a similar result using, for example, subprocess, but a better way would be to refactor PythonFile.py so that you can call a function from it, e.g.:
>>> import PythonFile
>>> PythonFile.run_with_input('TextFile.txt')
If you post the contents of PythonFile.py, we might be able to help you do this.
I'm trying to track down a Python UnicodeDecodeError in the following log line:
10.210.141.123 - - [09/Nov/2011:14:41:04 -0800] "gfR\x15¢\x09ì|Äbk\x0F[×ÐÖà\x11CEÐÌy\x5C¿DÌj\x08Ï ®At\x07å!;f>\x08éPW¤\x1C\x02ö*6+\x5C\x15{,ªIkCRA\x22 xþP9â\x13h\x01¢è´\x1DzõWiË\x5C\x10sòʨR)¶²\x1F8äl¾¢{ÆNw\x08÷#ï" 400 166 0.000 "-" "-"
I opened the entire log file in Vim, and then yanked the line into a new file so I could test just the one line. However, my parsing script works OK with the new file - it doesn't throw a UnicodeDecodeError. I don't understand why the one file would generate an error and the other one would not, when they are (on the surface) identical.
Here's what I tried: running enca to determine the file encoding, which complained that it Cannot determine (or understand) your language preferences. file -i says that both files are Regular files. I also deleted every other line in the original log file and still got the error in one file and no error in the other. I tried deleting
set encoding=utf-8
from my .vimrc, writing the file again, and I still got the error in one file and not in the other.
The logs are nginx logs. Nginx has this note in their release notes:
*) Change: now the 0x00-0x1F, '"' and '\' characters are escaped as \xXX
in an access_log.
Thanks to Maxim Dounin.
My Python script has with open('log_file') as f and the error comes up when I try to call json.dumps on a dict.
How can I track this down?
Your question: How can I track this down?
Answer:
(1) Show us the full text of the error message that you got -- without knowing what encoding that you were trying to use, we can't tell you anything. A traceback and a snippet of code that reads the file and reproduces the error would also be handy.
(2) Write a tiny Python script to find the line in the file and then do:
print repr(the_line) # Python 2.X
print ascii(the_line) # Python 3.x
and copy/paste the result into an edit of your question, so that we can see unambiguously what is in the line.
(3) It does look like random gibberish except for the but do tell us whether you expect that line to be text (if so, in what human language?).