We Keep Coding

Handling errors in Python in the console

I am currently working with a program and doing a lot of testing on it. I am using the Python logging library and write all my errors to the library. However, the errors also print directly to the console and show the entire stack trace which I do not want. Instead I would rather it just simply say error in the console and print out to the error logs.
I found this similar question here (How to make python gracefully fail?) but all answers talked about using try catches. Is there anyway to set up errors being handled this way throughout the entire program without using try-catches? I foresee them being difficult to use, for example when I quit using Control + C, which I could do at any time and may not be able to catch in a try-catch block. I would love to be able to tell my program to handle errors a certain way and have that reflected throughout the entire program.

How to programmatically execute/step through Python code line by line

I am trying to find a way that I can have a program step through Python code line by line and do something with the results of each line. In effect a debugger that could be controlled programmatically rather than manually. pdb would be exactly what I am looking for if it returned its output after each step as a string and I could then call pdb again to pickup where I left off. However, instead it outputs to stdout and I have to manually input "step" via the keyboard.
Things I have tried:
I am able to redirect pdb's stdout. I could redirect it to a second
Python program which would then process it. However, I cannot
figure out how to have the second Python program tell pdb to
step.
Related to the previous one, if I could get pdb to step all the way
through to the end (perhaps I could figure out something to spoof a
keyboard repeatedly entering "step"?) and redirect the output to a
file, I could then write another program that acted like it was
stepping through the program when it was actually just reading the
file line by line.
I could use exec to manually run lines of Python code. However,
since I would be looking at one line at a time, I would need to
manually detect and handle things like conditionals, loops, and
function calls which quickly gets very complicated.
I read some posts that say that pdb is implemented using
sys.settrace. If nothing else works I should be able to recreate
the behavior I need using this.
Is there any established/straight forward way to implement the behavior that I am looking for?

sys.settrace() is the fundamental building block for stepping through Python code. pdb is implemented entirely in Python, so you can just look at the module to see how it does things. It also has various public functions/methods for stepping under program control, read the library reference for your version of Python for details.

I read some posts that say that pdb is implemented using sys.settrace.
If nothing else works I should be able to recreate the behavior I need
using this.
Don't view this as a last resort. I think it's the best approach for what you want to accomplish.

Application logs for support and analysis purpose

I am analyzing an existing Python code that runs into hundreds of line. Adding log per line to capture flow / understanding run time processing is painful - but then the current application logging is very poor by just using print data.
Hence for support purpose these are not enough as its difficult to understand without looking into code.
What is the best way of change these unstandard logs into at least something like -
Class Name - Method Name - Error Details additional more details
With small modifications - I also run into risk of breaking the flow if not dealt carefully.
Please let me know which application mechanism logging would be the best?

I would advise you to type "Python /?" in a command prompt and see which possibilities you have (e.g. python -v gives a verbose output on the import statements in your code). Like this you might find a way of having more information without needing to modify your source code. Obviously I don't know if the information you get from python -v is the one you're looking for.

I think probably decorators are your best option so you touch the code as less as possible.
First link redirects the standard stdout to a python standard logging module, so it would have the format you want if you specify it within the logger properties.
https://wiki.python.org/moin/PythonDecoratorLibrary#Redirects_stdout_printing_to_python_standard_logging.
https://wiki.python.org/moin/PythonDecoratorLibrary#Logging_decorator_with_specified_logger_.28or_default.29

How do you avoid leaving debugger print/pdb statements in python open source apps when you're debugging them and just finish?

Sometimes when developing using open source software, you need to read it's source code (specially zope/plone). A lot of times I need to write print statements, or debug calls (import pdb), or comment try/except clauses, you name it.
Sometimes I have a lot of files opened when trying to find an issue, and sometimes I forget to remove these print/debug alterations.
So, my question is: how do you keep yourself organized when doing it? Do you write "TODOs" along the modifications and search for them later, do you keep everything opened in your editor and when you find what you were looking for you just revert the files (This approach isn't useful when you're searching for a really big problem that needs days, you need to turn off your computer and return the other day)? Or you just don't do nothing since print statements in development environment is nothing to worry about?
I'm using Vim. I'm just interested to know how other programmers treat this issue.

I used to run into that problem a lot. Now, as part of my check-in process, I run a find/grep script combo that looks for my debugging statements. The only caveat is that I must keep my added debugging statements consistent so grep can find them all.
something like this:
## pre-checkin_scan.bin
find . -name "*.py" -exec grep -H --file=/homes/js/bin/pre-checkin_scan_regexp_list.grep {} \;
## pre-checkin_scan_regexp_list.grep
## (The first pattern is to ignore Doxygen comments)
^##[^#]
pdb
^ *print *( *" *Dbg
^ *print *( *" *Debug
^ *debug

In case of my own projects, the source code is always in version control. Before committing, I always check the graphical diff so that I can see what has changed, what the commit message should be and whether I can split up into smaller commits. That way, I almost always recognize temporary garbage like print statements. If not, I usually notice it shortly afterwards and can do an uncommit if I haven't yet pushed (works for DVCS like git and bzr, not with subversion).
Concerning problems that take multiple days, it's just the same thing. I don't commit until the problem is solved and then look at the diff again.
A text editor that allows editing within the graphical diff view would be really helpful in these cases, but I'm mostly using Eclipse, which doesn't support that.

Well +1 for starting this discussion. Yes sometime this happen to me. I left those pdb and commit the code to the central code base, git. I use 'emacs'. So, Before commit the code I usually search for pdb in the file. But it is hectic checking each file.So, Before committing the code I usually check the diff very carefully. I am also finding the better way to resolve this issue.

I also develop Python with Vim. I have never had to substantially modify the source code for debugging. I do sometimes put debugging print statements, and I have the habit of putting "# XXX" after every one. Then when I want to remove them (before a commit), and just search for the XXX and delete those lines.
For exceptions, I have arranged to run my code in the Vim buffer with an external interpreter that is set up to automatically enter the debugger on any uncaught exception. Then I'm placed automatically in the code at the point the exception occured. I use a modified debugger that can also signal Vim (actually GTK Gvim) to open that source at that line.
Caught exceptions should report meaningful errors, anyway. It is considered by many to be bad practice to do things like:
try:
... some code
except:
handle everything
Since you probably aren't actually handling every possible error case. By not doing that you also enable the automatic debugging.

I can give you three suggestions:
Do not remove debugger statements. By this, I mean leave them in, but make them conditional on being in debug mode:
# Set this to True to enable Debug code
XYZ_Debug = False
if XYZ_Debug:
do_debugging()
Oh, and if the debugging code is just to print things out, you should get familiar with logging (PyMOTW). If you are using logging, you could:
import logging
# Set this to True to enable debug
XYZ_Debug = False
log = logging.getLogger("XYZ")
log.setLevel(logging.DEBUG if XYZ_Debug else logging.INFO)
log.debug("debug output")
Put the same unique tag (in a comment) after each line, or near each block:
do_debug_code() # XYZZY
I then use Emacs' Ibuffer feature, mark all Python buffers then search for occurrences of this tag. Using some combination of find/grep/sed as in other answers would work as well.
If you are using Mercurial and know Mercurial Queues (or might want to learn them), maintain the debug code as a patch in your queue. When you are ready for "production"; or push of the current changes; pop the patch containing the debug code and go. You could achieve something like this outside of version control with diff and patch.

python coding speed and cleanest

Python is pretty clean, and I can code neat apps quickly.
But I notice I have some minor error someplace and I dont find the error at compile but at run time. Then I need to change and run the script again. Is there a way to have it break and let me modify and run?
Also, I dislike how python has no enums. If I were to write code that needs a lot of enums and types, should I be doing it in C++? It feels like I can do it quicker in C++.

"I don't find the error at compile but at run time"
Correct. True for all non-compiled interpreted languages.
"I need to change and run the script again"
Also correct. True for all non-compiled interpreted languages.
"Is there a way to have it break and let me modify and run?"
What?
If it's a run-time error, the script breaks, you fix it and run again.
If it's not a proper error, but a logic problem of some kind, then the program finishes, but doesn't work correctly. No language can anticipate what you hoped for and break for you.
Or perhaps you mean something else.
"...code that needs a lot of enums"
You'll need to provide examples of code that needs a lot of enums. I've been writing Python for years, and have no use for enums. Indeed, I've been writing C++ with no use for enums either.
You'll have to provide code that needs a lot of enums as a specific example. Perhaps in another question along the lines of "What's a Pythonic replacement for all these enums."
It's usually polymorphic class definitions, but without an example, it's hard to be sure.

With interpreted languages you have a lot of freedom. Freedom isn't free here either. While the interpreter won't torture you into dotting every i and crossing every T before it deems your code worthy of a run, it also won't try to statically analyze your code for all those problems. So you have a few choices.
1) {Pyflakes, pychecker, pylint} will do static analysis on your code. That settles the syntax issue mostly.
2) Test-driven development with nosetests or the like will help you. If you make a code change that breaks your existing code, the tests will fail and you will know about it. This is actually better than static analysis and can be as fast. If you test-first, then you will have all your code checked at test runtime instead of program runtime.
Note that with 1 & 2 in place you are a bit better off than if you had just a static-typing compiler on your side. Even so, it will not create a proof of correctness.
It is possible that your tests may miss some plumbing you need for the app to actually run. If that happens, you fix it by writing more tests usually. But you still need to fire up the app and bang on it to see what tests you should have written and didn't.

You might want to look into something like nosey, which runs your unit tests periodically when you've saved changes to a file. You could also set up a save-event trigger to run your unit tests in the background whenever you save a file (possible e.g. with Komodo Edit).
That said, what I do is bind the F7 key to run unit tests in the current directory and subdirectories, and the F6 key to run pylint on the current file. Frequent use of these allows me to spot errors pretty quickly.

Python is an interpreted language, there is no compile stage, at least not that is visible to the user. If you get an error, go back, modify the script, and try again. If your script has long execution time, and you don't want to stop-restart, you can try a debugger like pdb, using which you can fix some of your errors during runtime.
There are a large number of ways in which you can implement enums, a quick google search for "python enums" gives everything you're likely to need. However, you should look into whether or not you really need them, and if there's a better, more 'pythonic' way of doing the same thing.