We Keep Coding

Development vs Release Python Code

I am currently developing a Python application which I continually performance test, simply by recording the runtime of various parts.
A lot of the code is related only to the testing environment and would not exist in the real world application, I have these separated into functions and at the moment I comment out these calls when testing. This requires me to remember which calls refer to test only components (they are quite interleaved so I cannot group the functionality).
I was wondering if there was a better solution to this, the only idea I have had so far is creation of a 'mode' boolean and insertion of If statements, though this feels needlessly messy. I was hoping there might be some more standardised testing method that I am naive of.
I am new to python so I may have overlooked some simple solutions.
Thank you in advance

There are libraries for testing like those in the development-section of the standard library. If you did not use such tools yet, you should start to do so - they help a lot with testing. (especially unittest).
Normally Python runs programs in debug mode with __debug__ set to True (see docs on assert) - you can switch off debug mode by setting the command-line switches -O or -OO for optimization (see docs).
There is something about using specifically assertions in the Python Wiki

I'd say if you're commenting out several parts of your code when switching between debug&release mode I think you're doing wrong. Take a look for example to the logging library, as you can see, with that library you can specify the logging level you want to use only by changing a single parameter.
Try to avoid commenting specific parts of your debug code by having one or more variables which controls the mode (debug, release, ...) your script will run. You could also use some builtin ones python already provides

Execution Code Tracking - How to know which code has been executed in project?

Let say that I have open source project from which I would like to borrow some functionality. Can I get some sort of report generated during execution and/or interaction of this project?
Report should contain e.g.:
which functions has been called,
in which order,
which classes has been instantiated etc.?
Would be nice to have some graphic output for that... you know, if else tree and highlighted the executed branch etc.
I am mostly interested in python and C (perl would be fine too) but if there is any universal tool that cover multiple languages (or one tool per language) for that, it would be very nice.
PS: I am familiar with debuggers but I do not want to step every singe line of code and check if this is the correct instruction. I'm assuming that if functions/methods/classes etc. are properly named then one can get some hints about where to find desired piece of code. But only naming is not enough because you do not know (from brief overview of code) if hopefully looking function foo() does not require some data that was generated by obscure function bar() etc. For that reason I am looking for something that can visualize relations between running code.
PS: Do not know if this is question for SO or programmers.stackexchange. Feel free to move if you wish. PS: I've noticed that tags that I've used are not recommended but execution flow tracking is the best phrase to describe this process

Check out Ned Batchelder's coverage and perhaps the graphviz/dot library called pycallgraph. May not be exactly what you need and also (python-only) but in the ballpark.
Pycallgraph is actually likelier to be of interest because it shows the execution path, not just what codelines got executed. It only renders to PDF normally, but it wasn't too difficult to get it to do SVG instead (dot/graphviz supports svg and other formats, pycallgraph was hardcoding pdf rendering).
Neither will do exactly what you want but they are a start.

Debugging with pycharm, how to step into project, without entering django libraries

Think about this scenario:
I debug my Django project and I step through the code (in and out). The debugger sometimes enters Django libraries or other external libraries.
Does anyone know how to prevent the debugger from entering external code? Or at least a 'big' step out to get the debugger back to the project code?

Does anyone know how to prevent the debugger from entering external code?
Yes, Dmitry Trofimov knows;
(...) add modules you don't want to trace to the dict DONT_TRACE in <pycharm-distr>/helpers/pydev/pydevd.py
That is a hacky solution (...)
If you want this feature to be less hacky you can vote on it by visiting issue
PY-9101 Implement "Do not step into the classes" option for Python debugger
Those using pdb might be interested to know there is such a feature in pdb;
Starting with Python 3.1, Pdb class has a new argument called skip -
class pdb.Pdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False)
The skip argument, if given, must be an iterable of glob-style module
name patterns. The debugger will not step into frames that originate
in a module that matches one of these patterns. 1
1 Whether a frame is considered to originate in a certain module is
determined by the __name__ in the frame globals.
The example given in the docs shows how to skip Django's packages -
import pdb; pdb.Pdb(skip=['django.*']).set_trace()

Everything looks the same to the debugger, it can't distinguish between your code or Django's code – it's all Python. So it will run everything, however if you want to stop it from drilling down so low you'll have to start “stepping over” lines of code instead of “stepping into” them.
According to the PyCharm docs you'll want to use F8 when ever you see a line of code that looks like it could be a gateway into Django's internals. If you accidently find yourself inside Django's source code you can hit Shift+F8 until you're out of it.

How to automate python logging

I need to add logging to a milter that I wrote a few months back. It is occasionally rejecting some messages, but I'm not sure why. I know how to add logging to a Python script from the HowTo, but is it necessary for me to add log output commands at every point in my script, or is there a way Python automatically handles that?
Basically, I don't know where in the script it fails and don't want to add the overhead of 60 logging lines. I'm looking for the simplest method of doing this.

If you have no idea where it fails you could run a debugging session with input that you know causes the error, and step through the code if that is an option.
Another pretty obvious option is to log all exceptions at the entrance of your script and then drill down from there, but I honestly don't think that there is a way that will find the right places to log for you - if this would be the case that program could just as well track the bug down on itself.

How can I write to the previous line in a log file using Python's Logging module?

long-time lurker here, finally emerging from the woodwork.
Essentially, what I'm trying to do is have my logger write data like this to the logfile:
Connecting to database . . . Done.
I'd like the 'Connecting to database . . . ' to be written when the function is called, and the 'Done' written after the function has successfully executed.
I'm using Python 2.6 and the logging module. Also, I'd really like to avoid using decorators for this. Any help would be most appreciated!

Writing to a log is, and must be, an atomic action -- this is crucial, and a key feature of any logging package (including the one in Python's standard library) that distinguishes logging from the simple appending of information to files (where bits of things being written by different processes and threads might well "interleave" -- one of them writing some part of a line but not the line-end, just as you desire, and then maybe another one interposing something right afterwards, before the first task writes what it thinks will be the last part of the line but actually ends up on another line... utter confusion often results;-).
It's not inevitable that the atomic unit be "a line" (logs can be recorded elsewhere than to a text file, of course, and some of the things that are acceptable "sinks" for logs won't even have the concept of "a line"!), but, for logging, atomic units there must be. If you want something entire non-atomic then you don't really want logging but simple appends to a file or other stream (and, watch out for the likely confusion mentioned in the first paragraph;-).
For transient updates about what your task is doing (in the middle of X, X done, starting Y, etc), you could think of a specialized log-handler that (for example) interprets such streams of updates by taking the first word as a subtask-identifier (incrementally building up and displaying somewhere the composite message about the "current subtask", recognizing when the subtask identifier changes that the previous subtask is finished or taking an explicit "subtask finished" message, and only writing persistent log entries on subtask-finished events).
It's a pretty specialized requirement so you're not likely to find a pre-made tool for this, but rather you'll have to roll your own. To help you with that, it's crucial to understand exactly what you're trying to accomplish (why would you want non-atomic logging entries, if such a concept even made any sense -- what deployment or system administration task are you trying to ameliorate by using such a hypothetical tool?) so that the specialized subsystem can be tailored to your actual needs. So, can you please expand on this?

I don't believe Python's logger supports that.
However, would it not be better to aggree on a Log format so that the log file can be easily parsed when you want analyse the data where ; is any deliminator you want:
DateTime;LogType;string
That could be parsed easiily by a script and then you could do analysis on the logs
If you use:
Connecting to database . . . Done.
Then you won't be able to analyse how long the transaction took

I don't think you should go down this route. A logging methodolgy with entry:
Time;functionName()->
And exit logging is more useful for troubleshooting:
Time;functionName()<-