Related question on SO (by myself earlier today): Why does error traceback show edited script instead of what actually ran? Now I know why it happens, then I want to now how I can deal with it.
I see some questions like How do I debug efficiently with spyder in Python? and How do I print debug messages in the Google Chrome JavaScript Console? well received, so I suppose asking about debugging practices is on-topic, right?
Background
I write a script that raises exception at line n, run it from terminal, add a line in the middle while the script is still running, and save the modified file. So the script file is modified while the interpreter is running it. Especially the line number of the very line that will raise exception has changed. The error traceback report by the Python interpreter shows me line n of the "modified" version of the script, not of the actual "running" version.
Minimal Example
Let's say I run a script:
import time
time.sleep(5)
raise Exception
and while the interpreter is stuck at time.sleep(5), I add a line after that one.
So now I have:
import time
time.sleep(5)
print("Hello World")
raise Exception
Then the interpreter wakes up from sleep, the next command, raise Exception is executed, and the program terminates with the following traceback.
Traceback (most recent call last):
File "test/minimal_error.py", line 4, in <module>
print("Hello World")
Exception
So it correctly reports the line number (from the original script, so actually useless if we only have the modified script) and the error message ("Exception"). But it shows a totally wrong line of code that actually raised the error; if it were to be of any help, raise Exception should be displayed, not print("Hello World"), which wasn't even executed by the interpreter.
Why this matters
In real practice, I implement one part of a program, run it to see if that part is doing fine, and while it is still running, I move on to the next thing I have to implement. And when the script throws an error, I have to find which actual line of code caused the error. I usually just read the error message and try to deduce the original code that caused it.
Sometimes it isn't easy to guess, so I copy the script to clipboard and rollback the code by undoing what I've written after running the script, check the line that caused error, and paste back from clipboard. Sometimes this is very annoying because it isn't always possible to remember the exact state of the script when I ran it. ("Do I need to undo more to rollback? Or is this the exact script I ran?")
Sometimes the script will run for more than 10 minutes, or even an hour before it raises an exception. In such case, "rollback by undo" is practically impossible. Sometimes I even don't know how long script will run before actually running it. I apparently can't just sit and keep my script unmodified before it terminates.
Question
By what practice can I correctly track down the command that caused the exception?
One hypothetical solution is to copy the script to a new file every time I want to run it, run the copied version, and keep editing the original one. But I think this is too bothersome to do every ten minutes, whenever I need to run a script to see if it works well.
Another way is to git-commit every time I want to run it, so that I can come back and see the original version when I need to, but this will make the commit history very dirty, so I think this is even worse than the other one.
I also tried python -m pdb -m script.py, but it shows the same "modified version of line n", just as the plain traceback does.
So is there any practical solution I can practice, say, every ten minutes?
Instead of committing every time you run the script, simply use git stashing, this way you will not add dirty commits to your history.
So before you run a script, git stash your local changes, inspect the error, then git stash pop.
Read more about git stash here.
This solution assumes that the script running is at the HEAD of the current branch,
Another solution if the above condition doesn't apply, is to create an arbitrary branch, call it (running-script), git stash your local changes that are not yet commited, checkout to this new branch, git apply stash and run the script. Then checkout back to your original branch, re-apply the stash and resume your work.
You could simply write a bash script file that automates this process as follows
git stash
git checkout -b running-script # potential param
git stash apply stash
RUN script # replace with the actual command to run the script in the background
git checkout original-branch # potential param
git stash apply stash
You could have the running-script and original-branch passed to the bash file as params.
#chepner's comment is valid:
I'm pretty sure the practical solution is "don't do this". Don't modify running code.
As a relatively simple workaround, you could accomplish this with a bash script (or similar scripted approach in whatever environment you are using if bash isn't available).
For bash, a script like the one below would work. It takes the filename as a parameter and uses date to create a unique temporary filename, then copies the file to it and executes it. In this manner, you always have a static copy of the running code and you can use aliasing to make it trivial to use:
filename=$1
# extract file name and extension
extension="${filename##*.}"
filename="${filename%.*}"
# create a unique temporary name (using date)
today=`date +%Y-%m-%d-%H:%M:%S` # or whatever pattern you desire
newname="$filename-$today.$extension"
# copy and run the python script
cp $1 $newname
echo "Executing from $newname..."
/path/to/python $newname
# clean it up when done, if you care to
rm $newname
You could then alias this to python if you want so you don't have to think about doing it, with something like this in your .bashrc or .bash_aliases:
alias python="source path/to/copy_execute.sh"
Although it may be better to give it a different name, like
alias mypy="source path/to/copy_execute.sh"
Then, you can run your script, modify, and run some more with mypy myscript.py and you won't ever be editing the currently executing code.
One drawback is that while this script will clean up and delete the files after it is done running, it will create a lot of temp files that will be around while it runs. To get around this, you could always copy to somewhere in /tmp or elsewhere where the temporary files won't get in the way. Another issue is that this get's more complicated for large code bases which you may not want to copy all over the place. I'll leave that one to you.
A similar approach could be crafted for Windows with powershell or cmd.
I'm probably going to give an oversimplified answer, and may not be applicable in every scenario.
Use PyCharm
I usually work with code that takes from minutes to hours to finish and I need to constantly run it to see how it is performing and I continue coding while it runs. If it fails, I receive the original line that threw the error.
I also have to run it in an GUI-less Ubuntu server, so this is how I do it to receive the right error every time:
Code in Pycharm.
Test in PyCharm and continue coding. (I get the right error if it
fails)
Once I'm comfortable with performance, I move it to the server and
run it again (I also get the right error here)
I am not saying that it will be completely avoided, but you may reduce this error.
If you are coding all your logic in a single file then stop doing it.
Here are the few recommendation..
split your code logic into multiple files. examples ..
utility,
helper,
model,
component,
train,
test,
feature
make your function as small as 10 lines (if possible)
if you are using class that should not be more that 125 lines
file size should not cross 150 line
Now if there is any Exceptions occur then it traceback might spread into more number of file and i guess not all files get modified in one shot to implement your changes.
Good news is, if your exception started from a file which you have not changed then its easy to catch that line and fix it, else it will be a minimum effort to find exact line.
if you are also using git and you have not committed then you can also compare revision to get exact code which might causing error.
Hope this minimize your problem.
I want to execute a testrun via bash, if the test needs too much time. So far, I found some good solutions here. But since the command kill does not work properly (when I use it correctly it says it is not used correctly), I decided to solve this problem using python. This is the Execution call I want to monitor:
EXE="C:/program.exe"
FILE="file.tpt"
HOME_DIR="C:/Home"
"$EXE" -vm-Xmx4096M --run build "$HOME_DIR/test/$FILE" "Auslieferung (ML) Execute"
(The opened *.exe starts a testrun which includes some simulink simulation runs - sometimes there are simulink errors - in this case, the execution time of the tests need too long and I want to restart the entire process).
First, I came up with the idea, calling a shell script containing these lines within a subprocess from python:
import subprocess
import time
process = subprocess.Popen('subprocess.sh', shell = True)
time.sleep(10)
process.terminate()
But when I use this, *.terminate() or *.kill() does not close the program I started with the subprocess call.
That´s why I am now trying to implement the entire call in python language. I got the following so far:
import subprocess
file = "somePath/file.tpt"
p = subprocess.Popen(["C:/program.exe", file])
Now I need to know, how to implement the second call "Auslieferung (ML) Execute" of the bash function. This call starts an intern testrun named "Auslieferung (ML) Execute". Any ideas? Or is it better to choose one of the other ways? Or can I get the "kill" option for bash somewhere, somehow?
So this one is a doozie, and a little too specific to find an answer online.
I am writing to a file in C++ and reading that file in Python at the same time to move a robot. Or trying to.
When I try running both programs at the same time, the C++ one runs first and then the Python one runs.
Here's the command I use:
./ColorFollow & python fileToHex.py
This happens even if I switch the order of commands.
Even if I run them in different terminals (which is the same thing, just covering all bases).
Both the Python and C++ code read / write in 'infinite' loops, so these two should run until I say stop.
The code works fine; when the Python script finally runs the robot moves as intended. It's just that the code doesn't run at the same time.
Is there a way to make this happen, or is this impossible?
If you need more information, lemme know, but the code is pretty much what you'd expect it to be.
If you are using Linux, & will release bash session and in this case, CollorFlow and fileToXex.py will run in different bash sessions.
At the same time, composition ./ColorFollow | python fileToHex.py looks interesting, cause you redirect stdout of ColorFollow to fileToHex.py stdin - it can syncronize scripts by printing some code string upon exit, then reading it by fileToHex.py and exit as well.
I would create some empty file like /var/run/ColorFollow.flag and write there 1 when one of processes exit. Not a pipe - cause we do not care which process will start first. So, if next loop step of ColorFollow sees 1 in the file, it deletes it and exits (means that fileToHex already exited). The same - for fileToHex - check flag file each loop step and exit if it exists, after deleting flag file.
I have an application that allows you to use a Python startup script. I want to run one part of the code after the application is fully initialized, but there is no way to know this.
I used time.sleep() but then the application hangs till it executes that code. I also transferred the code to another file and used execfile but same result.
So basically I want something like this:
startup script:
my code
code to run 5 seconds later
What's the easiest way to achieve this? Do I have to use threading for this?
How about at the end of your startup script it writes to or creates a log file contains something like 'initiation complete'.
So another code will check if the file is created or the text is written after each x millisecond and then execute the rest of it.
So far I have been getting a lot of help and been able to successfully put together a Python script. The script basically calls a Windows executable and then does some action like pulling down some files from a remote server. And at the end of the script I have a function which does a compression and moves the retrieved files to another server. So far the script was working great, but now looks like I have hit a road hurdle.
The script basically accepts a ParentNumber as a input and finds 1 or more ChildNumbers. Once the list of ChildNumbers are gathered the script goes calls the windows executable with the number repeatedly till it completes pulling data for all of them.
As mentioned above the Function I have built to Archive, Move Files and Email Notification is being called at end of the script, the function works perfectly fine if there is only one ChildNumber. If there are many ChildNumbers and when the executable moves on the 2nd ChildNumber the command line kinda treats it as end and stats with new line something like below:
.........
C:\Scripts\startscript.py
Input> ParentNumber
Retrieval started
Retrieval finished
**Email Sent Successfully**
Preparing ParentNumber #childNumber
C:\Scritps\ParentNumber123\childNumber2
Retrieval Started
Retrieval finished
.........
`
If you see the script flow above the Email Sent successfully message shows up under first ChildNumber only, which means it's called way before the completion of the script.
The actual behavior I want is that all ArchiveMoveEmailFunction should be called once all of the ChildNumbers are processed, but not sure where it's going wrong.
My function for the ArchiveMoveEmailFunction as below and it's at ending of all other lines in the script:
def archiveMoveEmailNotification(startTime, sender, receivers):
"""
Function to archive, move and email
"""
Code for Archive
Code for Move to remote server
Code for email
archiveMoveEmailNotification(startTime, sender,receivers)
Please let me know if I am missing something here to specify on when exactly this function should be executed. As mentioned it works totally fine if the ParentNumber has only 1 ChildNumber, so not sure if the second retrieval jump is causing some issue here. Is there a way I can just have this function wait till rest of the functions in the script are called or would be be logical to move this function to another script completely and call that function from the master script?
Here is the exe call part:
def execExe(childNumb):
cmd = "myExe retriveeAll -u \"%s\" -l \"%s\"" % (childNum.Url(),childAccount.workDir))
return os.system(cmd)
def retriveChildNumb(childNumb):
#Run the retrive
if not (execExe(childNumb)==0):
DisplayResult(childNumb,3)
else:
DisplayResult(childNumb,0)
return 0
Any inputs thoughts on this is very helpful.
Your question is verbose but hard to understand; providing the code would make this much easier to troubleshoot.
That said, my suspicion is that the code you're using to call the Windows executable is asynchronous, meaning your program continues (and finishes) without waiting for the executable to return a value.