I've got a Python program which sits on a remote server which uploads a file to an AWS bucket when run. If I ssh onto the server and run it with the command sudo python3 /path/to/backup.py it works as expected.
I'm writing a Python program to automate a bigger process which includes running backup.py. I created a function to do this using the paramiko library. This is where the command gets run
ssh_stdin, ssh_stdout, ssh_stderr = self.ssh.exec_command('sudo python3 /path/to/backup.py', 1800)
logging.debug(f'ssh_stdout: {ssh_stdout.readline()}')
logging.debug(f'ssh_stderr: {ssh_stderr.readline()}')
My automation gives me this output:
ssh_stdout: Tue, 19 May 2020 14:36:43 INFO The COS endpoint is 9.11.200.206, writing to vault: SD_BACKUP_4058
The program doesn't do anything after that. When I log onto the server and check the logs of backup.py, I can see that it is still running and seems to be sitting at the file upload. This is the code it's stuck at:
s3_client.upload_file(
Filename=BACKUP,
Bucket=BUCKET_NAME,
Key=SPLIT_FILE_NAME,
Callback=pp(BACKUP),
Config=config)
I can't understand why it's getting stuck here when started by my automation program and not when I run it from a command line in the terminal. I can't see anything in the logs which help me. It just seems to be stuck at that point in its execution. Could it be something to do with the callback not getting returned?
You read only one line of the output.
logging.debug(f'ssh_stdout: {ssh_stdout.readline()}')
If the remote program produces lot of output, as soon as its output buffer fills in, the program hangs on the next attempt to write some output.
If you want the program to finish, you have to keep reading the output.
The simplest way is to use readlines or read:
print(stdout.read())
But that's inefficient for large outputs like yours.
Instead you can read the output line by line:
for line in stdout:
print(line.strip())
It gets more complicated, when the commands produces also an error output, as then you have to read both output streams.
See Paramiko ssh die/hang with big output.
And you should check the error output in any case, for good error handling. See also:
Command executed with Paramiko does not produce any output
Related question on SO (by myself earlier today): Why does error traceback show edited script instead of what actually ran? Now I know why it happens, then I want to now how I can deal with it.
I see some questions like How do I debug efficiently with spyder in Python? and How do I print debug messages in the Google Chrome JavaScript Console? well received, so I suppose asking about debugging practices is on-topic, right?
Background
I write a script that raises exception at line n, run it from terminal, add a line in the middle while the script is still running, and save the modified file. So the script file is modified while the interpreter is running it. Especially the line number of the very line that will raise exception has changed. The error traceback report by the Python interpreter shows me line n of the "modified" version of the script, not of the actual "running" version.
Minimal Example
Let's say I run a script:
import time
time.sleep(5)
raise Exception
and while the interpreter is stuck at time.sleep(5), I add a line after that one.
So now I have:
import time
time.sleep(5)
print("Hello World")
raise Exception
Then the interpreter wakes up from sleep, the next command, raise Exception is executed, and the program terminates with the following traceback.
Traceback (most recent call last):
File "test/minimal_error.py", line 4, in <module>
print("Hello World")
Exception
So it correctly reports the line number (from the original script, so actually useless if we only have the modified script) and the error message ("Exception"). But it shows a totally wrong line of code that actually raised the error; if it were to be of any help, raise Exception should be displayed, not print("Hello World"), which wasn't even executed by the interpreter.
Why this matters
In real practice, I implement one part of a program, run it to see if that part is doing fine, and while it is still running, I move on to the next thing I have to implement. And when the script throws an error, I have to find which actual line of code caused the error. I usually just read the error message and try to deduce the original code that caused it.
Sometimes it isn't easy to guess, so I copy the script to clipboard and rollback the code by undoing what I've written after running the script, check the line that caused error, and paste back from clipboard. Sometimes this is very annoying because it isn't always possible to remember the exact state of the script when I ran it. ("Do I need to undo more to rollback? Or is this the exact script I ran?")
Sometimes the script will run for more than 10 minutes, or even an hour before it raises an exception. In such case, "rollback by undo" is practically impossible. Sometimes I even don't know how long script will run before actually running it. I apparently can't just sit and keep my script unmodified before it terminates.
Question
By what practice can I correctly track down the command that caused the exception?
One hypothetical solution is to copy the script to a new file every time I want to run it, run the copied version, and keep editing the original one. But I think this is too bothersome to do every ten minutes, whenever I need to run a script to see if it works well.
Another way is to git-commit every time I want to run it, so that I can come back and see the original version when I need to, but this will make the commit history very dirty, so I think this is even worse than the other one.
I also tried python -m pdb -m script.py, but it shows the same "modified version of line n", just as the plain traceback does.
So is there any practical solution I can practice, say, every ten minutes?
Instead of committing every time you run the script, simply use git stashing, this way you will not add dirty commits to your history.
So before you run a script, git stash your local changes, inspect the error, then git stash pop.
Read more about git stash here.
This solution assumes that the script running is at the HEAD of the current branch,
Another solution if the above condition doesn't apply, is to create an arbitrary branch, call it (running-script), git stash your local changes that are not yet commited, checkout to this new branch, git apply stash and run the script. Then checkout back to your original branch, re-apply the stash and resume your work.
You could simply write a bash script file that automates this process as follows
git stash
git checkout -b running-script # potential param
git stash apply stash
RUN script # replace with the actual command to run the script in the background
git checkout original-branch # potential param
git stash apply stash
You could have the running-script and original-branch passed to the bash file as params.
#chepner's comment is valid:
I'm pretty sure the practical solution is "don't do this". Don't modify running code.
As a relatively simple workaround, you could accomplish this with a bash script (or similar scripted approach in whatever environment you are using if bash isn't available).
For bash, a script like the one below would work. It takes the filename as a parameter and uses date to create a unique temporary filename, then copies the file to it and executes it. In this manner, you always have a static copy of the running code and you can use aliasing to make it trivial to use:
filename=$1
# extract file name and extension
extension="${filename##*.}"
filename="${filename%.*}"
# create a unique temporary name (using date)
today=`date +%Y-%m-%d-%H:%M:%S` # or whatever pattern you desire
newname="$filename-$today.$extension"
# copy and run the python script
cp $1 $newname
echo "Executing from $newname..."
/path/to/python $newname
# clean it up when done, if you care to
rm $newname
You could then alias this to python if you want so you don't have to think about doing it, with something like this in your .bashrc or .bash_aliases:
alias python="source path/to/copy_execute.sh"
Although it may be better to give it a different name, like
alias mypy="source path/to/copy_execute.sh"
Then, you can run your script, modify, and run some more with mypy myscript.py and you won't ever be editing the currently executing code.
One drawback is that while this script will clean up and delete the files after it is done running, it will create a lot of temp files that will be around while it runs. To get around this, you could always copy to somewhere in /tmp or elsewhere where the temporary files won't get in the way. Another issue is that this get's more complicated for large code bases which you may not want to copy all over the place. I'll leave that one to you.
A similar approach could be crafted for Windows with powershell or cmd.
I'm probably going to give an oversimplified answer, and may not be applicable in every scenario.
Use PyCharm
I usually work with code that takes from minutes to hours to finish and I need to constantly run it to see how it is performing and I continue coding while it runs. If it fails, I receive the original line that threw the error.
I also have to run it in an GUI-less Ubuntu server, so this is how I do it to receive the right error every time:
Code in Pycharm.
Test in PyCharm and continue coding. (I get the right error if it
fails)
Once I'm comfortable with performance, I move it to the server and
run it again (I also get the right error here)
I am not saying that it will be completely avoided, but you may reduce this error.
If you are coding all your logic in a single file then stop doing it.
Here are the few recommendation..
split your code logic into multiple files. examples ..
utility,
helper,
model,
component,
train,
test,
feature
make your function as small as 10 lines (if possible)
if you are using class that should not be more that 125 lines
file size should not cross 150 line
Now if there is any Exceptions occur then it traceback might spread into more number of file and i guess not all files get modified in one shot to implement your changes.
Good news is, if your exception started from a file which you have not changed then its easy to catch that line and fix it, else it will be a minimum effort to find exact line.
if you are also using git and you have not committed then you can also compare revision to get exact code which might causing error.
Hope this minimize your problem.
I've written a python script to check the status of the smoke detectors in my house. it runs every second and returns a value of 0,1,2. It uses a separate function on its own thread to recognise the difference between low battery and alarm beeps.
I need a bash script to listen for these conditions and send a netcat packet to my main server if one of the conditions for an alert is met.
I tried using
alertvalue = `python alarm.py`
and running that in a while loop, this echos nothing.
I thought of redirecting the python output to a file but some reading said that it would fail if the script wrote to the file at the same time the bash script would attempt to read from it.
I'm wondering if I can send and receive netcat information directly within the python script?
Could you point me in the right direction here?
I work with Gaussian, which is a program for molecular geometry optimization, among other applications. Gaussian can take days to end a single optimization so I decided to make a program on Python to send me an e-mail when it finishes running. The e-mail sending I figured out. The problem is that Gaussian automatically generates a log file and a chk file, which contains the actual results of the process and by using subprocess.call(['command'], shell=False) both files are not generated.
I also tried to solve the problem with os.system(command), which gives me the .log file and the .chk file, but the e-mail is sent without waiting for the optimization completion.
Another important thing, I have to run the entire process in the background, because as I said at the beginning it might take days to be over and I can't leave the terminal open that long.
by using subprocess.call(['command'], shell=False) both files are not generated.
Your comment suggests that you are trying to run subprocess.call(['g09 input.com &'], shell=False) that is wrong.
Your code should raise FileNotFoundError. If you don't see it; it means stderr is hidden. You should fix it (make sure that you can see the output of sys.stderr.write('stderr\n')). By default, stderr is not hidden i.e., the way you start your parent script is broken. To be able to disconnect from the session, try:
$ nohup python /path/to/your_script.py &>your_script.log &
or use screen, tmux.
shell=False (btw, it is default—no need to pass it explicitly) should hint strongly that call() function does not expect a shell command. And indeed, subprocess.call() accepts an executable and its parameters as a list instead—it does not run the shell:
subprocess.check_call(['g09', 'input.com', 'arg 2', 'etc'])
Note: check_call() raises an exception if g09 returns with a non-zero exit code (it indicates an error usually).
So far I have been getting a lot of help and been able to successfully put together a Python script. The script basically calls a Windows executable and then does some action like pulling down some files from a remote server. And at the end of the script I have a function which does a compression and moves the retrieved files to another server. So far the script was working great, but now looks like I have hit a road hurdle.
The script basically accepts a ParentNumber as a input and finds 1 or more ChildNumbers. Once the list of ChildNumbers are gathered the script goes calls the windows executable with the number repeatedly till it completes pulling data for all of them.
As mentioned above the Function I have built to Archive, Move Files and Email Notification is being called at end of the script, the function works perfectly fine if there is only one ChildNumber. If there are many ChildNumbers and when the executable moves on the 2nd ChildNumber the command line kinda treats it as end and stats with new line something like below:
.........
C:\Scripts\startscript.py
Input> ParentNumber
Retrieval started
Retrieval finished
**Email Sent Successfully**
Preparing ParentNumber #childNumber
C:\Scritps\ParentNumber123\childNumber2
Retrieval Started
Retrieval finished
.........
`
If you see the script flow above the Email Sent successfully message shows up under first ChildNumber only, which means it's called way before the completion of the script.
The actual behavior I want is that all ArchiveMoveEmailFunction should be called once all of the ChildNumbers are processed, but not sure where it's going wrong.
My function for the ArchiveMoveEmailFunction as below and it's at ending of all other lines in the script:
def archiveMoveEmailNotification(startTime, sender, receivers):
"""
Function to archive, move and email
"""
Code for Archive
Code for Move to remote server
Code for email
archiveMoveEmailNotification(startTime, sender,receivers)
Please let me know if I am missing something here to specify on when exactly this function should be executed. As mentioned it works totally fine if the ParentNumber has only 1 ChildNumber, so not sure if the second retrieval jump is causing some issue here. Is there a way I can just have this function wait till rest of the functions in the script are called or would be be logical to move this function to another script completely and call that function from the master script?
Here is the exe call part:
def execExe(childNumb):
cmd = "myExe retriveeAll -u \"%s\" -l \"%s\"" % (childNum.Url(),childAccount.workDir))
return os.system(cmd)
def retriveChildNumb(childNumb):
#Run the retrive
if not (execExe(childNumb)==0):
DisplayResult(childNumb,3)
else:
DisplayResult(childNumb,0)
return 0
Any inputs thoughts on this is very helpful.
Your question is verbose but hard to understand; providing the code would make this much easier to troubleshoot.
That said, my suspicion is that the code you're using to call the Windows executable is asynchronous, meaning your program continues (and finishes) without waiting for the executable to return a value.