We Keep Coding

Conditionally run subprocess over ssh, while appending output to a (potentially remote) file

I have a script which can run on my host machine and several other servers. I want to launch this script as a background process on my host machine along with the remote machine using ssh and output the stdout/stderr to host machine for my host machine background process and on the remote machines for remote machine background tasks.
I tried with
subprocess.check_output(['python' ,'script.py' ,'arg_1', ' > file.log ', ' & echo -ne $! ']
but it doesn't work. it doesnt give me the pid nor write into the file. It works with shell=True but then I read it is not good to use shell=True for security reasons.
then I tried
p = subprocess.Popen(['python' ,'script.py' ,'arg_1', ' > file.log ']
Now i can get the process pid but the output is not writing in the remote log file.
using stdout/stderr arguments like suggested below will open the log file in my host machine not the remote machine. i want to log on the remote machine instead.
append subprocess.Popen output to file?
Could someone please suggest me a single command that works both on my host machine and also ssh's to remote server and launches the background process there? and write to output file ?
<HOW_TO_GET_PID> = subprocess.<WHAT>( ([] if 'localhost' else ['ssh','<remote_server>']) + ['python', 'script.py', 'arg_1' <WHAT>] )
Someone could please finish the above psudo code ?
Thanks,

You're not going to get something that's safe and correct in a one-liner without making it unreadable; better not to try.
Note that we're using a shell here: In the local case we explicitly call shell=True, whereas in the remote case ssh always, implicitly starts a shell.
import shlex
import subprocess
def startBackgroundCommand(argv, outputFile, remoteHost=None, andGetPID=False):
cmd_str = ' '.join(shlex.quote(word) for word in argv)
if outputFile != None:
cmd_str += ' >%s' % (shlex.quote(outputFile),)
if andGetPID:
cmd_str += ' & echo "$!"'
if remoteHost != None:
p = subprocess.Popen(['ssh', remoteHost, cmd_str], stdout=subprocess.PIPE)
else:
p = subprocess.Popen(cmd_str, stdout=subprocess.PIPE, shell=True)
return p.communicate()[0]
# Run your command locally
startBackgroundCommand(['python', 'script.py', 'arg_1'],
outputFile='file.log', andGetPID=True)
# Or run your command remotely
startBackgroundCommand(['python', 'script.py', 'arg_1'],
remoteHost='foo.example.com', outputFile='file.log', andGetPID=True)

# At the beginning you can even program automatic daemonizing
# Using os.fork(), otherwise, you run it with something like:
# nohup python run_my_script.py &
# This will ensure that it continues running even if SSH connection breaks.
from subprocess import Popen, PIPE, STDOUT
p = Popen(["python", "yourscript.py"], stdout=PIPE, stderr=STDOUT, stdin=PIPE)
p.stdin.close()
log = open("logfile.log", "wb")
log.write(b"PID: %i\n\n" % p.pid)
while 1:
line = p.stdout.readline()
if not line: break
log.write(line)
log.flush()
p.stdout.close()
log.write(b"\nExit status: %i" % p.poll())
log.close()

Python how to make a ping -t script looping indefinitely

I want to run a python script file with this ping -t www.google.com command.
So far I did one with the ping www.google.com command which works, but I didn't suceed to do on with the ping -t looping indefinitely.
You can find below my ping.py script:
import subprocess
my_command="ping www.google.com"
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
result, error = my_process.communicate()
result = result.strip()
error = error.strip()
print ("\nResult of ping command\n")
print("-" *22)
print(result.decode('utf-8'))
print(error.decode('utf-8'))
print("-" *22)
input("Press Enter to finish...")
I want the command box to stay open when finished. I am using Python 3.7.

If you want to keep the process open and communicate with it all the time, you can use my_process.stdout as input and e.g. iterate over it's lines. With "communicate" you wait until the process completes, which would be bad for an indefinitely running process :)
import subprocess
my_command=["ping", "www.google.com"]
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
print( my_process.stdout.readline() )
EDIT
In this version we use re to only get the "time=xxms" part from the output:
import subprocess
import re
my_command=["ping", "-t", "www.google.com"]
my_process=subprocess.Popen(my_command, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
while True:
line = my_process.stdout.readline() #read a line - one ping in this case
line = line.decode("utf-8") #decode the byte literal to string
line = re.sub("(?s).*?(time=.*ms).*", "\\1", line) #find time=xxms in the string and use only that
print(line)

Start background process in python via subprocess and write output to file

Dear stackoverflow users,
I'm looking for a solution for a probably quite easy problem. I want to automate some quantum chemical calculations and ran into a small problem.
Normally you start your quantum chemical programm (in my case it's called orca) with your input file (*.inp) on a remote server as a background process and pipe the output into an outputfile (*.out) via
nohup orca H2.inp >& H2.out &
or something similar.
Now I wanted to use a python script (with some templating) to write the input file automatically. At the end the script should start the calculation in a way that I could log out of the server without stopping orca. I tried that with
subprocess.run(["orca", input_file], stdout=output_file)
but so far it did not work. How do I "emulate" the command given at the top with the subprocess module?
Regards
Update
I have one file that is called H2.xyz. The script reads and splits the filename by the point and creates an input file name H2.inp and the output should be written into the file H2.out.
Update 2
The input file is derived from the *xyz file via
xyzfile = str(sys.argv[1])
input_file = xyzfile.split(".")[0] + ".inp"
output_file = xyzfile.split(".")[0] + ".out"
and is created within the script via templating. In the end I want to run the script in the following way:
python3 script.py H2_0_1.xyz

Why not simply:
subprocess.Popen(f'orca {input_file} >& {output_file}',
shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)
More info:
Run Process and Don't Wait

For me (Windows, Python 2.7) the method call works very fine like this:
with open('H2.out', 'a') as out :
subprocess.call(['orca', infile], stdout=out,
stderr=out,
shell=True) # Yes, I know. But It's Windows.
On Linux you maybe do not need shell=True for a list of arguments.

Is the usage of subprocess important? If not, you could use os.system.
The Python call would get really short, in your case
os.system("nohup orca H2.inp >& H2.out &")
should do the trick.

I had the same problem not long ago.
Here is my solution:
commandLineCode = "nohup orca H2.inp >& H2.out &"
try:
proc = subprocess.Popen(commandLineCode,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd = workingDir)
except OSError:
print("Windows Error occured")
print(traceback.format_exc())
timeoutInSeconds = 100
try:
outs, errs = proc.communicate(timeout = timeoutInSeconds)
except subprocess.TimeoutExpired:
print("timeout")
proc.kill()
outs, errs = proc.communicate()
stdoutDecode = outs.decode("utf-8")
stderrDecode = errs.decode("utf-8")
for line in stdoutDecode.splitlines():
# write line to outputFile
if stderrDecode:
for line in stderrDecode.splitlines():
# write line to error log
The OSError exception is pretty important since you never now what your OS might do wrong.
For more on the communicate() command which actually starts the process read:
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate

Python Subprocess not printing vnstat process

Thank you for taking the time to read this post.
I'm trying to do live bandwidth monitoring in python using vnstat. Unfortunately, It is not printing the output that i want, and i cant seem to figure out why. This is my code below.
from subprocess import Popen, PIPE
import time
def run(command):
process = Popen(command, stdout=PIPE, bufsize=1, shell=True ,universal_newlines=True)
while True:
line = process.stdout.readline().rstrip()
print(line)
if __name__ == "__main__":
run("sudo vnstat -l -i wlan1")
When i run this code in the terminal , this is the output i get :
sudo python testingLog.py
Monitoring wlan1... (press CTRL-C to stop)
It does not show me the desired output when running "vnstat -l -i wlan1" in the terminal.
Desired Output :
Monitoring wlan1... (press CTRL-C to stop)
rx: 0 kbit/s 0 p/s tx: 0 kbit/s 0 p/s
What happens when i run vnstat -l -i wlan1 is that it will update it and be running live, so i suspect that my printing is wrong as it does not print the desired output but i cant seem to figure out why.

It's not that your printing is wrong, it's the fact that vnstat keeps updating the same line without issuing a new line so process.stdout.readline() hangs at one point, waiting for a new line that never comes.
If you just want to redirect vnstat STDOUT to Python's STDOUT you can just pipe it, i.e.:
import subprocess
import sys
import time
proc = subprocess.Popen(["vnstat", "-l", "-i", "wlan1"], stdout=sys.stdout)
while proc.poll() is None: # loop until the process ends (kill vnstat to see the effect)
time.sleep(1) # wait a second...
print("\nProcess finished.")
If you want to capture the output and deal with it yourself, however, you'll have to stream the STDOUT a character (or safe buffer) at the time to capture whatever vnstat publishes and then decide what to do with it. For example, to simulate the above piping but with you in the driver's seat you can do something like:
import subprocess
import sys
proc = subprocess.Popen(["vnstat", "-l", "-i", "wlan1"], stdout=subprocess.PIPE)
while True: # a STDOUT read loop
output = proc.stdout.read(1) # grab one character from vnstat's STDOUT
if output == "" and proc.poll() is not None: # process finished, exit the loop
break
sys.stdout.write(output) # write the output to Python's own STDOUT
sys.stdout.flush() # flush it...
# of course, you can collect the output instead of printing it to the screen...
print("\nProcess finished.")

Communicate with python subprocess while it is running

I running a subprocess that run a software in "command" mode. (This software is Nuke by The Foundy, in case you know that software)
When in command mode, this software is waiting for user input. This mode allow to create compositing scripts without any UI.
I have done this bit of code that start the process, find when the application is done starting then I try to send the process some commands, but the stdin doesn't seem to be sending the commands properly.
Here the sample code I did to test this process.
import subprocess
appPath = '/Applications/Nuke6.3v3/Nuke6.3v3.app/Nuke6.3v3' readyForCommand = False
commandAndArgs = [appPath, '-V', '-t']
commandAndArgs = ' '.join(commandAndArgs)
process = subprocess.Popen(commandAndArgs,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
shell=True, )
while True:
if readyForCommand:
print 'trying to send command to nuke...'
process.stdin.write('import nuke')
process.stdin.write('print nuke')
process.stdin.write('quit()')
print 'done sending commands'
readyForCommand = False
else:
print 'Reading stdout ...'
outLine = process.stdout.readline().rstrip()
if outLine:
print 'stdout:', outLine
if outLine.endswith('getenv.tcl'):
print 'setting ready for command'
readyForCommand = True
if outLine == '' and process.poll() != None:
print 'in break!'
break
print('return code: %d' % process.returncode)
when I run nuke in a shell and send the same commands here is what I get:
sylvain.berger core/$ nuke -V -t
[...]
Loading /Applications/Nuke6.3v3/Nuke6.3v3.app/Contents/MacOS/plugins/getenv.tcl
>>> import nuke
>>> print nuke
<module 'nuke' from '/Applications/Nuke6.3v3/Nuke6.3v3.app/Contents/MacOS/plugins/nuke/__init__.pyc'>
>>> quit()
sylvain.berger core/$
Any idea why the stdin is not sending the commands properly?
Thanks

your code will send the text
import nukeprint nukequit()
with no newline, thus the python instance will not try to execute anything, everything is just sitting in a buffer waiting for a newline

The subprocess module is not intended for interactive communication with a process. At best, you can give it a single pre-computed standard input string and then read its stdout and stderr:
p = Popen(..., stdin=PIPE, stdout=PIPE, stderr=PIPE)
out, err = p.communicate(predefined_stdin)
If you actually need interaction, consider using pexpect.