I need to execute multiple shell commands in a ssh session using the subprocess module.
I am able to execute one command at a time with:
subprocess.Popen(["ssh", "-o UserKnownHostsFile=/dev/null", "-o StrictHostKeyChecking=no", "%s" % <HOST>, <command>])
But is there a way to execute multiple shell commands with subprocess in a ssh session? If possible, I don't want to use packages.
Thank you very much!
Strictly speaking you are only executing one command with Popen no matter how many commands you execute on the remote server. That command is ssh.
To have multiple commands executed on the remote server just pass them in your command string seperated by ;s:
commands = ["echo 'hi'", "echo 'another command'"]
subprocess.Popen([
"ssh",
"-o UserKnownHostsFile=/dev/null",
"-o StrictHostKeyChecking=no",
";".join(commands)
])
You could alternatively join commands on && if you wanted each command to only execute if the previous command had succeeded.
If you have many commands and you are concerned that you might exceed the command line limits, you could execute sh (or bash) with the -s option on the remote server, which will execute commands one-by-one as you send them:
p = subprocess.Popen([
"ssh",
"-o UserKnownHostsFile=/dev/null",
"-o StrictHostKeyChecking=no",
"sh -s",
], stdin=subprocess.PIPE)
for command in commands:
p.stdin.write(command)
p.stdin.write("\n")
p.flush()
p.communicate()
Note that in Python3 you will need to encode the command to a byte string (command.encode("utf8")) before writing it to the stdin of the subprocess.
I feel like this is overkill for most simple situations though where the initial suggest is simplest.
Related
I am using a python script to restrict the commands usage using the command argument in the authorized_keys file.
command:
ssh host-name bash --login -c 'exec $0 "$#"' mkdir -p hello
My script is performing required actions to restrict the commands. After filtering, the python script does sys.exit(1) for error and sys.exit(0) for success. After the return value the above ssh command is not getting executed at the end. Is there something else I need to send from the python script to SSH daemon?
The command modifier in the authorized_keys is not (only) used to validate the users command, but that command is run instead of the command provided by the user. This means calling sys.exit(0) from there prevents running the user-provided command.
In that script, after you validate the command, you need to run it too!
I think changing it to
ssh host-name bash --login -c 'exec $0 "$#" && mkdir -p hello'
should do the trick, otherwise bash will assume only the part in the single quotes is the command to execute.
If the second part should be executed even if the first part fails, replace the && with ;
I have a python script(myscript.py) as follows:
#!/bin/python
import os
import optparse
import subprocess
import sys
sys.stdout.flush()
print("I can see this message on Jenkins console output")
cmd="sshpass -p 'xxx' ssh test#testmachine 'cmd /c cd C:\stage && test.bat'"
retval=subprocess.call(cmd,shell=True)
print retval
In jenkins, I have a job with execute shell as follows:
#!/bin/sh
./myscript.py
Problem:
Jenkins console shows only "I can see this message on Jenkins console output".
If there is any output from the subprocess call, it does not print it out on the console.
If I putty to Server A and run the same command (./myscript.py) on shell, I can see the output of subprocess call.
How can I print this output of subprocess call on Jenkins console?
FYI: As you can see from my command, the subprocess call is running a batch file on windows; Jenkins is running on Linux; There is ssh setup between the two machines..
Edit:
My test.bat looks like this:
echo off
RMDIR /S /Q C:\Test
IF %ERRORLEVEL% NEQ 0 (
ECHO Could not delete
EXIT /b %ERRORLEVEL%
)
if I run this batch file locally on windows server, it returns a 1 ( because am holding a file open in Test folder )
But when the python script calls this batch file using the subprocess call, all i get is a Zero for retval.
Why is this and how to fix this? If I can capture the correct retval, I can make the Jenkins job fail.
Edit 12/12:
Helllo!! Anybody! Somebody! Help!
I wonder if it has to do anything with stdout being buffered
Can you try setting PYTHONUNBUFFERED before running your command?
export PYTHONUNBUFFERED=true
In my Jenkins environment, executing python scripts with the unbuffered argument makes the output appear immediately. Like this:
python3 -u some_script.py
More information comes from the help menu (python3 --help):
-u : force the stdout and stderr streams to be unbuffered;
this option has no effect on stdin; also PYTHONUNBUFFERED=x
TL; DR
The fix is to use some conditional execution (the || operator) on rmdir to fix the errorlevel being returned.
Investigation
This was a corker of a bug, with quite a few twists and turns! We initially suspected that the stdout chain was broken somehow, so looked into that through explicit use of pipes in Popen and then removing sshpass from your command and so using the output from ssh directly.
However, that didn't do the trick, so we moved on to looking at the return code of the command. With sshpass removed, ssh should return the result of the command that was run. However, this was always 0 for you.
At this point, I found a known bug in Windows that rmdir (which is the same as rd) doesn't always set errorlevel correctly. The fix is to use some conditional execution (the || operator) on rmdir to fix up the errorlevel.
See batch: Exit code for "rd" is 0 on error as well for full details.
When you execute your script in a shell, Python sets your shell's STDOUT as the subprocess's STDOUT, so everything that gets executed gets printed to your terminal. I'm not sure why, but when you're executing in Jenkins the subprocess is not inheriting the shell's STDOUT so its output is not displayed.
In all likelihood, the best way to solve your problem will be to PIPE the STDOUT (and STDERR for good measure) and print it after the process ends. Also, if you exit with the exit code of your subprocess and the exit code is not 0, it will likely terminate your Jenkins job.
p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE, shell=True)
exit_code = p.wait() # wait for it to end
print('Got the following output from the script:\n', p.stdout.read().decode())
print('Got the following errors from the script:\n', p.stderr.read().decode())
print('Script returned exit code:', exit_code)
sys.exit(exit_code)
Trying to monitor the available physical disc space of a remote machine using a python script, which executes the df -h . command using subprocess.popen.
import subprocess
import time
command = 'ssh remoteserver "df -h ."'
while True:
proc = subprocess.Popen(command,shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,err=proc.communicate()
print output
print err
time.sleep(60)
The script runs fine and prints the output to the terminal when run from command line
$> python2.7 script.py
Filesystem Size Used Avail Use% Mounted on
remoteserver:/home/user
555G 447G 109G 81% /home
The scripts does not produce any output or seems to be blocking when the script is started with nohup command.
$> nohup python2.7 script.py &
Would like the script to work and fetch the disc space of remote machine using the above script when started in nohup.
I'm not 100% sure of the underlying issue here, but when you invoke NOHUP in the shell, it's disconnected some of the STDIN/STDOUT from the terminal process, which I suspect it causing some of this interactions you're seeing.
Given that you're doing this from a remote machine, I'd actually recommend you look at using something like Fabric as a library to do what you're after. It's pretty straightforward, and does most of the handling of terminal sessions as well as closing things down nicely for you when you're complete.
something like:
from fabric import api
from fabric.api import env
import fabric
env.host_string = '%s#%s' % (username, remote_host)
env.disable_known_hosts = True
env.password = password
fabric.state.output['stdout'] = False
fabric.state.output['stderr'] = False
results = api.run('df -h')
You might try sending stdin=subprocess.PIPE to the subprocess command, then calling proc.stdin.close() on the next line, before the communicate() call. Or you can try changing the command to 'ssh remoteserver "df -h ." </dev/null'. Others report using FNULL = open(os.devnull, 'r') and passing in FNULL to the stdin= argument, but I'm not sure if you need to call FNULL.close() after or not.
SSH is most likely waiting for input for some reason when it is run from nohup. Perhaps it is unable to authenticate in the nohup environment and is asking for password input?
To make sure SSH is not waiting for input, try adding -o "BatchMode yes" to the ssh command and see if there are some clues in the output/error from the subprocess communicate call.
I need to run some bash commands via Fabric API (ssh).
I have the following String in my Python module:
newCommand = command + "'`echo -ne '\\015'"
When I print this string directly in Python the output is the expected:
command'`echo -ne '\015'
However, if I try to run this command via the Fabric API the command is somehow modified into this:
/bin/bash -l -c "command'\`echo -ne '\015'"
Notice the '\' before 'echo'. Why is this happenning? The '\' is breaking my command and I can't successfuly run the command.
ps: The prefix "/bin/bash -l -c" is expected since that's how Fabric works with SSH
This is not a valid shell command:
command'`echo -ne '\015'
Even if you add the missing backtick and single quote, it's nothing like writing "command" and pressing enter.
The context your command will be run in is basically what you'd get if you'd ssh and paste a command:
clientprompt$ ssh host
Welcome to Host, User
hostprompt$ <COMMAND HERE>
You should focus your efforts on finding a single command that does what you want, and not a series of keypresses that you could write to do it (that's not how ssh works).
I use python module pysftp to connect to remote server. Below you can see python code :
import pysftp
import sys
import sqr_common
srv = pysftp.Connection(host="xxxxxx", username="xxxx",
password="xxxxx")
command = "/usr/bin/bash"
command2="APSHOME=/all/aps/msc_2012; export APSHOME; "
srv.execute(command)
srv.execute(command2)
srv.close()
Problem is that command /usr/bin/bash is an infinite process , so my script will never be executed. Can anyone help me how to choose shell on remote server for example bash and execute command in bash on remote server?? Is there any pysftp function that allows me chosing shell??
try this
/usr/bin/bash -c "APSHOME=/all/aps/msc_2012; export APSHOME; "
This problem is not specific to Python, but more like how to execute commands under specific shell.
If you need to run only single command you can run using bash -c switch
bash -c "echo 123"
You can run multiple commands ; separated
bash -c "echo 123 ; echo 246"
If you need to many commands under a specific shell, remotely create a shell script file (.bash file) an execute it
bash myscript.bash