I have one issue with subprocess.run.
This command in a Bash shell works without any problem:
tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1
But if I execute it through Python:
cmd = "tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
The execution works without errors but the --exclude clause is not considered.
Why?
Whether or not curly brace expansion is handled correctly depends on what the standard system shell is. By default, subprocess.run() invokes /bin/sh. On systems like Linux, /bin/sh is bash. On others, such as FreeBSD, it's a different shell that doesn't support brace expansion.
To ensure the subprocess runs with a shell that can handle braces properly, you can tell subprocess.run() what shell to use with the executable argument:
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, executable='/bin/bash')
As a simple example of this, here's a system where /bin/sh is bash:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo=a foo=b
and one where it's not:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo={a,b}
but specifying another shell works:
>>> subprocess.run("echo foo={a,b}", shell=True, executable='/usr/pkg/bin/bash')
foo=a foo=b
Bash curly expansion doesn't work inside Python and will be sent by subprocess as they are - they will not be expanded, regardless of the arguments you use on run().
Edit: unless of course the argument executable='/bin/bash' as stated on the other answer which seems to work after all
In a bash shell,
--exclude {'/home/user1/.cache','/home/user1/.config'}
becomes:
--exclude=/home/user1/.cache --exclude=/home/user1/.config
So to achieve the same result, in Python it must be expressed like this (one of the possible ways) before sending the command string to subprocess.run:
' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']])
cmd = "tar -C '/home/' " + ' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']]) + " -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
print(cmd) # output: "tar -C '/home/' --exclude=/home/user1/.cache --exclude=/home/user1/.config -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
Related
I'm trying to run a for loop in a shell through python. os.popen runs it fine, but is deprecated on 3.x and I want the stderr. Following the highest-voted answer on How to use for loop in Subprocess.run command results in Syntax error: "do" unexpected, with which shellcheck concurs:
import subprocess
proc = subprocess.run(
"bash for i in {1..3}; do echo ${i}; done",
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE, )
print(proc.stderr)
I'm ultimately trying to reset all usbs by calling this shell code https://unix.stackexchange.com/a/611305/362437 through python, so any alternate approaches to doing that would be appreciated too.
When you do
subprocess.run('foo', shell=True)
it actually runs the equivalent of
/bin/sh -c 'foo'
(except that it magically gets all quotes right :-) ). So, in your case, it executes
/bin/sh -c "bash for i in {1..3}; do echo ${i}; done"
So the "command" given with the -c switch is actually a list of three commands: bash for i in {1..3}, do echo ${i}, and done. This is going to leave you with a very confused shell.
The easiest way of fixing this is probably to remove that bash from the beginning of the string. That way, the command passed to /bin/sh makes some sense.
If you want to run bash explicitly, you're probably better off using shell=False and using a list for the first argument to preserve your quoting sanity. Something like
import subprocess
proc = subprocess.run(
['/bin/bash', '-c', 'for i in {1..3}; do echo ${i}; done'],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE, )
I am trying to run the shell command
echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest -
from jupyter notebook using subprocesses
I have tried the code
p = subprocess.Popen('''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''', shell=True)
p.communicate()
and some iterations with run() or call(), but everytime the output is
-e 'FROM busybox
It seems that the new line character \n causes the problem. Any ideas to solve the problem?
The \n gets parsed by Python into a literal newline. You can avoid that by using a raw string instead,
p = subprocess.run(
r'''echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -t myimage:latest - ''',
shell=True, check=True)
but I would recommend running a single process and passing in the output from Python; this also avoids a shell, which is generally desirable.
p = subprocess.run(['docker', 'build', '-t', 'myimage:latest', '-'],
input='FROM busybox\nRUN echo "hello world"',
text=True, check=True)
Notice also how we prefer subprocess.run() over the more primitive subprocess.Popen(); as suggested in the documentation, you want to avoid this low-level function whenever you can. With check=True we also take care to propagate any subprocess errors up to the Python parent process.
As an aside, printf is both more versatile and more portable than echo -e; I would generally recommend you to avoid echo -e altogether.
This ideone demo with nl instead of docker build demonstrates the variations, and coincidentally proves why you want to avoid echo -e even if your login shell is e.g. Bash (in which case you'd think it should be supported; but subprocess doesn't use your login shell).
So I have a remote Linux Server and would like to run a Python Script on my local machine to list all files and their modification dates in a specific folder on that remote server. That is my code so far:
command = "find \""+to_directory+'''\"* -type f -exec sh -c \"stat -c \'%y:%n\' \'{}\'\" \;'''
scp_process_ = subprocess.run("ssh "+to_user+"#"+to_host+" '"+command+"' ", shell=True, capture_output=False, text=True)
Now running the command
find "/shares/Public/Datensicherung/"* -type f -exec sh -c "stat -c '%y:%n' '{}'" \;
on the server itself works fine without any error.
But as soon I use a subprocess to run it remotely over ssh it has a problem with a file in a folder with spaces: "/shares/Public/Datensicherung/New folder/hi.txt" with a space in it:
stat: can't stat '/shares/Public/Datensicherung/New': No such file or directory
stat: can't stat 'folder/hi.txt': No such file or directory
I know it is messed up, but that is the best solution I could build.
I would like to stick with subprocess and ssh but if you have a better solution feel free to post it.
With shell=True you are invoking three shell instances, each of which requires a layer of quoting. This is possible to do, of course, but there are many reasons to avoid it if at all possible.
First off, you can easily avoid the local shell=True and this actually improves the robustness and clarity of your Python code.
command = "find \""+to_directory+'''\"* -type f -exec sh -c \"stat -c \'%y:%n\' \'{}\'\" \;'''
scp_process_ = subprocess.run(
["ssh", to_user+"#"+to_host, command],
capture_output=False, text=True)
Secondly, stat can easily accept multiple arguments, so you can take out the sh -c '...' too.
command = 'find "' + to_directory + '" -type f -exec stat -c "%y:%n" {} +'
The optimization also switches + for \; (so the sh -c '' wrapper was doubly useless anyway).
Sometimes the issue happening because malformed command string. For purpose of comunication with Unix shell was craeted shlex module. So basically you wrap your code with shlex and then pass it into supbrocess.run.
I don't see the actual final cmd to call but you could split it to proper command with shlex.split by yourself.
From your example it would be something like:
from shlex import join
cmd = join(['ssh',
f'{to_user}#{to_host}',
'find',
f'{to_directory}*',
'-type',
'f',
'-exec',
'sh',
'-c',
"stat -c '%y:%n' '{}'",
';']))
scp_process_ = subprocess.run(cmd, shell=True, capture_output=False, text=True)
Also, you maybe want to play around with shell=True option.
My .profile defines a function
myps () {
ps -aef|egrep "a|b"|egrep -v "c\-"
}
I'd like to execute it from my python script
import subprocess
subprocess.call("ssh user#box \"$(typeset -f); myps\"", shell=True)
Getting an error back
bash: -c: line 0: syntax error near unexpected token `;'
bash: -c: line 0: `; myps'
Escaping ; results in
bash: ;: command not found
script='''
. ~/.profile # load local function definitions so typeset -f can emit them
ssh user#box ksh -s <<EOF
$(typeset -f)
myps
EOF
'''
import subprocess
subprocess.call(['ksh', '-c', script]) # no shell=True
There are a few pertinent items here:
The dotfile defining this function needs to be locally invoked before you run typeset -f to dump the function's definition over the wire. By default, a noninteractive shell does not run the majority of dotfiles (any specified by the ENV environment variable is an exception).
In the given example, this is served by the . ~/profile command within the script.
The shell needs to be one supporting typeset, so it has to be bash or ksh, not sh (as used by script=True by default), which may be provided by ash or dash, lacking this feature.
In the given example, this is served by passing ['ksh', '-c'] is the first two arguments to the argv array.
typeset needs to be run locally, so it can't be in an argv position other than the first with script=True. (To provide an example: subprocess.Popen(['''printf '%s\n' "$#"''', 'This is just literal data!', '$(touch /tmp/this-is-not-executed)'], shell=True) evaluates only printf '%s\n' "$#" as a shell script; This is just literal data! and $(touch /tmp/this-is-not-executed) are passed as literal data, so no file named /tmp/this-is-not-executed is created).
In the given example, this is mooted by not using script=True.
Explicitly invoking ksh -s (or bash -s, as appropriate) ensures that the shell evaluating your function definitions matches the shell you wrote those functions against, rather than passing them to sh -c, as would happen otherwise.
In the given example, this is served by ssh user#box ksh -s inside the script.
I ended up using this.
import subprocess
import sys
import re
HOST = "user#" + box
COMMAND = 'my long command with many many flags in single quotes'
ssh = subprocess.Popen(["ssh", "%s" % HOST, COMMAND],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
result = ssh.stdout.readlines()
The original command was not interpreting the ; before myps properly. Using sh -c fixes that, but... ( please see Charles Duffy comments below ).
Using a combination of single/double quotes sometimes makes the syntax easier to read and less prone to mistakes. With that in mind, a safe way to run the command ( provided the functions in .profile are actually accessible in the shell started by the subprocess.Popen object ):
subprocess.call('ssh user#box "$(typeset -f); myps"', shell=True),
An alternative ( less safe ) method would be to use sh -c for the subshell command:
subprocess.call('ssh user#box "sh -c $(echo typeset -f); myps"', shell=True)
# myps is treated as a command
This seemingly returned the same result:
subprocess.call('ssh user#box "sh -c typeset -f; myps"', shell=True)
There are definitely alternative methods for accomplishing these type of tasks, however, this might give you an idea of what the issue was with the original command.
Hi I have to execute a shell command :diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)
I tried
cmd="diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
However I am getting an error diff: extra operand cat
I am pretty new to python. Any help would be appreciated
You are using <(...) (process substitution) syntax, which is interpreted by the shell. Provide shell=True to Popen to get it to use a shell:
cmd = "diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
output,error = subprocess.Popen(cmd, shell=True, executable="/bin/bash", stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
Since you don't want the Bourne shell (/bin/sh), use the executable argument to determine the shell to use.
You are using a special syntax called process substitiution in your command line. This is supported by most modern shells (bash, zsh), but not by /bin/sh. Therefore, the method suggested by Ned might not work. (It could, if another shell provides /bin/sh and does not "correctly emulate" sh's behaviour, but it is not guaranteed to).
try this instead:
cmd = "diff <(ssh -n root#10.22.254.34 cat /vms/cloudburst.qcow2.*) <(ssh -n root#10.22.254.101 cat /vms/cloudburst.qcow2)"
output,error = subprocess.Popen(['/bin/bash', '-c', cmd], stdout=subprocess.PIPE, stderr=subprocess.PIPE).communicate()
This is basically what the shell=True parameter does, but with /bin/bash instead of /bin/sh (as described in the subprocess docs).