We Keep Coding

Python subprocess.run ignores --exclude clause

I have one issue with subprocess.run.
This command in a Bash shell works without any problem:
tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1
But if I execute it through Python:
cmd = "tar -C '/home/' --exclude={'/home/user1/.cache','/home/user1/.config'} -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)
The execution works without errors but the --exclude clause is not considered.
Why?

Whether or not curly brace expansion is handled correctly depends on what the standard system shell is. By default, subprocess.run() invokes /bin/sh. On systems like Linux, /bin/sh is bash. On others, such as FreeBSD, it's a different shell that doesn't support brace expansion.
To ensure the subprocess runs with a shell that can handle braces properly, you can tell subprocess.run() what shell to use with the executable argument:
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE, executable='/bin/bash')
As a simple example of this, here's a system where /bin/sh is bash:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo=a foo=b
and one where it's not:
>>> subprocess.run("echo foo={a,b}", shell=True)
foo={a,b}
but specifying another shell works:
>>> subprocess.run("echo foo={a,b}", shell=True, executable='/usr/pkg/bin/bash')
foo=a foo=b

Bash curly expansion doesn't work inside Python and will be sent by subprocess as they are - they will not be expanded, regardless of the arguments you use on run().
Edit: unless of course the argument executable='/bin/bash' as stated on the other answer which seems to work after all
In a bash shell,
--exclude {'/home/user1/.cache','/home/user1/.config'}
becomes:
--exclude=/home/user1/.cache --exclude=/home/user1/.config
So to achieve the same result, in Python it must be expressed like this (one of the possible ways) before sending the command string to subprocess.run:
' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']])
cmd = "tar -C '/home/' " + ' '.join(["--exclude=" + path for path in ['/home/user1/.cache','/home/user1/.config']]) + " -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
print(cmd) # output: "tar -C '/home/' --exclude=/home/user1/.cache --exclude=/home/user1/.config -caf '/transito/user1.tar' '/home/user1' > /dev/null 2>&1"
subprocess.run(cmd, shell=True, stdout=subprocess.PIPE)

for loop in `Subprocess.run` results in `Syntax error: "do" unexpected`

I'm trying to run a for loop in a shell through python. os.popen runs it fine, but is deprecated on 3.x and I want the stderr. Following the highest-voted answer on How to use for loop in Subprocess.run command results in Syntax error: "do" unexpected, with which shellcheck concurs:
import subprocess
proc = subprocess.run(
"bash for i in {1..3}; do echo ${i}; done",
shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE, )
print(proc.stderr)
I'm ultimately trying to reset all usbs by calling this shell code https://unix.stackexchange.com/a/611305/362437 through python, so any alternate approaches to doing that would be appreciated too.

When you do
subprocess.run('foo', shell=True)
it actually runs the equivalent of
/bin/sh -c 'foo'
(except that it magically gets all quotes right :-) ). So, in your case, it executes
/bin/sh -c "bash for i in {1..3}; do echo ${i}; done"
So the "command" given with the -c switch is actually a list of three commands: bash for i in {1..3}, do echo ${i}, and done. This is going to leave you with a very confused shell.
The easiest way of fixing this is probably to remove that bash from the beginning of the string. That way, the command passed to /bin/sh makes some sense.
If you want to run bash explicitly, you're probably better off using shell=False and using a list for the first argument to preserve your quoting sanity. Something like
import subprocess
proc = subprocess.run(
['/bin/bash', '-c', 'for i in {1..3}; do echo ${i}; done'],
shell=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE, )

How to pass this tar command as argument to python subprocess

If I am running the command that is specified in the args on the terminal then it goes successfully on terminal but doing the same in python program is not working; I am seeing junk characters in the screen to the size of the input tar file and lot of xterm words too;
I feel the problem is handling the ' ' letters in the args;
import subprocess
try:
args = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz".split()
subprocess.check_call(args)
except subprocess.CalledProcessError as e:
print e

I am not specialist, but this i found - this commands not working in sh, but working in bash:
$ sh -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz'
$
$ bash -c "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
$
Thats a reason why it not work in subprocess directly. This code looks work fine:
import subprocess
command = "cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz"
subprocess.Popen(command, shell=True, executable='/bin/bash')

I tried a number of alternatives, and none of them were satisfactory. The best I found was switching over to Popen.
# this should have the a similar signature to check_call
def run_in_shell(*args):
# unfortunately, `args` won't be escaped as it is actually a string argument to bash.
proc = subprocess.Popen(['/bin/bash', '-c', ' '.join(args)])
# This will also work, though I have found users who had problems with it.
# proc = subprocess.Popen(' '.join(args), shell=True, executable='/bin/bash')
stat = proc.wait()
if stat != 0:
subprocess.CalledProcessError(returncode=stat, cmd=command)
return stat
run_in_shell("cat parsing.tgz <(echo -n ''| gzip)> new-file.tgz")
As a note: /bin/sh has problems with the unescaped parentheses. If you don't want to specify '/bin/bash' above, then you will need to escape the paren:
args = 'cat parsing.tgz <\\(echo -n ''| gzip\\)> new-file.tgz'

Python3 Run Alias Bash Commands

I have the following code that works great to run the ls command. I have a bash alias that I use alias ll='ls -alFGh' is it possible to get python to run the bash command without python loading my bash_alias file, parsing, and then actually running the full command?
import subprocess
command = "ls" # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)
#Launch the shell command:
output = process.communicate()
print (output[0])
Trying with command = "ll" the output I get is:
/bin/sh: ll: command not found
b''

You cannot. When you run a python process it has no knowledge of a shell alias. There are simple ways of passing text from parent to child process (other than IPC), the command-line and through environment (i.e. exported) variables. Bash does not support exporting aliases.
From the man bash pages: For almost every purpose, aliases are superseded by shell functions.
Bash does support exporting functions, so I suggest you make your alias a simple function instead. That way it is exported from shell to python to shell. For example:
In the shell:
ll() { ls -l; }
export -f ll
In python:
import subprocess
command = "ll" # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)
output = process.communicate()
print(output[0].decode()) # Required if using Python 3
Since you are using the print() function I have assumed you are using python 3. In which case you need the .decode(), since a bytes object is returned.
With a bit of hackery it is possible to create and export shell functions from python as well.

how to integrate "at" command in python

I need to integrate " echo /bin/meteo | at 23:00 today " in to a python script.
In the python script the command "at 23:00 today" should call the bash script /bin/meteo
I did install plumbum and intergrated this in my python scrip.
from plumbum.cmd import echo, grep
Unfortunately I have no clue how to proceed from here.
I tryed:
#!/usr/bin/python2.7
if pfd.input_pins[0].value ==0:
cmd = "echo /bin/meteo | at 06:36 today"
subprocess.Popen(cmd, shell=True)
but the lights in /bin/meteo are randomly swiching on and off (not blinking as they should)
They do it from 06:36 until 06:37 and not only 5 times.
/bin/meteo:
#!/bin/bash -x
for i in {1..5}; do #blink 5x
echo -n -e "\x37\x00\x55" | nc -u -q 1 192.168.0.6 8899 #Zone 3 on
sleep 0.1
echo -n -e "\x3A\x00\x55" | nc -u -q 1 192.168.0.6 8899 #Zone 3 off
done
sleep 0.1
exit

subprocess.Popen will run the command:
import subprocess
cmd = "echo /bin/meteo | at 23:00 today "
subprocess.Popen(cmd, shell=True)
Execute a child program in a new process. On Unix, the class uses os.execvp()-like behavior to execute the child program. On Windows, the class uses the Windows CreateProcess() function. The arguments to Popen are as follows.
args should be a sequence of program arguments or else a single string. By default, the program to execute is the first item in args if args is a sequence. If args is a string, the interpretation is platform-dependent and described below. See the shell and executable arguments for additional differences from the default behavior. Unless otherwise stated, it is recommended to pass args as a sequence.
It is not totally clear what you want but you can run any commands like:
In [9]: cmd = "date"
In [10]: subprocess.call(cmd, shell=True)
Sun Jul 6 22:30:47 IST 2014
Or using sudo:
import subprocess
cmd = "sudo which python"
my_pass="xxxx"
subprocess.call('echo {} | sudo -S {}'.format(my_pass,cmd), shell=True)
In [29]: subprocess.call('echo {} | sudo -S {}'.format(my_pass,cmd), shell=True)
/usr/local/bin/python
Out[29]: 0

With Python 3.4, it's easy to call a command and exchange input/output in a bulk:
subprocess.check_output(["at", "23:00", "today"], input="/bin/meteo")
Therefore in this very case, shell=True shouldn't be needed as we just call the at command with arguments and give it the script on input.
With older versions of python, this needs to be rewritten as:
process = subprocess.Popen(["at", "23:00", "today"])
process.communicate(input="/bin/meteo")
With the plumbum module, you could instead use:
from plumbum.cmd import at, echo
(echo["/bin/meteo"] | at["23:30", "today"])()
But I don't believe that it's very useful.