seems like I can't use apostrophe, the command fails with no informative error.
I'm trying to execute the following:
secretLsCmd = subprocess.Popen(('docker', 'secret', 'ls') , stdout=subprocess.PIPE)
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "') , stdout=subprocess.PIPE, stdin=secretLsCmd.stdout)
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2') , stdin=oneWhitespaceCmd.stdout)
in a normal Linux terminal, it would execute the following command:
docker secret ls | tr -s " " | cut -d' ' -f2
Running this command in CLI works fine, but once I put it in python it isn't working. The 2 first commands in the pipe are working fine (i have checked), the last command is not working, exiting with error code 1... I'm almost 100% sure it is the -d' ' , but how can I fix that? any idea?
This line:
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "'), ...)
is actually running:
tr -s '" "'
so you want to lose the extra double quotes there: Python will quote any arguments that it need to quote for the shell.
This command:
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2'), ...)
is equivalent to the shell command:
cut '-d'"'"' '"'"'' -f2
so again, you probably just want to lose the quotes round the whitespace:
onlySecretsCmd = subprocess.check_output(('cut', "-d ", '-f2'), ...)
and leave it to Python to insert quotes where required. This will actually run (what should be equivalent to what you want though not identical):
cut '-d ' -f2
I used shlex.quote() to create a shell equivalent to the commands you are running, though in practice unless you tell subprocess to invoke a shell it will just be executing the equivalent of the command bash would execute after parsing all the escape quote marks. So internally the escaping isn't happening but the quotes to distinguish the arguments aren't needed either.
Related
I have a function that takes in a command as a string, and runs it on an AWS container using the aws binary.
The command adds a few extra symbols before and after the command the user requested to run - I won't get into why.
def run_command_on_aws_container(command: str, aws_info):
full_command = f"bash -c 'echo -n \"|\"; {command}; echo -n \"|\"'"
subprocess.run(["aws", ..., "--command", full_command], ...)
...
command_to_run_on_aws_machine = 'python -c "for i in range(10): print(i)"'
run_command_on_aws_container(command_to_run_on_aws_machine, aws_info)
This works, but only if my command_to_run_on_aws_machine doesn't contain single-quotes. If my command_to_run_on_aws_machine was this for example:
command_to_run_on_aws_machine = "python -c 'for i in range(10): print(i)'"
which is exactly the same command, but using single quotes instead of double quotes, the whole thing crashes. Or at least it doesn't do what you'd expect.
Is there a way to make my run_command_on_aws_container function work with both strings, so that as long as the command arg is a proper bash command, it will run? Ideally not just blindly converting all single quotes to double quotes in a string, but in a way where if the command includes properly escaped quotes, it will still work?
Note: All commands being sent to run_command_on_aws_container as the command arg are hard-coded into the program. There's no security issue here of executing arbitrary commands on a remote system. This is only for convenience, so that code being written outside of the function doesn't need to worry about how to properly use a string.
shlex.quote() is designed specifically for this:
full_command = "bash -c " + shlex.quote(f'echo -n "|"; {command}; echo -n "|"')
My goal is to execute the following bash command in Python and store its output:
echo 'sudo ./run_script.sh -dates \\{\\'2017-11-16\\',\\'2017-11-29\\'\\}'|sed 's;\\\\;\\;'
When I run this command in bash, the output is: sudo ./run_script.sh -dates \{\'2019-10-05\',\'2019-10-04\'\}
My initial idea was to replace the double backslash by a single backslash in Python. As ridiculous as it seems, I couldn't do it in Python (only when using print() the output is as I would like but I can't store the output of print() and str() doesn't convert \ to . So I decided to do it in bash.
import subprocess
t= 'some \\ here'
cmd = "echo \'"+ t+"\'|sed 's;\\\\;\\;'"
ps = subprocess.run(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
ps.stdout
Out[6]: b"sed: -e expression #1, char 7: unterminated `s' command\n"
Running Python 3.6.8 on Ubuntu 18
Try using subprocess.check_output instead. You're also forgetting an extra backslash for every backslash in your command.
import subprocess
command = "echo 'some \\\\here'|sed 's;\\\\\\\\;\\\\;'"
output = subprocess.check_output(command, shell=True).decode()
print(output) # prints your expect result "some \here"
After re-reading your question I kinda understood what you wanted.
a = r'some \here'
print(a) #some \here
Again, raw string literals...
I am trying to integrate a Python script into a bash script. However when I use the input() function, I am getting an EOFError. How can I fix this problem?
#!/bin/bash
python3 <<END
print(input(">>> "))
END
You cannot source both the script and the user input through the program's standard input. (That's in effect what you're trying to do. << redirects the standard input.)
Ideally, you would provide the script as command line argument instead of stdin using -c SCRIPT instead of <<EOF heredoc EOF:
#!/bin/bash
python3 -c 'print(input(">>> "))'
Note that you may need to mind your quoting and escaping in case you have a more complicated Python script with nested quotes.
You can still let the script run over multiple lines, if you need to:
#!/bin/bash
python3 -c '
import os.path
path_name = input("enter a path name >>> ")
file_exists = os.path.exists(path_name)
print("file " + path_name + " " +
("exists" if file_exists else "does not exist"))
'
Note that you will get into trouble when you want to use single quotes in your Python script, as happens when you want to print doesn't instead of does not.
You can work around that using several approaches. The one I consider most flexible (apart from putting you into quoting hell) is surrounding the Python script with double quotes instead and properly escape all inner double quotes and other characters that the shell interprets:
#!/bin/bash
python3 -c "
print(\"It doesn't slice your bread.\")
print('But it can', 'unsliced'[2:7], 'your strings.')
print(\"It's only about \$0. Neat, right?\")
"
Note that I also escaped $, as the shell would otherwise interpret it inside the surrounding double quotes and the result may not be what you wanted.
I am generating a command that comprises multi nested quotes and then execute:
cmd = "python datax.py -p \"\"\" -Dpsql=\"DELETE FROM xxx WHERE date(yyy) = \'2019-02-02\'\" \"\"\" "
os.system(cmd)
The cmd can be generated as expected:
python datax.py -p """ -Dpsql="DELETE FROM xxx WHERE date(yyy) = '2019-02-02'" """
But it seems this does not work well. System still returns exceptions like '(' unexpected... etc.
I also tried '''...''' and doesn't work either. How to address this issue and is there any other way to deal with this case? Any help is appreciated.
You need to add an escaped backslash \\ for each nesting level, because your shell most likely does not support triple quotes. Therefore it parses The first two off your triple quotes and " -Dpsql=" as quoted strings and
DELETE FROM xxx WHERE date(yyy) = '2019-02-02' as shell code. But date(yyy) is not valid shell syntax
An example to reproduce this without having your datax script:
cmd = "python -c \"print(\"Hello (world!)\")\""
os.system(cmd)
fails with
syntax error near unexpected token `('
But
cmd = "python -c \"print(\\\"Hello (world!)\\\")\""
os.system(cmd)
outputs
Hello (world!)
I suggest to change your code to:
cmd = "python datax.py -p \" -Dpsql=\\\"DELETE FROM xxx WHERE date(yyy) = \'2019-02-02\'\\\"\" "
Now your command looks like:
python datax.py -p " -Dpsql=\"DELETE FROM xxx WHERE date(yyy) = '2019-02-02'\""
You see, there is an outer quotation level for your -p argument and an inner quotation level for the sql code.
If you print cmd you get:
python datax.py -p """ -Dpsql="DELETE FROM xxx WHERE date(yyy) = '2019-02-02'" """
What os.system does is just open a shell with a basic .sh interpreter and runs that. Stuff like """ are big no-nos for sh (mainly because you're not sure what it does - something like an empty string with another start of one) and of course, the parenthesis are also bad for sh,
You have to escape or make sure they are in a string. You are going to have to escape everything or make sure a proper sh string is passed. This is one way:
cmd = r"""python datax.py -p ' -Dpsql=\" DELETE FROM xxx WHERE date(yyy) = \\'2019-02-02\\' \" ' """
r"" conserves backslashes as is. Printing this results in:
python datax.py -p ' -Dpsql=\" DELETE FROM xxx WHERE date(yyy) = 2019-02-02 \" '
A better option I think would be to just use:
cmd = r"""python datax.py -p -Dpsql=" DELETE FROM xxx WHERE date(yyy) = '2019-02-02' " """
So I dropped one level of '. Now just fix datax.py to parse argv to better suite your needs (take everything after the -p here). This would make it easier to call it directly as well. When printed:
python datax.py -p -Dpsql=" DELETE FROM xxx WHERE date(yyy) = '2019-02-02' "
argparse handles these things well.
I use libtmux and libtmux uses subprocess.Popen() to invoke Tmux to control session, windows and panes. To set a pane titel Tmux requires to send an escape sequens. This works on command line as expected:
tmux send-keys -t 0 " printf '\033]2;%s\033\\' 'Pane Title'" C-m
When this command is issued by subprocess.Popen() or os.system() - may be others too, I've only tested these two - the escape squence does not make it to the shell. I logg the command sent to a file and the values in the log file are correct but when sent only this survives:
printf '2;%s' 'Pane Title'
I've tested this by executing this command:
echo "printf '\033]2;%s\033\\' 'Pane Title'" > /tmp/setname
The content of /tmp/setname is the above.
The methode finally used is tmux_cmd and I call it via send_keys like this:
p.send_keys("printf '\033]2;%s\033\\' '" + names[i] + "'")
where "p" is the pane object. See my post here.
My question is: How to issue shell commands with escape sequences in Python?
"raw prefixes" was what I've missed! Thanks Jean-François Fabre!
This works:
p.send_keys(r"printf '\033]2;%s\033\\' '" + names[i] + "'")