I have a function that takes in a command as a string, and runs it on an AWS container using the aws binary.
The command adds a few extra symbols before and after the command the user requested to run - I won't get into why.
def run_command_on_aws_container(command: str, aws_info):
full_command = f"bash -c 'echo -n \"|\"; {command}; echo -n \"|\"'"
subprocess.run(["aws", ..., "--command", full_command], ...)
...
command_to_run_on_aws_machine = 'python -c "for i in range(10): print(i)"'
run_command_on_aws_container(command_to_run_on_aws_machine, aws_info)
This works, but only if my command_to_run_on_aws_machine doesn't contain single-quotes. If my command_to_run_on_aws_machine was this for example:
command_to_run_on_aws_machine = "python -c 'for i in range(10): print(i)'"
which is exactly the same command, but using single quotes instead of double quotes, the whole thing crashes. Or at least it doesn't do what you'd expect.
Is there a way to make my run_command_on_aws_container function work with both strings, so that as long as the command arg is a proper bash command, it will run? Ideally not just blindly converting all single quotes to double quotes in a string, but in a way where if the command includes properly escaped quotes, it will still work?
Note: All commands being sent to run_command_on_aws_container as the command arg are hard-coded into the program. There's no security issue here of executing arbitrary commands on a remote system. This is only for convenience, so that code being written outside of the function doesn't need to worry about how to properly use a string.
shlex.quote() is designed specifically for this:
full_command = "bash -c " + shlex.quote(f'echo -n "|"; {command}; echo -n "|"')
Related
I'm passing the result of the execution of a command to python as input, like so:
$ python parse_ips.py "$(kubectl get configmap ...)"
This works fine when executing from the command line, however I'm now trying to edit the file using PyCharm. Therefore I need the escaped version of the result of this command which I can paste into PyCharm's debug configuration, as I can't execute the command in real-time like I can do on the command line.
However, I am struggling to find a way to replicate the escaping bash does behind the scenes, so I can use the result as an argument within the PyCharm configuration. Running the above kubectl command results in a multi-line string which includes spaces and quotes. When I paste this into PyCharm it just interprets it as multiple arguments. I'm looking for the escaped result, which I could paste directly into the command line, or into PyCharm's debug configuration, to achieve the same result with a fixed parameter for testing.
Any help would be greatly appreciated!
Edit: To clarify, I mean on the command line the result of the $(kubectl ...) command is passed into the python program as a single command line argument when it is surrounded by quotes ("$(kubectl ...)"). So in the python program, you can access sys.argv[1] and it will contain the entire execution output of $(kubectl get configmap ...). However, if I execute that command myself on the command line, the result is a multi-line string.
If I then copy the result of that into PyCharm (or even on the command line again), it is interpreted as many command line arguments. E.g. it would look something like this:
$ python parse_ips.py apiVersion: v1
data:
item1: ifconfig-push 127.0.0.0 255.255.0.0
item2: ifconfig-push 127.0.0.1 255.255.0.0
item3: ifconfig-push 127.0.0.2 255.255.0.0
...
And so on. This obviously doesn't work in the same way as it did before. So I am unable to test my program without making the kubectl call from the command line each time. I was looking to replicate what "$(kubectl ...)" gets converted into so it is able to pass the entire output as a single command line entry.
I am struggling to find a way to replicate the escaping bash does behind the scenes
Typically use printf "%q" to escape stuff.
printf "%q" "$(kubectl get configmap ....)"
This is printf as the bash builtin command. It differs from coreutils printf, and newest ones also support %q with different quoting style:
/usr/bin/printf "%q" "$(kubectl get configmap ....)"
Modern bash also has quoting expansion:
var="$(kubectl get configmap ....)"
echo "${var#Q}"
And there is also the quoting style outputted by set -x.
I would suggest to use a file:
kubectl get configmap ... > /tmp/tempfile
python parse_ips.py "$(cat /tmp/tempfile)"
With xclip you can copy command output straight to the X server clipboard, which is handy:
printf "%q" "$(kubectl get configmap ...)" | xclip -selection clipboard
# then in another window:
python parse_ips.py <right mouse click><select paste>
How do I approach converting this:
echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -
into sh (https://github.com/amoffat/sh) syntax.
The issue I have is that the args are tokenized and quoted. So docker isn't able to pass stdin onto the end of the command since - has single quotes placed around it. Digging in the docs I'm not seeing a way to support this.
Thanks for any help you can provide (alternatively if you have good examples of doing this in subprocess, that would work.)!
The issue I have is that the args are tokenized and quoted. So docker isn't able to pass stdin onto the end of the command since - has single quotes placed around it.
Like I commented,
- isn’t special shell syntax. It should work fine as a normal argument.
Still, here’s how it looks like you do it, going by the docs. Using _in,
dockerfile = r"""
FROM busybox
RUN echo "hello world"
"""
docker.build("-", _in=dockerfile)
I am trying to integrate a Python script into a bash script. However when I use the input() function, I am getting an EOFError. How can I fix this problem?
#!/bin/bash
python3 <<END
print(input(">>> "))
END
You cannot source both the script and the user input through the program's standard input. (That's in effect what you're trying to do. << redirects the standard input.)
Ideally, you would provide the script as command line argument instead of stdin using -c SCRIPT instead of <<EOF heredoc EOF:
#!/bin/bash
python3 -c 'print(input(">>> "))'
Note that you may need to mind your quoting and escaping in case you have a more complicated Python script with nested quotes.
You can still let the script run over multiple lines, if you need to:
#!/bin/bash
python3 -c '
import os.path
path_name = input("enter a path name >>> ")
file_exists = os.path.exists(path_name)
print("file " + path_name + " " +
("exists" if file_exists else "does not exist"))
'
Note that you will get into trouble when you want to use single quotes in your Python script, as happens when you want to print doesn't instead of does not.
You can work around that using several approaches. The one I consider most flexible (apart from putting you into quoting hell) is surrounding the Python script with double quotes instead and properly escape all inner double quotes and other characters that the shell interprets:
#!/bin/bash
python3 -c "
print(\"It doesn't slice your bread.\")
print('But it can', 'unsliced'[2:7], 'your strings.')
print(\"It's only about \$0. Neat, right?\")
"
Note that I also escaped $, as the shell would otherwise interpret it inside the surrounding double quotes and the result may not be what you wanted.
I am trying to get output of subprocess.Popen in variable.
It is working fine for pwd command, but not working for pwdx $(pgrep -U $USER -f SimpleHTTPServer) command.
This works:
(Pdb++) p = subprocess.Popen("pwd", stdout=subprocess.PIPE)
(Pdb++) result = p.communicate()[0]
(Pdb++) result
'xyz'
This is not working:
(Pdb++) subprocess.Popen("pwdx $(pgrep -U $USER -f SimpleHTTPServer)", stdout=subprocess.PIPE)
*** OSError: [Errno 2] No such file or directory
Can someone please let me know how can I save the output of it to a variable?
If you want to pass a command with arguments to Popen(), you have to pass it as a list, like so:
subprocess.Popen(['/bin/ls', '-lat'])
If you just pass a single string as in your example, it assumes the entire thing is the command name, and obviously there is no command literally named pwdx $(pgrep -U $USER -f SimpleHTTPServer).
As the docs and previous answers already state the command you want to execute via subprocess.Popen() needs to be passed as a list.
From the docs:
Note in particular that options (such as -input) and arguments (such
as eggs.txt) that are separated by whitespace in the shell go in
separate list elements, while arguments that need quoting or backslash
escaping when used in the shell (such as filenames containing spaces) are single list elements.
Another useful tip you can get from the docs is to use shlex.split() to help you to properly split your command into a list.
Beware though that the use of special shell parameters (e.g. $USER) might not work well with subprocess.Popen() unless you would set the shell=True option, which you shouldn't do without reading the doc's Security Considerations.
seems like I can't use apostrophe, the command fails with no informative error.
I'm trying to execute the following:
secretLsCmd = subprocess.Popen(('docker', 'secret', 'ls') , stdout=subprocess.PIPE)
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "') , stdout=subprocess.PIPE, stdin=secretLsCmd.stdout)
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2') , stdin=oneWhitespaceCmd.stdout)
in a normal Linux terminal, it would execute the following command:
docker secret ls | tr -s " " | cut -d' ' -f2
Running this command in CLI works fine, but once I put it in python it isn't working. The 2 first commands in the pipe are working fine (i have checked), the last command is not working, exiting with error code 1... I'm almost 100% sure it is the -d' ' , but how can I fix that? any idea?
This line:
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "'), ...)
is actually running:
tr -s '" "'
so you want to lose the extra double quotes there: Python will quote any arguments that it need to quote for the shell.
This command:
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2'), ...)
is equivalent to the shell command:
cut '-d'"'"' '"'"'' -f2
so again, you probably just want to lose the quotes round the whitespace:
onlySecretsCmd = subprocess.check_output(('cut', "-d ", '-f2'), ...)
and leave it to Python to insert quotes where required. This will actually run (what should be equivalent to what you want though not identical):
cut '-d ' -f2
I used shlex.quote() to create a shell equivalent to the commands you are running, though in practice unless you tell subprocess to invoke a shell it will just be executing the equivalent of the command bash would execute after parsing all the escape quote marks. So internally the escaping isn't happening but the quotes to distinguish the arguments aren't needed either.