We Keep Coding

What's the best way to escape a bash command in python?

I have a function that takes in a command as a string, and runs it on an AWS container using the aws binary.
The command adds a few extra symbols before and after the command the user requested to run - I won't get into why.
def run_command_on_aws_container(command: str, aws_info):
full_command = f"bash -c 'echo -n \"|\"; {command}; echo -n \"|\"'"
subprocess.run(["aws", ..., "--command", full_command], ...)
...
command_to_run_on_aws_machine = 'python -c "for i in range(10): print(i)"'
run_command_on_aws_container(command_to_run_on_aws_machine, aws_info)
This works, but only if my command_to_run_on_aws_machine doesn't contain single-quotes. If my command_to_run_on_aws_machine was this for example:
command_to_run_on_aws_machine = "python -c 'for i in range(10): print(i)'"
which is exactly the same command, but using single quotes instead of double quotes, the whole thing crashes. Or at least it doesn't do what you'd expect.
Is there a way to make my run_command_on_aws_container function work with both strings, so that as long as the command arg is a proper bash command, it will run? Ideally not just blindly converting all single quotes to double quotes in a string, but in a way where if the command includes properly escaped quotes, it will still work?
Note: All commands being sent to run_command_on_aws_container as the command arg are hard-coded into the program. There's no security issue here of executing arbitrary commands on a remote system. This is only for convenience, so that code being written outside of the function doesn't need to worry about how to properly use a string.

shlex.quote() is designed specifically for this:
full_command = "bash -c " + shlex.quote(f'echo -n "|"; {command}; echo -n "|"')

How to use input() function of Python in bash script?

I am trying to integrate a Python script into a bash script. However when I use the input() function, I am getting an EOFError. How can I fix this problem?
#!/bin/bash
python3 <<END
print(input(">>> "))
END

You cannot source both the script and the user input through the program's standard input. (That's in effect what you're trying to do. << redirects the standard input.)
Ideally, you would provide the script as command line argument instead of stdin using -c SCRIPT instead of <<EOF heredoc EOF:
#!/bin/bash
python3 -c 'print(input(">>> "))'
Note that you may need to mind your quoting and escaping in case you have a more complicated Python script with nested quotes.
You can still let the script run over multiple lines, if you need to:
#!/bin/bash
python3 -c '
import os.path
path_name = input("enter a path name >>> ")
file_exists = os.path.exists(path_name)
print("file " + path_name + " " +
("exists" if file_exists else "does not exist"))
'
Note that you will get into trouble when you want to use single quotes in your Python script, as happens when you want to print doesn't instead of does not.
You can work around that using several approaches. The one I consider most flexible (apart from putting you into quoting hell) is surrounding the Python script with double quotes instead and properly escape all inner double quotes and other characters that the shell interprets:
#!/bin/bash
python3 -c "
print(\"It doesn't slice your bread.\")
print('But it can', 'unsliced'[2:7], 'your strings.')
print(\"It's only about \$0. Neat, right?\")
"
Note that I also escaped $, as the shell would otherwise interpret it inside the surrounding double quotes and the result may not be what you wanted.

using apostrophe with python's subprocess

seems like I can't use apostrophe, the command fails with no informative error.
I'm trying to execute the following:
secretLsCmd = subprocess.Popen(('docker', 'secret', 'ls') , stdout=subprocess.PIPE)
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "') , stdout=subprocess.PIPE, stdin=secretLsCmd.stdout)
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2') , stdin=oneWhitespaceCmd.stdout)
in a normal Linux terminal, it would execute the following command:
docker secret ls | tr -s " " | cut -d' ' -f2
Running this command in CLI works fine, but once I put it in python it isn't working. The 2 first commands in the pipe are working fine (i have checked), the last command is not working, exiting with error code 1... I'm almost 100% sure it is the -d' ' , but how can I fix that? any idea?

This line:
oneWhitespaceCmd = subprocess.Popen(('tr', '-s','" "'), ...)
is actually running:
tr -s '" "'
so you want to lose the extra double quotes there: Python will quote any arguments that it need to quote for the shell.
This command:
onlySecretsCmd = subprocess.check_output(('cut', "-d' '", '-f2'), ...)
is equivalent to the shell command:
cut '-d'"'"' '"'"'' -f2
so again, you probably just want to lose the quotes round the whitespace:
onlySecretsCmd = subprocess.check_output(('cut', "-d ", '-f2'), ...)
and leave it to Python to insert quotes where required. This will actually run (what should be equivalent to what you want though not identical):
cut '-d ' -f2
I used shlex.quote() to create a shell equivalent to the commands you are running, though in practice unless you tell subprocess to invoke a shell it will just be executing the equivalent of the command bash would execute after parsing all the escape quote marks. So internally the escaping isn't happening but the quotes to distinguish the arguments aren't needed either.

Writing python scripts directly on the command line

When using bash shell commands it would sometimes be usefull to pipe in python and write a short program and then maybe pipe that into something else. Im not finding a lot of documentation about writing python programs like this although it looks like the "-c" option is the option to use..but when writing even the simplest python program the compiler or should i say interpreter complains. See example below:
$ python -c "
import os
if os.path.isfile("test"):
print "test is a file"
else:
print "test is not a file"
"
When entering the last " the interpretor complains. This runs fine if i put it in a file but if i type it like that on the command line i get errors.
$ python -c "
import os
if os.path.isfile("test"):
print "test is a file"
else:
print "test is not a file"
"
Traceback (most recent call last):
File "<string>", line 4, in <module>
NameError: name 'test' is not defined
I have no idea why the interpretor is complaining here. Does someone know why this isnt working ?
What im really after is something like this:
$ cat somefile | python -c "
import re
check = re.search(pattern, <file input>)
"
I dont know how to access the output of cat in this situation so i just wrote it literally.

You are using double quotes inside double quotes which is ending the quoted string you are passing to python, in a place where you don't expect. Try replacing the outer quotes with single quotes, like I did here:
python -c '
import os
if os.path.isfile("test"):
print "test is a file"
else:
print "test is not a file"
'
If you are using single quotes to terminate the string you are passing to python, make sure to only use double quotes in your code. Additionally, if you can guarantee the availability of Bash as your shell, you can gain added awesome points by using heredoc format instead:
$ python <<EOF
> print "I can put python code here"
> EOF
I can put python code here

Another solution is to escape your inner double quotes so bash doesn't parse them. Like this:
$ python -c "
import os
if os.path.isfile(\"test\"):
print \"test is a file\"
else:
print \"test is not a file\"
"

Either use single quotes to enclose your short program or, if you want to use double quotes to enclose it, escape the quotes with \.
Examples:
1. Escaping quotes
$ python -c "
print \"hello\"
for i in (1,2,3):
print i
"
Output:
hello
1
2
3
2. With single quotes
$ python -c '
print "hello"
for i in (1,2,3):
print i
'
Of course, if you use single quotes to enclose your program and you want to use single quotes inside your python code, you'll have to escape them with \ ;-).
The output is the same.

You can use what is commonly called a "here document" (as in "use the document that is right here"). This avoids all quoting problems when using python -c "..." or python -c '...'
For example:
#!/bin/sh
python <<EOF
print "hello"
for i in (1,2,3):
print i
EOF
The "here document" takes an arbitrary marker ("EOF" is a common choice, but it can be any string you know doesn't occur anywhere else in the data), and accepts all data up unto it finds a line that contains that marker.

Why, does it seem, that I need to use two double quotes to delimit file paths?

I'm writing a script in python that runs a program. Spaces in the path or arguments need to be quoted, but for some reason it won't seem to work unless I use double quoted strings e.g.
DEFAULT_CLIENT = '""C:/Program Files/Borland/StarTeam SDK 14.0/bin""'
command =('stjava.exe -autologon -is -p ""'+src+'"" -fp ""'+dst+'"" '
'-filter "OI" "*" -f ""'+dst+'/config/bc.stjava""')
if not "StarTeam Cross-Platform Client" in os.getenv("PATH"):
print "[INFO] Star Team Client not found in path defaulting to: "+DEFAULT_CLIENT
command = DEFAULT_CLIENT+"/"+command
os.system(command)
Why is this? If I am doing something wrong here then how should I be doing it?
[Update] When calling os.system(command) This finds the application correctly:
command = r'"C:/Program Files/Borland/StarTeam SDK 14.0/bin/stjava.exe" '
command += '-autologon -is -p '
But this doesn't:
command = '"C:/Program Files/Borland/StarTeam SDK 14.0/bin/stjava.exe" '
command += '-autologon -is -p '
command += '"C:\a\b c\d"'
It says 'C:\Program' is not a recognised command. The paths in the arguments seems to be an issue, but I don't know why.

As per my understanding you need following line as command
'"C:/Program Files/Borland/StarTeam SDK 14.0/bin/stjava.exe" -autologon -is -p "abc def" -fp "aa bb" -filter "OI" "*" -f "aa bb/config/bc.stjava"'
which you can achieve by following:
>>> import os
>>>
>>> DEFAULT_CLIENT = '"C:/Program Files/Borland/StarTeam SDK 14.0/bin"'
>>>
>>> DEFAULT_CLIENT
'"C:/Program Files/Borland/StarTeam SDK 14.0/bin"'
>>>
>>> src = "abc def"
>>> dst = "aa bb"
>>>
>>>
>>> command =('stjava.exe" -autologon -is -p "'+src+'" -fp "'+dst+'" -filter "OI" "*" -f "'+dst+'/config/bc.stjava"')
>>>
>>>
>>>
>>> DEFAULT_CLIENT[:-1] + os.sep + command
'"C:/Program Files/Borland/StarTeam SDK 14.0/bin/stjava.exe" -autologon -is -p "abc def" -fp "aa bb" -filter "OI" "*" -f "aa bb/config/bc.stjava"'
>>>
>>>

I am going to go out on a limb and say that, for some reason, the command-line parser you are using or the Windows or DOS command line interpreter needs quoted file names to have "escaped quotation marks". I tried various combinations with the DOS DIR command with zero, one, and two double quotes and it worked every time.

Initial answer:
You want C:/Program Files/Borland/StarTeam SDK 14.0/bin/stjava.exe to be your full binary path, right ?
As the path has spaces in it, quotes are necessary.
Using single quotes, after concatenation it will be something like: "C:/Program Files/Borland/StarTeam SDK 14.0/bin"stjava.exe.
Have you tried to use proper file-path separators ?
As you added some more information, I have another suggestion:
I see that the last parameter in your command string contains backslash characters. Those proved to be problematic in many situations. Can you try to escape those, or double them, into something like "C:\\a\\b c\\d"?