We Keep Coding

How to make YAML-embedded shell script safe against problems with different quotes?

I have some app which gets configured using a YAML file. That app does some processing and supports hooks to execute shell code before and after the processing. This is mostly meant to execute external script files doing the real business, but one can e.g. export environment variables as well. It seems like things are simply forwarded to a shell call with the configured string.
The important thing to note is that one hook is specially called in case ANYTHING goes wrong during processing. In that case the app provides some additional error details to the configured shell script. This is done by reading the necessary part of the YAML config and doing a simple string replacement of special keywords to what is actually available on runtime. Those keywords follow the syntax {...}. Things look like the following in my config:
on_error:
- |
export BGM_ERR_VAR_CONFIG_PATH="{configuration_filename}"
export BGM_ERR_VAR_REPO="{repository}"
export BGM_ERR_VAR_ERROR_MSG="{error}"
export BGM_ERR_VAR_ERROR_OUT="{output}"
'/path/to/script.sh 'some_arg' '[...]' [...]
Originally those keywords were expected to be forwarded as arguments in the called script, but my script needs some other arguments already, so I decided to forward things using environment variables. Shouldn't make too much of a difference regarding my problem, though.
That problem is that really ANYTHING can got wrong and especially the placeholder {output} can contain arbitrary complex error messages. It's most likely a mixture of executed shell commands, using single quotes in most cases, and stacktraces of the programming language the app is implemented in, using double quotes. With my config above this leads to invalid shell code being executed in the end:
[2021-10-12 07:18:46,073] ERROR: /bin/sh: 13: Syntax error: Unterminated quoted string
The following is what the app logs as being executed at all:
[2021-10-12 07:18:46,070] DEBUG: export BGM_ERR_VAR_CONFIG_PATH="/path/to/some.yaml"
export BGM_ERR_VAR_REPO="HOST:PARENT/CHILD"
export BGM_ERR_VAR_ERROR_MSG="Command 'borg check --prefix arch- --debug --show-rc --umask 0007 HOST:PARENT/CHILD' returned non-zero exit status 2."
export BGM_ERR_VAR_ERROR_OUT="using builtin fallback logging configuration
35 self tests completed in 0.04 seconds
SSH command line: ['ssh', '-F', '/[...]/.ssh/config', 'HOST', 'borg', 'serve', '--umask=007', '--debug']
RemoteRepository: 169 B bytes sent, 66 B bytes received, 3 messages sent
Connection closed by remote host
Traceback (most recent call last):
File "borg/archiver.py", line 177, in wrapper"
'/path/to/script.sh '[...]' '[...]' '[...]' '[...]'
The args to my own script are safe regarding quoting, those are only hard-coded paths, keywords etc., nothing dynamic in any way. The problem should be the double quotes used for the path to the python file throwing the exception. OTOH, if I only use single quotes with my environment variables, those would break because the output shell command invoked uses single quotes as well.
So, how do I implement a safe forwarding of {output} into the environment variable in this context?
I thought of using some subshell ="$(...)" and sed to normalize quotes, but everything I came up with resulted in a command line with exactly the same quoting problems like before. Same goes for printf and its %q to escape quotes. It seems I need something which is able to deal with arbitrary individual arguments and joining those to some string again or something like that. Additionally, things should not be too complex to not bloat the YAML config in the end.
The following might work, but loses the double quotes:
export BGM_ERR_VAR_ERROR_OUT="$(echo "{output}")"
How about that?
export BGM_ERR_VAR_ERROR_OUT="$(cat << EOT
{output}
EOT
)"
Anything else? Thanks!

To avoid all the replacement problems, I suggest not using replacements, and forwarding the values as environment variables instead. This assumes you have control over the calling code, which I assume is correct from your explanation.
Since environment variables are by convention uppercase, putting your values in lowercase names is quite safe, and then you can simply do
on_error:
- |
export BGM_ERR_VAR_CONFIG_PATH="$configuration_filename"
export BGM_ERR_VAR_REPO="$repository"
export BGM_ERR_VAR_ERROR_MSG="$error"
export BGM_ERR_VAR_ERROR_OUT="$output"
'/path/to/script.sh 'some_arg' '[...]' [...]
The calling code would need to modify the environment accordingly so that it will contain the expected values. This is the safest way to forward the values, since it guarantees not to interpret the values as bash syntax at all.
If this is not possible, the next best thing is probably to use a heredoc, albeit one with quotes to avoid processing anything in the content – you can use read to avoid the unnecessary cat:
on_error:
- |
read -r -d '' BGM_ERR_VAR_CONFIG_PATH <<'EOF'
{configuration_filename}
EOF
export BGM_ERR_VAR_CONFIG_PATH
# ... snip: other variables ...
'/path/to/script.sh 'some_arg' '[...]' [...]
The only thing you need to be aware of here is that the content may not include a line reading EOF. The calling code needs to ensure this.

python escape markquotes for bash script [duplicate]

When using os.system() it's often necessary to escape filenames and other arguments passed as parameters to commands. How can I do this? Preferably something that would work on multiple operating systems/shells but in particular for bash.
I'm currently doing the following, but am sure there must be a library function for this, or at least a more elegant/robust/efficient option:
def sh_escape(s):
return s.replace("(","\\(").replace(")","\\)").replace(" ","\\ ")
os.system("cat %s | grep something | sort > %s"
% (sh_escape(in_filename),
sh_escape(out_filename)))
Edit: I've accepted the simple answer of using quotes, don't know why I didn't think of that; I guess because I came from Windows where ' and " behave a little differently.
Regarding security, I understand the concern, but, in this case, I'm interested in a quick and easy solution which os.system() provides, and the source of the strings is either not user-generated or at least entered by a trusted user (me).

shlex.quote() does what you want since python 3.
(Use pipes.quote to support both python 2 and python 3,
though note that pipes has been deprecated since 3.10
and slated for removal in 3.13)

This is what I use:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
The shell will always accept a quoted filename and remove the surrounding quotes before passing it to the program in question. Notably, this avoids problems with filenames that contain spaces or any other kind of nasty shell metacharacter.
Update: If you are using Python 3.3 or later, use shlex.quote instead of rolling your own.

Perhaps you have a specific reason for using os.system(). But if not you should probably be using the subprocess module. You can specify the pipes directly and avoid using the shell.
The following is from PEP324:
Replacing shell pipe line
-------------------------
output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Maybe subprocess.list2cmdline is a better shot?

Note that pipes.quote is actually broken in Python 2.5 and Python 3.1 and not safe to use--It doesn't handle zero-length arguments.
>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1 arg3
See Python issue 7476; it has been fixed in Python 2.6 and 3.2 and newer.

I believe that os.system just invokes whatever command shell is configured for the user, so I don't think you can do it in a platform independent way. My command shell could be anything from bash, emacs, ruby, or even quake3. Some of these programs aren't expecting the kind of arguments you are passing to them and even if they did there is no guarantee they do their escaping the same way.

Notice: This is an answer for Python 2.7.x.
According to the source, pipes.quote() is a way to "Reliably quote a string as a single argument for /bin/sh". (Although it is deprecated since version 2.7 and finally exposed publicly in Python 3.3 as the shlex.quote() function.)
On the other hand, subprocess.list2cmdline() is a way to "Translate a sequence of arguments into a command line string, using the same rules as the MS C runtime".
Here we are, the platform independent way of quoting strings for command lines.
import sys
mswindows = (sys.platform == "win32")
if mswindows:
from subprocess import list2cmdline
quote_args = list2cmdline
else:
# POSIX
from pipes import quote
def quote_args(seq):
return ' '.join(quote(arg) for arg in seq)
Usage:
# Quote a single argument
print quote_args(['my argument'])
# Quote multiple arguments
my_args = ['This', 'is', 'my arguments']
print quote_args(my_args)

The function I use is:
def quote_argument(argument):
return '"%s"' % (
argument
.replace('\\', '\\\\')
.replace('"', '\\"')
.replace('$', '\\$')
.replace('`', '\\`')
)
that is: I always enclose the argument in double quotes, and then backslash-quote the only characters special inside double quotes.

On UNIX shells like Bash, you can use shlex.quote in Python 3 to escape special characters that the shell might interpret, like whitespace and the * character:
import os
import shlex
os.system("rm " + shlex.quote(filename))
However, this is not enough for security purposes! You still need to be careful that the command argument is not interpreted in unintended ways. For example, what if the filename is actually a path like ../../etc/passwd? Running os.system("rm " + shlex.quote(filename)) might delete /etc/passwd when you only expected it to delete filenames found in the current directory! The issue here isn't with the shell interpreting special characters, it's that the filename argument isn't interpreted by the rm as a simple filename, it's actually interpreted as a path.
Or what if the valid filename starts with a dash, for example, -f? It's not enough to merely pass the escaped filename, you need to disable options using -- or you need to pass a path that doesn't begin with a dash like ./-f. The issue here isn't with the shell interpreting special characters, it's that the rm command interprets the argument as a filename or a path or an option if it begins with a dash.
Here is a safer implementation:
if os.sep in filename:
raise Exception("Did not expect to find file path separator in file name")
os.system("rm -- " + shlex.quote(filename))

I think these answers are a bad idea for escaping command-line arguments on Windows. Based on the results: people are trying to apply a black-list approach to filtering 'bad' characters, assuming (and hoping) they got them all. Windows is very complex and there could be all manner of characters found in the future that might allow an attacker to hijack command line arguments.
I've already seen some answers neglect to filter basic meta-characters in Windows (like the semi-colon.) The approach I take is far simpler:
Make a list of allowed ASCII characters.
Remove all chars that aren't in that list.
Escape slashes and double-quotes.
Surround entire command with double quotes so the command argument cannot be maliciously broken and commandeered with spaces.
A basic example:
def win_arg_escape(arg, allow_vars=0):
allowed_list = """'"/\\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-. """
if allow_vars:
allowed_list += "~%$"
# Filter out anything that isn't a
# standard character.
buf = ""
for ch in arg:
if ch in allowed_list:
buf += ch
# Escape all slashes.
buf = buf.replace("\\", "\\\\")
# Escape double quotes.
buf = buf.replace('"', '""')
# Surround entire arg with quotes.
# This avoids spaces breaking a command.
buf = '"%s"' % (buf)
return buf
The function has an option to enable use of environmental variables and other shell variables. Enabling this poses more risk so its disabled by default.

Passing double quotation marks to a command from Python using the subprocess module

I have a command line program I'm using from within a Python script to take care of stamping a build number onto an executable.
This is the command line program: http://www.elphin.com/downloads/stampver/
My problem is that the program takes double quotes (") in one of its arguments and the Python subprocess module I'm using to call this program keeps prepending a backslash onto the double quotes when executing the program. This causes the program to fail due to incorrect syntax.
It is expecting a command like: StampVer.exe -nopad -k -f"0.1.4491.0" <ExeFile>.exe
And instead Python is executing: StampVer.exe -nopad -k -f\"0.1.4491.0\" <ExeFile>.exe
I've tried a few things that I've found here on StackOverflow for similar sounding problems such as marking the string as raw or adding backslashes before the quotes in Python; which just results in triple backslashes on the command line instead of one, because Python then tries to escape the backslash as well as the double quote.
Should I be using something other than the subprocess module to accomplish this or do all these types of modules in Python follow the same rules for escaping characters? Is there something I can do to tell subprocess to strip escape characters or to not add them at all?
EDIT
This is how I'm calling subprocess from Python:
def set_file_version(version_number, filepath):
try:
file_version_arg = '-f"{0}"'.format(version_number)
cmd_line = ["StampVer.exe", "-nopad", "-k", file_version_arg, filepath]
subprocess.check_output(cmd_line)
except subprocess.CalledProcessError as e:
if e.returncode == 1:
pass
else:
raise e
StampVer then returns this:
error: Invalid -f parameter. Try -f"0.0.0.0" Use StampVer -? for help

try this script sub.py:
#! /usr/bin/python
import sys
from subprocess import check_output
if len(sys.argv) > 1:
print sys.argv
else:
print check_output((sys.argv[0], '-f"1234"'))
then run it:
./sub.py
it return what we gave:
['./sub.py', '-f"1234"']
So I guess check_output works just fine, the problem may came from how StampVer.exe handle parameter, you can try
file_version_arg = '-f{0}'.format(version_number)

My solution ended up being kind of a cop-out. Despite the documentation for StampVer showing the format above for the version number in all examples, it turns out you can just leave the quotes out all together and even space it out from the -f switch and it will still be accepted.
I'm going to call this my answer but I still think being able to pass quotes through subprocess is a worthwhile problem to figure out. If anyone has an answer that will actually solve the initial problem then please post it and I'll mark it instead.

How to pass escaped string to shell script in Python

I am attempting to create a Python script that in turn runs the shell script "js2coffee" to convert some javascript into coffeescript.
From the command line I can run this, and get coffeescript back again...
echo "var myNumber = 100;" | js2coffee
What I need to do is use this same pattern from Python.
In Python, I've come to something like this:
command = "echo '" + myJavscript + "' | js2coffee"
result = os.popen(command).read()
This works sometimes, but there are issues related to special characters (mostly quotes, I think) not being properly escaped in the myJavascript. There has got to be a standard way of doing this. Any ideas? Thanks!

Use the input stream of a process to feed it the data, that way you can avoid the shell and you don't need to escape your javascript. Additionally, you're not vulnerable to shell injection attacks;
pr = subprocess.Popen(['js2coffee'],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
result, stderrdata = pr.communicate('var myNumber = 100;')

subprocess module is the way to go:
http://docs.python.org/library/subprocess.html#frequently-used-arguments
be kindly noted the following:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names)

grep command called from python

Platform: Windows
Grep: http://gnuwin32.sourceforge.net/packages/grep.htm
Python: 2.7.2
Windows command prompt used to execute the commands.
I am searching for the for the following pattern "2345$" in a file.
Contents of the file are as follows:
abcd 2345
2345
abcd 2345$
grep "2345$" file.txt
grep returns 2 lines (first and second) successfully.
When I try to run the above command through python I don't see any output.
Python code snippet is as follows:
temp = open('file.txt', "r+")
grep_cmd = []
grep_cmd.extend([grep, '"2345$"' ,temp.name])
print grep_cmd
p = subprocess.Popen(grep_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdoutdata = p.communicate()[0]
print stdoutdata
If I have
grep_cmd.extend([grep, '2345$' ,temp.name])
in my python script, I get the correct answer.
The questions is why the grep command with "
grep_cmd.extend([grep, '"2345$"' ,temp.name])
executed from python fails. Isn't python supposed to execute
the command as it is.
Thanks
Gudge.

Do not put double quotes around your pattern. It is only needed on the command line to quote shell metacharacters. When calling a program from python, you do not need this.
You also do not need to open the file yourself - grep will do that:
grep_cmd.extend([grep, '2345$', 'file.txt'])
To understand the reason for the double quotes not being needed and causing your command to fail, you need to understand the purpose of the double quotes and how they are processed.
The shell uses double quotes to prevent special processing of some shell metacharacters. Shell metacharacters are those characters that the shell handles specially and does not pass literally to the programs it executes. The most commonly used shell metacharacter is "space". The shell splits a command on space boundaries to build an argument vector to execute a program with. If you want to include a space in an argument, it must be quoted in some way (single or double quotes, backslash, etc). Another is the dollar sign ($), which is used to signify variable expansion.
When you are executing a program without the shell involved, all these rules about quoting and shell metacharacters are not relevant. In python, you are building the argument vector yourself, so the relevant quoting rules are python quoting rules (e.g. to include a double quote inside a double-quoted string, prefix the double quote with a backslash - the backslash will not be in the final string). The characters in each element of the argument vector when you have completed constructing it are the literal characters that will be passed to the program you are executing.
Grep does not treat double quotes as special characters, so if grep gets double quotes in its search pattern, it will attempt to match double quotes from its input.
My original answer's reference to shell=True was incorrect - first I did not notice that you had originally specified shell=True, and secondly I was coming from the perspective of a Unix/Linux implementation, not Windows.
The python subprocess module page has this to say about shell=True and Windows:
On Windows: the Popen class uses CreateProcess() to execute the child child program, which operates on strings. If args is a sequence, it will be converted to a string in a manner described in Converting an argument sequence to a string on Windows.
That linked section on converting an argument sequence to a string on Windows does not make sense to me. First, a string is a sequence, and so is a list, yet the Frequently Used Arguments section says this about arguments:
args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names).
This contradicts the conversion process described in the Python documentation, and given the behaviour you have observed, I'd say the documentation is wrong, and only applied to a argument string, not an argument vector. I cannot verify this myself as I do not have Windows or the source code for Python lying around.
I suspect that if you call subprocess.Popen like:
p = subprocess.Popen(grep + ' "2345$" file.txt', stdout=..., shell_True)
you may find that the double quotes are stripped out as part of the documented argument conversion.

You can use python-textops3 :
from textops import *
print('\n'.join(cat('file.txt') | grep('2345$')))
with python-textops3 you can use unix-like commands with pipes within python
so no need to fork a process which is very heavy