We Keep Coding

How to make YAML-embedded shell script safe against problems with different quotes?

I have some app which gets configured using a YAML file. That app does some processing and supports hooks to execute shell code before and after the processing. This is mostly meant to execute external script files doing the real business, but one can e.g. export environment variables as well. It seems like things are simply forwarded to a shell call with the configured string.
The important thing to note is that one hook is specially called in case ANYTHING goes wrong during processing. In that case the app provides some additional error details to the configured shell script. This is done by reading the necessary part of the YAML config and doing a simple string replacement of special keywords to what is actually available on runtime. Those keywords follow the syntax {...}. Things look like the following in my config:
on_error:
- |
export BGM_ERR_VAR_CONFIG_PATH="{configuration_filename}"
export BGM_ERR_VAR_REPO="{repository}"
export BGM_ERR_VAR_ERROR_MSG="{error}"
export BGM_ERR_VAR_ERROR_OUT="{output}"
'/path/to/script.sh 'some_arg' '[...]' [...]
Originally those keywords were expected to be forwarded as arguments in the called script, but my script needs some other arguments already, so I decided to forward things using environment variables. Shouldn't make too much of a difference regarding my problem, though.
That problem is that really ANYTHING can got wrong and especially the placeholder {output} can contain arbitrary complex error messages. It's most likely a mixture of executed shell commands, using single quotes in most cases, and stacktraces of the programming language the app is implemented in, using double quotes. With my config above this leads to invalid shell code being executed in the end:
[2021-10-12 07:18:46,073] ERROR: /bin/sh: 13: Syntax error: Unterminated quoted string
The following is what the app logs as being executed at all:
[2021-10-12 07:18:46,070] DEBUG: export BGM_ERR_VAR_CONFIG_PATH="/path/to/some.yaml"
export BGM_ERR_VAR_REPO="HOST:PARENT/CHILD"
export BGM_ERR_VAR_ERROR_MSG="Command 'borg check --prefix arch- --debug --show-rc --umask 0007 HOST:PARENT/CHILD' returned non-zero exit status 2."
export BGM_ERR_VAR_ERROR_OUT="using builtin fallback logging configuration
35 self tests completed in 0.04 seconds
SSH command line: ['ssh', '-F', '/[...]/.ssh/config', 'HOST', 'borg', 'serve', '--umask=007', '--debug']
RemoteRepository: 169 B bytes sent, 66 B bytes received, 3 messages sent
Connection closed by remote host
Traceback (most recent call last):
File "borg/archiver.py", line 177, in wrapper"
'/path/to/script.sh '[...]' '[...]' '[...]' '[...]'
The args to my own script are safe regarding quoting, those are only hard-coded paths, keywords etc., nothing dynamic in any way. The problem should be the double quotes used for the path to the python file throwing the exception. OTOH, if I only use single quotes with my environment variables, those would break because the output shell command invoked uses single quotes as well.
So, how do I implement a safe forwarding of {output} into the environment variable in this context?
I thought of using some subshell ="$(...)" and sed to normalize quotes, but everything I came up with resulted in a command line with exactly the same quoting problems like before. Same goes for printf and its %q to escape quotes. It seems I need something which is able to deal with arbitrary individual arguments and joining those to some string again or something like that. Additionally, things should not be too complex to not bloat the YAML config in the end.
The following might work, but loses the double quotes:
export BGM_ERR_VAR_ERROR_OUT="$(echo "{output}")"
How about that?
export BGM_ERR_VAR_ERROR_OUT="$(cat << EOT
{output}
EOT
)"
Anything else? Thanks!

To avoid all the replacement problems, I suggest not using replacements, and forwarding the values as environment variables instead. This assumes you have control over the calling code, which I assume is correct from your explanation.
Since environment variables are by convention uppercase, putting your values in lowercase names is quite safe, and then you can simply do
on_error:
- |
export BGM_ERR_VAR_CONFIG_PATH="$configuration_filename"
export BGM_ERR_VAR_REPO="$repository"
export BGM_ERR_VAR_ERROR_MSG="$error"
export BGM_ERR_VAR_ERROR_OUT="$output"
'/path/to/script.sh 'some_arg' '[...]' [...]
The calling code would need to modify the environment accordingly so that it will contain the expected values. This is the safest way to forward the values, since it guarantees not to interpret the values as bash syntax at all.
If this is not possible, the next best thing is probably to use a heredoc, albeit one with quotes to avoid processing anything in the content – you can use read to avoid the unnecessary cat:
on_error:
- |
read -r -d '' BGM_ERR_VAR_CONFIG_PATH <<'EOF'
{configuration_filename}
EOF
export BGM_ERR_VAR_CONFIG_PATH
# ... snip: other variables ...
'/path/to/script.sh 'some_arg' '[...]' [...]
The only thing you need to be aware of here is that the content may not include a line reading EOF. The calling code needs to ensure this.

Python3 - how to deal with different strings elements (quotes, slash and such) between Windows and other OS

I am passing as parameter a string; to a Python3 function.
This string is composed by different variables; and in few cases, I need to have the quotes in the string, so I did concatenate various strings to make it work.
This is the string I use in my code, to call an application called conn.app, which use a file called lac.tsd to perform various operations. The whole code logic is not relevant for the purpose of this question; since the root cause of the errors is the fact that path strings and quoted strings behave differently between OS (like Windows and Unix-like systems for example)
execute_string = '-b -m path="/Users/user/lac.tsd" -a app="/build/deploy/conn.app" -o output=/Users/user/out/'
This works without problems on OSX and Linux; but when I run this on Windows machines, I get an error
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"'
After digging a bit in the differences between OS, it is clear to see that my path for the tsd file is pointing at /Users/.. which is a pure unix construct for the file system; so that has been replaced when I run the code on Windows.
Although, there is still the problem with conn.app, which on windows is conn.exe; the location is the same, so I did fix that too.
Last but not least, 2 issues tied to the OS itself: the / is \ in Windows, so that has to change; and same goes for quotes and double quotes.
Does Python have any construct that help programmers to handle cases like mine, where you build a string of parameters to pass to a function, and have in it single and double quotes, paths and such?
To expand on the original point: how do you handle this between OS? Beside a simple RE to replace a character with another; is there any construct used in Python that allow code to operate independently from the OS used, when dealing with paths, quoted strings and multiple nested quoted strings?

I think the problem may well be that the string is being passed directly to the command interpreter. In UNIX this will work the way you expect because it treats quotes as special characters, but on Windows, depending on the shell, it probably will behave in very different ways. One possible solution is to break the command you want to run into separate argument strings:
execute_args = [
"-b", "-m",
"path=/Users/user/lac.tsd",
"-a",
"app=/build/deploy/conn.app",
"-o",
"output=/Users/user/out/"
]
And then use one of the many functions in subprocess to execute the command directly instead of using the shell (i.e. shell=False, which is the default for most of them).

How to programmatically escape quotes andd slashes

I need a python script to call a bash script on windows.
So basically I must make a subprocess call form python, that will call cygwin with the -c option that will call the script I need,
The problem is that this script takes a few arguments and that these arguments are full os spaces and quotes and slashes.
I'm using code like the following
arq_saida_unix = arq_saida.replace("\\","/")
subprocess.call("C:\\cygwin64\\bin\\bash \".\\retirarVirgula.sh\\ \""+arq_saida+"\"")
Or I'm directly escaping, which sometimes takes me to as much as 8 backslashes in a row, for a backslash to get to my script must be escaped i) in bash ii) in cmd.exe iii) in python
all of this is error prone and takes quite some time every time to get it right.
Is there a better way of doing it? Ideally I wouldn't have any escaping backslashes, but anything that avoids the triple-slash double quote above would be nice.
I tried to use re.escape, but could figure out how exactly to use it , except as a replacement to .replace("\","/") and similar.

Don't pass a single string to call; instead, pass a list consisting of the command name and one argument per element. This saves you from needing to protect special characters from shell interpretation.
subprocess.call(["retirarVirgula.sh", arq_saida], executable=r"C:\cygwin64\bin\bash")
Note: I'm assuming arq_saida contains the single argument to pass to the script; if the script takes multiple arguments, then arc_saida should probably be built as a list as well:
arq_saida = ["arg", "arg two", "arg three"]
subprocess.call(["retirarVirgula.sh"] + arq_saida, executable=r"C:\cygwin64\bin\bash")

Python system argument reading '&'

I want to pass url to my python via the console and then do the appropriate tasks. many of the links contain the character '&' in the link. python interprets that as ending the argument this however is not what I want. here is a sample of the code
import sys
external_id = sys.argv[1].encode("utf-8")
print external_id
And when I run the following:
python graph.py 2%60&7
I get:
2%60
How do I make python interpret the '&' as nothing more than another character in the url?

This is not python, it's bash. You need to escape it:
python graph.py 2%60\&7

Quoting this answer:
The & informs the shell to put the command in the background.
Python never receives that character because the shell takes care of it, thus this isn't an issue of Python. You can also wrap 2%60&7 with quotes to make it work
me#pc:~$ python3 script.py '2%60&7'
b'2%60&7'
Sometimes escaping & is a lot harder than doing this.

subprocess.call vs os.system python

First time questioning here:
I have a need to map a network drive in windows. The location is an internal sharepoint document library.
In the cmd window:
net use g: http://na.com/DMP/DMP/programming/
is successfull --> the command completed succeffuly
os.system('"net use k: http://na.com/DMP/DMP/programming/"')
is also successful.
However i would like to use subprocess.call in the event that the drive is already mapped - i would like to try another drive
call(["net", "use", ":q", '"http://na.com/DMP/DMP/programming/"'])
This fails with "System error 67 has occured. The network name cannot be found"
I have tried many options for the last list item with no luck.
Any idea what I can stuff in there to have this complete successfully or a different method to map drives.

There are at least two problems in your code:
call(["net", "use", ":q", '"http://na.com/DMP/DMP/programming/"'])
First, you've got ":q" where you meant "q:". This might cause the net command to interpret :q as your network location instead of your target drive, which could cause an error 67.
Second, you've got an extra set of quotes around the URL: '"http://na.com/DMP/DMP/programming/"' where you should be using 'http://na.com/DMP/DMP/programming/'. When subprocess builds the string to pass to CreateProcess, it already quotes each of your parameters. So, if you quote them yourself, you end up double-quoting the parameters. There are some cases where this is actually not possible in Windows, so you end up with garbage, but I don't think that's the case here. You will successfully get this quoted string to net, telling it that you want to open either a relative path starting with "http: or a URL with protocol "http, or something like that. Whatever it is, it's not a usable network location, which most likely will cause an error 67.
As Ben pointed out, your system call has a similar problem—you put an extra pair of quotes around the entire string. If you really wanted to figure it out, there probably is some reason that this worked… but I don't think you want to figure it out. Just take it as "I did the wrong thing, but I got lucky", and don't do it that way in the future.
Finally, as the documentation says:
On Windows, an args sequence is converted to a string that can be parsed
This means that, if you already have a working command line for Windows, you're better off just using it as a string, than trying to break it down into a sequence for subprocess to reassemble.
(Keep in mind that this is only true for Windows! On other platforms, instead of building a command line string to pass to a function in the CreateProcess family, subprocess builds an array of strings to pass to a function in the exec family.)
So, just do this:
call("net use g: http://na.com/DMP/DMP/programming/")