We Keep Coding

Python3 - how to deal with different strings elements (quotes, slash and such) between Windows and other OS

I am passing as parameter a string; to a Python3 function.
This string is composed by different variables; and in few cases, I need to have the quotes in the string, so I did concatenate various strings to make it work.
This is the string I use in my code, to call an application called conn.app, which use a file called lac.tsd to perform various operations. The whole code logic is not relevant for the purpose of this question; since the root cause of the errors is the fact that path strings and quoted strings behave differently between OS (like Windows and Unix-like systems for example)
execute_string = '-b -m path="/Users/user/lac.tsd" -a app="/build/deploy/conn.app" -o output=/Users/user/out/'
This works without problems on OSX and Linux; but when I run this on Windows machines, I get an error
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '"'
After digging a bit in the differences between OS, it is clear to see that my path for the tsd file is pointing at /Users/.. which is a pure unix construct for the file system; so that has been replaced when I run the code on Windows.
Although, there is still the problem with conn.app, which on windows is conn.exe; the location is the same, so I did fix that too.
Last but not least, 2 issues tied to the OS itself: the / is \ in Windows, so that has to change; and same goes for quotes and double quotes.
Does Python have any construct that help programmers to handle cases like mine, where you build a string of parameters to pass to a function, and have in it single and double quotes, paths and such?
To expand on the original point: how do you handle this between OS? Beside a simple RE to replace a character with another; is there any construct used in Python that allow code to operate independently from the OS used, when dealing with paths, quoted strings and multiple nested quoted strings?

I think the problem may well be that the string is being passed directly to the command interpreter. In UNIX this will work the way you expect because it treats quotes as special characters, but on Windows, depending on the shell, it probably will behave in very different ways. One possible solution is to break the command you want to run into separate argument strings:
execute_args = [
"-b", "-m",
"path=/Users/user/lac.tsd",
"-a",
"app=/build/deploy/conn.app",
"-o",
"output=/Users/user/out/"
]
And then use one of the many functions in subprocess to execute the command directly instead of using the shell (i.e. shell=False, which is the default for most of them).

How can I use ssh with print of python?

I would like to run ssh with print of python.
The followings are my test code.
import subprocess
# case1:
command_str = "\"print(\'test\')\""
# case 2:
# command_str = "\\\"print(\'test\')\\\""
ssh_command = ['ssh', 'USER_X#localhost', 'python', '-c']
ssh_command.append(command_str)
process = subprocess.run(ssh_command, stdout=subprocess.PIPE)
print(process.stdout)
case 1 and case 2 did not work.
The outputs are followings,
case 1:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c print('test')'
b''
case 2:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c \"print('test')\"'
b''
Please let me know how it works.

It should work with
command_str = "'print(\"test\")'"
or equivalently
command_str = '\'print("test")\''
Explanation
The outermost quotes and the escaping are for the local Python. So in either case, the local Python string will be 'print("test")'.
There is no quoting or escaping required for the local shell, as subcommand.run(...) won't invoke it unless shell=True is passed.
Thus the single quotes within the python string are for the remote shell (presumably bash or other sh-compatible shell). The argument passed to the remote Python is thus print("test"). (And the double quotes in there are to signify the string literal to print to the remote python.)
Can we do without escaping (without \)?
As there are three levels involved (local Python, remote shell, remote Python), I don't think so.
Can we do with a single type of quotes?
Yes, with a bit more escaping. Let's build this from behind (or inside-out).
We want to print
test
This needs to be escaped for the remote Python (to form a string literal instead of an identifier):
"test"
Call this with the print() function:
print("test")
Quite familiar so far.
Now we want to pass this as an argument to python -c on a sh-like shell. To protect the ( and ) to be interpreted by that, we quote the whole thing. For the already present " not to terminate the quotation, we escape them:
"print(\"test\")"
You can try this in a terminal:
$> echo "print(\"test\")"
print("test")
Perfect!
Now we have to represent the whole thing in (the local) Python. We wrap another layer of quotes around it, have to escape the four(!) existing quotation marks as well as the two backslashes:
"\"print(\\\"test\\\")\""
(Done. This can also be used as command_str.)
Can we do with only single quotes (') and escaping?
I don't know, but at least not as easily. Why? Because, other than to Python, double and single quotes aren't interchangeable to sh and bash: Within single quotes, these shells assume a raw string without escaping until the closing ' occurs.
My brain hurts!
If literally, go see a doctor. If figuratively, yeah, mine too. And your code's future readers (including yourself) will probably feel the same, when they try to untangle that quoting-escaping-forest.
But there's a painless alternative in our beloved Python standard library!
import shlex
command_str = shlex.quote('print("test")')
This is much easier to understand. The inner quotes (double quotes here, but doesn't really matter: shlex.quote("print('test')") works just as fine) are for the remote Python. The outer quotes are obviously for the local Python. And all the quoting and escaping beyond that for the remote shell is taken care of by this utility function.

The correct syntax for python 2 and 3 is:
python -c 'print("test")'

python escape markquotes for bash script [duplicate]

When using os.system() it's often necessary to escape filenames and other arguments passed as parameters to commands. How can I do this? Preferably something that would work on multiple operating systems/shells but in particular for bash.
I'm currently doing the following, but am sure there must be a library function for this, or at least a more elegant/robust/efficient option:
def sh_escape(s):
return s.replace("(","\\(").replace(")","\\)").replace(" ","\\ ")
os.system("cat %s | grep something | sort > %s"
% (sh_escape(in_filename),
sh_escape(out_filename)))
Edit: I've accepted the simple answer of using quotes, don't know why I didn't think of that; I guess because I came from Windows where ' and " behave a little differently.
Regarding security, I understand the concern, but, in this case, I'm interested in a quick and easy solution which os.system() provides, and the source of the strings is either not user-generated or at least entered by a trusted user (me).

shlex.quote() does what you want since python 3.
(Use pipes.quote to support both python 2 and python 3,
though note that pipes has been deprecated since 3.10
and slated for removal in 3.13)

This is what I use:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
The shell will always accept a quoted filename and remove the surrounding quotes before passing it to the program in question. Notably, this avoids problems with filenames that contain spaces or any other kind of nasty shell metacharacter.
Update: If you are using Python 3.3 or later, use shlex.quote instead of rolling your own.

Perhaps you have a specific reason for using os.system(). But if not you should probably be using the subprocess module. You can specify the pipes directly and avoid using the shell.
The following is from PEP324:
Replacing shell pipe line
-------------------------
output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Maybe subprocess.list2cmdline is a better shot?

Note that pipes.quote is actually broken in Python 2.5 and Python 3.1 and not safe to use--It doesn't handle zero-length arguments.
>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1 arg3
See Python issue 7476; it has been fixed in Python 2.6 and 3.2 and newer.

I believe that os.system just invokes whatever command shell is configured for the user, so I don't think you can do it in a platform independent way. My command shell could be anything from bash, emacs, ruby, or even quake3. Some of these programs aren't expecting the kind of arguments you are passing to them and even if they did there is no guarantee they do their escaping the same way.

Notice: This is an answer for Python 2.7.x.
According to the source, pipes.quote() is a way to "Reliably quote a string as a single argument for /bin/sh". (Although it is deprecated since version 2.7 and finally exposed publicly in Python 3.3 as the shlex.quote() function.)
On the other hand, subprocess.list2cmdline() is a way to "Translate a sequence of arguments into a command line string, using the same rules as the MS C runtime".
Here we are, the platform independent way of quoting strings for command lines.
import sys
mswindows = (sys.platform == "win32")
if mswindows:
from subprocess import list2cmdline
quote_args = list2cmdline
else:
# POSIX
from pipes import quote
def quote_args(seq):
return ' '.join(quote(arg) for arg in seq)
Usage:
# Quote a single argument
print quote_args(['my argument'])
# Quote multiple arguments
my_args = ['This', 'is', 'my arguments']
print quote_args(my_args)

The function I use is:
def quote_argument(argument):
return '"%s"' % (
argument
.replace('\\', '\\\\')
.replace('"', '\\"')
.replace('$', '\\$')
.replace('`', '\\`')
)
that is: I always enclose the argument in double quotes, and then backslash-quote the only characters special inside double quotes.

On UNIX shells like Bash, you can use shlex.quote in Python 3 to escape special characters that the shell might interpret, like whitespace and the * character:
import os
import shlex
os.system("rm " + shlex.quote(filename))
However, this is not enough for security purposes! You still need to be careful that the command argument is not interpreted in unintended ways. For example, what if the filename is actually a path like ../../etc/passwd? Running os.system("rm " + shlex.quote(filename)) might delete /etc/passwd when you only expected it to delete filenames found in the current directory! The issue here isn't with the shell interpreting special characters, it's that the filename argument isn't interpreted by the rm as a simple filename, it's actually interpreted as a path.
Or what if the valid filename starts with a dash, for example, -f? It's not enough to merely pass the escaped filename, you need to disable options using -- or you need to pass a path that doesn't begin with a dash like ./-f. The issue here isn't with the shell interpreting special characters, it's that the rm command interprets the argument as a filename or a path or an option if it begins with a dash.
Here is a safer implementation:
if os.sep in filename:
raise Exception("Did not expect to find file path separator in file name")
os.system("rm -- " + shlex.quote(filename))

I think these answers are a bad idea for escaping command-line arguments on Windows. Based on the results: people are trying to apply a black-list approach to filtering 'bad' characters, assuming (and hoping) they got them all. Windows is very complex and there could be all manner of characters found in the future that might allow an attacker to hijack command line arguments.
I've already seen some answers neglect to filter basic meta-characters in Windows (like the semi-colon.) The approach I take is far simpler:
Make a list of allowed ASCII characters.
Remove all chars that aren't in that list.
Escape slashes and double-quotes.
Surround entire command with double quotes so the command argument cannot be maliciously broken and commandeered with spaces.
A basic example:
def win_arg_escape(arg, allow_vars=0):
allowed_list = """'"/\\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-. """
if allow_vars:
allowed_list += "~%$"
# Filter out anything that isn't a
# standard character.
buf = ""
for ch in arg:
if ch in allowed_list:
buf += ch
# Escape all slashes.
buf = buf.replace("\\", "\\\\")
# Escape double quotes.
buf = buf.replace('"', '""')
# Surround entire arg with quotes.
# This avoids spaces breaking a command.
buf = '"%s"' % (buf)
return buf
The function has an option to enable use of environmental variables and other shell variables. Enabling this poses more risk so its disabled by default.

How to open a file in its default program with python

I want to open a file in python 3.5 in its default application, specifically 'screen.txt' in Notepad.
I have searched the internet, and found os.startfile(path) on most of the answers. I tried that with the file's path os.startfile(C:\[directories n stuff]\screen.txt) but it returned an error saying 'unexpected character after line continuation character'. I tried it without the file's path, just the file's name but it still didn't work.
What does this error mean? I have never seen it before.
Please provide a solution for opening a .txt file that works.
EDIT: I am on Windows 7 on a restricted (school) computer.

It's hard to be certain from your question as it stands, but I bet your problem is backslashes.
[EDITED to add:] Or actually maybe it's something simpler. Did you put quotes around your pathname at all? If not, that will certainly not work -- but once you do, you will find that then you need the rest of what I've written below.
In a Windows filesystem, the backslash \ is the standard way to separate directories.
In a Python string literal, the backslash \ is used for putting things into the string that would otherwise be difficult to enter. For instance, if you are writing a single-quoted string and you want a single quote in it, you can do this: 'don\'t'. Or if you want a newline character, you can do this: 'First line.\nSecond line.'
So if you take a Windows pathname and plug it into Python like this:
os.startfile('C:\foo\bar\baz')
then the string actually passed to os.startfile will not contain those backslashes; it will contain a form-feed character (from the \f) and two backspace characters (from the \bs), which is not what you want at all.
You can deal with this in three ways.
You can use forward slashes instead of backslashes. Although Windows prefers backslashes in its user interface, forward slashes work too, and they don't have special meaning in Python string literals.
You can "escape" the backslashes: two backslashes in a row mean an actual backslash. os.startfile('C:\\foo\\bar\\baz')
You can use a "raw string literal". Put an r before the opening single or double quotes. This will make backslashes not get interpreted specially. os.startfile(r'C:\foo\bar\baz')
The last is maybe the nicest, except for one annoying quirk: backslash-quote is still special in a raw string literal so that you can still say 'don\'t', which means you can't end a raw string literal with a backslash.

The recommended way to open a file with the default program is os.startfile. You can do something a bit more manual using os.system or subprocess though:
os.system(r'start ' + path_to_file')
or
subprocess.Popen('{start} {path}'.format(
start='start', path=path_to_file), shell=True)
Of course, this won't work cross-platform, but it might be enough for your use case.

For example I created file "test file.txt" on my drive D: so file path is 'D:/test file.txt'
Now I can open it with associated program with that script:
import os
os.startfile('d:/test file.txt')

Python system argument reading '&'

I want to pass url to my python via the console and then do the appropriate tasks. many of the links contain the character '&' in the link. python interprets that as ending the argument this however is not what I want. here is a sample of the code
import sys
external_id = sys.argv[1].encode("utf-8")
print external_id
And when I run the following:
python graph.py 2%60&7
I get:
2%60
How do I make python interpret the '&' as nothing more than another character in the url?

This is not python, it's bash. You need to escape it:
python graph.py 2%60\&7

Quoting this answer:
The & informs the shell to put the command in the background.
Python never receives that character because the shell takes care of it, thus this isn't an issue of Python. You can also wrap 2%60&7 with quotes to make it work
me#pc:~$ python3 script.py '2%60&7'
b'2%60&7'
Sometimes escaping & is a lot harder than doing this.