We Keep Coding

How can I use ssh with print of python?

I would like to run ssh with print of python.
The followings are my test code.
import subprocess
# case1:
command_str = "\"print(\'test\')\""
# case 2:
# command_str = "\\\"print(\'test\')\\\""
ssh_command = ['ssh', 'USER_X#localhost', 'python', '-c']
ssh_command.append(command_str)
process = subprocess.run(ssh_command, stdout=subprocess.PIPE)
print(process.stdout)
case 1 and case 2 did not work.
The outputs are followings,
case 1:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c print('test')'
b''
case 2:
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `python -c \"print('test')\"'
b''
Please let me know how it works.

It should work with
command_str = "'print(\"test\")'"
or equivalently
command_str = '\'print("test")\''
Explanation
The outermost quotes and the escaping are for the local Python. So in either case, the local Python string will be 'print("test")'.
There is no quoting or escaping required for the local shell, as subcommand.run(...) won't invoke it unless shell=True is passed.
Thus the single quotes within the python string are for the remote shell (presumably bash or other sh-compatible shell). The argument passed to the remote Python is thus print("test"). (And the double quotes in there are to signify the string literal to print to the remote python.)
Can we do without escaping (without \)?
As there are three levels involved (local Python, remote shell, remote Python), I don't think so.
Can we do with a single type of quotes?
Yes, with a bit more escaping. Let's build this from behind (or inside-out).
We want to print
test
This needs to be escaped for the remote Python (to form a string literal instead of an identifier):
"test"
Call this with the print() function:
print("test")
Quite familiar so far.
Now we want to pass this as an argument to python -c on a sh-like shell. To protect the ( and ) to be interpreted by that, we quote the whole thing. For the already present " not to terminate the quotation, we escape them:
"print(\"test\")"
You can try this in a terminal:
$> echo "print(\"test\")"
print("test")
Perfect!
Now we have to represent the whole thing in (the local) Python. We wrap another layer of quotes around it, have to escape the four(!) existing quotation marks as well as the two backslashes:
"\"print(\\\"test\\\")\""
(Done. This can also be used as command_str.)
Can we do with only single quotes (') and escaping?
I don't know, but at least not as easily. Why? Because, other than to Python, double and single quotes aren't interchangeable to sh and bash: Within single quotes, these shells assume a raw string without escaping until the closing ' occurs.
My brain hurts!
If literally, go see a doctor. If figuratively, yeah, mine too. And your code's future readers (including yourself) will probably feel the same, when they try to untangle that quoting-escaping-forest.
But there's a painless alternative in our beloved Python standard library!
import shlex
command_str = shlex.quote('print("test")')
This is much easier to understand. The inner quotes (double quotes here, but doesn't really matter: shlex.quote("print('test')") works just as fine) are for the remote Python. The outer quotes are obviously for the local Python. And all the quoting and escaping beyond that for the remote shell is taken care of by this utility function.

The correct syntax for python 2 and 3 is:
python -c 'print("test")'

python escape markquotes for bash script [duplicate]

When using os.system() it's often necessary to escape filenames and other arguments passed as parameters to commands. How can I do this? Preferably something that would work on multiple operating systems/shells but in particular for bash.
I'm currently doing the following, but am sure there must be a library function for this, or at least a more elegant/robust/efficient option:
def sh_escape(s):
return s.replace("(","\\(").replace(")","\\)").replace(" ","\\ ")
os.system("cat %s | grep something | sort > %s"
% (sh_escape(in_filename),
sh_escape(out_filename)))
Edit: I've accepted the simple answer of using quotes, don't know why I didn't think of that; I guess because I came from Windows where ' and " behave a little differently.
Regarding security, I understand the concern, but, in this case, I'm interested in a quick and easy solution which os.system() provides, and the source of the strings is either not user-generated or at least entered by a trusted user (me).

shlex.quote() does what you want since python 3.
(Use pipes.quote to support both python 2 and python 3,
though note that pipes has been deprecated since 3.10
and slated for removal in 3.13)

This is what I use:
def shellquote(s):
return "'" + s.replace("'", "'\\''") + "'"
The shell will always accept a quoted filename and remove the surrounding quotes before passing it to the program in question. Notably, this avoids problems with filenames that contain spaces or any other kind of nasty shell metacharacter.
Update: If you are using Python 3.3 or later, use shlex.quote instead of rolling your own.

Perhaps you have a specific reason for using os.system(). But if not you should probably be using the subprocess module. You can specify the pipes directly and avoid using the shell.
The following is from PEP324:
Replacing shell pipe line
-------------------------
output=`dmesg | grep hda`
==>
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

Maybe subprocess.list2cmdline is a better shot?

Note that pipes.quote is actually broken in Python 2.5 and Python 3.1 and not safe to use--It doesn't handle zero-length arguments.
>>> from pipes import quote
>>> args = ['arg1', '', 'arg3']
>>> print 'mycommand %s' % (' '.join(quote(arg) for arg in args))
mycommand arg1 arg3
See Python issue 7476; it has been fixed in Python 2.6 and 3.2 and newer.

I believe that os.system just invokes whatever command shell is configured for the user, so I don't think you can do it in a platform independent way. My command shell could be anything from bash, emacs, ruby, or even quake3. Some of these programs aren't expecting the kind of arguments you are passing to them and even if they did there is no guarantee they do their escaping the same way.

Notice: This is an answer for Python 2.7.x.
According to the source, pipes.quote() is a way to "Reliably quote a string as a single argument for /bin/sh". (Although it is deprecated since version 2.7 and finally exposed publicly in Python 3.3 as the shlex.quote() function.)
On the other hand, subprocess.list2cmdline() is a way to "Translate a sequence of arguments into a command line string, using the same rules as the MS C runtime".
Here we are, the platform independent way of quoting strings for command lines.
import sys
mswindows = (sys.platform == "win32")
if mswindows:
from subprocess import list2cmdline
quote_args = list2cmdline
else:
# POSIX
from pipes import quote
def quote_args(seq):
return ' '.join(quote(arg) for arg in seq)
Usage:
# Quote a single argument
print quote_args(['my argument'])
# Quote multiple arguments
my_args = ['This', 'is', 'my arguments']
print quote_args(my_args)

The function I use is:
def quote_argument(argument):
return '"%s"' % (
argument
.replace('\\', '\\\\')
.replace('"', '\\"')
.replace('$', '\\$')
.replace('`', '\\`')
)
that is: I always enclose the argument in double quotes, and then backslash-quote the only characters special inside double quotes.

On UNIX shells like Bash, you can use shlex.quote in Python 3 to escape special characters that the shell might interpret, like whitespace and the * character:
import os
import shlex
os.system("rm " + shlex.quote(filename))
However, this is not enough for security purposes! You still need to be careful that the command argument is not interpreted in unintended ways. For example, what if the filename is actually a path like ../../etc/passwd? Running os.system("rm " + shlex.quote(filename)) might delete /etc/passwd when you only expected it to delete filenames found in the current directory! The issue here isn't with the shell interpreting special characters, it's that the filename argument isn't interpreted by the rm as a simple filename, it's actually interpreted as a path.
Or what if the valid filename starts with a dash, for example, -f? It's not enough to merely pass the escaped filename, you need to disable options using -- or you need to pass a path that doesn't begin with a dash like ./-f. The issue here isn't with the shell interpreting special characters, it's that the rm command interprets the argument as a filename or a path or an option if it begins with a dash.
Here is a safer implementation:
if os.sep in filename:
raise Exception("Did not expect to find file path separator in file name")
os.system("rm -- " + shlex.quote(filename))

I think these answers are a bad idea for escaping command-line arguments on Windows. Based on the results: people are trying to apply a black-list approach to filtering 'bad' characters, assuming (and hoping) they got them all. Windows is very complex and there could be all manner of characters found in the future that might allow an attacker to hijack command line arguments.
I've already seen some answers neglect to filter basic meta-characters in Windows (like the semi-colon.) The approach I take is far simpler:
Make a list of allowed ASCII characters.
Remove all chars that aren't in that list.
Escape slashes and double-quotes.
Surround entire command with double quotes so the command argument cannot be maliciously broken and commandeered with spaces.
A basic example:
def win_arg_escape(arg, allow_vars=0):
allowed_list = """'"/\\abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-. """
if allow_vars:
allowed_list += "~%$"
# Filter out anything that isn't a
# standard character.
buf = ""
for ch in arg:
if ch in allowed_list:
buf += ch
# Escape all slashes.
buf = buf.replace("\\", "\\\\")
# Escape double quotes.
buf = buf.replace('"', '""')
# Surround entire arg with quotes.
# This avoids spaces breaking a command.
buf = '"%s"' % (buf)
return buf
The function has an option to enable use of environmental variables and other shell variables. Enabling this poses more risk so its disabled by default.

How to remove whitespaces from multiline string pass it to another program and then add back whitespaces?

Summary
I need to detect indentation level of the first line in multiline string passed to a script. Store it. Remove this indent from other lines. Pass the multiline string with removed indent level to another program (that I've figured how to do) add back indent to all lines in multiline string and print it to stdout (that I also know how to do).
To be specific I have a problem with vim and Python formatter YAPF.
The way yapf works is that if python file is incorrect formatting would result in error.
So imagine this
def f():
# imagine some very very long lines here that we want to reformat
If I would select this imagined lines in vim and then press gq (I've set formatprg=yapf) vim would substitute this lines with a traceback of yapf which is no good of course. But If I would select the whole function it would do the job perfectly.
You can test this with
echo ' fooo = 1' | yapf
This would result in IndentationError
While echo 'fooo = 1' | yapf would work
So what I think is a very nice workaround is to remove indentation store the indent level of the first line, pass string without indentation to yapf somehow and then add indent to the result. The problem with this is I'd like this to be a one liner or close to that so that it could be stored directly in my vimrc. So python isn't a good match for that because I would need at least to import re package etc.
So I thought about perl.
The only problem is that I don't know perl much.
So for now my experiment looks like this
$a = " foo = 1\n bar = '1'";
my ($indent, $text) = $a =~ m/^(\s+)(.*)$/m;
$command = "echo " . $text;
$out = `$command`;
print "$out\n";
print "$text\n";
I will be glad for any help. Maybe there is more easy way to do this, I don't know.

Since you seem to be familiar with Python already I would recommend using its textwrap module, which contains dedent and (in version 3.3 and later) indent functions that can do most of the job for you:
import re
from textwrap import dedent, indent
whitespace = re.compile('\s+')
test_string = ''' while True:
pass'''
leading_whitespace = whitespace.match(test_string)
dedented_text = dedent(test_string)
# Do whatever you want with dedented_text
indented_text = indent(dedented_text, leading_whitespace.group(0))

python multiline command running from bash

I'm trying to run this:
python -c "for i in range(10):\n print i"
but I get an error:
File "<string>", line 1
for i in range(10):\n print i
^
SyntaxError: unexpected character after line continuation character
According to this I assume that bash should have processed (namely, newline symbol) command line arguments but the returned error shows the opposite case. Where am I wrong, and why does this happen?
P.S. python-2.7
EDIT
Let me explain my motivation a bit.
This code example is definitely pretty silly. Since the doc says that "command can be one or more statements separated by newlines, with significant leading whitespace as in normal module code", I was interested in how should I bring those mentioned newlines to the command properly.
The proposed solutions here are:
Use ; to distinguish several commands inside the loop. Yes, that works but it still is a one-liner, I can not use it If I want to run some commands after the loop. ; is not a replacement for a newline.
Type ^M where newline is needed. This hits the goal more precisely but unfortunately, to my point of view, this basically ruins the whole idea of running a python code from the command line because it requires interactive mode. As I understand it's the same as entering a command ant hitting Enter key. So no difference to typing python and working in its shell. That said, I cannot write this in a bash script. Or may I?
Probably the question really should have been splitted into two ones:
Bash escaping:
Enclosing characters in double quotes (‘"’) preserves the literal value of all characters within the quotes, with the exception of ‘$’, ‘’, ‘\’, and, when history expansion is enabled, ‘!’. The characters ‘$’ and ‘’ retain their special meaning within double quotes (see Shell Expansions). The backslash retains its special meaning only when followed by one of the following characters: ‘$’, ‘`’, ‘"’, ‘\’, or newline.
How does this correspond to the case described? How does bash handles newlines? I found that putting the command into unary quotes makes no change.
How to pass a newline to python in a non-interactive way. (You may say -- why don't you write an ordinary python file with all newlines you want -- you are right but I'm interested in what is exactly meant in the documentation since it quotes newline)

You actually would need to transform the \n part into an actual newline. That can be done with the $'' syntax:
python -c $'for i in range(10):\n print i'
0
1
2
3
4
5
6
7
8
9
You can also reach that result with echo -e or printf
$ python -c "$(echo -e "for i in range(10):\n print i")"
You could also use a here string:
$ python <<< $(echo -e "for i in range(10):\n print i")
See section 3.1.2.4 ANSI-C Quoting of the Bash Manpage for more information.

Remove \n
python -c "for i in range(10): print i"
Or
You can use ; for using multiple line in for loop
python -c "for i in range(10): print '1st newline';print '2nd newline';print i"

You can run a multi-line python -c statement by adding CR characters in your line:
python -c "for i in range(10):^M print (i)^M print ('Hello:' + str(i*i))"
where ^M is not actually ^ followed by M, it is actually the character you get when you type [CTRL-v][CTRL-m]. Notice the space after this character, which means there are two print statements in the for loop, and it should print:
0
Hello:0
1
Hello:1
....
9
Hello:81
You can do this in a bash script too:
#!/bin/bash
A="python -c \"for i in range(10):^M print (i)^M print ('Hello:' + str(i*i))\""
eval $A

bug in "django-admin.py makemessages" or xgettext call? -> "warning: unterminated string"

django-admin.py makemessages dies with errors "warning: unterminated string" on cases where really long strings are wrapped:
string = "some text \
more text\
and even more"
These strings don't even need to be translated - e.g. sql query strings.
The problem goes away when I concatenate the string, but the result looks ugly and it takes time to join them...
Does anyone have a problem like this? Have you found a way to fix it?
I have the following versions of the tools involved:
xgettext-0.17, gettext-0.17, django-1.0.2, python-2.6.2
There was a ticket on this issue, but it was closed probably because the error appears only in some combination of component versions.
EDIT: found the source of problem - xgettext prints warning messages to sterr and django takes them as fatal errors and quits.
return status of xgettext call is 0 - "success". I guess that django should recognize it as success and not quit because of warnings.
Interestinly xgettext still extracts backslash-wrapped strings if they need to be translated, but gives warnings in stderr ("unterminated string") and .po file ("internationalized messages should not contain the `\r' escape sequence")
xgettext call is the following:
xgettext -d django -L Python --keyword=gettext_noop \
--keyword=gettext_lazy --keyword=ngettext_lazy:1,2 \
--keyword=ugettext_noop --keyword=ugettext_lazy \
--keyword=ungettext_lazy:1,2
--from-code UTF-8 -o - source_file.py
called from django/core/management/commands/makemessages.py

I can think of two possibilities: you might have an extra space after your backslash at the end of the line; or you might be somehow ending up with the wrong line-ending characters in your source (e.g. Windows-style when your Python is expecting Unix-style, thus disabling the backslashes).
Either way, I would take advantage of C-style automatic string concatenation:
>>> string = ("some text "
... "more text "
... "and even more")
>>> string
'some text more text and even more'
Alternatively, if you don't mind newlines ending up in there, use multi-line strings:
>>> string = """some text
... more text
... and even more"""
IMO these look much nicer, and are much less fragile when refactoring.
Does this help?