Related
I'm using Python with -c to execute a one-liner loop, i.e.:
python -c "for r in range(10): print 'rob'"
This works fine. However, if I import a module before the for loop, I get a syntax error:
python -c "import sys; for r in range(10): print 'rob'"
File "<string>", line 1
import sys; for r in range(10): print 'rob'
^
SyntaxError: invalid syntax
How can this be fixed?
It's important to me to have this as a one-liner so that I can include it in a Makefile.
You could do
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Or without pipes:
python -c "exec(\"import sys\nfor r in range(10): print 'rob'\")"
Or
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
Or SilentGhost's answer or Crast's answer.
This style can be used in makefiles too (and in fact it is used quite often).
python - <<EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
Or with hard tabs:
python - <<-EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
# Important: Replace the indentation above w/ hard tabs.
In above case, leading TAB characters are removed too (and some structured outlook can be achieved).
Instead of EOF can stand any marker word not appearing in the here document at a beginning of a line (see also here documents in the bash man page or here).
The issue is not actually with the import statement. It's with anything being before the for loop. Or more specifically, anything appearing before an inlined block.
For example, these all work:
python -c "import sys; print 'rob'"
python -c "import sys; sys.stdout.write('rob\n')"
If import being a statement were an issue, this would work, but it doesn't:
python -c "__import__('sys'); for r in range(10): print 'rob'"
For your very basic example, you could rewrite it as this:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
However, lambdas can only execute expressions, not statements or multiple statements, so you may still be unable to do the thing you want to do. However, between generator expressions, list comprehension, lambdas, sys.stdout.write, the "map" builtin, and some creative string interpolation, you can do some powerful one-liners.
The question is, how far do you want to go, and at what point is it not better to write a small .py file which your makefile executes instead?
- To make this answer work with Python 3.x as well, print is called as a function: in 3.x, only print('foo') works, whereas 2.x also accepts print 'foo'.
- For a cross-platform perspective that includes Windows, see kxr's helpful answer.
In bash, ksh, or zsh:
Use an ANSI C-quoted string ($'...'), which allows using \n to represent newlines that are expanded to actual newlines before the string is passed to python:
python -c $'import sys\nfor r in range(10): print("rob")'
Note the \n between the import and for statements to effect a line break.
To pass shell-variable values to such a command, it is safest to use arguments and access them via sys.argv inside the Python script:
name='rob' # value to pass to the Python script
python -c $'import sys\nfor r in range(10): print(sys.argv[1])' "$name"
See below for a discussion of the pros and cons of using an (escape sequence-preprocessed) double-quoted command string with embedded shell-variable references.
To work safely with $'...' strings:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by $'...' (see man printf for the supported escapes)
Escape ' instances as \'.
If you must remain POSIX-compliant:
Use printf with a command substitution:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob")')"
To work safely with this type of string:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by printf (see man printf for the supported escape sequences).
Pass a single-quoted string to printf %b and escape embedded single quotes as '\'' (sic).
Using single quotes protects the string's contents from interpretation by the shell.
That said, for short Python scripts (as in this case) you can use a double-quoted string to incorporate shell variable values into your scripts - as long as you're aware of the associated pitfalls (see next point); e.g., the shell expands $HOME to the current user's home dir. in the following command:
python -c "$(printf %b "import sys\nfor r in range(10): print('rob is $HOME')")"
However, the generally preferred approach is to pass values from the shell via arguments, and access them via sys.argv in Python; the equivalent of the above command is:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob is " + sys.argv[1])')" "$HOME"
While using a double-quoted string is more convenient - it allows you to use embedded single quotes unescaped and embedded double quotes as \" - it also makes the string subject to interpretation by the shell, which may or may not be the intent; $ and ` characters in your source code that are not meant for the shell may cause a syntax error or alter the string unexpectedly.
Additionally, the shell's own \ processing in double-quoted strings can get in the way; for instance, to get Python to produce literal output ro\b, you must pass ro\\b to it; with a '...' shell string and doubled \ instances, we get:
python -c "$(printf %b 'import sys\nprint("ro\\\\bs")')" # ok: 'ro\bs'
By contrast, this does not work as intended with a "..." shell string:
python -c "$(printf %b "import sys\nprint('ro\\\\bs')")" # !! INCORRECT: 'rs'
The shell interprets both "\b" and "\\b" as literal \b, requiring a dizzying number of additional \ instances to achieve the desired effect:
python -c "$(printf %b "import sys\nprint('ro\\\\\\\\bs')")"
To pass the code via stdin rather than -c:
Note: I'm focusing on single-line solutions here; xorho's answer shows how to use a multi-line here-document - be sure to quote the delimiter, however; e.g., <<'EOF', unless you explicitly want the shell to expand the string up front (which comes with the caveats noted above).
In bash, ksh, or zsh:
Combine an ANSI C-quoted string ($'...') with a here-string (<<<...):
python - <<<$'import sys\nfor r in range(10): print("rob")'
- tells python explicitly to read from stdin (which it does by default).
- is optional in this case, but if you also want to pass arguments to the scripts, you do need it to disambiguate the argument from a script filename:
python - 'rob' <<<$'import sys\nfor r in range(10): print(sys.argv[1])'
If you must remain POSIX-compliant:
Use printf as above, but with a pipeline so as to pass its output via stdin:
printf %b 'import sys\nfor r in range(10): print("rob")' | python
With an argument:
printf %b 'import sys\nfor r in range(10): print(sys.argv[1])' | python - 'rob'
Any idea how this can be fixed?
Your problem is created by the fact that Python statements, separated by ;, are only allowed to be "small statements", which are all one-liners. From the grammar file in the Python docs:
stmt: simple_stmt | compound_stmt
simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
Compound statements can't be included on the same line with other statements via semicolons - so doing this with the -c flag becomes very inconvenient.
When demonstrating Python while in a bash shell environment, I find it very useful to include compound statements. The only simple way of doing this reliably is with heredocs (a posix shell thing).
Heredocs
Use a heredoc (created with <<) and Python's command line interface option, -:
$ python - <<-"EOF"
import sys # 1 tab indent
for r in range(10): # 1 tab indent
print('rob') # 1 tab indent and 4 spaces
EOF
Adding the - after << (the <<-) allows you to use tabs to indent (Stackoverflow converts tabs to spaces, so I've indented 8 spaces to emphasize this). The leading tabs will be stripped.
You can do it without the tabs with just <<:
$ python - << "EOF"
import sys
for r in range(10):
print('rob')
EOF
Putting quotes around EOF prevents parameter and arithmetic expansion. This makes the heredoc more robust.
Bash multiline strings
If you use double-quotes, you'll get shell-expansion:
$ python -c "
> import sys
> for p in '$PATH'.split(':'):
> print(p)
> "
/usr/sbin
/usr/bin
/sbin
/bin
...
To avoid shell expansion use single-quotes:
$ python -c '
> import sys
> for p in "$PATH".split(":"):
> print(p)
> '
$PATH
Note that we need to swap the quote characters on the literals in Python - we basically can't use quote character being interpreted by BASH. We can alternate them though, like we can in Python - but this already looks quite confusing, which is why I don't recommend this:
$ python -c '
import sys
for p in "'"$PATH"'".split(":"):
print(p)
'
/usr/sbin
/usr/bin
/sbin
/bin
...
Critique of the accepted answer (and others)
This is not very readable:
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Not very readable, and additionally difficult to debug in the case of an error:
python -c "exec(\"import sys\\nfor r in range(10): print 'rob'\")"
Perhaps a bit more readable, but still quite ugly:
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
You'll have a bad time if you have "'s in your python:
$ python -c "import sys
> for r in range(10): print 'rob'"
Don't abuse map or list comprehensions to get for-loops:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
These are all sad and bad. Don't do them.
Just use Return and type it on the next line:
python -c "import sys
for r in range(10): print 'rob'"
rob
rob
...
python2.6 -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
works fine.
Use "[ ]" to inline your for loop.
The problem is not with the import statement. The problem is that the control flow statements don't work inlined in a Python interpreter command. Replace that import statement with any other statement, and you'll see the same problem.
Think about it: Python can't possibly inline everything. It uses indentation to group control-flow.
If your system is POSIX.2-compliant it should supply the printf utility:
printf "print 'zap'\nfor r in range(3): print 'rob'" | python
zap
rob
rob
rob
This variant is most portable for putting multi-line scripts on the command-line on Windows and Unix-like systems, Python 2 and Python 3, without pipes:
python -c "exec(\"import sys \nfor r in range(10): print('rob') \")"
(None of the other examples seen here so far did so.)
Neat on Windows is:
python -c exec"""import sys \nfor r in range(10): print 'rob' """
python -c exec("""import sys \nfor r in range(10): print('rob') """)
Neat on Bash on Unix-like systems is:
python -c $'import sys \nfor r in range(10): print("rob")'
This function turns any multiline-script into a portable command-one-liner:
def py2cmdline(script):
exs = 'exec(%r)' % re.sub('\r\n|\r', '\n', script.rstrip())
print('python -c "%s"' % exs.replace('"', r'\"'))
Usage:
>>> py2cmdline(getcliptext())
python -c "exec('print \'AA\tA\'\ntry:\n for i in 1, 2, 3:\n print i / 0\nexcept:\n print \"\"\"longer\nmessage\"\"\"')"
The input was:
print 'AA A'
try:
for i in 1, 2, 3:
print i / 0
except:
print """longer
message"""
I'm not really a big Pythoner - but I found this syntax once, forgot where from, so I thought I'd document it:
If you use sys.stdout.write instead of print (the difference being, sys.stdout.write takes arguments as a function, in parentheses - whereas print doesn't), then for a one-liner, you can get away with inverting the order of the command and the for, removing the semicolon, and enclosing the command in square brackets, i.e.:
python -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
I have no idea how this syntax would be called in Python :)
These square brackets in one-liners are "list comprehensions"; note this in Python 2.7:
STR=abc
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); print a"
Output:
<generator object <genexpr> at 0xb771461c>
So the command in round brackets/parenthesis is seen as a "generator object"; if we "iterate" through it by calling next() - then the command inside the parenthesis will be executed (note the "abc" in the output):
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); a.next() ; print a"
Output:
abc
<generator object <genexpr> at 0xb777b734>
If we now use square brackets - note that we don't need to call next() to have the command execute, it executes immediately upon assignment; however, later inspection reveals that a is None:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; print a"
Output:
abc
[None]
This doesn't leave much information to look for - for the square brackets case - but I stumbled upon this page which I think explains:
Python Tips And Tricks – First Edition - Python Tutorials | Dream.In.Code:
If you recall, the standard format of a single line generator is a kind of one line 'for' loop inside brackets. This will produce a 'one-shot' iterable object which is an object you can iterate over in only one direction and which you can't re-use once you reach the end.
A 'list comprehension' looks almost the same as a regular one-line generator, except that the regular brackets - ( ) - are replaced by square brackets - [ ]. The major advantage of alist comprehension is that produces a 'list', rather than a 'one-shot' iterable object, so that you can go back and forth through it, add elements, sort, etc.
And indeed it is a list - it's just its first element becomes none as soon as it is executed:
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin].__class__"
Output:
abc
<type 'list'>
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin][0]"
Output:
abc
None
List comprehensions are otherwise documented in 5. Data Structures: 5.1.4. List Comprehensions — Python v2.7.4 documentation as "List comprehensions provide a concise way to create lists"; presumably, that's where the limited "executability" of lists comes into play in one-liners.
Well, hope I'm not terribly too off the mark here ...
And here is a one-liner command line with two non-nested for-loops; both enclosed within "list comprehension" square brackets:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; b=[sys.stdout.write(str(x)) for x in range(2)] ; print a ; print b"
Output:
abc
01[None]
[None, None]
Notice that the second "list" b now has two elements, since its for loop explicitly ran twice; however, the result of sys.stdout.write() in both cases was (apparently) None.
single/double quotes and backslash everywhere:
python -c 'exec("import sys\nfor i in range(10): print \"bob\"")'
Much better:
python -c '
import sys
for i in range(10):
print("bob")
'
Note: On some systems executable python is for Python 2 and may not be available. And executable python3 is for Python 3.
This script provides a Perl-like command line interface:
Pyliner - Script to run arbitrary Python code on the command line (Python recipe)
I wanted a solution with the following properties:
Readable
Read standard input for processing output of other tools
Both requirements were not provided in the other answers, so here's how to read standard input while doing everything on the command line:
grep special_string -r | sort | python3 <(cat <<EOF
import sys
for line in sys.stdin:
tokens = line.split()
if len(tokens) == 4:
print("%-45s %7.3f %s %s" % (tokens[0], float(tokens[1]), tokens[2], tokens[3]))
EOF
)
When I needed to do this, I used:
python -c "$(echo -e "import sys\nsys.stdout.write('Hello, World!\\\n')")"
Note the triple backslash for the newline in the sys.stdout.write statement.
If you don't want to touch standard input and simulate as if you had passed "python cmdfile.py", you can do the following from a Bash shell:
python <(printf "word=raw_input('Enter word: ')\nimport sys\nfor i in range(5):\n print(word)")
As you can see, it allows you to use standard input for reading input data. Internally the shell creates the temporary file for the input command contents.
Use python -c with triple quotes:
python -c """
import os
os.system('pwd')
os.system('ls -l')
print('Hello, World!')
for _ in range(5):
print(_)
"""
There is one more option. sys.stdout.write returns None, which keeps the list empty:
cat somefile.log|python -c "import sys;[line for line in sys.stdin if sys.stdout.write(line*2)]"
I'm using Python with -c to execute a one-liner loop, i.e.:
python -c "for r in range(10): print 'rob'"
This works fine. However, if I import a module before the for loop, I get a syntax error:
python -c "import sys; for r in range(10): print 'rob'"
File "<string>", line 1
import sys; for r in range(10): print 'rob'
^
SyntaxError: invalid syntax
How can this be fixed?
It's important to me to have this as a one-liner so that I can include it in a Makefile.
You could do
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Or without pipes:
python -c "exec(\"import sys\nfor r in range(10): print 'rob'\")"
Or
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
Or SilentGhost's answer or Crast's answer.
This style can be used in makefiles too (and in fact it is used quite often).
python - <<EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
Or with hard tabs:
python - <<-EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
# Important: Replace the indentation above w/ hard tabs.
In above case, leading TAB characters are removed too (and some structured outlook can be achieved).
Instead of EOF can stand any marker word not appearing in the here document at a beginning of a line (see also here documents in the bash man page or here).
The issue is not actually with the import statement. It's with anything being before the for loop. Or more specifically, anything appearing before an inlined block.
For example, these all work:
python -c "import sys; print 'rob'"
python -c "import sys; sys.stdout.write('rob\n')"
If import being a statement were an issue, this would work, but it doesn't:
python -c "__import__('sys'); for r in range(10): print 'rob'"
For your very basic example, you could rewrite it as this:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
However, lambdas can only execute expressions, not statements or multiple statements, so you may still be unable to do the thing you want to do. However, between generator expressions, list comprehension, lambdas, sys.stdout.write, the "map" builtin, and some creative string interpolation, you can do some powerful one-liners.
The question is, how far do you want to go, and at what point is it not better to write a small .py file which your makefile executes instead?
- To make this answer work with Python 3.x as well, print is called as a function: in 3.x, only print('foo') works, whereas 2.x also accepts print 'foo'.
- For a cross-platform perspective that includes Windows, see kxr's helpful answer.
In bash, ksh, or zsh:
Use an ANSI C-quoted string ($'...'), which allows using \n to represent newlines that are expanded to actual newlines before the string is passed to python:
python -c $'import sys\nfor r in range(10): print("rob")'
Note the \n between the import and for statements to effect a line break.
To pass shell-variable values to such a command, it is safest to use arguments and access them via sys.argv inside the Python script:
name='rob' # value to pass to the Python script
python -c $'import sys\nfor r in range(10): print(sys.argv[1])' "$name"
See below for a discussion of the pros and cons of using an (escape sequence-preprocessed) double-quoted command string with embedded shell-variable references.
To work safely with $'...' strings:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by $'...' (see man printf for the supported escapes)
Escape ' instances as \'.
If you must remain POSIX-compliant:
Use printf with a command substitution:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob")')"
To work safely with this type of string:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by printf (see man printf for the supported escape sequences).
Pass a single-quoted string to printf %b and escape embedded single quotes as '\'' (sic).
Using single quotes protects the string's contents from interpretation by the shell.
That said, for short Python scripts (as in this case) you can use a double-quoted string to incorporate shell variable values into your scripts - as long as you're aware of the associated pitfalls (see next point); e.g., the shell expands $HOME to the current user's home dir. in the following command:
python -c "$(printf %b "import sys\nfor r in range(10): print('rob is $HOME')")"
However, the generally preferred approach is to pass values from the shell via arguments, and access them via sys.argv in Python; the equivalent of the above command is:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob is " + sys.argv[1])')" "$HOME"
While using a double-quoted string is more convenient - it allows you to use embedded single quotes unescaped and embedded double quotes as \" - it also makes the string subject to interpretation by the shell, which may or may not be the intent; $ and ` characters in your source code that are not meant for the shell may cause a syntax error or alter the string unexpectedly.
Additionally, the shell's own \ processing in double-quoted strings can get in the way; for instance, to get Python to produce literal output ro\b, you must pass ro\\b to it; with a '...' shell string and doubled \ instances, we get:
python -c "$(printf %b 'import sys\nprint("ro\\\\bs")')" # ok: 'ro\bs'
By contrast, this does not work as intended with a "..." shell string:
python -c "$(printf %b "import sys\nprint('ro\\\\bs')")" # !! INCORRECT: 'rs'
The shell interprets both "\b" and "\\b" as literal \b, requiring a dizzying number of additional \ instances to achieve the desired effect:
python -c "$(printf %b "import sys\nprint('ro\\\\\\\\bs')")"
To pass the code via stdin rather than -c:
Note: I'm focusing on single-line solutions here; xorho's answer shows how to use a multi-line here-document - be sure to quote the delimiter, however; e.g., <<'EOF', unless you explicitly want the shell to expand the string up front (which comes with the caveats noted above).
In bash, ksh, or zsh:
Combine an ANSI C-quoted string ($'...') with a here-string (<<<...):
python - <<<$'import sys\nfor r in range(10): print("rob")'
- tells python explicitly to read from stdin (which it does by default).
- is optional in this case, but if you also want to pass arguments to the scripts, you do need it to disambiguate the argument from a script filename:
python - 'rob' <<<$'import sys\nfor r in range(10): print(sys.argv[1])'
If you must remain POSIX-compliant:
Use printf as above, but with a pipeline so as to pass its output via stdin:
printf %b 'import sys\nfor r in range(10): print("rob")' | python
With an argument:
printf %b 'import sys\nfor r in range(10): print(sys.argv[1])' | python - 'rob'
Any idea how this can be fixed?
Your problem is created by the fact that Python statements, separated by ;, are only allowed to be "small statements", which are all one-liners. From the grammar file in the Python docs:
stmt: simple_stmt | compound_stmt
simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
Compound statements can't be included on the same line with other statements via semicolons - so doing this with the -c flag becomes very inconvenient.
When demonstrating Python while in a bash shell environment, I find it very useful to include compound statements. The only simple way of doing this reliably is with heredocs (a posix shell thing).
Heredocs
Use a heredoc (created with <<) and Python's command line interface option, -:
$ python - <<-"EOF"
import sys # 1 tab indent
for r in range(10): # 1 tab indent
print('rob') # 1 tab indent and 4 spaces
EOF
Adding the - after << (the <<-) allows you to use tabs to indent (Stackoverflow converts tabs to spaces, so I've indented 8 spaces to emphasize this). The leading tabs will be stripped.
You can do it without the tabs with just <<:
$ python - << "EOF"
import sys
for r in range(10):
print('rob')
EOF
Putting quotes around EOF prevents parameter and arithmetic expansion. This makes the heredoc more robust.
Bash multiline strings
If you use double-quotes, you'll get shell-expansion:
$ python -c "
> import sys
> for p in '$PATH'.split(':'):
> print(p)
> "
/usr/sbin
/usr/bin
/sbin
/bin
...
To avoid shell expansion use single-quotes:
$ python -c '
> import sys
> for p in "$PATH".split(":"):
> print(p)
> '
$PATH
Note that we need to swap the quote characters on the literals in Python - we basically can't use quote character being interpreted by BASH. We can alternate them though, like we can in Python - but this already looks quite confusing, which is why I don't recommend this:
$ python -c '
import sys
for p in "'"$PATH"'".split(":"):
print(p)
'
/usr/sbin
/usr/bin
/sbin
/bin
...
Critique of the accepted answer (and others)
This is not very readable:
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Not very readable, and additionally difficult to debug in the case of an error:
python -c "exec(\"import sys\\nfor r in range(10): print 'rob'\")"
Perhaps a bit more readable, but still quite ugly:
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
You'll have a bad time if you have "'s in your python:
$ python -c "import sys
> for r in range(10): print 'rob'"
Don't abuse map or list comprehensions to get for-loops:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
These are all sad and bad. Don't do them.
Just use Return and type it on the next line:
python -c "import sys
for r in range(10): print 'rob'"
rob
rob
...
python2.6 -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
works fine.
Use "[ ]" to inline your for loop.
The problem is not with the import statement. The problem is that the control flow statements don't work inlined in a Python interpreter command. Replace that import statement with any other statement, and you'll see the same problem.
Think about it: Python can't possibly inline everything. It uses indentation to group control-flow.
If your system is POSIX.2-compliant it should supply the printf utility:
printf "print 'zap'\nfor r in range(3): print 'rob'" | python
zap
rob
rob
rob
This variant is most portable for putting multi-line scripts on the command-line on Windows and Unix-like systems, Python 2 and Python 3, without pipes:
python -c "exec(\"import sys \nfor r in range(10): print('rob') \")"
(None of the other examples seen here so far did so.)
Neat on Windows is:
python -c exec"""import sys \nfor r in range(10): print 'rob' """
python -c exec("""import sys \nfor r in range(10): print('rob') """)
Neat on Bash on Unix-like systems is:
python -c $'import sys \nfor r in range(10): print("rob")'
This function turns any multiline-script into a portable command-one-liner:
def py2cmdline(script):
exs = 'exec(%r)' % re.sub('\r\n|\r', '\n', script.rstrip())
print('python -c "%s"' % exs.replace('"', r'\"'))
Usage:
>>> py2cmdline(getcliptext())
python -c "exec('print \'AA\tA\'\ntry:\n for i in 1, 2, 3:\n print i / 0\nexcept:\n print \"\"\"longer\nmessage\"\"\"')"
The input was:
print 'AA A'
try:
for i in 1, 2, 3:
print i / 0
except:
print """longer
message"""
I'm not really a big Pythoner - but I found this syntax once, forgot where from, so I thought I'd document it:
If you use sys.stdout.write instead of print (the difference being, sys.stdout.write takes arguments as a function, in parentheses - whereas print doesn't), then for a one-liner, you can get away with inverting the order of the command and the for, removing the semicolon, and enclosing the command in square brackets, i.e.:
python -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
I have no idea how this syntax would be called in Python :)
These square brackets in one-liners are "list comprehensions"; note this in Python 2.7:
STR=abc
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); print a"
Output:
<generator object <genexpr> at 0xb771461c>
So the command in round brackets/parenthesis is seen as a "generator object"; if we "iterate" through it by calling next() - then the command inside the parenthesis will be executed (note the "abc" in the output):
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); a.next() ; print a"
Output:
abc
<generator object <genexpr> at 0xb777b734>
If we now use square brackets - note that we don't need to call next() to have the command execute, it executes immediately upon assignment; however, later inspection reveals that a is None:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; print a"
Output:
abc
[None]
This doesn't leave much information to look for - for the square brackets case - but I stumbled upon this page which I think explains:
Python Tips And Tricks – First Edition - Python Tutorials | Dream.In.Code:
If you recall, the standard format of a single line generator is a kind of one line 'for' loop inside brackets. This will produce a 'one-shot' iterable object which is an object you can iterate over in only one direction and which you can't re-use once you reach the end.
A 'list comprehension' looks almost the same as a regular one-line generator, except that the regular brackets - ( ) - are replaced by square brackets - [ ]. The major advantage of alist comprehension is that produces a 'list', rather than a 'one-shot' iterable object, so that you can go back and forth through it, add elements, sort, etc.
And indeed it is a list - it's just its first element becomes none as soon as it is executed:
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin].__class__"
Output:
abc
<type 'list'>
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin][0]"
Output:
abc
None
List comprehensions are otherwise documented in 5. Data Structures: 5.1.4. List Comprehensions — Python v2.7.4 documentation as "List comprehensions provide a concise way to create lists"; presumably, that's where the limited "executability" of lists comes into play in one-liners.
Well, hope I'm not terribly too off the mark here ...
And here is a one-liner command line with two non-nested for-loops; both enclosed within "list comprehension" square brackets:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; b=[sys.stdout.write(str(x)) for x in range(2)] ; print a ; print b"
Output:
abc
01[None]
[None, None]
Notice that the second "list" b now has two elements, since its for loop explicitly ran twice; however, the result of sys.stdout.write() in both cases was (apparently) None.
single/double quotes and backslash everywhere:
python -c 'exec("import sys\nfor i in range(10): print \"bob\"")'
Much better:
python -c '
import sys
for i in range(10):
print("bob")
'
Note: On some systems executable python is for Python 2 and may not be available. And executable python3 is for Python 3.
This script provides a Perl-like command line interface:
Pyliner - Script to run arbitrary Python code on the command line (Python recipe)
I wanted a solution with the following properties:
Readable
Read standard input for processing output of other tools
Both requirements were not provided in the other answers, so here's how to read standard input while doing everything on the command line:
grep special_string -r | sort | python3 <(cat <<EOF
import sys
for line in sys.stdin:
tokens = line.split()
if len(tokens) == 4:
print("%-45s %7.3f %s %s" % (tokens[0], float(tokens[1]), tokens[2], tokens[3]))
EOF
)
When I needed to do this, I used:
python -c "$(echo -e "import sys\nsys.stdout.write('Hello, World!\\\n')")"
Note the triple backslash for the newline in the sys.stdout.write statement.
If you don't want to touch standard input and simulate as if you had passed "python cmdfile.py", you can do the following from a Bash shell:
python <(printf "word=raw_input('Enter word: ')\nimport sys\nfor i in range(5):\n print(word)")
As you can see, it allows you to use standard input for reading input data. Internally the shell creates the temporary file for the input command contents.
Use python -c with triple quotes:
python -c """
import os
os.system('pwd')
os.system('ls -l')
print('Hello, World!')
for _ in range(5):
print(_)
"""
There is one more option. sys.stdout.write returns None, which keeps the list empty:
cat somefile.log|python -c "import sys;[line for line in sys.stdin if sys.stdout.write(line*2)]"
Say I want to print 1 + 1 on stdout (i.e. one-liner coding).
With awk I could do it simply with:
$ echo | awk '{print 1+1}'
2
How to do this with python?
you are looking for -c:
$ python -c 'print 1 + 1'
2
As ifLoop pointed out, what you're looking for here is -c.
But, as yo've discovered, python -c often isn't as useful as the corresponding awk (or sed or bash or perl or even ruby) for one-liners.
Python is explicitly designed to value readability over brevity and explicitness over implicitness (along with some correlated tradeoffs, like vocabulary over syntax, as little magic as possible, etc.). See the Zen of Python. There are intentional limits to what you can cram onto one line, and things like looping over stdin and/or command-line args have to be done explicitly with, e.g., sys.stdin and sys.argv, or fileinput.input().
That means that some very trivial scripts become less trivial to write in Python, but that's considered a good tradeoff for making even moderately non-trivial scripts easier to write, maintain, and understand.
The core developers understand this means you can't rewrite a lot of one-liners in Python. And if you asked them, most of them will ask why that's a problem at all.
If you know how to write something as a one-liner in a language like sed or awk, then you should be writing it as a one-liner in sed or awk. Those are perfectly good languages that are handy for all kinds of simple tasks, and there's no reason to avoid them just because Python is also a good language.
If you can't figure your way through the syntax to write that one-liner… well, it probably shouldn't be a one-liner. The only reason you want to try it in Python is that Python is generally easier to write and read, and the same reasons that's true are the same reasons Python won't let you write what you want without 3 lines. So just write the 3 lines. Python is great for that.
So, what you often really want is not -c, but a heredoc, or just a separate script that you run like any other program, or awk or perl instead of python.
Inspired by the answer by IfLoop I wondered about the handy BEGIN and END blocks in awk. I have found the pawk module
ls -l | awk 'BEGIN {c = 0} {c += $5} END {print c}'
ls -l | pawk -s -B 'c = 0' -E 'c' 'c += int(f[4])'
Looks promising, but I have never tried this (yet)
if you're in the shell, the next would basic integer math
echo $((1+1))
echo $(( 100 / 5 ))
etc...
for floating point, yes, you should to use awk, or bc, or dc, or any other language what knows floating point math...
also, read this: https://stackoverflow.com/a/450853/632407
Like SingleNegationElimination's answer recommend's, the -c flag is the right tool for the job.
Here are some examples using awk, ruby, and python:
echo foo bar baz | awk '{ split($0, arr, " "); print arr[NF] }'
baz
echo foo bar baz | ruby -e 'puts STDIN.read.chomp.split(" ")[-1]'
baz
echo foo bar baz | python -c 'import sys; print sys.stdin.read().rstrip().split(" ")[-1]'
baz
I have to fetch the 'string' which is stored in between the pattern '[ ]'. Example is given below
[string]_someOtherString
Length of string to be fetched is not fixed. I would appreciate if you can help me do this. I dont want to use the cut command multiple times. I want to see if there is a simple and elegant way. Its good if you can suggest me multiple ways to solve this(helps me learn and compare different techniques).
One way would be to use grep:
$ grep -oP '(?<=\[).*(?=\])' <<< '[string]_someOtherString'
string
using python you can do something like:
import re
stg = '[string]_someOtherString'
print re.findall('\[([^\[\]]+)\]', stg)
or in the shell:
python -c "import re, sys; print re.findall('\[([^\[\]]+)\]', sys.argv[1])" '[string]_someOtherString'
and if you know that you only have one mathcing string:
python -c "import re, sys; print re.findall('\[([^\[\]]+)\]', sys.argv[1])[0]" '[string]_someOtherString'
I'm using Python with -c to execute a one-liner loop, i.e.:
python -c "for r in range(10): print 'rob'"
This works fine. However, if I import a module before the for loop, I get a syntax error:
python -c "import sys; for r in range(10): print 'rob'"
File "<string>", line 1
import sys; for r in range(10): print 'rob'
^
SyntaxError: invalid syntax
How can this be fixed?
It's important to me to have this as a one-liner so that I can include it in a Makefile.
You could do
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Or without pipes:
python -c "exec(\"import sys\nfor r in range(10): print 'rob'\")"
Or
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
Or SilentGhost's answer or Crast's answer.
This style can be used in makefiles too (and in fact it is used quite often).
python - <<EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
Or with hard tabs:
python - <<-EOF
import random
for r in range(3): print(random.randint(1, 42))
EOF
# Important: Replace the indentation above w/ hard tabs.
In above case, leading TAB characters are removed too (and some structured outlook can be achieved).
Instead of EOF can stand any marker word not appearing in the here document at a beginning of a line (see also here documents in the bash man page or here).
The issue is not actually with the import statement. It's with anything being before the for loop. Or more specifically, anything appearing before an inlined block.
For example, these all work:
python -c "import sys; print 'rob'"
python -c "import sys; sys.stdout.write('rob\n')"
If import being a statement were an issue, this would work, but it doesn't:
python -c "__import__('sys'); for r in range(10): print 'rob'"
For your very basic example, you could rewrite it as this:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
However, lambdas can only execute expressions, not statements or multiple statements, so you may still be unable to do the thing you want to do. However, between generator expressions, list comprehension, lambdas, sys.stdout.write, the "map" builtin, and some creative string interpolation, you can do some powerful one-liners.
The question is, how far do you want to go, and at what point is it not better to write a small .py file which your makefile executes instead?
- To make this answer work with Python 3.x as well, print is called as a function: in 3.x, only print('foo') works, whereas 2.x also accepts print 'foo'.
- For a cross-platform perspective that includes Windows, see kxr's helpful answer.
In bash, ksh, or zsh:
Use an ANSI C-quoted string ($'...'), which allows using \n to represent newlines that are expanded to actual newlines before the string is passed to python:
python -c $'import sys\nfor r in range(10): print("rob")'
Note the \n between the import and for statements to effect a line break.
To pass shell-variable values to such a command, it is safest to use arguments and access them via sys.argv inside the Python script:
name='rob' # value to pass to the Python script
python -c $'import sys\nfor r in range(10): print(sys.argv[1])' "$name"
See below for a discussion of the pros and cons of using an (escape sequence-preprocessed) double-quoted command string with embedded shell-variable references.
To work safely with $'...' strings:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by $'...' (see man printf for the supported escapes)
Escape ' instances as \'.
If you must remain POSIX-compliant:
Use printf with a command substitution:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob")')"
To work safely with this type of string:
Double \ instances in your original source code.
\<char> sequences - such as \n in this case, but also the usual suspects such as \t, \r, \b - are expanded by printf (see man printf for the supported escape sequences).
Pass a single-quoted string to printf %b and escape embedded single quotes as '\'' (sic).
Using single quotes protects the string's contents from interpretation by the shell.
That said, for short Python scripts (as in this case) you can use a double-quoted string to incorporate shell variable values into your scripts - as long as you're aware of the associated pitfalls (see next point); e.g., the shell expands $HOME to the current user's home dir. in the following command:
python -c "$(printf %b "import sys\nfor r in range(10): print('rob is $HOME')")"
However, the generally preferred approach is to pass values from the shell via arguments, and access them via sys.argv in Python; the equivalent of the above command is:
python -c "$(printf %b 'import sys\nfor r in range(10): print("rob is " + sys.argv[1])')" "$HOME"
While using a double-quoted string is more convenient - it allows you to use embedded single quotes unescaped and embedded double quotes as \" - it also makes the string subject to interpretation by the shell, which may or may not be the intent; $ and ` characters in your source code that are not meant for the shell may cause a syntax error or alter the string unexpectedly.
Additionally, the shell's own \ processing in double-quoted strings can get in the way; for instance, to get Python to produce literal output ro\b, you must pass ro\\b to it; with a '...' shell string and doubled \ instances, we get:
python -c "$(printf %b 'import sys\nprint("ro\\\\bs")')" # ok: 'ro\bs'
By contrast, this does not work as intended with a "..." shell string:
python -c "$(printf %b "import sys\nprint('ro\\\\bs')")" # !! INCORRECT: 'rs'
The shell interprets both "\b" and "\\b" as literal \b, requiring a dizzying number of additional \ instances to achieve the desired effect:
python -c "$(printf %b "import sys\nprint('ro\\\\\\\\bs')")"
To pass the code via stdin rather than -c:
Note: I'm focusing on single-line solutions here; xorho's answer shows how to use a multi-line here-document - be sure to quote the delimiter, however; e.g., <<'EOF', unless you explicitly want the shell to expand the string up front (which comes with the caveats noted above).
In bash, ksh, or zsh:
Combine an ANSI C-quoted string ($'...') with a here-string (<<<...):
python - <<<$'import sys\nfor r in range(10): print("rob")'
- tells python explicitly to read from stdin (which it does by default).
- is optional in this case, but if you also want to pass arguments to the scripts, you do need it to disambiguate the argument from a script filename:
python - 'rob' <<<$'import sys\nfor r in range(10): print(sys.argv[1])'
If you must remain POSIX-compliant:
Use printf as above, but with a pipeline so as to pass its output via stdin:
printf %b 'import sys\nfor r in range(10): print("rob")' | python
With an argument:
printf %b 'import sys\nfor r in range(10): print(sys.argv[1])' | python - 'rob'
Any idea how this can be fixed?
Your problem is created by the fact that Python statements, separated by ;, are only allowed to be "small statements", which are all one-liners. From the grammar file in the Python docs:
stmt: simple_stmt | compound_stmt
simple_stmt: small_stmt (';' small_stmt)* [';'] NEWLINE
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
import_stmt | global_stmt | nonlocal_stmt | assert_stmt)
Compound statements can't be included on the same line with other statements via semicolons - so doing this with the -c flag becomes very inconvenient.
When demonstrating Python while in a bash shell environment, I find it very useful to include compound statements. The only simple way of doing this reliably is with heredocs (a posix shell thing).
Heredocs
Use a heredoc (created with <<) and Python's command line interface option, -:
$ python - <<-"EOF"
import sys # 1 tab indent
for r in range(10): # 1 tab indent
print('rob') # 1 tab indent and 4 spaces
EOF
Adding the - after << (the <<-) allows you to use tabs to indent (Stackoverflow converts tabs to spaces, so I've indented 8 spaces to emphasize this). The leading tabs will be stripped.
You can do it without the tabs with just <<:
$ python - << "EOF"
import sys
for r in range(10):
print('rob')
EOF
Putting quotes around EOF prevents parameter and arithmetic expansion. This makes the heredoc more robust.
Bash multiline strings
If you use double-quotes, you'll get shell-expansion:
$ python -c "
> import sys
> for p in '$PATH'.split(':'):
> print(p)
> "
/usr/sbin
/usr/bin
/sbin
/bin
...
To avoid shell expansion use single-quotes:
$ python -c '
> import sys
> for p in "$PATH".split(":"):
> print(p)
> '
$PATH
Note that we need to swap the quote characters on the literals in Python - we basically can't use quote character being interpreted by BASH. We can alternate them though, like we can in Python - but this already looks quite confusing, which is why I don't recommend this:
$ python -c '
import sys
for p in "'"$PATH"'".split(":"):
print(p)
'
/usr/sbin
/usr/bin
/sbin
/bin
...
Critique of the accepted answer (and others)
This is not very readable:
echo -e "import sys\nfor r in range(10): print 'rob'" | python
Not very readable, and additionally difficult to debug in the case of an error:
python -c "exec(\"import sys\\nfor r in range(10): print 'rob'\")"
Perhaps a bit more readable, but still quite ugly:
(echo "import sys" ; echo "for r in range(10): print 'rob'") | python
You'll have a bad time if you have "'s in your python:
$ python -c "import sys
> for r in range(10): print 'rob'"
Don't abuse map or list comprehensions to get for-loops:
python -c "import sys; map(lambda x: sys.stdout.write('rob%d\n' % x), range(10))"
These are all sad and bad. Don't do them.
Just use Return and type it on the next line:
python -c "import sys
for r in range(10): print 'rob'"
rob
rob
...
python2.6 -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
works fine.
Use "[ ]" to inline your for loop.
The problem is not with the import statement. The problem is that the control flow statements don't work inlined in a Python interpreter command. Replace that import statement with any other statement, and you'll see the same problem.
Think about it: Python can't possibly inline everything. It uses indentation to group control-flow.
If your system is POSIX.2-compliant it should supply the printf utility:
printf "print 'zap'\nfor r in range(3): print 'rob'" | python
zap
rob
rob
rob
This variant is most portable for putting multi-line scripts on the command-line on Windows and Unix-like systems, Python 2 and Python 3, without pipes:
python -c "exec(\"import sys \nfor r in range(10): print('rob') \")"
(None of the other examples seen here so far did so.)
Neat on Windows is:
python -c exec"""import sys \nfor r in range(10): print 'rob' """
python -c exec("""import sys \nfor r in range(10): print('rob') """)
Neat on Bash on Unix-like systems is:
python -c $'import sys \nfor r in range(10): print("rob")'
This function turns any multiline-script into a portable command-one-liner:
def py2cmdline(script):
exs = 'exec(%r)' % re.sub('\r\n|\r', '\n', script.rstrip())
print('python -c "%s"' % exs.replace('"', r'\"'))
Usage:
>>> py2cmdline(getcliptext())
python -c "exec('print \'AA\tA\'\ntry:\n for i in 1, 2, 3:\n print i / 0\nexcept:\n print \"\"\"longer\nmessage\"\"\"')"
The input was:
print 'AA A'
try:
for i in 1, 2, 3:
print i / 0
except:
print """longer
message"""
I'm not really a big Pythoner - but I found this syntax once, forgot where from, so I thought I'd document it:
If you use sys.stdout.write instead of print (the difference being, sys.stdout.write takes arguments as a function, in parentheses - whereas print doesn't), then for a one-liner, you can get away with inverting the order of the command and the for, removing the semicolon, and enclosing the command in square brackets, i.e.:
python -c "import sys; [sys.stdout.write('rob\n') for r in range(10)]"
I have no idea how this syntax would be called in Python :)
These square brackets in one-liners are "list comprehensions"; note this in Python 2.7:
STR=abc
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); print a"
Output:
<generator object <genexpr> at 0xb771461c>
So the command in round brackets/parenthesis is seen as a "generator object"; if we "iterate" through it by calling next() - then the command inside the parenthesis will be executed (note the "abc" in the output):
echo $STR | python -c "import sys,re; a=(sys.stdout.write(line) for line in sys.stdin); a.next() ; print a"
Output:
abc
<generator object <genexpr> at 0xb777b734>
If we now use square brackets - note that we don't need to call next() to have the command execute, it executes immediately upon assignment; however, later inspection reveals that a is None:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; print a"
Output:
abc
[None]
This doesn't leave much information to look for - for the square brackets case - but I stumbled upon this page which I think explains:
Python Tips And Tricks – First Edition - Python Tutorials | Dream.In.Code:
If you recall, the standard format of a single line generator is a kind of one line 'for' loop inside brackets. This will produce a 'one-shot' iterable object which is an object you can iterate over in only one direction and which you can't re-use once you reach the end.
A 'list comprehension' looks almost the same as a regular one-line generator, except that the regular brackets - ( ) - are replaced by square brackets - [ ]. The major advantage of alist comprehension is that produces a 'list', rather than a 'one-shot' iterable object, so that you can go back and forth through it, add elements, sort, etc.
And indeed it is a list - it's just its first element becomes none as soon as it is executed:
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin].__class__"
Output:
abc
<type 'list'>
echo $STR | python -c "import sys,re; print [sys.stdout.write(line) for line in sys.stdin][0]"
Output:
abc
None
List comprehensions are otherwise documented in 5. Data Structures: 5.1.4. List Comprehensions — Python v2.7.4 documentation as "List comprehensions provide a concise way to create lists"; presumably, that's where the limited "executability" of lists comes into play in one-liners.
Well, hope I'm not terribly too off the mark here ...
And here is a one-liner command line with two non-nested for-loops; both enclosed within "list comprehension" square brackets:
echo $STR | python -c "import sys,re; a=[sys.stdout.write(line) for line in sys.stdin]; b=[sys.stdout.write(str(x)) for x in range(2)] ; print a ; print b"
Output:
abc
01[None]
[None, None]
Notice that the second "list" b now has two elements, since its for loop explicitly ran twice; however, the result of sys.stdout.write() in both cases was (apparently) None.
single/double quotes and backslash everywhere:
python -c 'exec("import sys\nfor i in range(10): print \"bob\"")'
Much better:
python -c '
import sys
for i in range(10):
print("bob")
'
Note: On some systems executable python is for Python 2 and may not be available. And executable python3 is for Python 3.
This script provides a Perl-like command line interface:
Pyliner - Script to run arbitrary Python code on the command line (Python recipe)
I wanted a solution with the following properties:
Readable
Read standard input for processing output of other tools
Both requirements were not provided in the other answers, so here's how to read standard input while doing everything on the command line:
grep special_string -r | sort | python3 <(cat <<EOF
import sys
for line in sys.stdin:
tokens = line.split()
if len(tokens) == 4:
print("%-45s %7.3f %s %s" % (tokens[0], float(tokens[1]), tokens[2], tokens[3]))
EOF
)
When I needed to do this, I used:
python -c "$(echo -e "import sys\nsys.stdout.write('Hello, World!\\\n')")"
Note the triple backslash for the newline in the sys.stdout.write statement.
If you don't want to touch standard input and simulate as if you had passed "python cmdfile.py", you can do the following from a Bash shell:
python <(printf "word=raw_input('Enter word: ')\nimport sys\nfor i in range(5):\n print(word)")
As you can see, it allows you to use standard input for reading input data. Internally the shell creates the temporary file for the input command contents.
Use python -c with triple quotes:
python -c """
import os
os.system('pwd')
os.system('ls -l')
print('Hello, World!')
for _ in range(5):
print(_)
"""
There is one more option. sys.stdout.write returns None, which keeps the list empty:
cat somefile.log|python -c "import sys;[line for line in sys.stdin if sys.stdout.write(line*2)]"