This question already has answers here:
How can I store a command in a variable in a shell script?
(12 answers)
Closed 4 years ago.
These work as advertised:
grep -ir 'hello world' .
grep -ir hello\ world .
These don't:
argumentString1="-ir 'hello world'"
argumentString2="-ir hello\\ world"
grep $argumentString1 .
grep $argumentString2 .
Despite 'hello world' being enclosed by quotes in the second example, grep interprets 'hello (and hello\) as one argument and world' (and world) as another, which means that, in this case, 'hello will be the search pattern and world' will be the search path.
Again, this only happens when the arguments are expanded from the argumentString variables. grep properly interprets 'hello world' (and hello\ world) as a single argument in the first example.
Can anyone explain why this is? Is there a proper way to expand a string variable that will preserve the syntax of each character such that it is correctly interpreted by shell commands?
Why
When the string is expanded, it is split into words, but it is not re-evaluated to find special characters such as quotes or dollar signs or ... This is the way the shell has 'always' behaved, since the Bourne shell back in 1978 or thereabouts.
Fix
In bash, use an array to hold the arguments:
argumentArray=(-ir 'hello world')
grep "${argumentArray[#]}" .
Or, if brave/foolhardy, use eval:
argumentString="-ir 'hello world'"
eval "grep $argumentString ."
On the other hand, discretion is often the better part of valour, and working with eval is a place where discretion is better than bravery. If you are not completely in control of the string that is eval'd (if there's any user input in the command string that has not been rigorously validated), then you are opening yourself to potentially serious problems.
Note that the sequence of expansions for Bash is described in Shell Expansions in the GNU Bash manual. Note in particular sections 3.5.3 Shell Parameter Expansion, 3.5.7 Word Splitting, and 3.5.9 Quote Removal.
When you put quote characters into variables, they just become plain literals (see http://mywiki.wooledge.org/BashFAQ/050; thanks #tripleee for pointing out this link)
Instead, try using an array to pass your arguments:
argumentString=(-ir 'hello world')
grep "${argumentString[#]}" .
In looking at this and related questions, I'm surprised that no one brought up using an explicit subshell. For bash, and other modern shells, you can execute a command line explicitly. In bash, it requires the -c option.
argumentString="-ir 'hello world'"
bash -c "grep $argumentString ."
Works exactly as original questioner desired. There are two restrictions to this technique:
You can only use single quotes within the command or argument strings.
Only exported environment variables will be available to the command
Also, this technique handles redirection and piping, and other shellisms work as well. You also can use bash internal commands as well as any other command that works at the command line, because you are essentially asking a subshell bash to interpret it directly as a command line. Here's a more complex example, a somewhat gratuitously complex ls -l variant.
cmd="prefix=`pwd` && ls | xargs -n 1 echo \'In $prefix:\'"
bash -c "$cmd"
I have built command processors both this way and with parameter arrays. Generally, this way is much easier to write and debug, and it's trivial to echo the command you are executing. OTOH, param arrays work nicely when you really do have abstract arrays of parameters, as opposed to just wanting a simple command variant.
Related
This question already has answers here:
What's the difference between "here string" and echo + pipe
(1 answer)
Redirector "<<<" in Ubuntu?
(3 answers)
Closed 3 months ago.
Hello guys i wanna write a Shell script that runs Python code saved in variable called $code.
So i save the script in variable $code with this command:
$ export CODE='print("Hello world")'
To resolve the problem
I write the following script in a file called run:
#!/bin/bash
echo "$CODE" > main.py
python3 main.py
To running the shell script i use:
./run
and its work but
I found another answer which I don't understand:
python3 <<< $CODE
so what do we mean by using <<<?
In a lot of shells <<< denotes a here string and is a way to pass standard input to commands. <<< is used for strings, e.g.
$ python3 <<< 'print("hi there")'
hi there
It passes the word on the right to the standard input of the command on the left.
whereas << denotes a here document, e.g.
command <<MultiLineDoc
Standard Input
That
Streches many
Lines and preserves
indentation and
linebreaks
which is useful for passing many arguments to a command,
e.g. passing text to a program and preserving its indentation.
The beginning and ending _MultiLineDoc_ delimiter can be named any way wanted,
it can be considered the name of the document.
Important is that it repeats identically at
both beginning and end and nowhere else in the
document, everything between that delimiter is passed.
MultiLineDoc
< is used for passing the contents of a file, e.g. command < filename.txt
As for your example with <<< :
You could do the same with | but that's only OK if all your variables are defined in what you are passing. If you do have other variables that you have defined in your environment and which you wish to cross reference you would use a here-string as in your example, that lets you reference other variables within the content you are passing.
Please see: https://en.wikipedia.org/wiki/Here_document
https://linuxhint.com/bash-heredoc-tutorial/
In Bash, zsh (and some other shells) <<< is the here string operator. The code you’ve posted is roughly equivalent to
echo "$PYCODE" | python3
In bash (as started by Python), I want to print this string \033[31m so that I can use a pipe | operator after it, followed by a command to copy that string to the clipboard. This means that in practice, I'm trying to run something like:
os.system('echo \\033[31m | xsel -ib')
...but the xsel -ib part is working fine, so this question is focused specifically on the behavior of echo.
Most of my attempts have been similar to:
echo -e \\033[31m
I have tried it with single quotes, double quotes, no quotes, removing the -e flag, etc. The closest I got was:
echo -n "\\ 033[31m"
which prints this string \ 033[31m
I don't want that space between \ and 0
-n flag is used to not append a new line after the printed string
I use Ubuntu 20.04, and xsel is a selection and clipboard manipulation tool for the X11 Window System (which Ubuntu 20.04 uses).
echo is the wrong tool for the job. It's a shell builtin, and one for which the POSIX sh standard explicitly does not guarantee portable behavior for when escape sequences (such as \033) are present. system() starts /bin/sh instead of bash, so POSIX behavior -- not that of your regular interactive shell -- is expected.
Use subprocess.run() instead of os.system(), and you don't need echo in the first place.
If you want to put an escape sequence into the clipboard (so not \033 but instead the ESC key that this gets converted to by an echo with XSI extensions to POSIX):
# to store \033 as a single escape character, use a regular Python bytestring
subprocess.run(['xsel', '-ib'], input=b'\033[31m')
If you want to put the literal text without being interpreted (so there's an actual backslash and an actual zero), use a raw bytestring instead:
# to store \033 as four separate characters, use a raw string
subprocess.run(['xsel', '-ib'], input=rb'\033[31m')
For a more detailed description of why echo causes problems in this context, see the excellent answer by Stephane to the Unix & Linux Stack Exchange question Why is printf better than echo?.
If you for some reason do want to keep using a shell pipeline, switch to printf instead:
# to store \033 as four separate characters, use %s
subprocess.run(r''' printf '%s\n' '\033[31m' | xsel -ib ''', shell=True)
# to store \033 as a single escape character, use %b
subprocess.run(r''' printf '%b\n' '\033[31m' | xsel -ib ''', shell=True)
This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 1 year ago.
When trying to run the tasklist command with grep by using subprocess:
command = ("tasklist | grep edpa.exe | gawk \"{ print $2 }\"")
p = subprocess.Popen(command, stdout=subprocess.PIPE)
text = p.communicate(timeout=600)[0]
print(text)
I get this error:
ERROR: Invalid argument/option - '|'.
Type "TASKLIST /?" for usage.
It works fine when i run the command directly from cmd, but when using subprocess something goes wrong.
How can it be fixed? I need to use the output of the command so i can not use os.system
.
Two options:
Use the shell=True option of the Popen(); this will pass it through the shell, which is the part that interprets things like the |
Just run tasklist in the Popen(), then do the processing in Python rather than invoking grep and awk
Of the two, the latter is probably the better approach in this particular instance, since these grep and awk commands are easily translated into Python.
Your linters may also complain that shell=True is prone to security issues, although this particular usage would be OK.
In the absence of shell=True, subprocess runs a single subprocess. In other words, you are passing | and grep etc as arguments to tasklist.
The simplest fix is to add shell=True; but a much better fix is to do the trivial text processing in Python instead. This also coincidentally gets rid of the useless grep.
for line in subprocess.check_output(['tasklist'], timeout=600).splitlines():
if 'edpa.exe' in line:
text = line.split()[1]
print(text)
I have assumed you really want to match edpa.exe literally, anywhere in the output line; your regex would match edpa followed by any character followed by exe. The code could be improved by doing the split first and then look for the search string only in the process name field (if that is indeed your intent).
Perhaps notice also how you generally want to avoid the low-level Popen whenever you can use one of the higher-level functions.
This question already has answers here:
How can I store a command in a variable in a shell script?
(12 answers)
Closed 4 years ago.
These work as advertised:
grep -ir 'hello world' .
grep -ir hello\ world .
These don't:
argumentString1="-ir 'hello world'"
argumentString2="-ir hello\\ world"
grep $argumentString1 .
grep $argumentString2 .
Despite 'hello world' being enclosed by quotes in the second example, grep interprets 'hello (and hello\) as one argument and world' (and world) as another, which means that, in this case, 'hello will be the search pattern and world' will be the search path.
Again, this only happens when the arguments are expanded from the argumentString variables. grep properly interprets 'hello world' (and hello\ world) as a single argument in the first example.
Can anyone explain why this is? Is there a proper way to expand a string variable that will preserve the syntax of each character such that it is correctly interpreted by shell commands?
Why
When the string is expanded, it is split into words, but it is not re-evaluated to find special characters such as quotes or dollar signs or ... This is the way the shell has 'always' behaved, since the Bourne shell back in 1978 or thereabouts.
Fix
In bash, use an array to hold the arguments:
argumentArray=(-ir 'hello world')
grep "${argumentArray[#]}" .
Or, if brave/foolhardy, use eval:
argumentString="-ir 'hello world'"
eval "grep $argumentString ."
On the other hand, discretion is often the better part of valour, and working with eval is a place where discretion is better than bravery. If you are not completely in control of the string that is eval'd (if there's any user input in the command string that has not been rigorously validated), then you are opening yourself to potentially serious problems.
Note that the sequence of expansions for Bash is described in Shell Expansions in the GNU Bash manual. Note in particular sections 3.5.3 Shell Parameter Expansion, 3.5.7 Word Splitting, and 3.5.9 Quote Removal.
When you put quote characters into variables, they just become plain literals (see http://mywiki.wooledge.org/BashFAQ/050; thanks #tripleee for pointing out this link)
Instead, try using an array to pass your arguments:
argumentString=(-ir 'hello world')
grep "${argumentString[#]}" .
In looking at this and related questions, I'm surprised that no one brought up using an explicit subshell. For bash, and other modern shells, you can execute a command line explicitly. In bash, it requires the -c option.
argumentString="-ir 'hello world'"
bash -c "grep $argumentString ."
Works exactly as original questioner desired. There are two restrictions to this technique:
You can only use single quotes within the command or argument strings.
Only exported environment variables will be available to the command
Also, this technique handles redirection and piping, and other shellisms work as well. You also can use bash internal commands as well as any other command that works at the command line, because you are essentially asking a subshell bash to interpret it directly as a command line. Here's a more complex example, a somewhat gratuitously complex ls -l variant.
cmd="prefix=`pwd` && ls | xargs -n 1 echo \'In $prefix:\'"
bash -c "$cmd"
I have built command processors both this way and with parameter arrays. Generally, this way is much easier to write and debug, and it's trivial to echo the command you are executing. OTOH, param arrays work nicely when you really do have abstract arrays of parameters, as opposed to just wanting a simple command variant.
I'm trying to understand one of the answers to this question:
Cannot pass an argument to python with "#!/usr/bin/env python"
#!/bin/sh
''''exec python -u -- "$0" ${1+"$#"} # '''
This works well, but I do not understand why it works with four ticks at the beginning of that line rather than three.
In addition, why the hash near the end of that string?
Python supports triple-quoted strings:
'''something'''
Shell supports only single-quoted strings:
'something'
By using four quotes, sh sees that as 2 empty strings, but Python sees the first three as the start of a triple-quoted string, and includes the fourth as part of the string value.
The rest of the line is then interpreted as a command by sh, but as part of a string by Python.
The # then forms a comment as far as sh is concerned, but it is still a string to Python, closing it off with a closing triple-quote.
So, to summarize:
sh sees: empty string ('') - empty string ('') - command (exec python -u -- "$0" ${1+"$#"}) - comment (# ''')
Python sees: triple-quoted string literal (containing the characters 'exec python -u -- "$0" ${1+"$#"} #)
So sh executes that command, replacing itself with the python -u -- with the script name and the rest of the command line arguments, and Python reads this file and just sees an initial string literal that isn't going anywhere.
Because it is the first string literal in the file, it'll be set as the docstring for the __main__ module but that is hardly going to matter if this is the main script.
I just use:
#!/bin/sh
''':'
exec python -tt "$0" "$#"
'''
# The above shell shabang trick is more portable than /usr/bin/env and supports adding arguments to the interpreter (python -tt)