Quote of the script:
!/bin/sh
## repo default configuration
##
REPO_URL='https://android.googlesource.com/tools/repo'
REPO_REV='stable'
...
magic='--calling-python-from-/bin/sh--'
"""exec" python -E "$0" "$#" """#$magic"
if __name__ == '__main__':
import sys
if sys.argv[-1] == '#%s' % magic:
del sys.argv[-1]
del magic
..all python from here on..
How can they put bash and python in a single script and make it run?
This is how I understand it:
The script is first invoked as a shell script and then calls python on line 23, so if you invoked the sync command on repo it would do the following:
"""exec" python -E "$0" "$#" """#$magic"
which I believe turns into:
exec "python -E "repo" "sync" "#--calling-python-from-/bin/sh--"
This then calls the script repo as a python script. You will notice that all of the syntax is legal in bash and python down to the exec line. The triple quotes on line 23 are really sweet, they work in the shell script and then work as a doc string in python. How awesome is that!
In python """ begin and end a docstring which can span multiple lines. Very roughly, it's a very special kind of comment that the python help system can read. In bash, I believe that the double quote is matched to the next double quote the interpreter encounters. So the first two quotes, quote the null string and it disappears from the interpreter. In our example, the next quotes quote the exec keyword. The last set of triple quotes, """#$magic", first quote the null string and then quote the #$magic argument. Since double quotes allow the variable $magic to be expanded, it becomes, #--calling-python-from-/bin/sh-- and passed as an argument.
If you look at the script with a text editor with syntax highlighting as a shell script and then as a python script, it's even easier to see. This is an extremely clever, way of starting the shell script and then giving control to python. Hope this helps!
Related
I'm passing the result of the execution of a command to python as input, like so:
$ python parse_ips.py "$(kubectl get configmap ...)"
This works fine when executing from the command line, however I'm now trying to edit the file using PyCharm. Therefore I need the escaped version of the result of this command which I can paste into PyCharm's debug configuration, as I can't execute the command in real-time like I can do on the command line.
However, I am struggling to find a way to replicate the escaping bash does behind the scenes, so I can use the result as an argument within the PyCharm configuration. Running the above kubectl command results in a multi-line string which includes spaces and quotes. When I paste this into PyCharm it just interprets it as multiple arguments. I'm looking for the escaped result, which I could paste directly into the command line, or into PyCharm's debug configuration, to achieve the same result with a fixed parameter for testing.
Any help would be greatly appreciated!
Edit: To clarify, I mean on the command line the result of the $(kubectl ...) command is passed into the python program as a single command line argument when it is surrounded by quotes ("$(kubectl ...)"). So in the python program, you can access sys.argv[1] and it will contain the entire execution output of $(kubectl get configmap ...). However, if I execute that command myself on the command line, the result is a multi-line string.
If I then copy the result of that into PyCharm (or even on the command line again), it is interpreted as many command line arguments. E.g. it would look something like this:
$ python parse_ips.py apiVersion: v1
data:
item1: ifconfig-push 127.0.0.0 255.255.0.0
item2: ifconfig-push 127.0.0.1 255.255.0.0
item3: ifconfig-push 127.0.0.2 255.255.0.0
...
And so on. This obviously doesn't work in the same way as it did before. So I am unable to test my program without making the kubectl call from the command line each time. I was looking to replicate what "$(kubectl ...)" gets converted into so it is able to pass the entire output as a single command line entry.
I am struggling to find a way to replicate the escaping bash does behind the scenes
Typically use printf "%q" to escape stuff.
printf "%q" "$(kubectl get configmap ....)"
This is printf as the bash builtin command. It differs from coreutils printf, and newest ones also support %q with different quoting style:
/usr/bin/printf "%q" "$(kubectl get configmap ....)"
Modern bash also has quoting expansion:
var="$(kubectl get configmap ....)"
echo "${var#Q}"
And there is also the quoting style outputted by set -x.
I would suggest to use a file:
kubectl get configmap ... > /tmp/tempfile
python parse_ips.py "$(cat /tmp/tempfile)"
With xclip you can copy command output straight to the X server clipboard, which is handy:
printf "%q" "$(kubectl get configmap ...)" | xclip -selection clipboard
# then in another window:
python parse_ips.py <right mouse click><select paste>
How do I approach converting this:
echo -e 'FROM busybox\nRUN echo "hello world"' | docker build -
into sh (https://github.com/amoffat/sh) syntax.
The issue I have is that the args are tokenized and quoted. So docker isn't able to pass stdin onto the end of the command since - has single quotes placed around it. Digging in the docs I'm not seeing a way to support this.
Thanks for any help you can provide (alternatively if you have good examples of doing this in subprocess, that would work.)!
The issue I have is that the args are tokenized and quoted. So docker isn't able to pass stdin onto the end of the command since - has single quotes placed around it.
Like I commented,
- isn’t special shell syntax. It should work fine as a normal argument.
Still, here’s how it looks like you do it, going by the docs. Using _in,
dockerfile = r"""
FROM busybox
RUN echo "hello world"
"""
docker.build("-", _in=dockerfile)
Could someone explain how this shebang works?
#!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/python3/r,$_)
I've seen it posted twice here but not coming from Perl it looks like magic to me.
I ask as I would like to adjust the directory to a python environment relative to the script.
i.e #!../env/bin/python3 to (I'm just guessing here) #!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/env/bin/python3/r,$_)
Edit: I am trying to execute a simple ''Hello world" program.
#!/usr/bin/perl -e'$_=$ARGV[0];exec(s{\w+$}{exploit-env/bin/python3}r,$_)'
###############################
def main():
print('Hello world')
###############################
if __name__ == '__main__':
main()
Shebang handling isn't consistent across all systems[1], but your system apparently apparently executes what the following shell command would execute (assuming the file containing the shebang is /path/to/script):
/usr/bin/perl -e'$_=$ARGV[0];exec(s/\w+$/python3/r,$_)' /path/to/script
(The path to the script might not be an absolute path —it might be relative to the current directory— but doesn't matter here.)
The script produces /path/to/python3 from /path/to/script (by replacing the trailing "word" characters, which include letters, digits and underscore, but not /), then evaluates
exec('/path/to/python3', '/path/to/script')
The replaces the program running in the current process with a Python interpreter, passing the path to the script as an argument.
If I read between the lines correctly, you want to to run /path/to/../env/bin/python3 instead of /path/to/python3. In order to achieve that, use either of the following:
#!/usr/bin/perl -e$_=$ARGV[0];exec(s/\w+$/..\/env\/bin\/python3/r,$_)
or
#!/usr/bin/perl -e$_=$ARGV[0];exec(s{\w+$}{../env/bin/python3}r,$_)
/ needs to be escaped by \ when / is used as the delimiter (first solution), but we can change the delimiter to produce a more readable result (second solution).
That shebang you presented causes the arguments to be absorbed. Replace the $_ with #ARGV to pass them on.
#!/usr/bin/perl -e$_=$ARGV[0];exec(s{\w+$}{../env/bin/python3}r,#ARGV)
At least historically, some systems treat the entire sequence that follows #! as the path (i.e. no arguments allows), and some have very strict limits as to the length of the path.
I am trying to integrate a Python script into a bash script. However when I use the input() function, I am getting an EOFError. How can I fix this problem?
#!/bin/bash
python3 <<END
print(input(">>> "))
END
You cannot source both the script and the user input through the program's standard input. (That's in effect what you're trying to do. << redirects the standard input.)
Ideally, you would provide the script as command line argument instead of stdin using -c SCRIPT instead of <<EOF heredoc EOF:
#!/bin/bash
python3 -c 'print(input(">>> "))'
Note that you may need to mind your quoting and escaping in case you have a more complicated Python script with nested quotes.
You can still let the script run over multiple lines, if you need to:
#!/bin/bash
python3 -c '
import os.path
path_name = input("enter a path name >>> ")
file_exists = os.path.exists(path_name)
print("file " + path_name + " " +
("exists" if file_exists else "does not exist"))
'
Note that you will get into trouble when you want to use single quotes in your Python script, as happens when you want to print doesn't instead of does not.
You can work around that using several approaches. The one I consider most flexible (apart from putting you into quoting hell) is surrounding the Python script with double quotes instead and properly escape all inner double quotes and other characters that the shell interprets:
#!/bin/bash
python3 -c "
print(\"It doesn't slice your bread.\")
print('But it can', 'unsliced'[2:7], 'your strings.')
print(\"It's only about \$0. Neat, right?\")
"
Note that I also escaped $, as the shell would otherwise interpret it inside the surrounding double quotes and the result may not be what you wanted.
I don't know if this is a problem with python or with the shell (zsh on linux), I've an argument like this: "#xyz" that starts with a "#"
python the_script.py first_argument #second_argument third_arg
I tried to escape # with \ or \\, or use "" but the program doesn't start. If I leave the # from #second_arguments everything's ok.
Perhaps the "#" is a glob character in zsh, expanding to all symbolic links in the current directory. Try escaping it with "##"?
Try running the argument list with echo, i.e:
echo the_script.py first_argument #second_argument third_arg
That way, you can figure out if it was expanded or passed as-is to the script.