We Keep Coding

Bash script loop on python variable

I'm trying to do a simple script, but I don't get how to pass a variable to the command that I need:
#!/bin/bash
for i in 1 2 3
do
python -c 'print "a"*' $i
done

If you really want to go on with your solution using python -c, you would have to remove the space:
#!/bin/bash
for i in 1 2 3
do
python -c 'print "a"*'$i
done
But the approach suggested by Asmox makes more sense to me (and it has nothing to do with the question where you pipe the standard output to).

Maybe this topic will help:
How do I run Python script using arguments in windows command line
Have you considered making a whole script? Like this one:
import sys
def f(a):
print a
if __name__ == "__main__":
a = int(sys.argv[1])
f(a)
This example might seem overcomplicated, but I wanted to show you how to pass parameter from console to function

I have tried the solution provided above and had to modify for python3+
due to lack of sufficient points I am not allowed to make a comment, thats why I posted my solution separately
#!/bin/bash
for i in {1..3}
do
python -c "print ('a'* ${i})"
done
output~>
a
aa
aaa

Issues with Python Popen() (Syntax?)

have a small issue with a program I am trying to launch from a Python script via Popen() (I understand Popen() may not be ideal, but I am working with somewhat of a template used in other instances, and want to follow convention).
I am a bit confused, as I can't seem to get the following to run:
root = os.getcwd()
bin = 'my_executable.exe'
bin_fullpath = os.path.join(root,bin)
params = 'Option C -f Module -y -q'
p = subprocess.Popen([bin_fullpath,params])
out = p.communicate()
The program launches, but exits with error code 1 (I checked with check_call).
However, when I forgo the above method, and simply provide the entire string I need to run, as follows:
subprocess.Popen(r'C:\Users\me\Desktop\path\to\tool\my_executable.exe Option C -f Module -y -q')
The program executes as expected. Obviously I have something wrong with the sytntax, but I can't figure out what . . .
Any insight would be greatly appreciated!

When you use the "list of arguments" format, each one has to be its own string, as a separate member of the list, like this:
params = ['Option', 'C', '-f', 'Module', '-y', '-q']
p = subprocess.Popen([bin_fullpath, *params])
When you put them all in a single string, you're telling subprocess they're all one big argument.1 So, rather than being the rough equivalent of this command line:
C:\Users\me\Desktop\path\to\tool\my_executable.exe Option C -f Module -y -q
… it's the rough equivalent of this one:
C:\Users\me\Desktop\path\to\tool\my_executable.exe "Option C -f Module -y -q"
If it's not clear why those are different, consider these examples instead:
fix.exe "My Pictures\picture1.jpg"
fix.exe My Pictures\picture1.jpg
The first one is fixing one picture, My Pictures\picture1.jpg. The second is fixing two pictures, My, and Pictures\picture1.jpg.
For more details on the args argument, see Frequently Used Arguments. Notice the "one big string" version is actually not valid without shell=True—even though it happens to usually work on Windows.
1. Things are a little more complicated than this on Windows, because subprocess actually has to take all of the args and work out how to put them together in a string so that they can be parsed back into the actual separate values the way you asked for them. But never mind that.

When to use Shell=True for Python subprocess module [duplicate]

This question already has answers here:
Actual meaning of 'shell=True' in subprocess
(7 answers)
Closed 7 years ago.
It seems whenever I try to use Python's subprocess module, I find I still don't understand some things. Currently, I was trying to join 3 mp4 files from within a Python module.
When I tried
z ='MP4Box -cat test_0.mp4 -cat test_1.mp4 -cat test_2.mp4 -new test_012d.mp4'
subprocess.Popen(z,shell=True)
Everything worked.
When I tried
z = ['MP4Box', '-cat test_0.mp4', '-cat test_1.mp4', '-cat test_2.mp4', '-new test_012d.mp4']
subprocess.Popen(z,shell=False)
I got the following error:
Option -cat test_0.mp4 unknown. Please check usage
I thought that for shell=False I just needed to supply a list where the first element was the executable I wanted to run and each succeeding element was an argument to that executable. Am I mistaken in this belief, or is there a correct way to create the command I wanted to use?
Also, are there any rules for using Shell=True in subprocess.Popen? So far, all I really know(?) is "don't do it - you can expose your code to Shell injection attacks". Why does Shell=False avoid this problem? Is there ever an actual advantage to using 'Shell=True`?

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.
When shell=True is dangerous?
If we execute shell commands that might include unsanitized input from an untrusted source, it will make a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input
Eg. (Taken from docs)
>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / #
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly..

You have to give every single argument as one element of a list:
z = ['MP4Box', '-cat', 'test_0.mp4', '-cat', 'test_1.mp4', '-cat', 'test_2.mp4', '-new', 'test_012d.mp4']
subprocess.Popen(z,shell=False)
This is normally what you want to do, because you don't need to escape especial characters of the shell in filenames.

Python script argument conditional

Is anyone able to tell me how to write a conditional for an argument on a python script? I want it to print "Argument2 Entered" if it is run with a second command line arguments such as:
python script.py argument1 argument2
And print "No second argument" if it is run without command line arguments, like this:
python script.py argument1
Is this possible?

import sys
if len(sys.argv)==2: # first entry in sys.argv is script itself...
print "No second argument"
elif len(sys.argv)==3:
print "Second argument"

There are many answers to this, depending on what exactly you want to do and how much flexibility you are likely to need.
The simplest solution is to examine the variable sys.argv, which is a list containing all of the command-line arguments. (It also contains the name of the script as the first element.) To do this, simply look at len(sys.argv) and change behaviour based on its value.
However, this is often not flexible enough for what people expect command-line programs to do. For example, if you want a flag (-i, --no-defaults, ...) then it's not obvious how to write one with just sys.argv. Likewise for arguments (--dest-dir="downloads"). There are therefore many modules people have written to simplify this sort of argument parsing.
The built-in solution is argparse, which is powerful and pretty easy-to-use but not particularly concise.
A clever solution is plac, which inspects the signature of the main function to try to deduce what the command-line arguments should be.

There are many ways to do this simple thing in Python. If you are interested to know more than I recommend to read this article. BTW I am giving you one solution below:
import click
'''
Prerequisite: # python -m pip install click
run: python main.py ttt yyy
'''
#click.command(context_settings=dict(ignore_unknown_options=True))
#click.argument("argument1")
#click.argument("argument2")
def main(argument1, argument2):
print(f"argument1={argument1} and argument2={argument2}")
if __name__ == '__main__':
main()

Following block should be self explanatory
$ ./first.py second third 4th 5th
5
$ cat first.py
#!/usr/bin/env python
import sys
print (len(sys.argv))

This is related to many other posts depending upon where you are going with this, so I'll put four here:
What's the best way to grab/parse command line arguments passed to a Python script?
Implementing a "[command] [action] [parameter]" style command-line interfaces?
How can I process command line arguments in Python?
How do I format positional argument help using Python's optparse?
But the direct answer to your question from the Python docs:
sys.argv -
The list of command line arguments passed to a Python script. argv[0] is the script name (it is operating system dependent whether this is a full pathname or not). If the command was executed using the -c command line option to the interpreter, argv[0] is set to the string '-c'. If no script name was passed to the Python interpreter, argv[0] is the empty string.
To loop over the standard input, or the list of files given on the command line, see the fileinput module.

Using Python to parse complex arguments to shell script

When I'm writing shell scripts, I often find myself spending most of my time (especially when debugging) dealing with argument processing. Many scripts I write or maintain are easily more than 80% input parsing and sanitization. I compare that to my Python scripts, where argparse handles most of the grunt work for me, and lets me easily construct complex option structures and sanitization / string parsing behavior.
I'd love, therefore, to be able to have Python do this heavy lifting, and then get these simplified and sanitized values in my shell script, without needing to worry any further about the arguments the user specified.
To give a specific example, many of the shell scripts where I work have been defined to accept their arguments in a specific order. You can call start_server.sh --server myserver --port 80 but start_server.sh --port 80 --server myserver fails with You must specify a server to start. - it makes the parsing code a lot simpler, but it's hardly intuitive.
So a first pass solution could be something as simple as having Python take in the arguments, sort them (keeping their parameters next to them) and returning the sorted arguments. So the shell script still does some parsing and sanitization, but the user can input much more arbitrary content than the shell script natively accepts, something like:
# script.sh -o -aR --dir /tmp/test --verbose
#!/bin/bash
args=$(order.py "$#")
# args is set to "-a --dir /tmp/test -o -R --verbose"
# simpler processing now that we can guarantee the order of parameters
There's some obvious limitations here, notably that parse.py can't distinguish between a final option with an argument and the start of indexed arguments, but that doesn't seem that terrible.
So here's my question: 1) Is there any existing (Python preferably) utility to enable CLI parsing by something more powerful than bash, which can then be accessed by the rest of my bash script after sanitization, or 2) Has anyone done this before? Are there issues or pitfalls or better solutions I'm not aware of? Care to share your implementation?
One (very half-baked) idea:
#!/bin/bash
# Some sort of simple syntax to describe to Python what arguments to accept
opts='
"a", "append", boolean, help="Append to existing file"
"dir", str, help="Directory to run from"
"o", "overwrite", boolean, help="Overwrite duplicates"
"R", "recurse", boolean, help="Recurse into subdirectories"
"v", "verbose", boolean, help="Print additional information"
'
# Takes in CLI arguments and outputs a sanitized structure (JSON?) or fails
p=$(parse.py "Runs complex_function with nice argument parsing" "$opts" "$#")
if [ $? -ne 0 ]; exit 1; fi # while parse outputs usage to stderr
# Takes the sanitized structure and an argument to get
append=$(arg.py "$p" append)
overwrite=$(arg.py "$p" overwrite)
recurse=$(arg.py "$p" recurse)
verbose=$(arg.py "$p" verbose)
cd $(python arg.py "$p" dir)
complex_function $append $overwrite $recurse $verbose
Two lines of code, along with concise descriptions of the arguments to expect, and we're on to the actual script behavior. Maybe I'm crazy, but that seems way nicer than what I feel like I have to do now.
I've seen Parsing shell script arguments and things like this wiki page on easy CLI argument parsing, but many of these patterns feel clunky and error prone, and I dislike having to re-implement them every time I write a shell script, especially when Python, Java, etc. have such nice argument processing libraries.

You could potentially take advantage of associative arrays in bash to help obtain your goal.
declare -A opts=($(getopts.py $#))
cd ${opts[dir]}
complex_function ${opts[append]} ${opts[overwrite]} ${opts[recurse]} \
${opts[verbose]} ${opts[args]}
To make this work, getopts.py should be a python script that parses and sanitizes your arguments. It should print a string like the following:
[dir]=/tmp
[append]=foo
[overwrite]=bar
[recurse]=baz
[verbose]=fizzbuzz
[args]="a b c d"
You could set aside values for checking that the options were able to be properly parsed and sanitized as well.
Returned from getopts.py:
[__error__]=true
Added to bash script:
if ${opts[__error__]}; then
exit 1
fi
If you would rather work with the exit code from getopts.py, you could play with eval:
getopts=$(getopts.py $#) || exit 1
eval declare -A opts=($getopts)
Alternatively:
getopts=$(getopts.py $#)
if [[ $? -ne 0 ]]; then
exit 1;
fi
eval declare -A opts=($getopts)

Having the very same needs, I ended up writing an optparse-inspired parser for bash (which actually uses python internally); you can find it here:
https://github.com/carlobaldassi/bash_optparse
See the README at the bottom for a quick explanation. You may want to check out a simple example at:
https://github.com/carlobaldassi/bash_optparse/blob/master/doc/example_script_simple
From my experience, it's quite robust (I'm super-paranoid), feature-rich, etc., and I'm using it heavily in my scripts. I hope it may be useful to others. Feedback/contributions welcome.

Edit: I haven't used it (yet), but if I were posting this answer today I would probably recommend https://github.com/docopt/docopts instead of a custom approach like the one described below.
I've put together a short Python script that does most of what I want. I'm not convinced it's production quality yet (notably error handling is lacking), but it's better than nothing. I'd welcome any feedback.
It takes advantage of the set builtin to re-assign the positional arguments, allowing the remainder of the script to still handle them as desired.
bashparse.py
#!/usr/bin/env python
import optparse, sys
from pipes import quote
'''
Uses Python's optparse library to simplify command argument parsing.
Takes in a set of optparse arguments, separated by newlines, followed by command line arguments, as argv[2] and argv[3:]
and outputs a series of bash commands to populate associated variables.
'''
class _ThrowParser(optparse.OptionParser):
def error(self, msg):
"""Overrides optparse's default error handling
and instead raises an exception which will be caught upstream
"""
raise optparse.OptParseError(msg)
def gen_parser(usage, opts_ls):
'''Takes a list of strings which can be used as the parameters to optparse's add_option function.
Returns a parser object able to parse those options
'''
parser = _ThrowParser(usage=usage)
for opts in opts_ls:
if opts:
# yes, I know it's evil, but it's easy
eval('parser.add_option(%s)' % opts)
return parser
def print_bash(opts, args):
'''Takes the result of optparse and outputs commands to update a shell'''
for opt, val in opts.items():
if val:
print('%s=%s' % (opt, quote(val)))
print("set -- %s" % " ".join(quote(a) for a in args))
if __name__ == "__main__":
if len(sys.argv) < 2:
sys.stderr.write("Needs at least a usage string and a set of options to parse")
sys.exit(2)
parser = gen_parser(sys.argv[1], sys.argv[2].split('\n'))
(opts, args) = parser.parse_args(sys.argv[3:])
print_bash(opts.__dict__, args)
Example usage:
#!/bin/bash
usage="[-f FILENAME] [-t|--truncate] [ARGS...]"
opts='
"-f"
"-t", "--truncate",action="store_true"
'
echo "$(./bashparse.py "$usage" "$opts" "$#")"
eval "$(./bashparse.py "$usage" "$opts" "$#")"
echo
echo OUTPUT
echo $f
echo $#
echo $0 $2
Which, if run as: ./run.sh one -f 'a_filename.txt' "two' still two" three outputs the following (notice that the internal positional variables are still correct):
f=a_filename.txt
set -- one 'two'"'"' still two' three
OUTPUT
a_filename.txt
one two' still two three
./run.sh two' still two
Disregarding the debugging output, you're looking at approximately four lines to construct a powerful argument parser. Thoughts?

The original premise of my question assumes that delegating to Python is the right approach to simplify argument parsing. If we drop the language requirement we can actually do a decent job* in Bash, using getopts and a little eval magic:
main() {
local _usage='foo [-a] [-b] [-f val] [-v val] [args ...]'
eval "$(parse_opts 'f:v:ab')"
echo "f=$f v=$v a=$a b=$b -- $#: $*"
}
main "$#"
The implementation of parse_opts is in this gist, but the basic approach is to convert options into local variables which can then be handled like normal. All the standard getopts boilerplate is hidden away, and error handling works as expected.
Because it uses local variables within a function, parse_opts is not just useful for command line arguments, it can be used with any function in your script.
* I say "decent job" because Bash's getopts is a fairly limited parser and only supports single-letter options. Elegant, expressive CLIs are still better implemented in other languages like Python. But for reasonably small functions or scripts this provides a nice middle ground without adding too much complexity or bloat.