I have a main function which takes args from sys.argv using argparse:
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-arg1', '--argument1', type=str, required=True)
parser.add_argument('-arg2', '--argument2', type=str, required=True)
args = parser.parse_args()
The main function is called from another module as mod.main().
When i print what is passed to the main function the args are: ['-arg1', 'val1', '-arg2', 'val2'] which is the correct list of arguments.
However, when i run it it seems that the args are not passed correctly with argparse.
['-arg1', 'val1', '-arg2', 'val2']
usage: -arg1 [-h] -arg2 ARG2 -arg1 ARG1
-arg1: error: the following arguments are required: -arg1
The issue is that i am not running the run_module.py which calls the module_to_call.main() from the command line but passing it as args to spark-submit in order to run it on EMR.
"HadoopJarStep": {
"Args": [
"spark-submit",..., "run_module.py", "module_to_call", "-arg1", "val1", "-arg2", "val2",...
Here is a minimal example for you
#!/usr/bin/env python
# argparse_example.py
import sys
import argparse
def main():
print(sys.argv)
parser = argparse.ArgumentParser()
parser.add_argument('-arg1', '--argument1', type=str, required=True)
parser.add_argument('-arg2', '--argument2', type=str, required=True)
args = parser.parse_args()
print(args)
if __name__ == "__main__":
main()
You can run it with
python argparse_example.py -arg1 val1 -arg2 val2
And get
['argparse_example.py', '-arg1', 'val1', '-arg2', 'val2']
Namespace(argument1='val1', argument2='val2')
Or run it without arguments
python argparse_example.py
to get the help message:
['argg.py']
usage: argg.py [-h] -arg1 ARGUMENT1 -arg2 ARGUMENT2
argg.py: error: the following arguments are required: -arg1/--argument1, -arg2/--argument2
UPDATE
From comments by the OP I realize that sys.argv got mangled on the way to function main() by some enveloping code and the first argument got cut off.
The argument list ['-arg1', 'val1', '-arg2', 'val2'] is not the correct list of arguments as it is missing sys.argv[0] — the name of the python script being called from the command line. As argparse processes sys.argv[1:], it is unable to find '-arg1' and therefore fails with an error.
In this case you can add a dummy element to sys.argv to make sure that -arg1 will be passed to argparse when your code is being run as a module.
#!/usr/bin/env python
# argparse_example.py
import sys
import argparse
def main():
print(sys.argv)
if __name__ != "__main__":
# we are running as a module so add a dummy argument
sys.argv = ['argparse_example.py'] + sys.argv
parser = argparse.ArgumentParser()
parser.add_argument('-arg1', '--argument1', type=str, required=True)
parser.add_argument('-arg2', '--argument2', type=str, required=True)
args = parser.parse_args()
print(args)
if __name__ == "__main__":
main()
You can test it with the following enveloping code
#!/usr/bin/env python
# runner.py
import sys
import argparse_example
if __name__ == "__main__":
sys.argv = sys.argv[1:]
argparse_example.main()
And call it from the command line like this:
python runner.py -arg1 val1 -arg2 val2
The result should be the same as when calling argparse_example directly from the command line, even though sys.argv will be missing the first element:
['-arg1', 'val1', '-arg2', 'val2']
Namespace(argument1='val1', argument2='val2')
Related
In the argparse module, is it possible (easily) to print/log arguments that have been changed from default values when running a script? For example,
my_script.py
argparse.add_argument("--arg1", default="val1")
argparse.add_argument("--arg2", default="val2")
Running:
python my_script.py --arg2 newval2
Print:
Arguments changed:
arg2 : newval2
Once you have parsed your arguments you can, take a look to this script:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-c", dest="c", default=1)
def main():
options = parser.parse_args()
print(options.c)
if __name__ == "__main__":
main()
If you save this code in a script tmp.py and then call:
$>python3 tmp.py
$>1
$>python3 tmp.py -c 12
$>12
The idea is to add a flag (--slack, or -s) when running the script, so that I don't have to comment out the rep.post_report_to_slack() method every time I don't want to use it. When I run:
$ python my_script.py --slack
I get the error:
my_script.py: error: unrecognized arguments: --slack
Here's the code:
def main():
gc = Google_Connection()
meetings = gc.meetings
rep = Report(meetings)
if args.slack:
rep.post_report_to_slack()
print('posted to slack')
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-s', '--slack', help='post to slack',
action='store_true')
args = parser.parse_args()
main()
Your code works, but it relies on args being available in the module namespace, which isn't great because, for one thing, it means you can't use your function without calling the script from the command line. A more flexible and conventional approach would be to write the function to accept whatever arguments it needs, and then pass everything you get from argparse to the function:
# imports should usually go at the top of the module
import argparse
def get_meeting_report(slack=False):
gc = Google_Connection()
meetings = gc.meetings
rep = Report(meetings)
if slack:
rep.post_report_to_slack()
print('posted to slack')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('-s', '--slack', help='post to slack',
action='store_true')
args = parser.parse_args()
args = vars(args)
get_meeting_report(**args)
Now you can also more easily use your function outside of argparse by calling it directly.
After trying the solution offered by this thread:
What is the best way to call a Python script from another Python script?
I came along with another problem dealing with arguments.
I have:
test1.py
def some_func():
print 'in test 1, unproductive'
if __name__ == '__main__':
# test1.py executed as script
# do something
some_func()
service.py (with dummy arguments)
import argparse
import test1
actions = ['start', 'remove']
parser = argparse.ArgumentParser()
parser.add_argument("action", help="Possible actions are: "
"'{d[0]}|{d[1]}' ".format(d=actions))
parser.add_argument("-d", "--debug", help="Debug mode", action="store_true")
args = parser.parse_args()
def service_func():
print 'service func'
if __name__ == '__main__':
# service.py executed as script
# do something
service_func()
test1.some_func()
This works:
python service.py start
service func
in test 1, unproductive
However, if I want to add arguments in test1.py as well:
test1.py
import argparse
######## new stuff ##########
parser = argparse.ArgumentParser()
parser.add_argument("-d", "--debug", help="Debug mode", action="store_true")
args = parser.parse_args()
##############################
def some_func():
print 'in test 1, unproductive'
if __name__ == '__main__':
# test1.py executed as script
# do something
some_func()
Now I get:
python service.py start
usage: service.py [-h] [-d]
service.py: error: unrecognized arguments: start
Not sure why...
python service.py start - to argparse 'start' looks like a positional argument. It does not have a '--' to mark it as a flag string.
I'm guessing the parser in service.py is handling it ok, though you don't display or otherwise use the resulting args.
The usage in the error message is consistent with the parser in test1.py. It does not define a positional, just the --debug
usage: service.py [-h] [-d]
test1.py is using the same sys.argv list.
There are several solutions:
add a positional argument to test1
use parse_known_args instead of parse_args.
modify sys.argv in service.py (after parsing) to remove this 'start' string. A change in sys.argv in this script will carry over to test1. Print sys.argv in both to be sure.
use REMAINDER as documented in the docs
argparse.REMAINDER. All the remaining command-line arguments are gathered into a list. This is commonly useful for command line utilities that dispatch to other command line utilities:
Call
args, extras = parser.parse_known_args()
This would set
args to Namespace(debug=False) and extras to ['start']. Otherwise you get a tuple of these two values, (Namespace(debug=False), ['start']).
If you haven't modified sys.argv in the first script, the 2nd one will see the same list. Parsing does not modify this list.
I'm trying to use the Python argparse module to check argument parameters:
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Example with non-optional arguments')
parser.add_argument('count', action="store", type=int)
parser.add_argument('units', action="store")
print 'test'
When I run the script (python test.py some inches), it just prints the output 'test' but the argparse module is not being triggered.
You need to actually call it!
import argparse
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Example with non-optional arguments')
parser.add_argument('count', action="store", type=int)
parser.add_argument('units', action="store")
args = parser.parse_args()
print args
You have to parse the args for this to work. You either need to call parser.parse_args() or parser.parse_known_args(). More info can be found here:
https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args
I have two python scripts with the following structure:
# Script1.py
from optparse import OptionParser
def main():
parser = OptionParser()
parser.add_option("-a", "--add-foobar", action="store_true", help="set foobar true",
dest="foobar", default=False)
options, args = parser.parse_args()
print options.foobar
if __name__ == "__main__":
main()
# Script2.py
from Script1 import main as script1Main
def main():
script1Main()
Is there a way to pass command line arguments from script 2 to script 1? Script 1 in this example is immutable, therefore this must be done only thorough optparse.
If you don't pass any arguments to parse_args, it just uses the value of sys.argv[1:], which is going to be whatever arguments were passed when you called Script2.py. The fact that Script2.py calls Script1.main doesn't change that.
Firstly, maybe use argparse instead. You can process all arguments in script 2, then pass the argument handle to script 1.
# Script1.py
def main(args):
print args
# Script2.py
import argparse
from Script1 import main as script1Main
def main():
parser = argparse.ArgumentParser(
parser.add_option("-a", "--add-foobar", action="store_true", help="set foobar true", default=False)
args = parser.parse_args()
script1Main(args)
if __name__ == "__main__":
main()