We Keep Coding

How to hide args to argparse when running script with exec?

I have 2 files, runner.py that runs target.py with subprocess or exec.
They both have command line options.
If runner runs target with subprocess it's ok:
$ python runner.py
run target.py with subprocess...
target.py: running with dummy = False
If runner runs target code with exec (with the -e option):
$ python runner.py -e
run target.py with exec...
usage: runner.py [-h] [-d]
runner.py: error: unrecognized arguments: -e
the command line argument -e is "seen" by target.py code (which accepts only one --dummy option) and raises an error.
How can I hide args to argparse when running script with exec?
Here's the code:
runner.py
import subprocess
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-e", "--exec", help="run with exec", action="store_true")
args = parser.parse_args()
target_filename = "target.py"
if args.exec:
print("run target.py with exec...")
source_code = open(target_filename).read()
compiled = compile(source_code, filename=target_filename, mode="exec")
exec(compiled) # OPTION 1 - error on argparse
# exec(compiled, {}) # OPTION 2 - target does not go inside "if main"
# exec(compiled, dict(__name__="__main__")) # OPTION 3 - same error as OPTION 1
else:
print("run target.py with subprocess...")
subprocess.run(["python3", target_filename])
I tried to hide the globals with the commented options above, but without luck.
Seems related to how argparse works.
target.py
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("-d", "--dummy", help="a dummy option", action="store_true")
args = parser.parse_args()
print(f"target.py: running with dummy = {args.dummy}")

There is the argparse conflict_handler option so one can write
argparse.ArgumentParser(conflict_handler='resolve') in target script.
resolve removes the conflicting options, but that doesn't handle well similar cases where the options have the same name both in runner and target or the case where you can't or don't want to change the target file.
Here's the solution I have found.
Internally argparse uses sys.argv to retrieve options set with command line.
You can directly set sys.argv = [target_filename] which removes the options, but changing sys can give a lot of other problems.
Using unittest.mock.patch (python3.4+) the sys.argv can be securely altered like this:
from unittest.mock import patch
# [...]
source_code = open(target_filename).read()
compiled = compile(source_code, filename=target_filename, mode="exec")
# remove command args
with patch('sys.argv', [target_filename]):
exec(compiled)
So one can also run target script code with options:
# run target
with patch('sys.argv', [target_filename]):
exec(compiled)
# run target with -d
with patch('sys.argv', [target_filename, "-d"]):
exec(compiled)

How choose variable for python script from command line?

I need to run python test script for different environments (different urls). And I need to define which variable use from command line. In future this parameter will be used in Jenkins job.
script.py:
class TestLogin(unittest.TestCase):
#allure.step
def test_LoginValidation(self):
devURL = "http://url1/admin/login/"
stagingURL = "http://url2/admin/login/"
prodURL = "https://url3/admin/login"
driver.maximize_window()
driver.implicitly_wait(10)
driver.get(url)
lp = LoginPage(driver)
lp.login("login", "password")
time.sleep(2)
driver.quit()
In command line I need to write
python script.py stagingURL
In a result in method test_LoginValidation in driver.get(url) will be used url which I defined in command line.

You can use argparse to do this:
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Description')
parser.add_argument('--dev',
dest='dev',
action='store_true',
help="Help message")
parser.add_argument('--stage',
dest='stage',
action='store_true',
help="Help message")
parser.add_argument('--prod',
dest='prod',
action='store_true',
help="Help message")
parser.set_defaults(dev=True,
stage=False,
action=False)
args = parser.parse_args()
url = None
if args.dev:
url = "http://url1/admin/login/"
if args.stage:
url = "http://url2/admin/login/"
if args.prod:
url = "https://url3/admin/login"
# do something with the url
This is one way to do it. You are creating some arg parameters --dev, --stage, --prod and by default --dev is set to true. You can also have no default (just set dev=False).
So next time you can run:
python program.py --dev
python program.py --stage
python program.py --prod
You might want to handle the case where more than one flag is passed.
You can also do it this way:
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Description')
parser.add_argument("--env",
choices={"dev", "stage", "prod"},
help="Some help message.")
args = parser.parse_args()
url = None
if args.env == "dev":
url = "http://url1/admin/login/"
elif args.env == "stage":
url = "http://url2/admin/login/"
elif args.env == "prod":
url = "https://url3/admin/login"
else:
print("Please specify the environment using --env flag.")
if url is not None:
print(url)
Example:
$ python3 test2.py
Please specify the environment using --env flag.
$ python3 test2.py --env prod
https://url3/admin/login
$ python3 test2.py --env stage
http://url2/admin/login/
$ python3 test2.py --env dev
http://url1/admin/login/
$ python3 test2.py --env wrong
usage: test2.py [-h] [--env {stage,dev,prod}]
test2.py: error: argument --env: invalid choice: 'wrong' (choose from 'stage', 'dev', 'prod')
You can read more about argparse here.

I can recommend click package for creating CLI. It's really simple, well documented, has a lot of options and in my opinion much easier to use than argparse.
A dummy example:
import click
#click.command()
#click.option(
'--count',
default=1,
help='Number of greetings.'
)
#click.option(
'--name',
prompt='Your name',
help='The person to greet.'
)
def hello(**options):
"""Simple program that greets NAME for a total of COUNT times."""
for x in range(options['count']):
click.echo('Hello %s!' % options['name'])
if __name__ == '__main__':
hello()
And what it looks like when run:
$ python hello.py --count=3
Your name: John
Hello John!
Hello John!
Hello John!
It automatically generates nicely formatted help pages:
$ python hello.py --help
Usage: hello.py [OPTIONS]
Simple program that greets NAME for a total of COUNT times.
Options:
--count INTEGER Number of greetings.
--name TEXT The person to greet.
--help Show this message and exit.
You can get the library directly from PyPI:
pip install click
If you want to create CLI just to parametrize unit test you may consider using #pytest.mark.parametrize which allows one to define multiple sets of arguments and fixtures at the test function or class.
An example:
import pytest
class TestLogin(object):
#pytest.mark.parametrize("url", [
"http://url1/admin/login/",
"http://url2/admin/login/",
"https://url3/admin/login",
])
def test_LoginValidation(self, url):
driver.maximize_window()
driver.implicitly_wait(10)
driver.get(url)
lp = LoginPage(driver)
lp.login("login", "password")
time.sleep(2)
driver.quit()

What you're looking for is argparse. That should allow you to do exactly what you're looking for, for example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('url', help = 'The URL to use for ...', type = str)
This sets up the url as a required argument to be passed to the function, and sets its type to str (this is the default behavior, but being explicit is good).
You can then extract the arguments using:
args = parser.parse_args()
specified_url = args.url
From here you can proceed as you normally would. If you wish to make the argument optional but with a default value, that is also possible using argparse.
Using the environment variables works but is much harder to debug, especially if you expect this script to be run by another piece of software argparse is much more reliable.

It's almost certainly easier to do this in Jenkins than it is to do it in Python. Additionally it seems to make sense that your devops pipeline controls the location of dev, staging, and release URIs (at least as much as it is sensible to do that).
def targetUrl = ''
switch (env.TARGET) {
case 'dev':
targetUrl = "http://url1/admin/login/"
break
// etc
}
sh "python script.py ${targetUrl}"
then have the python script look at sys.argv[1] (which is the first argument passed to it) and use that URL directly.

Python argparse: args has no attribute func

Intro
I'm trouble for a school project. I'm making a testsuit and i'm needing bot a configuration generation interface and a test runner. For that i used the library argparse and two subparsers cgi and run
The issue itself
So here is the failing code section:
def CGI(args):
print("CGI: Work In Progress")
exit(0)
def runTest(args):
print("Run: Work in Progress")
exit(0)
parser = argparse.ArgumentParser()
subparser = parser.add_subparsers()
cgi = subparser.add_parser("cgi", help="CSV Generator Interface")
run = subparser.add_parser("run", help="Test running")
verbosity = parser.add_argument_group("Verbosity").add_mutually_exclusive_group()
check = run.add_argument_group("Checks")
# Arguments
#Run parser
run.set_defaults(func=runTest)
# Run argument declaration ...
# Verbosity argument declaration ...
# Check argument declaration ...
#CGI
cgi.set_defaults(func=CGI)
args = parser.parse_args()
args.func(args) # Error is here
Whenever i run this code i have the following error:
File "/home/thor/Projects/EPITA/TC/test/test.py", line 44, in main
args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'
Python version
$ python -V
Python 3.6.4
Argparse version
$ pip show argparse
Name: argparse
Version: 1.4.0
Summary: Python command-line parsing library
Home-page: https://github.com/ThomasWaldmann/argparse/
Author: Thomas Waldmann
Author-email: tw#waldmann-edv.de
License: Python Software Foundation License
Location: /usr/lib/python3.6/site-packages
Requires:
EDIT
If i install argparse manually it work sudo pip install argparse. But is there any native solution. I'm not sure it will work on school's computers (we can' install packages)
EDIT 2
OK my bad i've been a total idiot i didn't rewrited my running script so i forgot to input run or cgi
Thanks for reading my message and for your future help :)

This is a known bug in the Python 3 version of argparse (https://bugs.python.org/issue16308). In Python 2, if the script is called with no arguments whatsoever (i.e., no subcommand), it exits cleanly with "error: too few arguments". In Python3, however, you get an unhandled AttributeError. Luckily, the workaround is pretty straightforward:
try:
func = args.func
except AttributeError:
parser.error("too few arguments")
func(args)

parser = ArgumentParser()
parser.set_defaults(func=lambda args: parser.print_help())
imho better than try..except

Another solution could be:
if len(args.__dict__) <= 1:
# No arguments or subcommands were given.
parser.print_help()
parser.exit()

You would have to make subparsers required to call the script without arguments. To do that, you have to specify the dest and required arguments for the parser.add_subparsers:
parser = argparse.ArgumentParser()
subparser = parser.add_subparsers(dest='cmd', required=True)
Note that for Python 3.6 and earlier there's no required argument and you have to set it explicitly for the subparser object:
subparser.required = True
Details are available in this SO answer: Argparse with required subparser

Or, a combination of #simleo and #nehaljwani's answers:
# Parse the arguments and call the sub command
args = parser.parse_args()
try:
args.func(args)
except AttributeError:
parser.print_help()
parser.exit()

This error only happened when you run 'python script.py' directly.
'python script.py --help' works fine.
Add
args = parser.parse_args()
try:
args.func(args)
except AttributeError:
parser.print_help()
parser.exit()
will help you handle this case of running 'python script.py' directly.
It resolved my issue, thanks very much!

Python argparse versatility ability for true/false and string?

I have the following arguments parser using argparse in a python 2.7 script:
parser = argparse.ArgumentParser(description=scriptdesc)
parser.add_argument("-l", "--list", help="Show current running sesssions", dest="l_list", type=str, default=None)
I want to be able to run:
./script -l and ./script -l session_1
So that the script returns either all sessions or a single session without an extra parameter such as -s
However I can't find a way to do this in a single arg.

This is a bit of a hack since it relies on accessing sys.argv outside of any argparse function but you can do something like:
import argparse
import sys
parser = argparse.ArgumentParser(description='')
parser.add_argument("-l", "--list", help="Show current running sesssions", dest="l_list", nargs='?')
args = parser.parse_args()
if args.l_list == None:
if '-l' in sys.argv or '--list' in sys.argv:
print('display all')
else:
print('display %s only' %args.l_list)
And you would obviously replace the print statements with your actual code. This works by allowing 0 or 1 argument (using nargs='?'). This allows you to either pass an argument with -l or not. This means that in the args namespace, l_list can be None (the default) if you call -l without an argument OR if you don't use -l at all. Then later you can check if -l was called without an argument (if l_list == None and -l or --list is in sys.argv).
If I name this script test.py I get the following outputs when calling it from the command line.
$python test.py
$python test.py -l
display all
$python test.py -l session1
display session1 only
EDIT
I figured out an argparse only solution!! No relying on sys.argv:
import argparse
parser = argparse.ArgumentParser(description='')
parser.add_argument("-l", "--list", help="Show current running sesssions", dest="l_list", nargs='?', default=-1)
args = parser.parse_args()
if args.l_list == None:
print('display all')
elif args.l_list != -1:
print('display %s only' %args.l_list)
So it turns out that the default keyword in .add_argument only applies when the argument flag is not called at all. If the flag is used without anything following it, it will default to None regardless of what the default keyword is. So if we set the default to something that is not None and not an expected argument value (in this case I chose -1), then we can handle all three of your cases:
$ python test.py
$ python test.py -l
display all
$ python test.py -l session1
display session1 only

Defining Argument values with docopt

Im working on my first python "app" and after some advice from the participants on Stackoverflow. Ive decided to scrap what I had and start from scratch.
It seems to be parsing the arguments nicely for usage etc but im not sure how I am meant to assign the values to the args?
Do I have to create a nest of ifs? if so how do i do that for the args in docopt?
maybe like this?
if opt in ("-f", "--file"):
FWORD = arg
CODE
#!/usr/bin/python
"""
Basic domain bruteforcer
Usage:
your_script.py (-f <file>) (-d <domain>) [-t 10] [-v]
your_script.py -h | --help
Options:
-h --help Show this screen.
-f --file File to read potential Sub-domains from. (Required argument)
-p --proxy Proxy address and port. [default: http://127.0.0.1:8080] (Optional)
-d --domain Domain to bruteforce.(Required argument)
-t --thread Thread count. (Optional)
-v --verbose Turn debug on. (Optional)
"""
from docopt import docopt
def fread(FWORD, *args):
flist = open(FWORD).readlines()
return flist
if __name__ == "__main__":
arguments = docopt(__doc__, version='0.1a')
print fread(fword)

You almost got it. Your arguments variable contains the argument and you look them up as you would in a dict. So if you want to call the fread function with the file argument your main would look like this:
if __name__ == "__main__":
arguments = docopt(__doc__, version='0.1a')
fread(arguments['<file>'])
If you call the script like this:
> python your_script.py -f myfiles/file.txt -d google.com
Then your arguments will look like this:
>>> print arguments
{'--domain': True,
'--file': True,
'--help': False,
'--thread': False,
'--verbose': False,
'10': False,
'<domain>': 'google.com',
'<file>': 'myfiles/file.txt'}

You should take a look at argparse from the python standard library.