We Keep Coding

Imported written module overrides opt_parser

I am trying to write multiple files for specific tasks and in the end I would like to be able to use those functionalities in those files by importing them. But in the mean time, I also want to make other files runnable on their own. Therefore, for each file I have added options to them. But when I tried to run main where I imported all of the files, I saw that option parser of a file, overrides the option parser of the main. Code below is showing a similar case.
import optparse
def get_user_input():
parse_object = optparse.OptionParser()
parse_object.add_option("-t","--test",dest="test",help="test")
(user_input,arguments) = parse_object.parse_args()
return user_input
def test_func1():
print("hello")
def test11():
user_input = get_user_input()
test11()
This is the test1.py.
import optparse
import test1
def get_input():
sample = "[sudo] python3 main.py -[option] [argument]"
parse_object = optparse.OptionParser(usage=sample)
parse_object.add_option("-d","--discovery",dest="network_discovery",help="dd.")
parse_object.add_option("-p","--all-ports",dest = "all_ports",help = "pp")
parse_object.add_option("-i","--ip",dest="ip",help="ii.")
parse_object.add_option("-l","--list",dest="ip_list",help="ll")
parse_object.add_option("-o","--output",dest="save_output",help="oo")
parse_object.add_option("-c","--icmp",dest="icmp_scan",help="icmp")
(user_input,arguments) = parse_object.parse_args()
return user_input
user_input = get_input()
And this is the main.py.
I have realized that when I do not test11() in test1.py it does not override which simply means that when I don't call get_user_input(). But when I run main.py with help function the output is as below.
Usage: main.py [options]
Options:
-h, --help show this help message and exit
-t TEST, --test=TEST test
But the expected output is as follows:
Usage: [sudo] python3 main.py -[option] [argument]
Options:
-h, --help show this help message and exit
-d NETWORK_DISCOVERY, --discovery=NETWORK_DISCOVERY
dd.
-p ALL_PORTS, --all-ports=ALL_PORTS
pp
-i IP, --ip=IP ii.
-l IP_LIST, --list=IP_LIST
ll
-o SAVE_OUTPUT, --output=SAVE_OUTPUT
oo
-c ICMP_SCAN, --icmp=ICMP_SCAN
icmp
As I mentioned, this is just an example case. In the real project, I have six files which can be run individually and one main function to run all of them in a specific way and all files have option parser.
So far, I have tried test1.get_user_input = get_input, from test1 import test_func1 to see if I could exclude that call. This is the last stage of the project. So, If you could help me out it will be much appreciated.

How to run snscrape command from python script?

I'm trying to download some tweets with snscrape. After installing, I can run a command like the following to download a few tweets:
snscrape --jsonl --max-results 4 twitter-search "#SherlockHolmes since:2015-01-01 until:2015-01-15" > sherlock_tweets.json
Now I want to run this command from within a python script. As I understand it, the way to do this is using the subprocess.run method. I use the following code to run the command from python:
import subprocess
# Running this in a terminal works
cmd = '''snscrape --jsonl --max-results 4 twitter-search "#SherlockHolmes since:2015-01-01 until:2015-01-15" > sherlock_tweets.json'''
arglist = cmd.split(" ")
process = subprocess.run(arglist, shell=True)
Running this, however, gives the following error.
usage: snscrape [-h] [--version] [-v] [--dump-locals] [--retry N] [-n N] [-f FORMAT | --jsonl] [--with-entity] [--since DATETIME] [--progress]
{telegram-channel,weibo-user,vkontakte-user,instagram-user,instagram-hashtag,instagram-location,twitter-thread,twitter-search,reddit-user,reddit-subreddit,reddit-search,facebook-group,twitter-user,twitter-hashtag,twitter-list-posts,facebook-user,facebook-community,twitter-profile}
...
snscrape: error: the following arguments are required: scraper
Why is the behaviour not the same in these two cases? How do I accomplish running the command from a python script, getting the exact same behaviour as I would entering it in a terminal?

I don't know if you found the solution, but I ran this code and that worked for me :
import pandas as pd
import snscrape.modules.twitter as sntwitter
tweet_collection = pd.DataFrame({
'Username':[],
'Date'=[],
'Likes'=[],
'Content'=[]})
for tweet in sntwitter.TwitterSearchScraper(f'since:{date_beg} until:{date_end} from:{twitter_account}').get_items():
tweets_collection = tweets_candidats.append({
"Username":tweet.user.username,
"Date":tweet.date,
"Tweet":tweet.content,
"Likes":tweet.likeCount,},ignore_index=True)
tweets_candidats.to_csv('Path/file.csv')
You can find more detail in the code on git hub
Twitter snscrape arguments

How choose variable for python script from command line?

I need to run python test script for different environments (different urls). And I need to define which variable use from command line. In future this parameter will be used in Jenkins job.
script.py:
class TestLogin(unittest.TestCase):
#allure.step
def test_LoginValidation(self):
devURL = "http://url1/admin/login/"
stagingURL = "http://url2/admin/login/"
prodURL = "https://url3/admin/login"
driver.maximize_window()
driver.implicitly_wait(10)
driver.get(url)
lp = LoginPage(driver)
lp.login("login", "password")
time.sleep(2)
driver.quit()
In command line I need to write
python script.py stagingURL
In a result in method test_LoginValidation in driver.get(url) will be used url which I defined in command line.

You can use argparse to do this:
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Description')
parser.add_argument('--dev',
dest='dev',
action='store_true',
help="Help message")
parser.add_argument('--stage',
dest='stage',
action='store_true',
help="Help message")
parser.add_argument('--prod',
dest='prod',
action='store_true',
help="Help message")
parser.set_defaults(dev=True,
stage=False,
action=False)
args = parser.parse_args()
url = None
if args.dev:
url = "http://url1/admin/login/"
if args.stage:
url = "http://url2/admin/login/"
if args.prod:
url = "https://url3/admin/login"
# do something with the url
This is one way to do it. You are creating some arg parameters --dev, --stage, --prod and by default --dev is set to true. You can also have no default (just set dev=False).
So next time you can run:
python program.py --dev
python program.py --stage
python program.py --prod
You might want to handle the case where more than one flag is passed.
You can also do it this way:
import argparse
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Description')
parser.add_argument("--env",
choices={"dev", "stage", "prod"},
help="Some help message.")
args = parser.parse_args()
url = None
if args.env == "dev":
url = "http://url1/admin/login/"
elif args.env == "stage":
url = "http://url2/admin/login/"
elif args.env == "prod":
url = "https://url3/admin/login"
else:
print("Please specify the environment using --env flag.")
if url is not None:
print(url)
Example:
$ python3 test2.py
Please specify the environment using --env flag.
$ python3 test2.py --env prod
https://url3/admin/login
$ python3 test2.py --env stage
http://url2/admin/login/
$ python3 test2.py --env dev
http://url1/admin/login/
$ python3 test2.py --env wrong
usage: test2.py [-h] [--env {stage,dev,prod}]
test2.py: error: argument --env: invalid choice: 'wrong' (choose from 'stage', 'dev', 'prod')
You can read more about argparse here.

I can recommend click package for creating CLI. It's really simple, well documented, has a lot of options and in my opinion much easier to use than argparse.
A dummy example:
import click
#click.command()
#click.option(
'--count',
default=1,
help='Number of greetings.'
)
#click.option(
'--name',
prompt='Your name',
help='The person to greet.'
)
def hello(**options):
"""Simple program that greets NAME for a total of COUNT times."""
for x in range(options['count']):
click.echo('Hello %s!' % options['name'])
if __name__ == '__main__':
hello()
And what it looks like when run:
$ python hello.py --count=3
Your name: John
Hello John!
Hello John!
Hello John!
It automatically generates nicely formatted help pages:
$ python hello.py --help
Usage: hello.py [OPTIONS]
Simple program that greets NAME for a total of COUNT times.
Options:
--count INTEGER Number of greetings.
--name TEXT The person to greet.
--help Show this message and exit.
You can get the library directly from PyPI:
pip install click
If you want to create CLI just to parametrize unit test you may consider using #pytest.mark.parametrize which allows one to define multiple sets of arguments and fixtures at the test function or class.
An example:
import pytest
class TestLogin(object):
#pytest.mark.parametrize("url", [
"http://url1/admin/login/",
"http://url2/admin/login/",
"https://url3/admin/login",
])
def test_LoginValidation(self, url):
driver.maximize_window()
driver.implicitly_wait(10)
driver.get(url)
lp = LoginPage(driver)
lp.login("login", "password")
time.sleep(2)
driver.quit()

What you're looking for is argparse. That should allow you to do exactly what you're looking for, for example:
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('url', help = 'The URL to use for ...', type = str)
This sets up the url as a required argument to be passed to the function, and sets its type to str (this is the default behavior, but being explicit is good).
You can then extract the arguments using:
args = parser.parse_args()
specified_url = args.url
From here you can proceed as you normally would. If you wish to make the argument optional but with a default value, that is also possible using argparse.
Using the environment variables works but is much harder to debug, especially if you expect this script to be run by another piece of software argparse is much more reliable.

It's almost certainly easier to do this in Jenkins than it is to do it in Python. Additionally it seems to make sense that your devops pipeline controls the location of dev, staging, and release URIs (at least as much as it is sensible to do that).
def targetUrl = ''
switch (env.TARGET) {
case 'dev':
targetUrl = "http://url1/admin/login/"
break
// etc
}
sh "python script.py ${targetUrl}"
then have the python script look at sys.argv[1] (which is the first argument passed to it) and use that URL directly.

click no such command found

I am trying click (Command line interface package for Python), while running the following code I get error Error: No such command "abcd"
#click.group()
#click.option('--source', required=True)
#click.pass_context
def cli(ctx, source):
ctx.obj = "pass it"
#cli.command()
#click.argument('abcd')
#click.pass_context
def hello(ctx, abcd):
click.echo("Hello, World")
if __name__ == '__main__':
cli()
I am running it as follows
python playclick.py --source this abcd

"abcd" is being treated as a separate command because of the space (this is a characteristic of your shell, not of click specifically).
If you want the value of source to be "this abcd", use quotes:
python playclick.py --source "this abcd"
To actually provide the abcd argument, you need to call the hello command – the argument is for that command:
python playclick.py --source this hello 123456
The hello command will have an argument of 123456.
Breaking down the entire line:
--source this provides the source argument to the main cli command.
hello is the command to run (try python playclick.py --source this and you'll get an error because there is no command), and 123456 is the argument named abcd to that command.

For those who are only using #click.argument but still getting the same no such command found error, what finally helped me solve this issue with (7.1.2) was just to remove uppercase letters in the command name.

Set a default choice for optionparser when the option is given

I have a python option parsers that parses an optional --list-something option.
I also want the --list-something option to have an optional argument (an option)
Using the argument default="simple" does not work here, otherwise simple will always
be the default, not only when --list-something was given.
from optparse import OptionParser, OptionGroup
parser = OptionParser()
options = OptionGroup(parser, "options")
options.add_option("--list-something",
type="choice",
choices=["simple", "detailed"],
help="show list of available things"
)
parser.add_option_group(options)
opts, args = parser.parse_args()
print opts, args
The above code is producing this:
[jens#ca60c173 ~]$ python main.py --list-something simple
{'list_something': 'simple'} []
[jens#ca60c173 ~]$ python main.py --list-something
Usage: main.py [options]
main.py: error: --list-something option requires an argument
[jens#ca60c173 ~]$ python main.py
{'list_something': None} []
But I want this to hapen:
[jens#ca60c173 ~]$ python main.py --list-something simple
{'list_something': 'simple'} []
[jens#ca60c173 ~]$ python main.py --list-something
{'list_something': 'simple'} []
[jens#ca60c173 ~]$ python main.py
{'list_something': None} []
I would like something that works out of the box in python 2.4 up till 3.0 (3.0 not included)
Since argparse is only introduced in python 2.7 this is not something I could use.

Optparse does not have any options for doing this easily. Instead you'll have to create a custom callback for your option. The callback is triggered when your option is parsed, at which point, you can check the remaining args to see if the user put an argument for the option.
Check out the custom callback section of the docs, in particular Callback example 6: variable arguments.

There is no default for the Optparser in python.
However, you can use the follwing -
# show help as default
if len(sys.argv) == 1:
os.system(sys.argv[0] + " -h")
exit()
This will run the same script with the -h option, and exit.
please notice - you will need to import the sys module.