Given the following program:
#!/usr/bin/env python
import click
#click.command()
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
return
if __name__ == "__main__":
main()
I can run it with the arguments and options provided through command line.
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: None
config_file: my_config_file
How do I provide a configuration file (in ini? yaml? py? json?) to --config_file and accept the content as the value for the arguments and options?
For instance, I want my_config_file to contain
opt: my_opt
and have the output of the program show:
$ ./click_test.py my_arg --config_file my_config_file
arg: my_arg
opt: my_opt
config_file: my_config_file
I've found the callback function which looked to be useful but I couldn't find a way to modify the sibling arguments/options to the same function.
This can be done by over riding the click.Command.invoke() method like:
Custom Class:
def CommandWithConfigFile(config_file_param_name):
class CustomCommandClass(click.Command):
def invoke(self, ctx):
config_file = ctx.params[config_file_param_name]
if config_file is not None:
with open(config_file) as f:
config_data = yaml.safe_load(f)
for param, value in ctx.params.items():
if value is None and param in config_data:
ctx.params[param] = config_data[param]
return super(CustomCommandClass, self).invoke(ctx)
return CustomCommandClass
Using Custom Class:
Then to use the custom class, pass it as the cls argument to the command decorator like:
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
Test Code:
# !/usr/bin/env python
import click
import yaml
#click.command(cls=CommandWithConfigFile('config_file'))
#click.argument("arg")
#click.option("--opt")
#click.option("--config_file", type=click.Path())
def main(arg, opt, config_file):
print("arg: {}".format(arg))
print("opt: {}".format(opt))
print("config_file: {}".format(config_file))
main('my_arg --config_file config_file'.split())
Test Results:
arg: my_arg
opt: my_opt
config_file: config_file
I realize that this is way old, but since Click 2.0, there's a more simple solution. The following is a slight modification of the example from the docs.
This example takes explicit --port args, it'll take an environment variable, or a config file (with that precedence).
Command Groups
Our code:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
#click.group(context_settings={'auto_envvar_prefix': 'FOOP'}) # this allows for environment variables
#click.option('--config', default='~/config.yml', type=click.Path()) # this allows us to change config path
#click.pass_context
def foop(ctx, config):
if os.path.exists(config):
with open(config, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
#foop.command()
#click.option('--port', default=8000)
def runserver(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
if __name__ == '__main__':
foop()
Assuming our config file (~/config.yml) looks like:
runserver:
port: 5000
and we have a second config file (at ~/config2.yml) that looks like:
runserver:
port: 9000
Then if we call it from bash:
$ foop runserver
# ==> Serving on http://127.0.0.1:5000/
$ FOOP_RUNSERVER_PORT=23 foop runserver
# ==> Serving on http://127.0.0.1:23/
$ FOOP_RUNSERVER_PORT=23 foop runserver --port 34
# ==> Serving on http://127.0.0.1:34/
$ foop --config ~/config2.yml runserver
# ==> Serving on http://127.0.0.1:9000/
Single Commands
If you don't want to use command groups and want to have configs for a single command:
import os
import click
from yaml import load
try:
from yaml import CLoader as Loader
except ImportError:
from yaml import Loader
def set_default(ctx, param, value):
if os.path.exists(value):
with open(value, 'r') as f:
config = load(f.read(), Loader=Loader)
ctx.default_map = config
return value
#click.command(context_settings={'auto_envvar_prefix': 'FOOP'})
#click.option('--config', default='config.yml', type=click.Path(),
callback=set_default, is_eager=True, expose_value=False)
#click.option('--port')
def foop(port):
click.echo(f"Serving on http://127.0.0.1:{port}/")
will give similar behavior.
locust --no-web --client=1 --hatch-rate=1 --num-request=2 --host= http://localhost
I want to read --host value provided in cmd line in my HTTPLocust class. I am aware I can use host attribute for direct assignment but I do not want it. I want to read the value from cmd line with in HTTPLocust class. I am building custom logs and want to pass that value to the logs. I tried HTTPLocust.host but that returns none.
Also I want to read --web-port from python code.
New answer
There is a much more straightforward solution than my initial one below. Each TaskSet has a locust property that links back to their parent Locust locustinstance, so something like this will do exactly what you need:
class UserBehaviour(TaskSet):
def __init__(self, parent):
super().__init__(parent)
print(self.locust.host)
Old answer
Looking at the code for HttpSession, it seems base_url is what you want.
So something like this should give you the current host, either default or specified on the command line:
class HostGetter(locust.TaskSet):
#locust.Task()
def get_host(self):
print(self.client.base_url)
You can access the host variable by an instance of User()class. See the example below:
from locust import HttpLocust, TaskSet, task
import random, requests, time, os, inspect, json, sys
class UserBehaviour(TaskSet):
#task(1)
def test1(self):
user = User()
print(user.host)
self.client.get("/v3/User/ListOfCoupon/")
class User(HttpLocust):
task_set = UserBehaviour
min_wait = 1000
max_wait = 3000
see the log:
~/P/m/p/general (master ⚡↑) locust -f app_couponlist.py --host=http://www.google.com
[2017-09-19 14:33:13,020] Mesuts-MacBook.local/INFO/locust.main: Starting web monitor at *:8089
[2017-09-19 14:33:13,021] Mesuts-MacBook.local/INFO/locust.main: Starting Locust 0.8a3
[2017-09-19 14:33:22,281] Mesuts-MacBook.local/INFO/locust.runners: Hatching and swarming 5 clients at the rate 1 clients/s...
[2017-09-19 14:33:22,282] Mesuts-MacBook.local/INFO/stdout: http://www.google.com
[2017-09-19 14:33:22,282] Mesuts-MacBook.local/INFO/stdout:
[2017-09-19 14:33:23,285] Mesuts-MacBook.local/INFO/stdout: http://www.google.com
[2017-09-19 14:33:23,285] Mesuts-MacBook.local/INFO/stdout:
[2017-09-19 14:33:24,226] Mesuts-MacBook.local/INFO/stdout: http://www.google.com
This is my code:
import time,csv,argparse
class MySQLLocust(Locust):
parser = argparse.ArgumentParser()
parser.add_argument('--host', '--host')
args, unknown = parser.parse_known_args()
print("Host = " + args.host)
Now when I give:
locust -f mysql_locust.py --host=myhost-vm-101 --no-web --clients=2 --hatch-rate=10 --run-time=5m
I get the print statement result as expected:
Host = myhost-vm-101
You can access it via sys.argv
Or parse options via argparse
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-H', '--host')
args, unknown = parser.parse_known_args()
print str(args.host)
How can I run a playbook in python script? What is the equivalent of the following using ansible module in python:
ansible -i hosts dbservers -m setup
ansible-playbook -i hosts -vvvv -k site.yml
I was looking at their documenation in http://docs.ansible.com/developing_api.html but they have very limited examples.
Deprecation Notice: This post doesn't work as of ansible 2. The API was changed.
This covered in the Ansible documentation under "Python API."
For example, ansible -i hosts dbservers -m setup is implemented via:
import ansible.runner
runner = ansible.runner.Runner(
module_name='setup',
module_args='',
pattern='dbservers',
)
dbservers_get_facts = runner.run()
There are a bunch of non-documented parameters in the __init__ method of Runner (from ansible.runner). There's too many to list inline, but I've included some of the parameters in this post as a guess to what you're specifically looking for.
class Runner(object):
''' core API interface to ansible '''
# see bin/ansible for how this is used...
def __init__(self,
host_list=C.DEFAULT_HOST_LIST, # ex: /etc/ansible/hosts, legacy usage
module_path=None, # ex: /usr/share/ansible
module_name=C.DEFAULT_MODULE_NAME, # ex: copy
module_args=C.DEFAULT_MODULE_ARGS, # ex: "src=/tmp/a dest=/tmp/b"
...
pattern=C.DEFAULT_PATTERN, # which hosts? ex: 'all', 'acme.example.org'
remote_user=C.DEFAULT_REMOTE_USER, # ex: 'username'
remote_pass=C.DEFAULT_REMOTE_PASS, # ex: 'password123' or None if using key
remote_port=None, # if SSH on different ports
private_key_file=C.DEFAULT_PRIVATE_KEY_FILE, # if not using keys/passwords
sudo_pass=C.DEFAULT_SUDO_PASS, # ex: 'password123' or None
...
sudo=False, # whether to run sudo or not
sudo_user=C.DEFAULT_SUDO_USER, # ex: 'root'
module_vars=None, # a playbooks internals thing
play_vars=None, #
play_file_vars=None, #
role_vars=None, #
role_params=None, #
default_vars=None, #
extra_vars=None, # extra vars specified with he playbook(s)
is_playbook=False, # running from playbook or not?
inventory=None, # reference to Inventory object
...
su=False, # Are we running our command via su?
su_user=None, # User to su to when running command, ex: 'root'
su_pass=C.DEFAULT_SU_PASS,
vault_pass=None,
...
):
For instance, the above command that specifies a sudo user and pass would be:
runner = ansible.runner.Runner(
module_name='setup',
module_args='',
pattern='dbservers',
remote_user='some_user'
remote_pass='some_pass_or_python_expression_that_returns_a_string'
)
For playbooks, look into playbook.PlayBook, which takes a similar set of initializers:
class PlayBook(object):
'''
runs an ansible playbook, given as a datastructure or YAML filename.
...
'''
# *****************************************************
def __init__(self,
playbook = None,
host_list = C.DEFAULT_HOST_LIST,
module_path = None,
....
and can be executed with the .run() method. e.g.:
from ansible.playbook import PlayBook
pb = PlayBook(playbook='/path/to/book.yml, --other initializers--)
pb.run()
more robust usage can be found in the ansible-playbook file.
As far as I know, translating playbooks to Python modules is a bit more involved, but the documentation listed above should get you covered and you can reuse the YAML parser built into Ansible to convert playbooks to variables.
Just a quick code update that works on 2.8.3,
from ansible import context
from ansible.cli import CLI
from ansible.module_utils.common.collections import ImmutableDict
from ansible.executor.playbook_executor import PlaybookExecutor
from ansible.parsing.dataloader import DataLoader
from ansible.inventory.manager import InventoryManager
from ansible.vars.manager import VariableManager
loader = DataLoader()
context.CLIARGS = ImmutableDict(tags={}, listtags=False, listtasks=False, listhosts=False, syntax=False, connection='ssh',
module_path=None, forks=100, remote_user='xxx', private_key_file=None,
ssh_common_args=None, ssh_extra_args=None, sftp_extra_args=None, scp_extra_args=None, become=True,
become_method='sudo', become_user='root', verbosity=True, check=False, start_at_task=None)
inventory = InventoryManager(loader=loader, sources=('/xxx/inventory_file',))
variable_manager = VariableManager(loader=loader, inventory=inventory, version_info=CLI.version_info(gitinfo=False))
pbex = PlaybookExecutor(playbooks=['/xxx/playbook.yml'], inventory=inventory, variable_manager=variable_manager, loader=loader, passwords={})
results = pbex.run()
I have answered the question here
Posting this here cause posting links is discouraged in the community. Hope it helps.
The documentation is surprisingly lacking and you'll have to get started here
That being said, here is a quick script I hacked together that manages to run a playbook.
#!/usr/bin/env python
import os
import sys
from collections import namedtuple
from ansible.parsing.dataloader import DataLoader
from ansible.vars.manager import VariableManager
from ansible.inventory.manager import Inventory
from ansible.executor.playbook_executor import PlaybookExecutor
loader = DataLoader()
inventory = Inventory(loader=loader, sources='/home/slotlocker/hosts2')
variable_manager = VariableManager(loader=loader, inventory=inventory)
playbook_path = '/home/slotlocker/ls.yml'
if not os.path.exists(playbook_path):
print '[INFO] The playbook does not exist'
sys.exit()
Options = namedtuple('Options', ['listtags', 'listtasks', 'listhosts', 'syntax', 'connection','module_path', 'forks', 'remote_user', 'private_key_file', 'ssh_common_args', 'ssh_extra_args', 'sftp_extra_args', 'scp_extra_args', 'become', 'become_method', 'become_user', 'verbosity', 'check','diff'])
options = Options(listtags=False, listtasks=False, listhosts=False, syntax=False, connection='ssh', module_path=None, forks=100, remote_user='slotlocker', private_key_file=None, ssh_common_args=None, ssh_extra_args=None, sftp_extra_args=None, scp_extra_args=None, become=True, become_method='sudo', become_user='root', verbosity=None, check=False, diff=False)
variable_manager.extra_vars = {'hosts': 'mywebserver'} # This can accomodate various other command line arguments.`
passwords = {}
pbex = PlaybookExecutor(playbooks=[playbook_path], inventory=inventory, variable_manager=variable_manager, loader=loader, options=options, passwords=passwords)
results = pbex.run()
Ansible provides the ansible-runner Python package (PyPI, GitHub), which can be used for this.
Usage example from the documentation:
import ansible_runner
r = ansible_runner.run(private_data_dir='/tmp/demo', playbook='test.yml')
print("{}: {}".format(r.status, r.rc))
# successful: 0
for each_host_event in r.events:
print(each_host_event['event'])
print("Final status:")
print(r.stats)
Based on vigilander's answer, I took the liberty to upload a pkg to pypi and now you can use this simple ansible runner:
Just pip install ansible-playbook-runner and run:
from ansible_playbook_runner import Runner
Runner(['inventory_path'], 'playbook_path').run()
You're looking at something that isn't officially supported or recommended therefore little documentation to be had.
That said, if you really want to pursue that course I would start with cracking open the ansible-playbook script in bin and reverse engineering what you want to do.
I use Python 2.6 / Fabric 1.8.0 and I have several hosts containing different local path and variable.
import
....
def vmm_two_a():
env.hosts = 'xxx.xx.xx.xx'
env.user = 'tom'
env.password = password_from_netrc(env.hosts,env.user)
global local_path
local_path = '/home/vmm_two_a/binaries'
def vmm_two_mgmt():
env.hosts = 'xxx.xx.xx.xx'
env.user = 'john'
env.password = password_from_netrc(env.hosts,env.user)
global local_path
local_path = '/home/vmm_two_mgmt/binaries'
def get_etc_binaries():
''' Get all etc binaries deployed '''
with hide('stderr','stdout','output','running', 'warnings'):
settings(warn_only=True)
get('/usr/local/etc', local_path)
How to run this task against all hosts? I've tried fab -H with no success.
Thanks for your help
In this instance you'd need to call the task setting the env vars before the get_etc_binaries task each time. You might also make local_path be env.local_path, so it's shared easier.
Call like this would look like: fab vmm_twomgmt get_etc_binaries vmm_two_a get_etc_binaries
I need to use Fabric to do some operations in a website that use one machine for the filesystem and other machine to the database server. I need to handle two hosts. How can I do that?
I have some code but I cannot get the environment definition to work.
The idea is to connect to the remote Filesystem server and get the files and then connect to the remote Database server and get the database schema.
The code that I have for now is something like this:
from __future__ import with_statement
from fabric.api import *
from fabric.contrib.console import confirm
'''
Here I define where is my "aid"s file structure
'''
local_root = '/home/andre/test' # This is the root folder for the audits
code_location = '/remote_code' # This is the root folder dor the customer code inside each audit
#
# ENVIRONMENTS CONFIGURATIONS
#
'''
Here I configure where is the remote file server
'''
def file_server():
env.user = 'andre'
env.hosts = ['localhost']
'''
Here I configure where is the database server
'''
def database_server():
env.user = 'andre'
env.hosts = ['192.168.5.1']
#
# START SCRIPT
#
def get_install(remote_location, aid):
### I will get the files
'''
Here I need to load the file_server() definitions
'''
working_folder = local_root + '/%s' % aid # I will define the working folder
local('mkdir ' + working_folder) # I will create the working folder for this audit
local('mkdir ' + working_folder + code_location) # I will create the folder to receive the code
get(remote_location, working_folder + code_location) # I will download the code to my machine
### I will get the database
'''
Here I need to load the database_server() definitions
'''
local('dir') # Just to test
How can I inside get_install() define the environments file_server() and database_server() ?
Best Regards,
I don't understand exactly what you are trying to do, but maybe you can split up your get_install function into two functions each for every server.
Then limit those functions to the correct servers with fabric.decorators.hosts(*host_list) decorator:
For example, the following will ensure that, barring an override on the command line, my_func will be run on host1, host2 and host3, and with specific users on host1 and host3:
#hosts('user1#host1', 'host2', 'user2#host3')
def my_func():
pass
(For more info see http://readthedocs.org/docs/fabric/en/1.1.0/api/core/decorators.html#fabric.decorators.hosts)
And you can than call those 2 functions in one go by defining your get_install method as:
def get_install():
func1()
func2()
You should be able to do this with fab database_server get_install. Basically, fab [environment] [command] should do what you want.