By no means do I write scripts very often, but I am trying to write a Nagios plugin to check the status of a RAID controller on a remote host. The issue is that the command to get the output requires elevated privileges. What would be the correct, and most effective way to pull this off? The goal is to run:
'/opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary -a0'
on a remote host from the monitoring server,
and then follow the basic idea of this logic:
#Nagios Plugin for Testing LSI Raid Status
import os, sys
import argparse
import socket
import subprocess
#nagios exit codes do not change#
OK = 0
WARNING = 1
CRITICAL = 2
DEPENDENT = 3
UNKNOWN = 4
#nagios exit codes do not change#
#patterns to be searched
active = str("Active")
online = str("Online")
k = str("OK")
degrade = str("Degraded")
fail = str("Failed")
parser = argparse.ArgumentParser(description='Py3 script for monitoring RAID status.')
#arguments
parser.add_argument("--user",
metavar = '-U',
help = "username for remote connection")
parser.add_argument("--hostname",
metavar = '-H',
help = "hostname of the remote host")
args = parser.parse_args()
print(args)
#turning args into variables
hostname = args.hostname
user = args.user
ssh = subprocess.Popen(f"ssh {user}#{hostname} /opt/MegaRAID/MegaCli/MegaCli64 -ShowSummary -a0", shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
check = ssh.stdoutreadlines()
OK_STR = str("RAID is OK!")
WARN_STR = str("Warning! Something is wrong with the RAID!")
CRIT_STR = str("CRITICAL! THE RAID IS BROKEN")
UNK_STR = str("Uh oh! Something ain't right?")
if (degrade) in (check):
print(WARN_STR) and exit(WARNING)
elif (fail) in (check):
print (CRIT_STR) and exit(CRITICAL)
elif (active) or (online) or (k) in (check):
print(OK_STR) and exit(OK)
else:
print(UNK_STR) and exit(UNKNOWN)
Any thoughts? This is far from my forte (and also an unfinished script) so I apologize for the layman format and any confusion in my phrasing.
I am trying to write a Nagios plugin to check the status of a RAID controller on a remote host. The issue is that the command to get the output requires elevated privileges. What would be the correct, and most effective way to pull this off?
I would recommend running the script remotely over NRPE on the system in question, and then give the user the NRPE daemon is running as (probably nagios or similar) sudo permissions to run that script with some very exact parameters.
The nrpe.cfg file mentions this example:
# Usage scenario:
# Execute restricted commmands using sudo. For this to work, you need to add
# the nagios user to your /etc/sudoers. An example entry for alllowing
# execution of the plugins from might be:
#
# nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
...but there's no reason to be so forgiving, you can make it a lot safer by only allowing an exact command:
nagios ALL = NOPASSWD: /usr/sbin/megacli
Note that this allows any parameters with that command, this is even safer as it will not allow any other variants (example):
nagios ALL = NOPASSWD: /usr/sbin/megacli -a foo -b bar -c5 -w1
Then configure the nrpe command to run the above with sudo before it, and it should work. You can verify by su:ing to the nagios user and trying the sudo command yourself.
Also, note that there are very likely some available modules you can import for python nagios plugins that makes it easier for you, to get built-in support for things like thresholds and their syntax.
Related
I have python script which run during the start of the linux VM:
I have added it in the chkconfig 345
script is supposed to check the hostname and if it is localhost.localdom then it should exit
#!/usr/bin/python
import subprocess,platform,os,sys,logging,shlex
system_name = os.getenv('HOSTNAME')
if system_name == 'localhost.localdom':
logging.info('Please correct host name for proxies, it is showing localhost')
sys.exit()"
if I run it manually it works fine. But during the startup process, even though the hostname is localhost.localdom. It does not exit.
so it look like during the boot process,
os.getenv('HOSTNAME')
is not returning the localshot.localdom what I have set in condition.
Please help getting this to work during reboot.
Thanks,
Jitendra Singh
posting an answer using the info in Isedev's comment...
you could try getting the hostname by:
import os
system_name = os.popen('/bin/hostname')
if system_name.read().rstrip() == 'localhost.localdom':
logging.info('Please correct host name for proxies, it is showing localhost')
sys.exit()
I've some fabric tasks in my fabfile and I need to initialize, the env variable before their execution. I'm trying to use a decorator, it works but fabric always says "no host found Please specify (single)" however if I print the content of my variable "env" all seems good.
Also I call my tasks from another python script.
from fabric.api import *
from instances import find_instances
def init_env(func):
def wrapper(*args, **kwargs):
keysfolder = 'keys/'
env.user = 'admin'
env.key_filename = '%skey_%s_prod.pem'%(keysfolder, args[0])
env.hosts = find_instances(args[1])
return func(args[0], args[1])
return wrapper
#init_env
def restart_apache2(region, groupe):
print(env.hosts)
run('/etc/init.d/apache2 restart')
return True
My script which call the fabfile:
from fabfile import init_env, restart_apache2
restart_apache2('eu-west-1', 'apache2')
Output of print in restart apache2:
[u'10.10.0.1', u'10.10.0.2']
Any idea why my task restart_apache2 doesn't use the env variable?
Thanks
EDIT:
Which is interesting it's if in my script which calls the fabfile, I use settings from fabric.api and set a host ip, it works. This show that my decorator has well initialized the env variable because the key and user are send to fabric. It's only the env.hosts that's not read by fabric...
EDIT2:
I can reach my goal with using settings from fabric.api, like that:
#init_env
def restart_apache2(region, groupe):
for i in env.hosts:
with settings(host_string = '%s#%s' % (env.user, i)):
run('/etc/init.d/apache2 restart')
return True
Bonus question, has there a solution to use directly the env.hosts without settings?
I'm guessing here a little, but I'm assuming you've got into trouble because you're trying to solve two problems at once.
The first issue relates to the issue of multiple hosts. Fabric includes the concepts of roles, which are just groups of machines that you can issue commands to in one go. The information in the find_instances function could be used to populate this data.
from fabric import *
from something import find_instances
env.roledefs = {
'eu-west-1' : find_instances('eu-west-1'),
'eu-west-2' : find_instances('eu-west-2'),
}
#task
def restart_apache2():
run('/etc/init.d/apache2 restart')
The second issue is that you have different keys for different groups of servers. One way to resolve this problem is to use an SSH config file to prevent you from having to mix the details of the keys / users accounts with your fabric code. You can either add an entry per instance into your ~/.ssh/config, or you can use local SSH config (env.use_ssh_config and env.ssh_config_path)
Host instance00
User admin
IdentityFile keys/key_instance00_prod.pem
Host instance01
User admin
IdentityFile keys/key_instance01_prod.pem
# ...
On the command line, you should then be able to issue the commands like:
fab restart_apache2 -R eu-west-1
Or, you can still do single hosts:
fab restart_apache2 -H apache2
In your script, these two are equivalent to the execute function:
from fabric.api import execute
from fabfile import restart_apache2
execute(restart_apache2, roles = ['eu-west-1'])
execute(restart_apache2, hosts = ['apache2'])
We're experiencing some strange issues with Nagios. We wrote a script in Python which uses paramiko, httplib and re. When we comment out the code that is written to use paramiko the script returns OK in Nagios. When we uncomment the script the status just returns (null).
This is our code
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
#ssh.connect(self.get('hostname'),int(self.get('port')),self.get('username'),allow_agent=True)
ssh.connect('192.168.56.102' , 22 , 'oracle' ,allow_agent=True)
link = '127.0.0.1:4848'
stdin,stdout,stderr = ssh.exec_command('wget --no-proxy ' + link + ' 2>&1 | grep -i "failed\|error"')
result = stdout.readlines()
result = " ".join(result)
if result == "":
return MonitoringResult(MonitoringResult.OK,'Webservice up')
else:
return MonitoringResult(MonitoringResult.CRITICAL,'Webservice down %s' % result)
So when we comment out the part above
if result =="":
and add result="" above the if, it will just return 'Webservice up' in Nagios. When we enable the code above the if it will just return (null). Is there a conflict with paramiko or something?
When running the code in terminal it just returns the correct status but it doesn't show when implemented in Nagios
I found that although nagios runs as effective user "nagios" it is using the environment settings for user "root" and not able to read the root private key to make the connection.
Adding key_filename='/nagiosuserhomedir/.ssh/id_dsa' to the options for ssh.connect() solved this same problem for me (pass the private key for nagios user explicitly in the code).
This question is about best practices. I'm running a deployment script with Fabric. My deployment user 'deploy' needs sudo to restart services. So I am using the sudo function from fabric to run these commands in my script. This works fine but prompts for password during script execution. I DON'T want to type a password during deployments. What's the best practice here. The only solution I can think of is changing the sudo permissions to not require password for the commands my deployment user runs. This doesn't seem right to me.
The ideal solution is to create a user on your server that is used only for deployment (eg, deploy). Then, set env.user=deploy in your fabfile. Then on your servers, you can give the user the necessary permission on a command-by-command basis in a sudoers file:
IMPORTANT: Always use sudo visudo to modify a sudoers file
Cmnd_Alias RELOAD_SITE = /bin/bash -l -c supervisorctl*, /usr/bin/supervisorctl*
deploy ALL = NOPASSWD: RELOAD_SITE
You can add as many Cmnd_Alias directives as is needed by the deploy user, then grant NOPASSWD access for each of those commands. See man sudoers for more details.
I like to keep my deploy-specific sudoers config in /etc/sudoers.d/deploy and include that file from /etc/sudoers by adding: includedir /etc/suoders.d at the end.
You can use:
fabric.api import env
# [...]
env.password = 'yourpassword'
The best way to do this is with subtasks. You can prompt for a password in the fabfile and never expose any passwords, nor make reckless configuration changes to sudo on the remote system(s).
import getpass
from fabric.api import env, parallel, run, task
from fabric.decorators import roles
from fabric.tasks import execute
env.roledefs = {'my_role': ['host1', 'host2']}
#task
# #parallel -- uncomment if you need parallel execution, it'll work!
#roles('my_role')
def deploy(*args, **kwargs):
print 'deploy args:', args, kwargs
print 'password:', env.password
run('echo hello')
#task
def prompt(task_name, *args, **kwargs):
env.password = getpass.getpass('sudo password: ')
execute(task_name, *args, role='my_role', **kwargs)
Note that you can even combine this with parallel execution and the prompt task still only runs once, while the deploy task runs for each host in the role, in parallel.
Finally, an example of how you would invoke it:
$ fab prompt:deploy,some_arg,another_arg,key=value
Seems like sudo may not be that bad of an option after all. You can specify which commands a user can run and the arguments the command may take (man sudoers). If the problem is just having to type the password, an option would involve using the pexpect module to login automatically, maybe with a password that you could store encrypted:
import pexpect, sys
pwd = getEncryptedPassword()
cmd = "yourcommand"
sCmd = pexpect.spawn('sudo {0}'.format(cmd))
sCmd.logfile_read = sys.stdout
sCmd.expect('Password:')
sCmd.sendline(pwd)
sCmd.expect(pexpect.EOF)
Use the keyring module to store and access passwords securely.
Here's how I do it with Fabric 2:
from fabric import task
import keyring
#task
def restart_apache(connection):
# set the password with keyring.set_password('some-host', 'some-user', 'passwd')
connection.config.sudo.password = keyring.get_password(connection.host, 'some-user')
connection.sudo('service apache2 restart')
You could also use GPG or any other command-line password tool. For example:
connection.config.sudo.password = connection.local('gpg --quiet -d /path/to/secret.gpg', hide=True).strip()
The secret.gpg file can be generated with echo "mypassword" | gpg -e > secret.gpg. The hide argument avoids echoing the password to the console.
To retain support for --prompt-for-sudo-password, add a conditional:
if not connection.config.sudo.password:
connection.config.sudo.password = keyring.get_password(connection.host, 'some-user')
You can also use passwords for multiple machines:
from fabric import env
env.hosts = ['user1#host1:port1', 'user2#host2.port2']
env.passwords = {'user1#host1:port1': 'password1', 'user2#host2.port2': 'password2'}
See this answer: https://stackoverflow.com/a/5568219/552671
As Bartek also suggests, enable password-less sudo for the deployment 'user' in the sudoers file.
Something like:
run('echo "{0} ALL=(ALL) ALL" >> /etc/sudoers'.format(env.user))
How do you configure fabric to connect to remote hosts using SSH keyfiles (for example, Amazon EC2 instances)?
Finding a simple fabfile with a working example of SSH keyfile usage isn't easy for some reason. I wrote a blog post about it (with a matching gist).
Basically, the usage goes something like this:
from fabric.api import *
env.hosts = ['host.name.com']
env.user = 'user'
env.key_filename = '/path/to/keyfile.pem'
def local_uname():
local('uname -a')
def remote_uname():
run('uname -a')
The important part is setting the env.key_filename environment variable, so that the Paramiko configuration can look for it when connecting.
Also worth mentioning here that you can use the command line args for this:
fab command -i /path/to/key.pem [-H [user#]host[:port]]
Another cool feature available as of Fabric 1.4 - Fabric now supports SSH configs.
If you already have all the SSH connection parameters in your ~/.ssh/config file, Fabric will natively support it, all you need to do is add:
env.use_ssh_config = True
at the beginning of your fabfile.
For fabric2 in fabfile use the following:
from fabric import task, Connection
#task
def staging(ctx):
ctx.name = 'staging'
ctx.user = 'ubuntu'
ctx.host = '192.1.1.1'
ctx.connect_kwargs.key_filename = os.environ['ENV_VAR_POINTS_TO_PRIVATE_KEY_PATH']
#task
def do_something_remote(ctx):
with Connection(ctx.host, ctx.user, connect_kwargs=ctx.connect_kwargs) as conn:
conn.sudo('supervisorctl status')
and run it with:
fab staging do_something_remote
UPDATE:
For multiple hosts (one host will do also) you can use this:
from fabric2 import task, SerialGroup
#task
def staging(ctx):
conns = SerialGroup(
'user#10.0.0.1',
'user#10.0.0.2',
connect_kwargs=
{
'key_filename': os.environ['PRIVATE_KEY_TO_HOST']
})
ctx.CONNS = conns
ctx.APP_SERVICE_NAME = 'google'
#task
def stop(ctx):
for conn in ctx.CONNS:
conn.sudo('supervisorctl stop ' + ctx.APP_SERVICE_NAME)
and run it with fab or fab2:
fab staging stop
For me, the following didn't work:
env.user=["ubuntu"]
env.key_filename=['keyfile.pem']
env.hosts=["xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"]
or
fab command -i /path/to/key.pem [-H [user#]host[:port]]
However, the following did:
env.key_filename=['keyfile.pem']
env.hosts=["ubuntu#xxx-xx-xxx-xxx-southeast-1.compute.amazonaws.com"]
or
env.key_filename=['keyfileq.pem']
env.host_string="ubuntu#xxx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com"
I had to do this today, my .py file was as simple as possible, like the one posted in the answer of #YuvalAdam but still I kept getting prompted for a password...
Looking at the paramiko (the library used by fabric for ssh) log, I found the line:
Incompatible ssh peer (no acceptable kex algorithm)
I updated paramiko with:
sudo pip install paramiko --upgrade
And now it's working.
None of these answers worked for me on py3.7, fabric2.5.0 and paramiko 2.7.1.
However, using the PKey attribute in the documentation does work: http://docs.fabfile.org/en/2.5/concepts/authentication.html#private-key-objects
from paramiko import RSAKey
ctx.connect_kwargs.pkey = RSAKey.from_private_key_file('path_to_your_aws_key')
with Connection(ctx.host, user, connect_kwargs=ctx.connect_kwargs) as conn:
//etc....
As stated above, Fabric will support .ssh/config file settings after a fashion, but using a pem file for ec2 seems to be problematic. IOW a properly setup .ssh/config file will work from the command line via 'ssh servername' and fail to work with 'fab sometask' when env.host=['servername'].
This was overcome by specifying the env.key_filename='keyfile' in my fabfile.py and duplicating the IdentityFile entry already in my .ssh/config.
This could be either Fabric or paramiko, which in my case was Fabric 1.5.3 and Paramiko 1.9.0.