We Keep Coding

Edit YAML file with Bash

I'm trying to edit the following YAML file
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'password_admin'
To edit the host part, I got it working with
sed -i "/^\([[:space:]]*host: \).*/s//\1'$DNS_ENDPOINT'/" config.yml
But I can't find a way to update the password for main and admin (which are different values).
I tried to play around with \n and [[:space:]] and got different flavours of:
sed -i "/^\([[:space:]]*main:\n*[[:space:]]*password: \).*/s//\1'$DNS_ENDPOINT'/" config.yml
But never got it to work.
Any help greatly appreciated!
Edit - Requirement: no external binaries/tools. Just good ol' bash.

Since you don't want to install yq you could use python that you most probably already have installed.
Here are the fundamentals:
#!/usr/bin/python
import yaml
with open("config.yml") as f:
y = yaml.safe_load(f)
y['db']['admin']['password'] = 'new_admin_pass'
print(yaml.dump(y, default_flow_style=False, sort_keys=False))
Output:
db:
host: x.x.x.x.x
main:
password: password_main
admin:
password: new_admin_pass
A similar piece of python code as a one-liner that you can put in a bash script would look something like this (and produce the same output):
python -c 'import yaml;f=open("config.yml");y=yaml.safe_load(f);y["db"]["admin"]["password"] = "new_admin_pass"; print(yaml.dump(y, default_flow_style=False, sort_keys=False))'
If you'd like to save the output to a file, you can provide an output stream as the second argument to dump():
#!/usr/bin/python
import yaml
with open("config.yml") as istream:
ymldoc = yaml.safe_load(istream)
ymldoc['db']['admin']['password'] = 'new_admin_pass'
with open("modified.yml", "w") as ostream:
yaml.dump(ymldoc, ostream, default_flow_style=False, sort_keys=False)
If you'd like to overwrite the original file, I recommend writing to a temporary file first and only if that succeeds, use os.rename to move that file in place of the original one. That's to minimize the risk of creating a corrupt config.yml in case of problems.

Note: Using a YAML parser like yq (or yq) will be a way more reliable solution.
However, I've used the following 'technique' to alter a 'pre-defined' line though the help of grep and sed like so;
/tmp/config.yml
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'password_admin'
Get the line number where your 'old-password' is located:
grep -n 'password_admin' /tmp/config.yml | cut -d ':' -f1
6
Then, use sed to override that line with your new password:
sed -i '6s/.*/ password: \'new_admin_pass\'/' /tmp/config.yml
The new file now looks like this:
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'new_admin_pass'
Note
Keep in mind that any special chars (&, \, /) in the password will cause sed to misbehave!
This could fail if the indent changes, since YAML cares about indentation. Just like I mentioned above, using a YAML parser will be a much more reliable solution!

$ awk -v new="'sumthin'" 'prev=="main:"{sub(/\047.*/,""); $0=$0 new} {prev=$1} 1' file
db:
host: 'x.x.x.x.x'
main:
password: 'sumthin'
admin:
password: 'password_admin'
or if your new text can contain escape sequences that you don't want expanded (e.g. \t or \n), as seems likely when setting a password, then:
new="'sumthin'" awk 'prev=="main:"{sub(/\047.*/,""); $0=$0 ENVIRON["new"]} {prev=$1} 1' file
See How do I use shell variables in an awk script? for why/how I use ENVIRON[] to access a shell variable rather than setting an awk variable in that second script.

This is by no way as reliable as yq but you can use this awk if your yaml file structure is same as how it is shown in question:
pw='new_&pass'
awk -v pw="${pw//&/\\\\&}" '/^[[:blank:]]*main:/ {
print
if (getline > 0 && $1 == "password:")
sub(/\047[^\047]*\047/, "\047" pw "\047")
} 1' file
db:
host: 'x.x.x.x.x'
main:
password: 'new_&pass'
admin:
password: 'password_admin'

As mentioned by experts in other answers too, yq should be the proper way but in case someone doesn't have it then one could try following.
awk -v s1="'" -v new_pass="new_value_here" '
/main:/{
main_found=1
print
next
}
main_found && /password/{
next
}
/admin:/ && main_found{
print " password: " s1 new_pass s1 ORS $0
main_found=""
next
}
1
' Input_file
NOTE: In case you want to save output into Input_file itself then append > temp && mv temp Input_file to above solution.

Attempt to generate email with new lines from Python through Bash mailx command

Where I work, we have servers that are pre-configured for the use of the bash mail command to send attachments and messages. I'm working on a notification script that will monitor server activity and generate an email if it detects an issue. I'm using the subprocess.call function in order to send a bash command.
I am successful in sending messages, but in the body portion of the email, it is stringing each notification line together rather than putting each notification on a separate line. I have tried to append each line within the string with "\n" and "\r\n". I have to use double backslashes as python will interpret this as literal new lines when it sends the echo command. I also passed the command "shopt -s xpg_echo" before using the echo with pipe to mail using the double backspaces but this also had no effect. I also tried using echo without the "-e" option and this had no effect either.
The trick is that I need python to send the new line to bash and then somehow get bash to interpret this as a new line using echo piped through to mail. Here is a sample of the code:
import os
import shutil
import sys
import time
import re
import subprocess
import smtplib
serviceports["SCP Test"] = ["22"]
serviceports["Webtier"] = ["9282"]
bashCommand = "netstat -an | grep LISTEN | grep -v LISTENING"
netstat_results = subprocess.check_output(bashCommand, shell=True)
netstat_results = str(netstat_results)
#Iterate through all ports for each service and assign down ports to variable
for servicename, ports in serviceports.items():
for ind_port in ports:
ind_port_chk = ":" + ind_port
count = sum(1 for _ in re.finditer(r'\b%s\b' % re.escape(ind_port_chk), netstat_results))
if count == 0:
warning = servicename + " on port " + ind_port + " is currently down!"
report.append(warning)
for warning in report:
message = message + warning + "\\\n"
fromaddr=serveridsimp + "#xxxxx.com"
toaddr='email#xxxxx.com'
subject="Testing..."
body=message
cmd= cmd='echo -e '+body+' | mail -s '+subject+' -r '+fromaddr+' '+toaddr
send=subprocess.call(cmd,shell=True)
The code runs a netstat command and assigns it to a string. The code will then iterate through the specified ports and search for where that port doesn't exist in the netstat string (netstat_results). It then will create a list object (warning) containing all the ports not located in netstat_results and then append each line adding \n to a string called "message". It then sends an echo piped to the xmail command to generate an email to be sent containing all the ports not found. What happens currently is that I will get an email saying something like this:
SCP Test on port 22 is currently down!nOHS Webtier on port 9282 is currently down!n etc...
I want it to put each message on a new line like so:
SCP Test on port 22 is currently down!
Webtier on port 9282 is currently down!
I am trying to avoid writing the output to a file and then using bash to read it back into the mail command. Is this possible without having to create a file?

I was finally able to fix the issue by changing the command sent to bash and character being appended to the following:
message = message + warning + "\n"
cmd= cmd='echo -e '+'"'+body+'"'+'|awk \'{ print $0" " }\''+' | mail -s '+'"'+subject+'"'+' -r '+fromaddr+' '+toaddr

How do I change the hostname using Python on a Raspberry Pi

I tried using (going from memory, this may not be 100% accurate):
import socket
socket.sethostname("NewHost")
I got a permissions error.
How would I approach this entirely from within the Python program?

If you only need to do change the hostname until the next reboot, many linux system can change it with:
import subprocess
subprocess.call(['hostname', 'newhost'])
or with less typing but some potential pitfalls:
import os
os.system('hostname %s' % 'newhost')

I wanted to change the hostname permanently, which required making changes in a few places, so I made a shell script:
#!/bin/bash
# /usr/sbin/change_hostname.sh - program to permanently change hostname. Permissions
# are set so that www-user can `sudo` this specific program.
# args:
# $1 - new hostname, should be a legal hostname
sed -i "s/$HOSTNAME/$1/g" /etc/hosts
echo $1 > /etc/hostname
/etc/init.d/hostname.sh
hostname $1 # this is to update the current hostname without restarting
In Python, I ran the script with subprocess.run:
subprocess.run(
['sudo', '/usr/sbin/change_hostname.sh', newhostname])
This was happening from a webserver which was running as www-data, so I allowed it to sudo this specific script without a password. You can skip this step and run the script without sudo if you're running as root or similar:
# /etc.d/sudoers.d/099-www-data-nopasswd-hostname
www-data ALL = (root) NOPASSWD: /usr/sbin/change_hostname.sh

Here is a different approach
import os
def setHostname(newhostname):
with open('/etc/hosts', 'r') as file:
# read a list of lines into data
data = file.readlines()
# the host name is on the 6th line following the IP address
# so this replaces that line with the new hostname
data[5] = '127.0.1.1 ' + newhostname
# save the file temporarily because /etc/hosts is protected
with open('temp.txt', 'w') as file:
file.writelines( data )
# use sudo command to overwrite the protected file
os.system('sudo mv temp.txt /etc/hosts')
# repeat process with other file
with open('/etc/hostname', 'r') as file:
data = file.readlines()
data[0] = newhostname
with open('temp.txt', 'w') as file:
file.writelines( data )
os.system('sudo mv temp.txt /etc/hostname')
#Then call the def
setHostname('whatever')
At the next reboot the hostname will be set to the new name

Pass a variable to a path/command in python

I am new to Python. I want to execute a script which has parameters in python. Please have a look at below example on what I finally want to achieve.
I have a script with parameters:
PATH="ABCXYZ"
username = "admin"
password = "pass"
query = "select * from user;"
OUTPUT = /usr/local/<PATH>/dbscript -u username -p password -q query
How can I pass a PATH variable in "/usr/local/<PATH>/dbscript" here ?
I tried with os.system & subprocess, But We cannot use variable in between.
I want that OUTPUT variable data which will print yes or no or some number

Below must work.
import subprocess as sp
PATH="ABCXYZ"
username = "admin"
password = "pass"
query = "select * from user;"
command = "/usr/local/%s/dbscript -u %s -p %s -q %s" % (PATH, username, password, query)
#Next line print '/usr/local/ABCXYZ/dbscript -u admin -p pass -q select * from user;' for me.
print command #
sp.Popen(command.split(), shell=True)
Furthermore you should play with shell parameter and useful read it.

Change Unix password from command line over Python/Fabric

I would like a way to update my password on a remote Ubuntu 10.4 box with fabric.
I would expect my fabfile.py would look something like this:
def update_password(old_pw, new_pw):
# Connects over ssh with a public key authentication
run("some_passwd_cmd --old %s --new %s" % (old_pw, new_pd))
Unfortunately the only command I know of that lets one change the password is passwd, and on Ubuntu 10.4 there doesn't seem to be any way to pass in the new (or old) password as an argument to passwd.
What command could one use to change a user's password on Ubuntu 10.4 via fabric?
EDIT:
I've had a look at usermod -p, and that may work but it isn't recommended by the man page.
EDIT: For some reason usermod -p wasn't working either over fabric.
As well, I've tried a (somewhat insecure) variation on mikej's answer that did solve the problem:
# connecting & running as root.
from fabric.api import *
from fabric.contrib import files
files.append("%s\n%s" % (passwd, passwd), '.pw.tmp')
# .pw.tmp:
# PASSWD
# PASSWD
run("passwd %s < .pw.tmp" % user)
run("rm .pw.tmp")
It's not a very elegant solution, but it works.
Thank you for reading.
Brian

You could feed the new and old passwords into passwd using echo e.g.
echo -e "oldpass\\nnewpass\\nnewpass" | passwd
(the -e option for echo enables interpretation of backslash escapes so the newlines are interpreted as such)

The trick is to use a combination of usermod and Python’s crypt to change your password:
from crypt import crypt
from getpass import getpass
from fabric.api import *
def change_password(user):
password = getpass('Enter a new password for user %s:' % user)
crypted_password = crypt(password, 'salt')
sudo('usermod --password %s %s' % (crypted_password, user), pty=False)

I use chpasswd on Ubuntu 11.04
fabric.api.sudo('echo %s:%s | chpasswd' % (user, pass))
Note:
Normally this pattern doesn't work:
$ sudo echo bla | restricted_command
because only the 'echo' gets elevated privileges, not the 'restricted_command'.
However, here it works because when fabric.api.sudo is caled
with shell=True (the default), fabric assembles the command like this:
$ sudo -S -p <sudo_prompt> /bin/bash -l -c "<command>"
sudo spawns a new shell (/bin/bash), running with root privileges, and
then that escalated shell runs the command.
Another way to pipe with sudo is to use sudo tee:

Out of interest, I have to do a similar task on a collection of Solaris boxes (add a whole lot of users, set their password). Solaris usermod doesn't have a --password option, so in the past I've used Expect to do this, but writing Expect scripts can be painful.
So this time I'm going to use Python's crypt.crypt, edit /etc/shadow directly (with backups, of course). http://docs.python.org/release/2.6.1/library/crypt.html
Commenters have suggested using various echo incantations piped to passwd. AFAIK this will never work, as passwd is programmed to ignore input from stdin and only accept input from an interactive tty. See http://en.wikipedia.org/wiki/Expect

I had no luck with the other methods. Thought I would share my method that I used for a once-off throwaway script.
It uses auto-responder to type in passwords at the prompts. I then immediately expire all the passwords so that users have a chance to choose their own.
This is not the most secure method, but depending on your use case it may be useful.
from collections import namedtuple
from getpass import getpass
import hashlib
from invoke import Responder
import uuid
from fabric import Connection, Config
User = namedtuple('UserRecord', ('name', 'password'))
def set_passwords(conn, user):
print(f'Setting password for user, {user.name}')
responder = Responder(
pattern=r'(?:Enter|Retype) new UNIX password:',
response=f'{user.password}\n',
)
result = conn.sudo(f'passwd {user.name}', warn=True, hide='both',
user='root', pty=True, watchers = [responder])
if result.exited is not 0:
print(f'Error, could not set password for user, "{user.name}". command: '
f'{result.command}; exit code: {result.exited}; stderr: '
f'{result.stderr}')
else:
print(f'Successfully set password for {user.name}')
def expire_passwords(conn, user):
print(f'Expiring password for user, {user.name}')
cmd = f'passwd --expire {user.name}'
result = conn.sudo(cmd, warn=True, user='root')
if result.exited is not 0:
print(f'Error, could not expire password for user, "{user.name}". '
f'command: {result.command}; exit code: {result.exited}; stderr: '
f'{result.stderr}')
else:
print(f'Successfully expired password for {user.name}')
def gen_password(seed_string):
# Don't roll your own crypto. This is for demonstration only and it is
# expected to only create a temporary password that requires changing upon
# initial login. I am no cryptography expert, hence this alternative
# simplified answer to the one that uses crypt, salt, etc -
# https://stackoverflow.com/a/5137688/1782641.
seed_str_enc = seed_string.encode(encoding='UTF-8')
uuid_obj = uuid.UUID(int=int(hashlib.md5(seed_str_enc).hexdigest(), 16))
return str(uuid_obj)[:8]
def some_function_that_returns_something_secret(conn):
return f'dummy-seed-{conn}'
sudo_pass = getpass('Enter your sudo password:')
config = Config(overrides={'sudo': {'password': sudo_pass}})
with Connection('vm', config=config) as vm_conn:
print(f'Making a new connection to {vm_conn.host}.')
# I usually use the sudo connection here to run a command that returns a
# reproducible string that only the sudo user could get access to be used
# for user_record.password bellow. Proceed with caution, this is not a
# recommended approach
seed = some_function_that_returns_something_secret(vm_conn)
user_record = User(name='linux_user', password=gen_password(seed))
set_passwords(vm_conn, user_record)
expire_passwords(vm_conn, user_record)
print(f'Done! Disconnecting from {vm_conn.host}.')
# So that you know the temporary password, print user_record or save to file
# `ssh linux_user#vm` and it should insist that you change password
print(user_record)