I'm trying to edit the following YAML file
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'password_admin'
To edit the host part, I got it working with
sed -i "/^\([[:space:]]*host: \).*/s//\1'$DNS_ENDPOINT'/" config.yml
But I can't find a way to update the password for main and admin (which are different values).
I tried to play around with \n and [[:space:]] and got different flavours of:
sed -i "/^\([[:space:]]*main:\n*[[:space:]]*password: \).*/s//\1'$DNS_ENDPOINT'/" config.yml
But never got it to work.
Any help greatly appreciated!
Edit - Requirement: no external binaries/tools. Just good ol' bash.
Since you don't want to install yq you could use python that you most probably already have installed.
Here are the fundamentals:
#!/usr/bin/python
import yaml
with open("config.yml") as f:
y = yaml.safe_load(f)
y['db']['admin']['password'] = 'new_admin_pass'
print(yaml.dump(y, default_flow_style=False, sort_keys=False))
Output:
db:
host: x.x.x.x.x
main:
password: password_main
admin:
password: new_admin_pass
A similar piece of python code as a one-liner that you can put in a bash script would look something like this (and produce the same output):
python -c 'import yaml;f=open("config.yml");y=yaml.safe_load(f);y["db"]["admin"]["password"] = "new_admin_pass"; print(yaml.dump(y, default_flow_style=False, sort_keys=False))'
If you'd like to save the output to a file, you can provide an output stream as the second argument to dump():
#!/usr/bin/python
import yaml
with open("config.yml") as istream:
ymldoc = yaml.safe_load(istream)
ymldoc['db']['admin']['password'] = 'new_admin_pass'
with open("modified.yml", "w") as ostream:
yaml.dump(ymldoc, ostream, default_flow_style=False, sort_keys=False)
If you'd like to overwrite the original file, I recommend writing to a temporary file first and only if that succeeds, use os.rename to move that file in place of the original one. That's to minimize the risk of creating a corrupt config.yml in case of problems.
Note: Using a YAML parser like yq (or yq) will be a way more reliable solution.
However, I've used the following 'technique' to alter a 'pre-defined' line though the help of grep and sed like so;
/tmp/config.yml
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'password_admin'
Get the line number where your 'old-password' is located:
grep -n 'password_admin' /tmp/config.yml | cut -d ':' -f1
6
Then, use sed to override that line with your new password:
sed -i '6s/.*/ password: \'new_admin_pass\'/' /tmp/config.yml
The new file now looks like this:
db:
host: 'x.x.x.x.x'
main:
password: 'password_main'
admin:
password: 'new_admin_pass'
Note
Keep in mind that any special chars (&, \, /) in the password will cause sed to misbehave!
This could fail if the indent changes, since YAML cares about indentation. Just like I mentioned above, using a YAML parser will be a much more reliable solution!
$ awk -v new="'sumthin'" 'prev=="main:"{sub(/\047.*/,""); $0=$0 new} {prev=$1} 1' file
db:
host: 'x.x.x.x.x'
main:
password: 'sumthin'
admin:
password: 'password_admin'
or if your new text can contain escape sequences that you don't want expanded (e.g. \t or \n), as seems likely when setting a password, then:
new="'sumthin'" awk 'prev=="main:"{sub(/\047.*/,""); $0=$0 ENVIRON["new"]} {prev=$1} 1' file
See How do I use shell variables in an awk script? for why/how I use ENVIRON[] to access a shell variable rather than setting an awk variable in that second script.
This is by no way as reliable as yq but you can use this awk if your yaml file structure is same as how it is shown in question:
pw='new_&pass'
awk -v pw="${pw//&/\\\\&}" '/^[[:blank:]]*main:/ {
print
if (getline > 0 && $1 == "password:")
sub(/\047[^\047]*\047/, "\047" pw "\047")
} 1' file
db:
host: 'x.x.x.x.x'
main:
password: 'new_&pass'
admin:
password: 'password_admin'
As mentioned by experts in other answers too, yq should be the proper way but in case someone doesn't have it then one could try following.
awk -v s1="'" -v new_pass="new_value_here" '
/main:/{
main_found=1
print
next
}
main_found && /password/{
next
}
/admin:/ && main_found{
print " password: " s1 new_pass s1 ORS $0
main_found=""
next
}
1
' Input_file
NOTE: In case you want to save output into Input_file itself then append > temp && mv temp Input_file to above solution.
Related
I have been asked to create a user and password from two files in ubuntu. Below is what i did in python.
import os
with open("users.txt",'r') as file, open("passwords.txt", 'r') as password:
for line in file:
pass
for passw in password:
os.system('useradd ' +line)
os.system('echo +line +passw | chpasswd')
Contents of users.txt
avinash
ananthram
Contents of passwords.txt
lifeisbeautiful
lifeisbeautifulagain
It gives me an error in the last line saying chpasswd: line 1: missing new password. I have tried using os.system("echo +line +passw | chpasswd") but still it gives me the error. Can someone help me through this? Thanks in advance
I am expecting to create users with the password from two files.
You have to loop through the lines in both files simultaneously. Code you provided reads the first line from user file, then loops through all the passwords trying to set those passwords for the first user. By the time you reach read second line in users file you have already reached EOF in password file.
Substitute line and passw in last command with those variables' values.
chpasswd expects user and password delimited by colon, not space
add -n to echo to suppress adding newline character at the end
import os
with open("users.txt",'r') as file, open("passwords.txt", 'r') as password:
for line, passw in zip(file, password):
os.system('useradd ' +line)
os.system(f'echo -n "{line}:{passw}" | chpasswd')
Try this
os.system(f'echo -e "{line}\n{passw}" | chpasswd')
Explanation - The echo command is used to write a string to the standard output, but you are using it to construct the command that you want to pass to chpasswd.
Instead of using echo, you can use the -e option of echo to include a new line in the string being passed to chpasswd.
I tried using (going from memory, this may not be 100% accurate):
import socket
socket.sethostname("NewHost")
I got a permissions error.
How would I approach this entirely from within the Python program?
If you only need to do change the hostname until the next reboot, many linux system can change it with:
import subprocess
subprocess.call(['hostname', 'newhost'])
or with less typing but some potential pitfalls:
import os
os.system('hostname %s' % 'newhost')
I wanted to change the hostname permanently, which required making changes in a few places, so I made a shell script:
#!/bin/bash
# /usr/sbin/change_hostname.sh - program to permanently change hostname. Permissions
# are set so that www-user can `sudo` this specific program.
# args:
# $1 - new hostname, should be a legal hostname
sed -i "s/$HOSTNAME/$1/g" /etc/hosts
echo $1 > /etc/hostname
/etc/init.d/hostname.sh
hostname $1 # this is to update the current hostname without restarting
In Python, I ran the script with subprocess.run:
subprocess.run(
['sudo', '/usr/sbin/change_hostname.sh', newhostname])
This was happening from a webserver which was running as www-data, so I allowed it to sudo this specific script without a password. You can skip this step and run the script without sudo if you're running as root or similar:
# /etc.d/sudoers.d/099-www-data-nopasswd-hostname
www-data ALL = (root) NOPASSWD: /usr/sbin/change_hostname.sh
Here is a different approach
import os
def setHostname(newhostname):
with open('/etc/hosts', 'r') as file:
# read a list of lines into data
data = file.readlines()
# the host name is on the 6th line following the IP address
# so this replaces that line with the new hostname
data[5] = '127.0.1.1 ' + newhostname
# save the file temporarily because /etc/hosts is protected
with open('temp.txt', 'w') as file:
file.writelines( data )
# use sudo command to overwrite the protected file
os.system('sudo mv temp.txt /etc/hosts')
# repeat process with other file
with open('/etc/hostname', 'r') as file:
data = file.readlines()
data[0] = newhostname
with open('temp.txt', 'w') as file:
file.writelines( data )
os.system('sudo mv temp.txt /etc/hostname')
#Then call the def
setHostname('whatever')
At the next reboot the hostname will be set to the new name
I have zero experience with bash so I'm struggling with the syntax - I gave up attempting in python as I thought that it might be easier. I would like to extract part of the filename (before the .xyz extension, and after the prefix), insert it on every line (starting from the third line) and pipe the output to a new file. I would also like to do this for multiple files, where the string changes.
My input files are as follows:
blahblah-0.xyz
blahblah-1.xyz
blahblah-2xyz
So far I know that I can do:
sed '3,$ s/^/500 /' file-500.xyz > output
and this will insert the string on every line. But I don't want to do this 100 times for each directory! I also tried the following from here: awk parse filename and add result to the end of each line:
for filename in ratio*; do
num=$(echo $filename | grep -Eo '[^ratio_distances]+\.xyz' | cut -d. -f1)
sed -i "s/\^/\t$num" $filename
done
Just to add, this is just being performed in the standard mac terminal, as I've had errors crop up in regards to the 'sed -i' command.
EDIT:
I got it to work in python, but I'd still be interested to know the bash commands. Python code should any one else be after the same thing:
import sys
import os
import glob
list_of_files = glob.glob("./blah-blah*.xyz")
for file in list of files:
for i in range (0, 80):
P = 10*i
if str(P) in file:
with open(file, 'r') as infile:
lines = infile.readlines()
lines[:]=lines[2:]
lines = [str(P)+' '+line for line in lines]
with open(file.replace('blahblah','output'),'w') as outfile:
outfile.writelines(lines)
infile.close()
outfile.close()
Thanks very much for any insight,
Anna
Assuming you can just prefix the old file names with "new_" to create the new file names:
awk '
FNR==1 { pfx = FILENAME; sub(/.*\./,"",pfx) }
FNR>=3 { $0 = pfx $0 }
{ print > ("new_"FILENAME) }
' ratio*
You can use bash's parameter expansion to extract the number from the file name. The -i is not supported in Mac's sed, so you have to use a temp file:
#! /bin/bash
for filename in ratio* ; do
num=${filename#ratio_distances-} # Remove from the left.
num=${num%.xyz} # Remove from the right.
sed "3,\$ s/^/$num /" "$filename" > new"$num"
mv new"$num" "$filename" # Replace the original with the tempfile.
done
#!/bin/bash
PrefixFile="blahblah"
awk -v "Prefix=${PrefixFile}" '
# At each new file starting
FNR == 1 {
# take number from current file name
gsub( "^" Prefix "-|[.]xyz$", "", Index = FILENAME)
}
# at each line (so for every files)
{
# print the index file (current) followed by original line
# to the (corresponding) filename.New
print Index $0 > ( FILENAME ".New" )
}
' ${PrefixFile}*.xyz
Using awk, using all files at once from shell expansion
assume prefix is without the - (easily adapteable if not)
ouput culd be any other file except itself (modification of aw could also change the name at the end but better in bash itself)
Thanks to #EdMorton for extra info about behaviour of awk about file redirection
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if /<name>/; print "$2\n" if /<(title|name)>(.*)<\/\1>/;'
I have this shell script which gets the Atom feed with command-line arguments for the username and password. I was wondering if this type of thing was possible in Python, and if so, how I would go about doing it. The atom feed is just regular XML.
Python does not lend itself to compact one liners quite as well as Perl. This is primarily for three reasons:
With Perl, whitespace is insignificant in almost all cases. In Python, whitespace is very significant.
Perl has some helpful shortcuts for one liners, such as perl -ne or perl -pe that put an implicit loop around the line of code.
There is a large body a cargo-cult Perl one liners to do useful things.
That all said, this python is close to what you posted in Perl:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" | python -c '
import sys
for s in sys.stdin:
s=s.strip()
if not s: print '\t',
else: print s
'
It is a little difficult to do better because, as stated in my comment, the Perl you posted is incomplete. You have:
perl -ne 'print "\t" if //; print "$2\n" if /(.*)/;'
Which is equivalent to:
LINE:
while (<>) {
print "\t" if //; # print a tab for a blank line
print "$2\n" if /(.*)/; # nonsensical. Print second group but only
# a single match group defined...
}
Edit
While it is trivial to rewrite that Perl in Python, here is something a bit better:
#!/usr/bin/python
from xml.dom.minidom import parseString
import sys
def get_XML_doc_stdin(f):
return xml.dom.minidom.parse(f)
def get_tagged_data2(tag, index=0):
xmlData = dom.getElementsByTagName(tag)[index].firstChild.data
return xmlData
data=sys.stdin.read()
dom = parseString(data)
ele2=get_tagged_data2('title')
print ele2
count=int(get_tagged_data2('fullcount'))
print count,"New Messages:"
for i in range(0,count):
nam=get_tagged_data2('name',i)
email=get_tagged_data2('email',i)
print " {0}: {1} <{2}>".format(i+1,nam,email)
Now save that in a text file, run chmod +x on it, then:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" |
/path/pythonfile.py
It produces this:
Gmail - Inbox for xxxxxxx#gmail.com
2 New Messages:
1: bob smith <bob#smith.com>
2: Google Alerts <googlealerts-noreply#google.com>
edit 2
And if you don't like that, here is the Python 1 line filter:
curl -u $1:$2 --silent "https://mail.google.com/mail/feed/atom" |python -c '
import sys, re
for t,m in re.findall(r"<(title|name)>(.*)<\/\1>",sys.stdin.read()):
print "\t",m
'
You may use an "URL opener" from the urllib2 standard Python module with a handler for authentication. For example:
#!/usr/bin/env python
import getpass
import sys
import urllib2
def main(program, username=None, password=None, url=None):
# Get input if any argument is missing
username = username or raw_input('Username: ')
password = password or getpass.getpass('Password: ')
url = url or 'https://mail.google.com/mail/feed/atom'
# Create password manager
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
password_mgr.add_password(None, url, username, password)
# Create HTTP Authentication handler and URL opener
authhandler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(authhandler)
# Fetch URL and print content
response = opener.open(url)
print response.read()
if __name__ == '__main__':
main(*sys.argv)
If you'd like to extract information from the feed too, you should check how to parse Password-Protected Feeds with feedparser.
I would like a way to update my password on a remote Ubuntu 10.4 box with fabric.
I would expect my fabfile.py would look something like this:
def update_password(old_pw, new_pw):
# Connects over ssh with a public key authentication
run("some_passwd_cmd --old %s --new %s" % (old_pw, new_pd))
Unfortunately the only command I know of that lets one change the password is passwd, and on Ubuntu 10.4 there doesn't seem to be any way to pass in the new (or old) password as an argument to passwd.
What command could one use to change a user's password on Ubuntu 10.4 via fabric?
EDIT:
I've had a look at usermod -p, and that may work but it isn't recommended by the man page.
EDIT: For some reason usermod -p wasn't working either over fabric.
As well, I've tried a (somewhat insecure) variation on mikej's answer that did solve the problem:
# connecting & running as root.
from fabric.api import *
from fabric.contrib import files
files.append("%s\n%s" % (passwd, passwd), '.pw.tmp')
# .pw.tmp:
# PASSWD
# PASSWD
run("passwd %s < .pw.tmp" % user)
run("rm .pw.tmp")
It's not a very elegant solution, but it works.
Thank you for reading.
Brian
You could feed the new and old passwords into passwd using echo e.g.
echo -e "oldpass\\nnewpass\\nnewpass" | passwd
(the -e option for echo enables interpretation of backslash escapes so the newlines are interpreted as such)
The trick is to use a combination of usermod and Python’s crypt to change your password:
from crypt import crypt
from getpass import getpass
from fabric.api import *
def change_password(user):
password = getpass('Enter a new password for user %s:' % user)
crypted_password = crypt(password, 'salt')
sudo('usermod --password %s %s' % (crypted_password, user), pty=False)
I use chpasswd on Ubuntu 11.04
fabric.api.sudo('echo %s:%s | chpasswd' % (user, pass))
Note:
Normally this pattern doesn't work:
$ sudo echo bla | restricted_command
because only the 'echo' gets elevated privileges, not the 'restricted_command'.
However, here it works because when fabric.api.sudo is caled
with shell=True (the default), fabric assembles the command like this:
$ sudo -S -p <sudo_prompt> /bin/bash -l -c "<command>"
sudo spawns a new shell (/bin/bash), running with root privileges, and
then that escalated shell runs the command.
Another way to pipe with sudo is to use sudo tee:
Out of interest, I have to do a similar task on a collection of Solaris boxes (add a whole lot of users, set their password). Solaris usermod doesn't have a --password option, so in the past I've used Expect to do this, but writing Expect scripts can be painful.
So this time I'm going to use Python's crypt.crypt, edit /etc/shadow directly (with backups, of course). http://docs.python.org/release/2.6.1/library/crypt.html
Commenters have suggested using various echo incantations piped to passwd. AFAIK this will never work, as passwd is programmed to ignore input from stdin and only accept input from an interactive tty. See http://en.wikipedia.org/wiki/Expect
I had no luck with the other methods. Thought I would share my method that I used for a once-off throwaway script.
It uses auto-responder to type in passwords at the prompts. I then immediately expire all the passwords so that users have a chance to choose their own.
This is not the most secure method, but depending on your use case it may be useful.
from collections import namedtuple
from getpass import getpass
import hashlib
from invoke import Responder
import uuid
from fabric import Connection, Config
User = namedtuple('UserRecord', ('name', 'password'))
def set_passwords(conn, user):
print(f'Setting password for user, {user.name}')
responder = Responder(
pattern=r'(?:Enter|Retype) new UNIX password:',
response=f'{user.password}\n',
)
result = conn.sudo(f'passwd {user.name}', warn=True, hide='both',
user='root', pty=True, watchers = [responder])
if result.exited is not 0:
print(f'Error, could not set password for user, "{user.name}". command: '
f'{result.command}; exit code: {result.exited}; stderr: '
f'{result.stderr}')
else:
print(f'Successfully set password for {user.name}')
def expire_passwords(conn, user):
print(f'Expiring password for user, {user.name}')
cmd = f'passwd --expire {user.name}'
result = conn.sudo(cmd, warn=True, user='root')
if result.exited is not 0:
print(f'Error, could not expire password for user, "{user.name}". '
f'command: {result.command}; exit code: {result.exited}; stderr: '
f'{result.stderr}')
else:
print(f'Successfully expired password for {user.name}')
def gen_password(seed_string):
# Don't roll your own crypto. This is for demonstration only and it is
# expected to only create a temporary password that requires changing upon
# initial login. I am no cryptography expert, hence this alternative
# simplified answer to the one that uses crypt, salt, etc -
# https://stackoverflow.com/a/5137688/1782641.
seed_str_enc = seed_string.encode(encoding='UTF-8')
uuid_obj = uuid.UUID(int=int(hashlib.md5(seed_str_enc).hexdigest(), 16))
return str(uuid_obj)[:8]
def some_function_that_returns_something_secret(conn):
return f'dummy-seed-{conn}'
sudo_pass = getpass('Enter your sudo password:')
config = Config(overrides={'sudo': {'password': sudo_pass}})
with Connection('vm', config=config) as vm_conn:
print(f'Making a new connection to {vm_conn.host}.')
# I usually use the sudo connection here to run a command that returns a
# reproducible string that only the sudo user could get access to be used
# for user_record.password bellow. Proceed with caution, this is not a
# recommended approach
seed = some_function_that_returns_something_secret(vm_conn)
user_record = User(name='linux_user', password=gen_password(seed))
set_passwords(vm_conn, user_record)
expire_passwords(vm_conn, user_record)
print(f'Done! Disconnecting from {vm_conn.host}.')
# So that you know the temporary password, print user_record or save to file
# `ssh linux_user#vm` and it should insist that you change password
print(user_record)