We Keep Coding

Get a string in Shell/Python with subprocess

After this topic Get a string in Shell/Python using sys.argv , I need to change my code, I need to use a subprocess in a main.py with this function :
def download_several_apps(self):
subproc_two = subprocess.Popen(["./readtext.sh", self.inputFileName_download], stdout=subprocess.PIPE)
Here is my file readtext.sh
#!/bin/bash
filename="$1"
counter=1
while IFS=: true; do
line=''
read -r line
if [ -z "$line" ]; then
break
fi
python3 ./download.py \
-c ./credentials.json \
--blobs \
"$line"
done < "$filename"
And my download.py file
if (len(sys.argv) == 2):
downloaded_apk_default_location = 'Downloads/'
else:
readtextarg = os.popen("ps " + str(os.getppid()) + " | awk ' { out = \"\"; for(i = 6; i <= NF; i++) out = out$i\" \" } END { print out } ' ").read()
textarg = readtextarg.split(" ")[1 : -1][0]
downloaded_apk_default_location = 'Downloads/'+textarg[1:]
How can I get and print self.inputFileName_download in my download.py file ?
I used sys.argv as answerd by #tripleee in my previous post but it doesn't work as I need.

Ok I changed the last line by :
downloaded_apk_default_location = 'Downloads/'+textarg.split("/")[-1]
to get the textfile name

The shell indirection seems completely superfluous here.
import download
with open(self.inputFileName_download) as apks:
for line in apks:
if line == '\n':
break
blob = line.rstrip('\n')
download.something(blob=blob, credentials='./credentials.json')
... where obviously I had to speculate about what the relevant function from downloads.py might be called.

Python to search keyword starts with and Replace in file

I have file1.txt which has below contents
if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
|| ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
|| uses_abstraction "${GRUB_DEVICE}" lvm; then
LINUX_ROOT_DEVICE=${GRUB_DEVICE}
else
LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
fi
GRUBFS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2>/dev/null || true`"
Linux_CMDLINE="nowatchdog rcupdate.rcu_cpu_stall_suppress=1"
I want to find string starts with Linux_CMDLINE=" and replace that line with Linux_CMDLINE=""
I tried below code and it is not working. Also I am thinking it is not best way to implement. Is there any easy method to achieve this?
with open ('/etc/grub.d/42_sgi', 'r') as f:
newlines = []
for line in f.readlines():
if line.startswith('Linux_CMDLINE=\"'):
newlines.append("Linux_CMDLINE=\"\"")
else:
newlines.append(line)
with open ('/etc/grub.d/42_sgi', 'w') as f:
for line in newlines:
f.write(line)
output expected:
if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
|| ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
|| uses_abstraction "${GRUB_DEVICE}" lvm; then
LINUX_ROOT_DEVICE=${GRUB_DEVICE}
else
LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
fi
GRUBFS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2>/dev/null || true`"
Linux_CMDLINE=""

repl = 'Linux_CMDLINE=""'
with open ('/etc/grub.d/42_sgi', 'r') as f:
newlines = []
for line in f.readlines():
if line.startswith('Linux_CMDLINE='):
line = repl
newlines.append(line)

Minimal code thanks to open file for both reading and writing?
# Read and write (r+)
with open("file.txt","r+") as f:
find = r'Linux_CMDLINE="'
changeto = r'Linux_CMDLINE=""'
# splitlines to list and glue them back with join
newstring = ''.join([i if not i.startswith(find) else changeto for i in f])
f.seek(0)
f.write(newstring)
f.truncate()

useradd script not working as intended

I am writing a script that allows a user to put information in a text file such as
Alice;McCormick;ballstate;2000;3457
using this format
FirstName;LastName;Password;UID;GID
import os
import hashlib
iFile = open(“NewUsers.txt”, “rt”)
fileContents = iFile.readlines()
Username =””
fname=””
lname=””
password=""
uid=""
gid=""
for line in fileContents:
items = line.split(‘;’)
fname = items[0].lower()
lname = items[1].lower()
username = fname[0]+lname[0:7]
password = hashlib.sha256(items[2]).hexdigest()
uid = items[3]
gid = items[4]
os.system("/usr/sbin/useradd -p " + password + " -u " + uid + " -g " + gid + username)
I created a group called 3000 that has a groupid of 3457 so it already exists. When I run the script I get the following output.
Usage: useradd [options] LOGIN
useradd -D
useradd -D [options]
Options: lists out all the available options you can use when doing useradd
below that I receive sh: 2: amccormi: not found

I have never saved a command line as a variable do you just save it as a string?
You're passing a string to os.system, so sure, you can just save that to a string:
cmd = "/usr/sbin/useradd -p " + password + " -u " + uid + " -g " + gid + username
And then you can print it out:
print "running command:", cmd
Before passing it to os.system:
os.system(cmd)
What you will find, ultimately, is that when you do this:
>>> iFile = open('data')
>>> fileContents = iFile.readlines()
>>> for line in fileContents:
... items = line.split(';')
... print items
The variable items will end up containing the following:
['Alice', 'McCormick', 'ballstate', '2000', '3457\n']
Look at the final item in that list, which is 3457\n. It contains a
newline character, which means when you build your command line like
this:
os.system("/usr/sbin/useradd -p " + password + " -u " + uid + " -g " + gid + username)
You end up passing the following to /bin/sh:
/usr/bin/useradd -p ballstate -u 2000 -g 3457
amccormi
Hopefully at this point it's clear why you're getting the error that
you've described.
There are a couple of ways to solve this problem. The simplest is
probably to call line.strip(), which will remove whitespace
--including newlines -- at the beginning and end of your string:
>>> for line in fileContents:
... items = line.strip().split(';')

Adding short-hostname into the /etc/hosts file with python

Currently my /etc/hosts file is missing the short-hostname(last column) is there a way to take the FQDN value in the file remove '.pdp.wdf.ltd' and add the hostname to the last column.
To reach till here I did write a small python script wrote it to a file, but unable to proceed to get the short-hostname added
#!/usr/bin/env python
import re,subprocess,os,socket
a=subprocess.Popen('ifconfig -a', stdout=subprocess.PIPE, shell=True)
_a, err= a.communicate()
_ou=dict(re.findall(r'^(\S+).*?inet addr:(\S+)', _a, re.S | re.M))
_ou=_ou.values()
_ou.remove('127.0.0.1')
y=[]
for i in _ou:
_z = '{0} ' .format (i), socket.getfqdn(i)
y.append(_z)
_y=dict(y)
_z=(' \n'.join('{0} \t {1}'.format(key, val)for (key,val) in _y.iteritems()))
cat /etc/hosts
#IP-Address Full-Qualified-Hostname Short-Hostname
10.68.80.28 dewdfgld00035.pdp.wdf.ltd
10.68.80.45 lddbrdb.pdp.wdf.ltd
10.68.80.46 ldcirdb.pdp.wdf.ltd
10.72.176.28 dewdfgfd00035b.pdp.wdf.ltd
Output needed in the /etc/hosts file
##IP-Address Full-Qualified-Hostname Short-Hostname
10.68.80.28 dewdfgld00035.pdp.wdf.ltd dewdfgld00035
10.68.80.45 lddbrdb.pdp.wdf.ltd lddbrdb
10.68.80.46 ldcirdb.pdp.wdf.ltd ldcirbd
10.72.176.28 dewdfgfd00035b.pdp.wdf.ltd dewdfgfd00035b

You can use the following to match (with global and multiline flags) :
(^[^\s#]+\s+([^.\n]+).*)
And replace with the following:
\1\2
See RegEX DEMO

Okies I got it but had to tweak around a bit.
#!/usr/bin/env python
import re,subprocess,os,socket,shutil
header= """#DO NOT EDIT MANUALLY ## File controlled by SaltStack#
# IP-Address Full-Qualified-Hostname Short-Hostname
#
::1 localhost loopback
127.0.0.1 localhost
"""
a=subprocess.Popen('ifconfig -a', stdout=subprocess.PIPE, shell=True)
_a, err= a.communicate()
_ou=dict(re.findall(r'^(\S+).*?inet addr:(\S+)', _a, re.S | re.M))
_ou=_ou.values()
_ou.remove('127.0.0.1')
y=[]
for i in _ou:
n = socket.getfqdn(i) +'\t'+ (socket.getfqdn(i).split("."))[0]
_z = '{0} ' .format (i), n
y.append(_z)
_y=dict(y)
_z=(' \n'.join('{0} \t {1}'.format(key, val)for (key,val) in _y.iteritems()))
_z = header + _z
def make_version_path(path, version):
if version == 0:
return path
else:
return path + "." + str(version)
def rotate(path,version=0):
old_path = make_version_path(path, version)
if not os.path.exists(old_path):
raise IOError, "'%s' doesn't exist" % path
new_path = make_version_path(path, version + 1)
if os.path.exists(new_path):
rotate(path, version + 1)
shutil.move(old_path, new_path)
_hosts_path = '/etc/hosts'
shutil.copy (_hosts_path, _hosts_path+'_salt_bak')
rotate(_hosts_path+'_salt_bak')
f = open(_hosts_path, "w")
f.write(_z);
f.close()
The change was done in the code
y=[]
for i in _ou:
n = socket.getfqdn(i) +'\t'+ (socket.getfqdn(i).split("."))[0]
_z = '{0} ' .format (i), n
y.append(_z)
_y=dict(y)
And it worked as expected.

How to use regex to parse a S3 bucket list of files - Python

I have the following method:
def scan_s3dir(dirname):
try:
cmd = "s3cmd ls {s3bucket} --recursive".format(s3bucket=dirname)
output = subprocess.check_output([cmd],
stdin=None,
shell=True)
#s3://dgsecure/test_data/
regex = "dgsecure/test_data/[^/]*/(\S+)*"
installers = re.findall(regex, output)
print installers
except Exception, e:
print e
sys.exit(2)
when I execute s3cmd ls /path/to/bucket --recursive I get:
2014-02-14 02:21 0 s3://path/to/bucket/
2014-02-14 17:32 236 s3://path/to/bucket/foo.txt
2014-02-26 23:31 6035 s3://path/to/bucket/bar.txt
2014-02-14 22:17 2960 s3://path/to/bucket/baz.txt
from that regular expression, I want to produce a list all the files, including the subdir present in //path/to/bucket/ for example like:
s3://path/to/bucket/hello/world.txt
The output I would like to have it returned is:
['s3://path/to/bucket/foo.txt', 's3://path/to/bucket/bar.txt', 's3:////path/to/bucket/baz.txt']
What am I missing in the regular expression?

Try running this command :
s3cmd ls {s3bucket} --recursive | tr -s ' ' | cut -d " " -f 4

Here's something I would do without regex:
def parse_dir (output):
if output == "":
return []
else:
dir_list = []
line = output.split('\n')
for var in line:
if var != "":
dir_list.append(var.split()[3])
return dir_list[1:]