Get a string in Shell/Python with subprocess - python

After this topic Get a string in Shell/Python using sys.argv , I need to change my code, I need to use a subprocess in a main.py with this function :
def download_several_apps(self):
subproc_two = subprocess.Popen(["./readtext.sh", self.inputFileName_download], stdout=subprocess.PIPE)
Here is my file readtext.sh
#!/bin/bash
filename="$1"
counter=1
while IFS=: true; do
line=''
read -r line
if [ -z "$line" ]; then
break
fi
python3 ./download.py \
-c ./credentials.json \
--blobs \
"$line"
done < "$filename"
And my download.py file
if (len(sys.argv) == 2):
downloaded_apk_default_location = 'Downloads/'
else:
readtextarg = os.popen("ps " + str(os.getppid()) + " | awk ' { out = \"\"; for(i = 6; i <= NF; i++) out = out$i\" \" } END { print out } ' ").read()
textarg = readtextarg.split(" ")[1 : -1][0]
downloaded_apk_default_location = 'Downloads/'+textarg[1:]
How can I get and print self.inputFileName_download in my download.py file ?
I used sys.argv as answerd by #tripleee in my previous post but it doesn't work as I need.

Ok I changed the last line by :
downloaded_apk_default_location = 'Downloads/'+textarg.split("/")[-1]
to get the textfile name

The shell indirection seems completely superfluous here.
import download
with open(self.inputFileName_download) as apks:
for line in apks:
if line == '\n':
break
blob = line.rstrip('\n')
download.something(blob=blob, credentials='./credentials.json')
... where obviously I had to speculate about what the relevant function from downloads.py might be called.

Related

Python append text between lines

The task:
I have list of IPs which needs to be added to the .htaccess files in this format:
##ip_access1
Require ip 127.0.0.1
Require ip 127.0.0.2
Require all denied
##ip_access2
The problem:
How to append text into .htaccess file with Python? I know how to do this with bash, but I need Python specifically for now.
Cases:
If tuple of IPs is empty, find pattern ##ip_access1 ##ip_access2 and delete everything between them including the pattern in the file;
If .htaccess file is not empty, append ##ip_access1 <...> ##ip_access2 to the bottom of the file with all IPs;
P.S. Bash implementation.
ip_access() {
local user=$1
local htaccess="/var/www/${user}/site/.htaccess"
local ips="$2"
# manipulate records in .htaccess
[ ! -f "${htaccess}" ] && touch "${htaccess}"
if [ -z "${ips}" ]; then
sed -i '/##ip_access1/,/##ip_access2/{d}' "${htaccess}"
chown "${user}":"${user}" "${htaccess}"
echo "IP access successfully reset!"
exit 0
fi
arrip=()
for ip in ${ips//,/ }; do
arrip+=("Require ip $ip\n")
done
# always inject fresh batch of ips
sed -i '/##ip_access1/,/##ip_access2/{d}' "${htaccess}"
{ echo -e "##ip_access1";\
echo -e "${arrip:?}" | head -c -1;\
echo -e "Require all denied";\
echo -e "##ip_access2"; } >> "${htaccess}"
chown "${user}":"${user}" "${htaccess}"
echo "IP access successfully set!"
}
This function is the bare bones of a possible solution. It doesn't perform any sanity checks so caution should be exercised.
import os
def ips_to_file(ips, file_path):
if len(ips) > 0:
ip_lines = ['##ip_access1'] + [f'Require ip {ip}' for ip in ips] + ['Require all denied', '##ip_access2']
else:
ip_lines = []
if os.path.isfile(file_path):
with open(file_path, 'r+') as fp:
lines = [line.strip() for line in fp.readlines()]
lines = lines[:lines.index('##ip_access1')] + ip_lines + lines[lines.index('##ip_access2')+1:]
fp.seek(0)
fp.truncate()
fp.writelines(lines)
Found solution with help of course:
from typing import *
ip_lst = ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
htaccess_file_contents = open("test.txt", "r").read()
def _generate_htaccess_compat_lst(lst) -> str:
to_return = []
for addr in lst:
to_return.append("Require ip " + addr)
return "\n{}\n".format("\n".join(to_return))
def _inject_between(start, end, to_manipulate, to_replace) -> str:
lines = to_manipulate.splitlines()
counter = 0
pos1, pos2 = -1, -1
# find lines between that we need to replace
for line in lines:
if start == line:
pos1 = counter
elif end == line:
pos2 = counter
counter += 1
# return null if we can't find text between
if pos1 == -1 or pos2 == -1:
return None
# +1 to offset the last line as the first index is inclusive
return "\n".join(lines[0:pos1]) + start + to_replace + end + "\n".join(lines[pos2 + 1:len(lines)])
tmp = _inject_between("##ip_access1", "##ip_access2",
htaccess_file_contents,
_generate_htaccess_compat_lst(ip_lst))
print(tmp)
# feel free to write tmp back to .htaccess

How to solve a subprocess that contains a '|'

This code does not work.
I wrote like this.
str = "curl -s 'URL_ADDRESS' | tail -1".split()
p = subprocess.Popen(str,stdout=subprocess.PIPE).stdout
data = p.read()
p.close()
print(data)
But the result is b''.
What's the problem with this?
If you use subprocess, use instead of '|' like this.
This will solve the problem.
str = "curl -s 'URL_ADDRESS'".split()
tail = "tail -1".split()
temp = subprocess.Popen(str, stdout=subprocess.PIPE).stdout
temp1 = subprocess.Popen(tail, stdin=temp, stdout=subprocess.PIPE).stdout
temp.close()
data = temp1.read()
temp1.close()

Comparing two files and removing all whitespaces

Is there a more elegant way of comparing these two files?
Right now I am getting the following error message: syntax error near unexpected token (... diff <( tr -d ' '.
result = Popen("diff <( tr -d ' \n' <" + file1 + ") <( tr -d ' \n' <"
+ file2 + ") | wc =l", shell=True, stdout=PIPE).stdout.read()
Python seems to read "\n" as a literal character.
The constructs you are using are interpreted by bash and do not form a standalone statement that you can pass to system() or exec().
<( ${CMD} )
< ${FILE}
${CMD1} | ${CMD2}
As such, you will need to wire-up the redirection and pipelines yourself, or call on bash to interpret the line for you (as #wizzwizz4 suggests).
A better solution would be to use something like difflib that will perform this internally to your process rather than calling on system() / fork() / exec().
Using difflib.unified_diff will give you a similar result:
import difflib
def read_file_no_blanks(filename):
with open(filename, 'r') as f:
lines = f.readlines()
for line in lines:
if line == '\n':
continue
yield line
def count_differences(diff_lines):
diff_count = 0
for line in diff_lines:
if line[0] not in [ '-', '+' ]:
continue
if line[0:3] in [ '---', '+++' ]:
continue
diff_count += 1
return diff_count
a_lines = list(read_file_no_blanks('a'))
b_lines = list(read_file_no_blanks('b'))
diff_lines = difflib.unified_diff(a_lines, b_lines)
diff_count = count_differences(diff_lines)
print('differences: %d' % ( diff_count ))
This will fail when you fix the syntax error because you are attempting to use bash syntax in what is implemented as a C system call.
If you wish to do this in this way, either write a shell script or use the following:
result = Popen(['bash', '-c',
"diff <( tr -d ' \n' <" + file1 + ") <( tr -d ' \n' <"
+ file2 + ") | wc =l"], shell=True, stdout=PIPE).stdout.read()
This is not an elegant solution, however, since it is relying on the GNU coreutils and bash. A more elegant solution would be pure Python. You could do this with the difflib module and the re module.

Python to search keyword starts with and Replace in file

I have file1.txt which has below contents
if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
|| ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
|| uses_abstraction "${GRUB_DEVICE}" lvm; then
LINUX_ROOT_DEVICE=${GRUB_DEVICE}
else
LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
fi
GRUBFS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2>/dev/null || true`"
Linux_CMDLINE="nowatchdog rcupdate.rcu_cpu_stall_suppress=1"
I want to find string starts with Linux_CMDLINE=" and replace that line with Linux_CMDLINE=""
I tried below code and it is not working. Also I am thinking it is not best way to implement. Is there any easy method to achieve this?
with open ('/etc/grub.d/42_sgi', 'r') as f:
newlines = []
for line in f.readlines():
if line.startswith('Linux_CMDLINE=\"'):
newlines.append("Linux_CMDLINE=\"\"")
else:
newlines.append(line)
with open ('/etc/grub.d/42_sgi', 'w') as f:
for line in newlines:
f.write(line)
output expected:
if [ "x${GRUB_DEVICE_UUID}" = "x" ] || [ "x${GRUB_DISABLE_LINUX_UUID}" = "xtrue" ] \
|| ! test -e "/dev/disk/by-uuid/${GRUB_DEVICE_UUID}" \
|| uses_abstraction "${GRUB_DEVICE}" lvm; then
LINUX_ROOT_DEVICE=${GRUB_DEVICE}
else
LINUX_ROOT_DEVICE=UUID=${GRUB_DEVICE_UUID}
fi
GRUBFS="`${grub_probe} --device ${GRUB_DEVICE} --target=fs 2>/dev/null || true`"
Linux_CMDLINE=""
repl = 'Linux_CMDLINE=""'
with open ('/etc/grub.d/42_sgi', 'r') as f:
newlines = []
for line in f.readlines():
if line.startswith('Linux_CMDLINE='):
line = repl
newlines.append(line)
Minimal code thanks to open file for both reading and writing?
# Read and write (r+)
with open("file.txt","r+") as f:
find = r'Linux_CMDLINE="'
changeto = r'Linux_CMDLINE=""'
# splitlines to list and glue them back with join
newstring = ''.join([i if not i.startswith(find) else changeto for i in f])
f.seek(0)
f.write(newstring)
f.truncate()

merge multiple lines into single line by value of column

I have a tab-delimited text file that is very large. Many lines in the file have the same value for one of the columns in the file. I want to put them into same line. For example:
a foo
a bar
a foo2
b bar
c bar2
After run the script it should become:
a foo;bar;foo2
b bar
c bar2
how can I do this in either a shell script or in Python?
thanks.
With awk you can try this
{ a[$1] = a[$1] ";" $2 }
END { for (item in a ) print item, a[item] }
So if you save this awk script in a file called awkf.awk and if your input file is ifile.txt, run the script
awk -f awkf.awk ifile.txt | sed 's/ ;/ /'
The sed script is to remove out the leading ;
Hope this helps
from collections import defaultdict
items = defaultdict(list)
for line in open('sourcefile'):
key, val = line.split('\t')
items[key].append(val)
result = open('result', 'w')
for k in sorted(items):
result.write('%s\t%s\n' % (k, ';'.join(items[k])))
result.close()
not tested
Tested with Python 2.7:
import csv
data = {}
reader = csv.DictReader(open('infile','r'),fieldnames=['key','value'],delimiter='\t')
for row in reader:
if row['key'] in data:
data[row['key']].append(row['value'])
else:
data[row['key']] = [row['value']]
writer = open('outfile','w')
for key in data:
writer.write(key + '\t' + ';'.join(data[key]) + '\n')
writer.close()
A Perl way to do it:
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
open my $fh, '<', 'path/to/file' or die "unable to open file:$!";
my %res;
while(<$fh>) {
my ($k, $v) = split;
push #{$res{$k}}, $v;
}
print Dumper \%res;
output:
$VAR1 = {
'c' => [
'bar2'
],
'a' => [
'foo',
'bar',
'foo2'
],
'b' => [
'bar'
]
};
#! /usr/bin/env perl
use strict;
use warnings;
# for demo only
*ARGV = *DATA;
my %record;
my #order;
while (<>) {
chomp;
my($key,$combine) = split;
push #order, $key unless exists $record{$key};
push #{ $record{$key} }, $combine;
}
print $_, "\t", join(";", #{ $record{$_} }), "\n" for #order;
__DATA__
a foo
a bar
a foo2
b bar
c bar2
Output (with tabs converted to spaces because Stack Overflow breaks the output):
a foo;bar;foo2
b bar
c bar2
def compress(infilepath, outfilepath):
input = open(infilepath, 'r')
output = open(outfilepath, 'w')
prev_index = None
for line in input:
index, val = line.split('\t')
if index == prev_index:
output.write(";%s" %val)
else:
output.write("\n%s %s" %(index, val))
input.close()
output.close()
Untested, but should work. Please leave a comment if there are any concerns

Categories