This code does not work.
I wrote like this.
str = "curl -s 'URL_ADDRESS' | tail -1".split()
p = subprocess.Popen(str,stdout=subprocess.PIPE).stdout
data = p.read()
p.close()
print(data)
But the result is b''.
What's the problem with this?
If you use subprocess, use instead of '|' like this.
This will solve the problem.
str = "curl -s 'URL_ADDRESS'".split()
tail = "tail -1".split()
temp = subprocess.Popen(str, stdout=subprocess.PIPE).stdout
temp1 = subprocess.Popen(tail, stdin=temp, stdout=subprocess.PIPE).stdout
temp.close()
data = temp1.read()
temp1.close()
Related
After this topic Get a string in Shell/Python using sys.argv , I need to change my code, I need to use a subprocess in a main.py with this function :
def download_several_apps(self):
subproc_two = subprocess.Popen(["./readtext.sh", self.inputFileName_download], stdout=subprocess.PIPE)
Here is my file readtext.sh
#!/bin/bash
filename="$1"
counter=1
while IFS=: true; do
line=''
read -r line
if [ -z "$line" ]; then
break
fi
python3 ./download.py \
-c ./credentials.json \
--blobs \
"$line"
done < "$filename"
And my download.py file
if (len(sys.argv) == 2):
downloaded_apk_default_location = 'Downloads/'
else:
readtextarg = os.popen("ps " + str(os.getppid()) + " | awk ' { out = \"\"; for(i = 6; i <= NF; i++) out = out$i\" \" } END { print out } ' ").read()
textarg = readtextarg.split(" ")[1 : -1][0]
downloaded_apk_default_location = 'Downloads/'+textarg[1:]
How can I get and print self.inputFileName_download in my download.py file ?
I used sys.argv as answerd by #tripleee in my previous post but it doesn't work as I need.
Ok I changed the last line by :
downloaded_apk_default_location = 'Downloads/'+textarg.split("/")[-1]
to get the textfile name
The shell indirection seems completely superfluous here.
import download
with open(self.inputFileName_download) as apks:
for line in apks:
if line == '\n':
break
blob = line.rstrip('\n')
download.something(blob=blob, credentials='./credentials.json')
... where obviously I had to speculate about what the relevant function from downloads.py might be called.
Is there a more elegant way of comparing these two files?
Right now I am getting the following error message: syntax error near unexpected token (... diff <( tr -d ' '.
result = Popen("diff <( tr -d ' \n' <" + file1 + ") <( tr -d ' \n' <"
+ file2 + ") | wc =l", shell=True, stdout=PIPE).stdout.read()
Python seems to read "\n" as a literal character.
The constructs you are using are interpreted by bash and do not form a standalone statement that you can pass to system() or exec().
<( ${CMD} )
< ${FILE}
${CMD1} | ${CMD2}
As such, you will need to wire-up the redirection and pipelines yourself, or call on bash to interpret the line for you (as #wizzwizz4 suggests).
A better solution would be to use something like difflib that will perform this internally to your process rather than calling on system() / fork() / exec().
Using difflib.unified_diff will give you a similar result:
import difflib
def read_file_no_blanks(filename):
with open(filename, 'r') as f:
lines = f.readlines()
for line in lines:
if line == '\n':
continue
yield line
def count_differences(diff_lines):
diff_count = 0
for line in diff_lines:
if line[0] not in [ '-', '+' ]:
continue
if line[0:3] in [ '---', '+++' ]:
continue
diff_count += 1
return diff_count
a_lines = list(read_file_no_blanks('a'))
b_lines = list(read_file_no_blanks('b'))
diff_lines = difflib.unified_diff(a_lines, b_lines)
diff_count = count_differences(diff_lines)
print('differences: %d' % ( diff_count ))
This will fail when you fix the syntax error because you are attempting to use bash syntax in what is implemented as a C system call.
If you wish to do this in this way, either write a shell script or use the following:
result = Popen(['bash', '-c',
"diff <( tr -d ' \n' <" + file1 + ") <( tr -d ' \n' <"
+ file2 + ") | wc =l"], shell=True, stdout=PIPE).stdout.read()
This is not an elegant solution, however, since it is relying on the GNU coreutils and bash. A more elegant solution would be pure Python. You could do this with the difflib module and the re module.
I have this function to get a file type :
def get_file_type():
try:
cmd = ['/usr/bin/file', '/home/user']
p = Popen(cmd, stdout=PIPE).communicate()[0]
p = str(p).split(':')[1:]
if len(p) > 1:
' : '.join(p).strip().replace('\\n', '')
else:
p = p[0].strip().replace('\\n', '')
print(p)
except CalledProcessError:
print('unknown')
But it returns this : directory'
The ending apostrophe is not a typo, it is what bothers me. And I don't understand why (not that it bothers me.. ;) )
Thank you
The problem is that you're treating bytes as a string and you're using Python3. So what you're getting when you call str(p) looks like this:
"b'/home/user: directory\\n'"
You could fix this by doing p.decode().split instead of str(p).split
I am very novice to python. I have code in bash need to convert it to python, tired with converter but getting syntax error. Will be very helpful if someone help me in finding error !
Error :
File "cp_file.py", line 75
print(myfilename.val)
SyntaxError : Invalid Syntax
Will be very helpful if someone convert following bash code to python without or help me in finding error !
Bash Code :
grep " 200 " /var/log/ba/access.log |awk '{print $7}'|sort|uniq > /tmp/read_log.txt
for i in $(cat /tmp/read_log.txt); do
echo $i
myfilename="$(echo ${i##*/})"
echo $myfilename
wget http://mydata.na.xyz/$i
curl -X POST -d #$myfilename http://xyz.xyz/ba/$i
done
Python Code :
#! /usr/bin/env python
from __future__ import print_function
import sys,os,subprocess
class Bash2Py(object):
__slots__ = ["val"]
def __init__(self, value=''):
self.val = value
def setValue(self, value=None):
self.val = value
return value
def GetVariable(name, local=locals()):
if name in local:
return local[name]
if name in globals():
return globals()[name]
return None
def Make(name, local=locals()):
ret = GetVariable(name, local)
if ret is None:
ret = Bash2Py(0)
globals()[name] = ret
return ret
def Str(value):
if isinstance(value, list):
return " ".join(value)
if isinstance(value, basestring):
return value
return str(value)
def Array(value):
if isinstance(value, list):
return value
if isinstance(value, basestring):
return value.strip().split(' ')
return [ value ]
_rc0 = _rcr1, _rcw1 = os.pipe()
if os.fork():
os.close(_rcw1)
os.dup2(_rcr1, 0)
_rcr2, _rcw2 = os.pipe()
if os.fork():
os.close(_rcw2)
os.dup2(_rcr2, 0)
_rcr3, _rcw3 = os.pipe()
if os.fork():
os.close(_rcw3)
os.dup2(_rcr3, 0)
subprocess.call("uniq",shell=True,stdout=file("/tmp/read_log.txt",'wb'))
else:
os.close(_rcr3)
os.dup2(_rcw3, 1)
subprocess.call(["sort"],shell=True)
sys.exit(0)
else:
os.close(_rcr2)
os.dup2(_rcw2, 1)
subprocess.call(["awk","{print $7}"],shell=True)
sys.exit(0)
else:
os.close(_rcr1)
os.dup2(_rcw1, 1)
subprocess.call(["grep","200","/var/log/ba/access.log"],shell=True)
sys.exit(0)
for Make("i").val in Array(os.popen("cat /tmp/read_log.txt").read().rstrip("\n")):
print(i.val)
Make("myfilename").setValue(os.popen("echo "+str(i.val##*/)).read().rstrip("\n"))
print(myfilename.val)
subprocess.call(["wget","http://xyz.xyz/"+str(i.val)],shell=True)
subprocess.call(["curl","-X","POST","-D","#"+str(myfilename.val),"http://xyz.xyz/ba/"+str(i.val)],shell=True)
That auto-generated Python code is horrible. You'd be much better off sticking with Bash. But best would be to actually migrate your code to Python using human understanding. For example, take just this part:
grep " 200 " /var/log/ba/access.log | awk '{print $7}'|sort|uniq > /tmp/read_log.txt
In Python that is something like:
with open('/var/log/ba/access.log') as infile, open('/tmp/read_log.txt', 'w') as outfile:
results = set()
for line in infile:
if ' 200 ' in line:
tokens = line.split()
results.add(tokens[6]) # 7th token
for result in sorted(results):
print >>outfile, result
For the HTTP part, use the Python module requests. It's easy to use. Quite possibly you won't need outfile anymore--you can just directly use for result in sorted(results) to make your HTTP requests.
First never read lines of a file with for, use while instead. See here why
It's a very small script, is more easy rewrite in python than use your converter.
If are in hurry and really need the script in python you can use linux commands inside python, is not the best way but is more faster and easy for someone who don't know python
import subprocess
p = subprocess.Popen(["curl","POST","-X", "YOUR_URL"],
stdout=subprocess.PIPE, shell=True) (output, err) = p.communicate()
I get couple of grep:write errors when I run this code.
What am I missing?
This is only part of it:
while d <= datetime.datetime(year, month, daysInMonth[month]):
day = d.strftime("%Y%m%d")
print day
results = [day]
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output1=first.communicate()[0]
d += delta
day = d.strftime("%Y%m%d")
second=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+"*.txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt", shell=True, stdout=subprocess.PIPE, )
output2=second.communicate()[0]
articleList = (output1.split('\n'))
articleList2 = (output2.split('\n'))
results.append( len(articleList)+len(articleList2))
w.writerow(tuple(results))
d += delta
When you do
A | B
in a shell, process A's output is piped into process B as input. If process B shuts down before reading all of process A's output (e.g. because it found what it was looking for, which is the function of the -l option), then process A may complain that its output pipe was prematurely closed.
These errors are basically harmless, and you can work around them by redirecting stderr in the subprocesses to /dev/null.
A better approach, though, may simply be to use Python's powerful regex capabilities to read the files:
def fileContains(fn, pat):
with open(file) as f:
for line in f:
if re.search(pat, line):
return True
return False
first = []
for file in glob.glob(monthDir +"/"+day+"*.txt"):
if fileContains(file, 'Algeria|Bahrain') and fileContains(file, 'Protest|protesters'):
file.append(first)
To find the files matching two patterns, the command structure should be:
grep -l pattern1 $(grep -l pattern2 files)
$(command) substitutes the output of the command into the command line.
So your script should be:
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' $("+ grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+"*.txt)", shell=True, stdout=subprocess.PIPE, )
and similarly for second
If you are just looking for whole words, you could use the count() member function;
# assuming names is a list of filenames
for fn in names:
with open(fn) as infile:
text = infile.read().lower()
# remove puntuation
text = text.replace(',', '')
text = text.replace('.', '')
words = text.split()
print "Algeria:", words.count('algeria')
print "Bahrain:", words.count('bahrain')
print "protesters:", words.count('protesters')
print "protest:", words.count('protest')
If you want more powerful filtering, use re.
Add stderr args in the Popen function based on the python version the stderr value will change. This will support if the python version is less than 3
first=subprocess.Popen("grep -Eliw 'Algeria|Bahrain' "+ monthDir +"/"+day+".txt | grep -Eliw 'Protest|protesters' "+ monthDir +"/"+day+".txt", shell=True, stdout=subprocess.PIPE, stderr = subprocess.STDOUT)