We Keep Coding

os.popen: How to integrate output and data from file?

I have a basic script that reads a file which has package names to build a command string and I store in a variable.
I then call a os.popen to run the command and store the output to a variable for further processing.
I loop over the variable looking for an 'Error:' string and if there is a match it prints it. All works good but it just prints the error which is what I want but I also want to know which package caused the error even if I include the package variable I only get the error.
Here are the contents of the file:
kernel-3.10.0-1160.el7
openshift-clients-4.3.7-202003130552.git.0.6027a27.el7
NetworkManager-config-server-1.18.8-1.el7
python2-psutil-5.6.6-1.el7ar
systemd-219-67.el7_7.1.x86_64
Here is the script:
import os
import sys
f=open("data1", "r")
for pkg in f:
#print(pkg)
command='yum --showduplicates list + ' +pkg
with os.popen(command) as results_in:
for item in results_in:
if 'Error:' in item:
print(item + "package name:" + pkg)
Here is the results of the script:
Error: No matching Packages to list
I was hoping to get the error + package name.
Can someone please tell me what I need to do to make the proper adjustments?

yum is writing the error message to stderr, not stdout. What you're seeing is the error message being printed by yum, not from your script.
You need to redirect stderr to stdout so you can capture it and check it.
It's also a good idea to remove the trailing newline from the line being read from the file, so do pkg = pkg.strip()
command=f'yum --showduplicates list ' + pkg + ' 2>&1'

I wrote the script another way to get the data I'm looking for. Thank you for your help! You sparked the idea of stderr so I chased that method to capture it and respond based on that.
import subprocess
import shlex
f=open("data1", "r")
for pkg in f:
command='yum list available ' + pkg
proc = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = proc.stdout.readline()
stdout, stderr= proc.communicate()
if 'Error' in str(stderr, 'utf-8').strip():
print("Error not available: "+pkg)
else:
print("Package available: "+pkg)

sh: 1: Syntax error: redirection unexpected Python/Bash

I have problem with my python cmd script.
I don't know why it does not work. Maybe something wrong with my code.
Im trying to run the program in cmdline through my python script.
And Im getting error in bash "sh: 1: Syntax error: redirection unexpected"
pls help Im just biologist :)
Im using spyder (anaconda)/Ubuntu
#!/usr/bin/python
import sys
import os
input_ = sys.argv[1]
output_file = open(sys.argv[2],'a+')
names = input_.rsplit('.')
for name in names:
os.system("esearch -db pubmed -query %s | efetch -format xml | xtract -pattern PubmedArticle -element AbstractText >> %s" % (name, output_file))
print("------------------------------------------")

output_file is a file object. When you do "%s" % output_file, the resulting string is something like "<open file 'filename', mode 'a+' at 0x7f1234567890>". This means that the os.system call is running a command like
command... >> <open file 'filename', mode 'a+' at 0x7f1234567890>
The < after the >> causes the "Syntax error: redirection unexpected" error message.
To fix that, don't open the output file in your Python script, just use the filename:
output_file = sys.argv[2]

I got similar error on following line:
os.system('logger Status changed on %s' %s repr(datetime.now())
Indeed, as nomadictype stated the problem is in running plain OS command. The command may include special characters. In my case this was <.
So instead of changing OS command significantly, I just added quotes and this works:
os.system('logger "Status changed on %s"' %s repr(datetime.now())
Quotes make content of passed parameter invisible for shell.

python script using subprocess, redirect ALL output to file

I am writing something for static analysis of source code in different languages. As anything has to be open source and callable from command line I now have downloaded one tool per language. So I decided to write a python script listing all source files in a project folder and calling the respective tool.
So part of my code looks like this:
import os
import sys
import subprocess
from subprocess import call
from pylint.lint import Run as pylint
class Analyser:
def __init__(self, source=os.getcwd(), logfilename=None):
# doing initialization stuff
self.logfilename = logfilename or 'CodeAnalysisReport.log'
self.listFiles()
self.analyseFiles()
def listFiles(self):
# lists all source files in the specified directory
def analyseFiles(self):
self.analysePythons()
self.analyseCpps()
self.analyseJss()
self.analyseJavas()
self.analyseCs()
if __name__ == '__main__':
Analyser()
Let's have at a look at the C++ files part (I use Cppcheck to analyse those):
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile])
The console output for one of the files (it's just a random downloaded file) is:
**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
(information) Cppcheck cannot find all the include files (use --check-config for details)
Line 1 and 2 coming from my script, lines 3 to 7 coming from Cppcheck.
And this is what I want to save to my log file, for all the other files too. Everything in one single file.
Of course I have searched SO and found some methods. But none is working completely.
First try:
Adding sys.stdout = open(self.logfilename, 'w') to my constructor. This makes line 1 and 2 of the above showed output be written to my log file. The rest is still shown on console.
Second try:
Additionaly, in analyseCpps I use:
call(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout)
This makes my log file to be:
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
and the console output is:
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
Not what I want.
Third try:
Using Popen with pipe. sys.stdout is back to default.
As preliminary work analyseCpps now is:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=subprocess.PIPE)
p.stdout.read()
p.stdout.read() shows only the last line of my desired output (line 7 in code box 3)
Fourth try:
Using subprocess.Popen(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=open(self.logfilename, 'a+')) just writes the one line Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc... to my logfile, the rest is shown on the console.
Fifth try:
Instead of subprocess.Popen I use os.system, so my calling command is:
os.system('C:\CodeAnalysis\cppcheck\cppcheck --enable=all %s >> %s' % (sourcefile, self.logfilename))
This results in the same log file as my fourth try. If I type the same command directly in the windows console the result is the same. So I guess it it is not exactly a python problem but still:
If it is on the console there must be a way to put it in a file. Any ideas?
E D I T
Foolish me. I'm still a noob so I forgot about the stderr. That's where the decisive messages are going to.
So now I have:
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stderr=subprocess.PIPE)
with open(self.logfilename, 'a+') as logfile:
logfile.write('%s\n%s\n' % ('*'*70, sourcefile))
for line in p.stderr.readlines():
logfile.write('%s\n' % line.strip())
and it's working fine.
ANOTHER EDIT
according to Didier's answer:
with sys.stdout = open(self.logfilename, 'w', 0) in my constructor:
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout, stderr=sys.stdout)

There are several problems:
you should redirect both stdout and stderr
you should use unbuffered files if you want to mix normal print and the output of launched commands.
Something like this:
import sys, subprocess
# Note the 0 here (unbuffered file)
sys.stdout = open("mylog","w",0)
print "Hello"
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
print "End"

You need to redirect stderr too, you can use STDOUT or pass the file object to stderr=:
from subprocess import check_call,STDOUT
with open("log.txt","w") as f:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
check_call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile],
stdout=f, stderr=STDOUT)

Try to redirect stdout and stderr to a logfile:
import subprocess
def analyseCpps(self):
with open("logfile.txt", "w") as logfile:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
call(['C:\\CodeAnalysis\\cppcheck\\cppcheck',
'--enable=all', sourcefile], stdout=logfile,
stderr=subprocess.STDOUT)
In this example the filename is hardcoded, but you should be able to change that easily (to your self.logfilename or similar).

python copy file in local network (linux -> linux) and output

I'm trying to write a script to copy files in my RaspberryPi, from my Desktop PC.
Here is my code: (a part)
print "start the copy"
path_pi = '//192.168.2.2:22/home/pi/Stock/'
file_pc = path_file + "/" + file
print "the file to copy is: ", file_pc
shutil.copy2(file_pc, path_pi + file_pi)
Actually I have this error: (in french)
IOError: [Errno 2] Aucun fichier ou dossier de ce type: '//192.168.2.2:22/home/pi/Stock/exemple.txt'
So, how could I proceed? Must I connect the 2 machines before trying to copy?
I have tryed with:
path_pi = r'//192.168.2.2:22/home/pi/Stock'
But the problem is the same. (And file_pc is a variable)
Thanks
Edit:
Ok, I found this:
command = 'scp', file_pc, file_pi
p = subprocess.Popen(command, stdout=subprocess.PIPE)
But no way to have the output... (work with Shell=False)

shutil.copy2() works with local files. 192.168.2.2:22 suggests that you want to copy files over ssh. You could mount the remote directory (RaspberryPi) onto a local directory on your desktop machine (sshfs) so that shutil.copy2() would work.
If you want to see the output of a command then don't set stdout=PIPE (note: if you set stdout=PIPE then you should read from p.stdout otherwise the process may block forever):
from subprocess import check_call
check_call(['scp', file_pc, file_pi])
scp will print to whatever places your parent Python script prints.
To get the output as a string:
from subprocess import check_output
output = check_output(['scp', file_pc, file_pi])
Though It looks like scp doesn't print anything by default if the output is redirected.
You could use pexpect to make scp think that it runs in a terminal:
import pipes
import re
import pexpect # $ pip install pexpect
def progress(locals):
# extract percents
print(int(re.search(br'(\d+)%[^%]*$', locals['child'].after).group(1)))
command = "scp %s %s" % tuple(map(pipes.quote, [file_pc, file_pi]))
status = pexpect.run(command, events={r'\d+%': progress}, withexitstatus=1)[1]
print("Exit status %d" % status)

Do you have SSH enabled? Something like this could help you:
import os
os.system("scp FILE USER#SERVER:PATH")

mp3info giving an error with variable filename

I have been using mp3info to calculate my file length using the following syntax:
mp3info -p "%S" /path/to/file
whenever I use the code with the filename I get the correct output:
mp3info -p "%S" far_from_love.mp3
However, on storing the filename in a string variable and then using the variable I get an error:
SyntaxError: invalid syntax
Could anyone tell me how to correctly use the command with a variable filename?
this is the python code which uses mp3info
listing=os.listdir("C:\\Python27")
for f in listing:
if fnmatch.fnmatch(f,'*.mp3'):
ext=f[:-4] #extract name of file without extension
WAVE_OUTPUT_FILENAME="%s.wav"%ext
print WAVE_OUTPUT_FILENAME#save output filename as wav extension
print f
x=os.popen('mp3info -p "%S" f).read()
print x

x=os.popen('mp3info -p "%S" f).read()
is missing a closing quote:
x=os.popen('mp3info -p "%S" ' + f).read()
You may also want to use the safer subprocess module:
import subprocess
x = subprocess.check_output(['mp3info', '-p', '%S', f])