We Keep Coding

Executing script inside python subprocess and getting the output

I am trying to get execute one script with some arguments and getting its out to a buffer for further processing. The original code snippet which works now as below-
cmd = "/pathToScript/myscript.sh --endpoint "+ ip_addr + " --register '"+ reg_token + "' --output-file " + pathToOutputFile
os.system(cmd)
The above cmd runs and creates the output file where it write the data generated by myhelper script.
Now I want to run the same script, but instead of writing to file, I am expecting the output to a buffer as returned value.
I rewrite the code line as below, but it does not generates the data into buffer.
creds = subprocess.run(
["/pathToScript/myscript.sh", "--endpoint {0}".format(endpoint_ipaddr), "--register {0}".format(reg_token)],
check=True,
stdout=subprocess.PIPE,
universal_newlines=True
).stdout
print("creds")
I tried with both subprocess.run() and call() command, but both did not return anything. Is my modified code is correct with respect to executing the script and getting the output collected to returned variable?
Appreciate your help on this.
thank you

Python can't read a temporary file

I have a python program, which is supposed to calculate changes based on a value written in a temporary file (eg. "12345\n"). It is always an integer.
I have tried different methods to read the file, but python wasn't able to read it. So then I had the idea to execute a shell command ("cat") that will return content. When I execute this in the shell it works fine, but python the feedback I get is empty. Then I tried writing a bash and then a php skript, which would read the file and then return the value. In python I called them over the shell and the feedback I get is empty as well.
I was wondering if that was a general problem in python and made my scripts return the content of other temporary files, which worked fine.
Inside my scripts I was able to do calculations with the value and in the shell the output is exactly as expected, but not when called via python. I also noticed that I don't get the value with my extra scripts when they are called by phython (I tried to write it into another file; it was updated but empty).
The file I am trying to read is in the /tmp directory and is written into serveral time per second by another script.
I am looking for a solution (open for new ideas) in which I end up having the value of the file in a python variable.
Thanks for the help
Here are my programs:
python:
# python script
import subprocess
stdout = subprocess.Popen(["php /path/to/my/script.php"], shell = True, stdout = subprocess.PIPE).communicate()[0].decode("utf-8")
# other things I tried
#with open("/tmp/value.txt", "r") as file:
# stdout = file.readline() # output = "--"
#stdout = os.popen("cat /tmp/value.txt").read() # output = "--"
#stdout = subprocess.check_output(["php /path/to/my/script.php"], shell = True, stdout = subprocess.PIPE).decode("utf-8") # output = "--"
print(str("-" + stdout + "-")) # output = "--"
php:
# php script
valueFile = fopen("/tmp/value.txt", "r");
value = trim(fgets($valueFile), "\n");
fclose($valueFile);
echo $value; # output in the shell is the value of $value
Edit: context: my python script is started by another python script, which listens for commands from an apache server on the pi. The value I want to read comes from a "1wire" device that listens for S0-signals.

Python file output adding weird characters

Turns out it is an error with my C program. I changed my printf to only print a preset string and redirected it to a file and the extra characters were still there. I still don't know why though.
Hi I'm writing a python script to run analysis on a C program I'm making parallel. Write now I have the number of processors used and the iterations I want to pass to my C program in a separate file called tests. I'm extremely new to Python, here's my sample code I wrote to figure out how to write results to a file which fill eventually be a .csv file.
#!/usr/bin/env python
import subprocess
mpiProcess = "runmpi"
piProcess = "picalc"
tests = open("tests.txt")
analysis = open("analysis.txt", "w")
def runPiCalc (numProcs, numIterations):
numProcs = str(numProcs)
numIterations = str(numIterations)
args = (mpiProcess, piProcess, numProcs, numIterations)
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
return output
def runTest (testArgs):
testProcs = testArgs[0]
testIterations = testArgs[1]
output = runPiCalc(testProcs,testIterations)
appendResults(output)
def appendResults (results):
print results
analysis.write(results + '\n')
for testLine in tests:
testArgs = testLine.split()
runTest(testArgs)
tests.close()
analysis.close()
My problem right now is when I "print results" to stdout the output comes out as expected and I get 3.14blablablablawhatever. When I check the analysis.txt file though I get [H[2J (weirder characters that are encoded as ESC not on the web) at the start of every line before my pi calculation shows up. I can't figure out why that is. Why would file.write have different output than print. Again this is my first time with Python so I'm probably just missing something easy.
This is on a ubuntu server I'm sshing to btw.
Here's the tests.txt and a picture of how the characters look on linux

The problem was I had a bash script executing my C program. The bash script was inserting the weird characters before the program output and adding it to its standard output. Putting the command I was calling inside the python script directly instead of calling a bash script fixed the problem.

Python: Write stdout to log file; output is hexadecimal not ascii

I'm working on a script to call an executable for i/o files. I'm using subprocess and trying to shell out the exe and the stdout to a log file. Problem is I would like to output a simple ascii file and I'm getting a hexadecimal file. Just really learning to program python (or any language for that matter) so, I'm assuming there some type of formatting I can do but I just don't get it. I've done a fair bit of searching on this site and others but I haven't anything like what I'm using subprocess for. The "outRadcorr" is what I need help on the most...Any ideas? More code on request.
Import system modules
import os, sys, string, traceback, time, datetime
import params
from subprocess import Popen, PIPE, STDOUT
...some code here.....
Write stdout to log file
rad_log_file = open(dsFolder + '\\radcorr.log', 'w')
# loop through the files in raw file list to run radiometric correction
for rawfiles in rawFolderList:
# Define the file base
rawBase = rawfiles.split(".")[0]
print ('\nProcessing file: %s \n')%( rawBase )
# define variables from raw file to process radcorr
radFile = rawfiles
pixFile = ('%s.pix')%( rawBase )
attFile = ('%s.att')%( rawBase )
# create windose bat file function
def rad_bat_writer( radcorr_bat ):
with open(dsFolder + '\\radcorr.bat', 'a') as rad_bat_file:
rad_bat_file.write(radcorr_bat + '\n')
# grab radcor input/output files and run radcorr.exe
radcorr = ('C:\\itres\\bin\\radcorr.exe -f 1 -j 100 -g 50 -s -1 -n -1 %s %s %s -i '+ rawFolder + '\%s,rb -o ' + radFolder + '\%s -a ' \
+ radFolder + '\%s -c C:\\itres\\rad_cal_files\\%s -I 0 -v 0 -r Y -R Y -^^ 2') %( sum,scatter,shift,radFile,pixFile,attFile,rad_prefix )
# print out radcorr command
print radcorr
# Execute radcorr and write stdout
outRadcorr = Popen("{};".format(radcorr), stdout=PIPE, stderr=STDOUT)
for line in outRadcorr.stdout:
sys.stdout.write(line)
rad_log_file.write(line)
# write output to log
rad_bat_writer( radcorr )
# Close out exe and log files
outRadcorr.wait()
rad_log_file.close()

I was using UltraEdit to view the file, which you can view hex files in. I was not looking in hex mode. I might be confusing the terminology here. File looks normal in NotePad/NotePad++/WordPad etc. As I ran the script in debug mode, I could hit the loop file by file. First two files (1GB/file), the output log looked fine. Once the radcorr.log file was over 10kb, I change from a normal ascii output to this binary file viewed that looked hex. I can't post images yet, but just google ultraedit hexadecimal.
Still not sure why it moved to this format. Finally, output size was 45kb. I change to view/edit hex mode in UltraEdit and it looks fine. Just wanted to get it out there to see if others had any ideas why when I specified the log to be 'w' and not 'wb', for instance.
I do appreciate all you help. #J.F. Sebastian I'll have to test the code you posted, probably help fix potential bugs down the road.

Python subprocess library: Running grep command from Python

I am trying to run grep command from my Python module using the subprocess library. Since, I am doing this operation on the doc file, I am using Catdoc third party library to get the content in a plan text file. I want to store the content in a file. I don't know where I am going wrong but the program fails to generate a plain text file and eventually to get the grep result. I have gone through the error log but its empty. Thanks for all the help.
def search_file(name, keyword):
#Extract and save the text from doc file
catdoc_cmd = ['catdoc', '-w' , name, '>', 'testing.txt']
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
output = catdoc_process.communicate()[0]
grep_cmd = []
#Search the keyword through the text file
grep_cmd.extend(['grep', '%s' %keyword , 'testing.txt'])
print grep_cmd
p = subprocess.Popen(grep_cmd,stdout=subprocess.PIPE,stderr=subprocess.PIPE, shell=True)
stdoutdata = p.communicate()[0]
print stdoutdata

On UNIX, specifying shell=True will cause the first argument to be treated as the command to execute, with all subsequent arguments treated as arguments to the shell itself. Thus, the > won't have any effect (since with /bin/sh -c, all arguments after the command are ignored).
Therefore, you should actually use
catdoc_cmd = ['catdoc -w "%s" > testing.txt' % name]
A better solution, though, would probably be to just read the text out of the subprocess' stdout, and process it using re or Python string operations:
catdoc_cmd = ['catdoc', '-w' , name]
catdoc_process = subprocess.Popen(catdoc_cmd, stdout=subprocess.PIPE,stderr=subprocess.PIPE)
for line in catdoc_process.stdout:
if keyword in line:
print line.strip()

I think you're trying to pass the > to the shell, but that's not going to work the way you've done it. If you want to spawn a process, you should arrange for its standard out to be redirected. Fortunately, that's really easy to do; all you have to do is open the file you want the output to go to for writing and pass it to popen using the stdout keyword argument, instead of PIPE, which causes it to be attached to a pipe which you can read with communicate().