Turns out it is an error with my C program. I changed my printf to only print a preset string and redirected it to a file and the extra characters were still there. I still don't know why though.
Hi I'm writing a python script to run analysis on a C program I'm making parallel. Write now I have the number of processors used and the iterations I want to pass to my C program in a separate file called tests. I'm extremely new to Python, here's my sample code I wrote to figure out how to write results to a file which fill eventually be a .csv file.
#!/usr/bin/env python
import subprocess
mpiProcess = "runmpi"
piProcess = "picalc"
tests = open("tests.txt")
analysis = open("analysis.txt", "w")
def runPiCalc (numProcs, numIterations):
numProcs = str(numProcs)
numIterations = str(numIterations)
args = (mpiProcess, piProcess, numProcs, numIterations)
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
popen.wait()
output = popen.stdout.read()
return output
def runTest (testArgs):
testProcs = testArgs[0]
testIterations = testArgs[1]
output = runPiCalc(testProcs,testIterations)
appendResults(output)
def appendResults (results):
print results
analysis.write(results + '\n')
for testLine in tests:
testArgs = testLine.split()
runTest(testArgs)
tests.close()
analysis.close()
My problem right now is when I "print results" to stdout the output comes out as expected and I get 3.14blablablablawhatever. When I check the analysis.txt file though I get [H[2J (weirder characters that are encoded as ESC not on the web) at the start of every line before my pi calculation shows up. I can't figure out why that is. Why would file.write have different output than print. Again this is my first time with Python so I'm probably just missing something easy.
This is on a ubuntu server I'm sshing to btw.
Here's the tests.txt and a picture of how the characters look on linux
The problem was I had a bash script executing my C program. The bash script was inserting the weird characters before the program output and adding it to its standard output. Putting the command I was calling inside the python script directly instead of calling a bash script fixed the problem.
Related
as an example I have two scripts, say script1.py
f = open("output1.txt", "w")
count = 1
for i in range(100):
f.write(str(count) + "\n")
print(str(count))
count +=1
f.close
This script prints numbers from 1 to 100 to a file and to standard output.
Then I have a second script, say script2.py
import sys
import time
stdin = sys.stdin
f1 = open("output2.txt", "w")
for line in stdin:
if len(line)>0:
print(line.strip())
time.sleep(0.05)
f1.write(line.strip() + "\n")
which reads data from standard input and prints them to a file. I added a time.sleep command to ensure the second script consumes data at a far lower rate than they are produced by the first one.
I run the scripts from the command line as
python3 script1.py | python3 script2.py
so redirecting the standard output of the first (so the print() command) to the standard input of the second one.
It works as somehow expected, two files are generated containing numbers from 1 to 100.
I am nevertheless wondering how the data transfer part works, from first to second script.
the first script generates data at a faster rate. Where are these data stored, waiting for the second script to access them?
Is there some sort of buffer that is put in place between the two process? Or what else?
Is Python responsible for this, or the OS?
Is the buffer limited in size? Can it be programmed (e.g. accessed to direct data to another target as well)?
Thanks a bunch
it is because of the pipe "|", more info here: https://ss64.com/nt/syntax-redirection.html
commandA | commandB Pipe the output from commandA into commandB
so the prints from your script1 are sent to your script2.
My guess on how it works is that every print is saved in memory as a big string and then sent back (as text) to the second that s why sys.stdin works
I have a python program, which is supposed to calculate changes based on a value written in a temporary file (eg. "12345\n"). It is always an integer.
I have tried different methods to read the file, but python wasn't able to read it. So then I had the idea to execute a shell command ("cat") that will return content. When I execute this in the shell it works fine, but python the feedback I get is empty. Then I tried writing a bash and then a php skript, which would read the file and then return the value. In python I called them over the shell and the feedback I get is empty as well.
I was wondering if that was a general problem in python and made my scripts return the content of other temporary files, which worked fine.
Inside my scripts I was able to do calculations with the value and in the shell the output is exactly as expected, but not when called via python. I also noticed that I don't get the value with my extra scripts when they are called by phython (I tried to write it into another file; it was updated but empty).
The file I am trying to read is in the /tmp directory and is written into serveral time per second by another script.
I am looking for a solution (open for new ideas) in which I end up having the value of the file in a python variable.
Thanks for the help
Here are my programs:
python:
# python script
import subprocess
stdout = subprocess.Popen(["php /path/to/my/script.php"], shell = True, stdout = subprocess.PIPE).communicate()[0].decode("utf-8")
# other things I tried
#with open("/tmp/value.txt", "r") as file:
# stdout = file.readline() # output = "--"
#stdout = os.popen("cat /tmp/value.txt").read() # output = "--"
#stdout = subprocess.check_output(["php /path/to/my/script.php"], shell = True, stdout = subprocess.PIPE).decode("utf-8") # output = "--"
print(str("-" + stdout + "-")) # output = "--"
php:
# php script
valueFile = fopen("/tmp/value.txt", "r");
value = trim(fgets($valueFile), "\n");
fclose($valueFile);
echo $value; # output in the shell is the value of $value
Edit: context: my python script is started by another python script, which listens for commands from an apache server on the pi. The value I want to read comes from a "1wire" device that listens for S0-signals.
I am attempting to write a (Bash) shell script that wraps around a third-party python script and captures all output (errors and stdout) into a log file, and also restarts the script with a new batch of data each time it completes successfully. I'm doing this on a standard Linux distribution, but hopefully this solution can be platform-independent.
So here's a simplified version of the shell script, omitting everything except the logging:
#!/bin/bash
/home/me/script.py &>> /home/me/logfile
The problem is the third-party python script's output is mostly on a single line, which is being refreshed periodically (~every 90 seconds) by use of a carriage return ("\r"). Here's an example of the type of output I mean:
#!/usr/bin/env python3
import time
tracker = 1
print("This line is captured in the logfile because it ends with a newline")
while tracker < 5:
print(" This output isn't captured in the log file. Tracker = " + str(tracker),end="\r")
tracker += 1
time.sleep(1)
print("This line does get captured. Script is done. ")
How can I write a simple shell script to capture the output each time it is refreshed, or at least to periodically capture the current output as it would appear on the screen if I were running the script in the terminal?
Obviously I could try to modify the python script to change its output behavior, but the actual script I'm using is very complex and I think beyond my abilities to do that easily.
The program should have disabled this behavior when output is not a tty.
The output is already captured completely, it's just that you see all the updates at once when you cat the file. Open it in a text editor and see for yourself.
To make the file easier to work with, you can just replace the carriage returns with line feeds:
/home/me/script.py | tr '\r' '\n'
If the process normally produces output right away, but not with this command, you can disable Python's output buffering.
Writing from file_A to file_B using IDLE always makes IDLE print out the lines as they are being written. If the file is very large, then the process would take hours to finish.
How can I make IDLE not print anything while the process of writing to a new file is ongoing, in order to speed things up?
A simple code to demonstrate that IDLE prints the lines as they are being written:
file = open('file.csv','r')
copy = open('copy.csv','w')
for i in file:
i = i.split()
copy.write(str(i))
I assume you are using Python3 where write returns the number of characters written to the file and IDLE's python shell prints this return value when you call it. In Python2 write returns None that is not printed by IDLE's shell.
The workaround is to assign the return value of write to a temporary dummy variable
dummy = f.write("my text")
For your example the following code should work
file = open('file.csv','r')
copy = open('copy.csv','w')
for i in file:
i = i.split()
dummy = copy.write(str(i))
I added two screenshots for all of you to see the difference between the writes in Python 2 and Python 3 on my system.
This is baffling me.
I have a python script that does some work on a Windows platform to generate an XML file, download some images and then call an external console application to generate a video using the xml and images the python script has generated.
The application I call with pOPen is supposed to return a status i.e. [success] or [invalid] or [fail] dependant on how it interprets the data I pass it.
If I use my generation script to generate the information and then call the console application separately in another script it works fine and I get success and a video is generated.
Successful code (please ignore the test prints!):
print ("running console app...")
cmd = '"C:/Program Files/PropertyVideos/propertyvideos.console.exe" -xml data/feed2820712.xml -mpeg -q normal'
print (cmd)
p = subprocess.Popen(cmd , stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = p.communicate()[0]
print ("\n----\n[" + output + "]\n----\n")
if output == "[success]":
print "\nHURRAHHHHH!!!!!!!"
print ("finished...")
But if I include the same code at the end of the script that generates the info to feed the console application then it runs for about 2 seconds and output = []
Same code, just ran at the end of a different script...
EDIT:
Thanks to Dave, Dgrant and ThomasK it seems to be that the generate script is not closing the file as redirecting strerr to stdout shows:
Unhandled Exception: System.IO.IOException: The process cannot access the file '
C:\videos\data\feed2820712.xml' because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, I
nt32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions o
ptions, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
However I AM closing the file:
Extract from the generation script:
xmlfileObj.write('</FeedSettings>\n') # write the last line
xmlfileObj.close
# sleep to allow files to close
time.sleep(10)
# NOW GENERATE THE VIDEO
print ("initialising video...")
cmd = '"C:/Program Files/PropertyVideos/propertyvideos.console.exe" -xml data/feed2820712.xml -mpeg -q normal'
print (cmd)
p = subprocess.Popen(cmd , stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate()[0]
print ("\n----\n[" + output + "]\n----\n")
if output == "[success]":
print "\nHURRAHHHHH!!!!!!!"
print ("finished...")
Any help would be appreciated.
You're not closing the file. Your code says:
xmlfileObj.close
when it should be
xmlfileObj.close()
Edit: Just to clarify - the code xmlfileObj.close is a valid python expression which returns a reference to the built in close method of a file (or file like) object. Since it is a valid python expression it is perfectly legal code, but it does not have any side effects. Specifically, it does not have the effect of actually calling the close() method. You need to include the open and close brackets to do that.