This question already has an answer here:
Why doesn't print output show up immediately in the terminal when there is no newline at the end?
(1 answer)
Closed last month.
import time
import sys
sys.stdout.write("1")
time.sleep(5)
print("2")
will print "12" after 5 seconds
import time
import sys
sys.stdout.write("1\n")
time.sleep(5)
print("2")
will print "1\n" right away, then "2" after 5 seconds
Why is this?
If you add "\n" then stream is flushed automaticaly, and it is not without new line at the end.
You can flush output with: sys.stdout.flush()
Because stdout is buffered. You may be able to force the output sooner with a sys.stdout.flush() call.
The sys.stdout.write command from the sys module purposefully prints out the statement without the \n character. This is how a normal call to the stdout stream works, such as in C++ or C, where the \n character must be added manually.
However the print command provided by Python automatically adds a \n character to the string, therefore simplifying the code and making it easier to read.
The reason the phenomenon in the first result happens is because the system is waiting for a flush to print out, which is provided by the \n character. You can avoid this by using this command, sys.stdout.flush(), which will flush the stdout stream which forces it to print.
Buffering. It's not really Python, but rather your operating system/terminal. Output from any program is sent to a buffer, a holding area of memory. When a whole line is collected, it's sent to the screen. There is usually a hook, a method named something like flush(), to force output of partial lines.
It's because output in Python is buffered by default - ordinarily you won't get the output until the buffer is full or something causes the buffer to be flushed. In this case the \n was sensed and caused an automatic flush.
See this question for ways around the problem: How to flush output of Python print?
Consider that when you type a command into a computer, it doesn't know you're finished until you press ENTER
Similarly, the newline tells Python you've finished that line.
Related
I have a simple script that reads values from a device and outputs them via print, and another script, which listens on stdin and interprets each number. The device outputs one number each second. Surprisingly, piping the scripts on my ubuntu box does not work. However, if the first script is made not to read from the device but generate random numbers as fast as it can, the second script successfully receives the data.
Below is a simplified example of my situation.
print.py:
#!/usr/bin/env python2
import time
import sys
while True:
time.sleep(1) # without this everything works
print "42"
sys.stdout.flush()
read.py:
#!/usr/bin/env python2
import sys
while True:
for str in sys.stdin:
print str
Command line invocation:
vorac#laptop:~/test$ ./print.py | ./read.py
Here is the end result. The first script reads from the device and the second graphs the data in two separate time frames (what is shown are random numbers).
Ah, now that is a tricky problem. It happens because the iterator method for sys.stdin (which is xreadlines()) is buffered. In other words, when your loop implicitly calls next(sys.stdin) to get the next line of input, Python tries to read from the real under-the-hood standard input stream until its internal buffer is full, and only once the buffer is full does it proceed through the body of the loop. The buffer size is 8 kilobytes, so this takes a while.
You can see this by decreasing the time delay in the sleep() call to 0.001 or some such value, depending on the capabilities of your system. If you hit the time just right, you'll see nothing for a few seconds, and then a whole block of 42s come out all at once.
To fix it, use sys.stdin.readline(), which is unbuffered.
while True:
line = sys.stdin.readline()
print line
You might also want to strip off the trailing newline before printing it, otherwise you'll get double line breaks. Use line.rstrip('\n'), or just print line, to suppress the extra newline that gets printed.
I changed your read.py and it worked for me :), you forget to .readline() from stdin.
import sys
while True:
line = sys.stdin.readline()
if line:
print line.strip()
else:
continue
Output is :
$ python write.py | python read.py
42
42
42
42
42
42
42
I have the following code in program.py:
from sys import stdin
for line in stdin:
print line
I run, enter lines, and then press Ctrl+D, but the program does not exit.
This does work:
$ printf "echo" | python program.py
Why does the program not exit when I press Ctrl+d?
I am using the Fedora 18 terminal.
Ctrl+D has a strange effect. It doesn't close the input stream, but only causes a C-level fread() to return an empty result. For regular files such a result means that the file is now at its end, but it's acceptable to read more, e.g. to check if someone else wrote more data to the file in the meantime.
In addition, there are issues of buffering --- three levels of them!
Python's iteration over a file does block buffering. Avoid it to read from interactive streams.
the C-level stdin file has, by default, a line buffer.
the terminal itself(!), in its default mode ("cooked mode"), reads one line of data before sending it to the process, which explains why typing Ctrl+D doesn't have any effect when typed in the middle of a line.
This example avoids the first issue, which is all you need if all you want is detecting Ctrl+D typed as its own line:
import sys
while True:
line = sys.stdin.readline()
print repr(line)
You get every line with a final '\n', apart from when the "line" comes from a Ctrl+D, in which case you get just '' (but reading continues, unless of course we add if line == '': break).
Everytime I'm executing my Python script, it appears to hang on this line:
lines = sys.stdin.readlines()
What should I do to fix/avoid this?
EDIT
Here's what I'm doing with lines:
lines = sys.stdin.readlines()
updates = [line.split() for line in lines]
EDIT 2
I'm running this script from a git hook so is there anyway around the EOF?
This depends a lot on what you are trying to accomplish. You might be able do:
for line in sys.stdin:
#do something with line
Of course, with this idiom as well as the readlines() method you are using, you need to somehow send the EOF character to your script so that it knows that the file is ready to read. (On unix Ctrl-D usually does the trick).
Unless you are redirecting something to stdin that would be expected behavior. That says to read input from stdin (which would be the console you are running the script from). It is waiting for your input.
See: "How to finish sys.stdin.readlines() input?
If you're running the program in an interactive session, then this line causes Python to read from standard input (i. e. your keyboard) until you send the EOF character (Ctrl-D (Unix/Mac) or Ctrl-Z (Windows)).
>>> import sys
>>> a = sys.stdin.readlines()
Test
Test2
^Z
>>> a
['Test\n', 'Test2\n']
I know this isn't directly answering your question, as others have already addressed the EOF issue, but typically what I've found that works best when reading live output from a long lived subprocess or stdin is the while/if line approach:
while True:
line = sys.stdin.readline()
if not line:
break
process(line)
In this case, sys.stdin.readline() will return lines of text before an EOF is returned. Once the EOF if given, the empty line will be returned which triggers the break from the loop. A hang can still occur here, as long as an EOF isn't provided.
It's worth noting that the ability to process the "live output", while the subprocess/stdin is still running, requires the writing application to flush it's output.
I have a script which executes some command using os.popen4. Problem is some time command being executed will require user input ("y" or "n"). I am reading stdout/stderr and printing it, but it seems question from command doesn't got printed and it hangs. To make it work, i had to write "n" to stdin blindly. Can some one please guide on how to handle it?
Code not working:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
print cmd_out
f_p_stdin.write("n")
f_p_stdin.close()
f_p_stdout_stderr.close()
Working Code:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
f_p_stdin.write("n")
f_p_stdin.close()
print cmd_out
f_p_stdout_stderr.close()
NOTE : I am aware that it is depreciated and subprocess module is used, but right now i don't know on how to use it. So i'll appreciate if some one will help me to handle it using os.popen4. I want to capture the question and handle the input from user and execute it.
readlines() : returns a list containing all the lines of data in the file. If reading from a process like in this case, there is a good chance it does not send a newline and/or flush the output. You should read characters from the input and process that to see if the question was posed.
It would help to know what cmd_exec looks like, so others can try and emulate what you tried.
Update:
I wrote a uncheckout command in Python:
#! /usr/bin/env python
# coding: utf-8
import sys
print 'Uncheckout of {} is irreversible'.format(sys.argv[1])
print 'Do you want to proceed? [y/N]',
sys.stdout.flush()
x = raw_input()
if x == 'y':
print sys.argv[1], "no longer checked out"
else:
print sys.argv[1], "still checked out"
I put the prompt string on purpose not as argument to raw_input, to be able to do the flush() explicitly.
Neither of your code snippets work with that (assuming cmd_exec to be ['./uncheckout', 'abc.txt'] or './uncheckout abc.txt', popen4() uses the shell in the latter case to start the program).
Only when I move the readlines() until after the write() and close() will the command continue.
That makes sense to me as the close() flushes the output. You are writing in text mode and that buffers normally until end-of-line, which is not in your .write('n').
To be able to check what the prompt is and test and react on that., the following works with the above uncheckout:
#! /usr/bin/env python
# coding: utf-8
import os
import sys
cmd_exec = ['./uncheckout', 'abc.txt']
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
line = ''
while True:
x = f_p_stdout_stderr.read(1)
if not x:
break
sys.stdout.write(x)
sys.stdout.flush()
if x == '\n':
line = ''
else:
line += x
if line.endswith('[y/N]'):
f_p_stdin.write("n\n")
f_p_stdin.flush()
sys.stdout.write('\n')
Maybe you can work backwards from that to make something that works for you. Make sure to keep flushes at appropriate places.
I am new to python, and I am sorry if what I am asking seems odd. I want to loop over each line on standard input and return a modified line to standard output immediately. I have code that works, mostly. However I do not know how to make this work completely.
I have the following code
while True:
line = sys.stdin.readline()
if not line:
break
sys.stdout.write(line)
When being used interactively this will exit if there is an EOF on a new line, however if there is text before I type Control-D I must give the code twice before it will exit the line, and then once more before the loop will exit.
How do I fix this.
I think my answer from here can be copied immediately:
It has to do with ^D really does: it just stops the current
read(2) call.
If the program does int rdbytes = read(fd, buffer, sizeof buffer);
and you press ^D inbetween, read() returns with the currently read
bytes in the buffer, returning their number. The same happens on line
termination; the \n at the end is always delivered.
So only a ^D at the start of a line or after another ^D has the
desired effect of having read() return 0, signalizing EOF.
And this behaviour, of course, affects Python code as well.
A strategy suggested in the python docs is:
for line in sys.stdin:
sys.stdout.write(line)
See the IO Tutorial.