I have a simple script that reads values from a device and outputs them via print, and another script, which listens on stdin and interprets each number. The device outputs one number each second. Surprisingly, piping the scripts on my ubuntu box does not work. However, if the first script is made not to read from the device but generate random numbers as fast as it can, the second script successfully receives the data.
Below is a simplified example of my situation.
print.py:
#!/usr/bin/env python2
import time
import sys
while True:
time.sleep(1) # without this everything works
print "42"
sys.stdout.flush()
read.py:
#!/usr/bin/env python2
import sys
while True:
for str in sys.stdin:
print str
Command line invocation:
vorac#laptop:~/test$ ./print.py | ./read.py
Here is the end result. The first script reads from the device and the second graphs the data in two separate time frames (what is shown are random numbers).
Ah, now that is a tricky problem. It happens because the iterator method for sys.stdin (which is xreadlines()) is buffered. In other words, when your loop implicitly calls next(sys.stdin) to get the next line of input, Python tries to read from the real under-the-hood standard input stream until its internal buffer is full, and only once the buffer is full does it proceed through the body of the loop. The buffer size is 8 kilobytes, so this takes a while.
You can see this by decreasing the time delay in the sleep() call to 0.001 or some such value, depending on the capabilities of your system. If you hit the time just right, you'll see nothing for a few seconds, and then a whole block of 42s come out all at once.
To fix it, use sys.stdin.readline(), which is unbuffered.
while True:
line = sys.stdin.readline()
print line
You might also want to strip off the trailing newline before printing it, otherwise you'll get double line breaks. Use line.rstrip('\n'), or just print line, to suppress the extra newline that gets printed.
I changed your read.py and it worked for me :), you forget to .readline() from stdin.
import sys
while True:
line = sys.stdin.readline()
if line:
print line.strip()
else:
continue
Output is :
$ python write.py | python read.py
42
42
42
42
42
42
42
Related
as an example I have two scripts, say script1.py
f = open("output1.txt", "w")
count = 1
for i in range(100):
f.write(str(count) + "\n")
print(str(count))
count +=1
f.close
This script prints numbers from 1 to 100 to a file and to standard output.
Then I have a second script, say script2.py
import sys
import time
stdin = sys.stdin
f1 = open("output2.txt", "w")
for line in stdin:
if len(line)>0:
print(line.strip())
time.sleep(0.05)
f1.write(line.strip() + "\n")
which reads data from standard input and prints them to a file. I added a time.sleep command to ensure the second script consumes data at a far lower rate than they are produced by the first one.
I run the scripts from the command line as
python3 script1.py | python3 script2.py
so redirecting the standard output of the first (so the print() command) to the standard input of the second one.
It works as somehow expected, two files are generated containing numbers from 1 to 100.
I am nevertheless wondering how the data transfer part works, from first to second script.
the first script generates data at a faster rate. Where are these data stored, waiting for the second script to access them?
Is there some sort of buffer that is put in place between the two process? Or what else?
Is Python responsible for this, or the OS?
Is the buffer limited in size? Can it be programmed (e.g. accessed to direct data to another target as well)?
Thanks a bunch
it is because of the pipe "|", more info here: https://ss64.com/nt/syntax-redirection.html
commandA | commandB Pipe the output from commandA into commandB
so the prints from your script1 are sent to your script2.
My guess on how it works is that every print is saved in memory as a big string and then sent back (as text) to the second that s why sys.stdin works
Hi I have a problem that I cannot seem to find any solution for.
(Maybe i'm just horrible at phrasing searches correctly in english)
I'm trying to execute a binary from python using pwntools and reading its output completely before sending some input myself.
The output from my binary is as follows:
Testmessage1
Testmessage2
Enter input: <binary expects me to input stuff here>
Where I would like to read the first line, the second line and the output part of the third line (with ':' being the last character).
The third line of the output does not contain a newline at the end and expects the user to make an input directly. However, I'm not able to read the output contents that the third line starts with, no matter what I try.
My current way of trying to achieve this:
from pwn import *
io = process("./testbin")
print io.recvline()
print io.recvline()
print io.recvuntil(":", timeout=1) # this get's stuck if I dont use a timeout
...
# maybe sending data here
# io.send(....)
io.close()
Do I missunderstand something about stdin and stdout? Is "Enter input:" of the third line not part of the output that I should be able to receive before making an input?
Thanks in advance
I finally figured it out.
I got the hint I needed from
https://github.com/zachriggle/pwntools-glibc-buffering/blob/master/demo.py
It seems that Ubuntu is doing lots of buffering on its own.
When manually making sure that pwnTools uses a pseudoterminal for stdin and stdout it works!
import * from pwn
pty = process.PTY
p = process(stdin=pty, stdout=pty)
You can use the clean function which is more reliable and which can be used for remote connections: https://docs.pwntools.com/en/dev/tubes.html#pwnlib.tubes.tube.tube.clean
For example:
def start():
p = remote("0.0.0.0", 4000)
return p
io = start()
io.send(b"YYYY")
io.clean()
io.send(b"ZZZ")
I have the following code in program.py:
from sys import stdin
for line in stdin:
print line
I run, enter lines, and then press Ctrl+D, but the program does not exit.
This does work:
$ printf "echo" | python program.py
Why does the program not exit when I press Ctrl+d?
I am using the Fedora 18 terminal.
Ctrl+D has a strange effect. It doesn't close the input stream, but only causes a C-level fread() to return an empty result. For regular files such a result means that the file is now at its end, but it's acceptable to read more, e.g. to check if someone else wrote more data to the file in the meantime.
In addition, there are issues of buffering --- three levels of them!
Python's iteration over a file does block buffering. Avoid it to read from interactive streams.
the C-level stdin file has, by default, a line buffer.
the terminal itself(!), in its default mode ("cooked mode"), reads one line of data before sending it to the process, which explains why typing Ctrl+D doesn't have any effect when typed in the middle of a line.
This example avoids the first issue, which is all you need if all you want is detecting Ctrl+D typed as its own line:
import sys
while True:
line = sys.stdin.readline()
print repr(line)
You get every line with a final '\n', apart from when the "line" comes from a Ctrl+D, in which case you get just '' (but reading continues, unless of course we add if line == '': break).
I have a script which executes some command using os.popen4. Problem is some time command being executed will require user input ("y" or "n"). I am reading stdout/stderr and printing it, but it seems question from command doesn't got printed and it hangs. To make it work, i had to write "n" to stdin blindly. Can some one please guide on how to handle it?
Code not working:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
print cmd_out
f_p_stdin.write("n")
f_p_stdin.close()
f_p_stdout_stderr.close()
Working Code:
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
cmd_out = f_p_stdout_stderr.readlines()
f_p_stdin.write("n")
f_p_stdin.close()
print cmd_out
f_p_stdout_stderr.close()
NOTE : I am aware that it is depreciated and subprocess module is used, but right now i don't know on how to use it. So i'll appreciate if some one will help me to handle it using os.popen4. I want to capture the question and handle the input from user and execute it.
readlines() : returns a list containing all the lines of data in the file. If reading from a process like in this case, there is a good chance it does not send a newline and/or flush the output. You should read characters from the input and process that to see if the question was posed.
It would help to know what cmd_exec looks like, so others can try and emulate what you tried.
Update:
I wrote a uncheckout command in Python:
#! /usr/bin/env python
# coding: utf-8
import sys
print 'Uncheckout of {} is irreversible'.format(sys.argv[1])
print 'Do you want to proceed? [y/N]',
sys.stdout.flush()
x = raw_input()
if x == 'y':
print sys.argv[1], "no longer checked out"
else:
print sys.argv[1], "still checked out"
I put the prompt string on purpose not as argument to raw_input, to be able to do the flush() explicitly.
Neither of your code snippets work with that (assuming cmd_exec to be ['./uncheckout', 'abc.txt'] or './uncheckout abc.txt', popen4() uses the shell in the latter case to start the program).
Only when I move the readlines() until after the write() and close() will the command continue.
That makes sense to me as the close() flushes the output. You are writing in text mode and that buffers normally until end-of-line, which is not in your .write('n').
To be able to check what the prompt is and test and react on that., the following works with the above uncheckout:
#! /usr/bin/env python
# coding: utf-8
import os
import sys
cmd_exec = ['./uncheckout', 'abc.txt']
(f_p_stdin, f_p_stdout_stderr) = os.popen4(cmd_exec,"t")
line = ''
while True:
x = f_p_stdout_stderr.read(1)
if not x:
break
sys.stdout.write(x)
sys.stdout.flush()
if x == '\n':
line = ''
else:
line += x
if line.endswith('[y/N]'):
f_p_stdin.write("n\n")
f_p_stdin.flush()
sys.stdout.write('\n')
Maybe you can work backwards from that to make something that works for you. Make sure to keep flushes at appropriate places.
This question already has an answer here:
Why doesn't print output show up immediately in the terminal when there is no newline at the end?
(1 answer)
Closed last month.
import time
import sys
sys.stdout.write("1")
time.sleep(5)
print("2")
will print "12" after 5 seconds
import time
import sys
sys.stdout.write("1\n")
time.sleep(5)
print("2")
will print "1\n" right away, then "2" after 5 seconds
Why is this?
If you add "\n" then stream is flushed automaticaly, and it is not without new line at the end.
You can flush output with: sys.stdout.flush()
Because stdout is buffered. You may be able to force the output sooner with a sys.stdout.flush() call.
The sys.stdout.write command from the sys module purposefully prints out the statement without the \n character. This is how a normal call to the stdout stream works, such as in C++ or C, where the \n character must be added manually.
However the print command provided by Python automatically adds a \n character to the string, therefore simplifying the code and making it easier to read.
The reason the phenomenon in the first result happens is because the system is waiting for a flush to print out, which is provided by the \n character. You can avoid this by using this command, sys.stdout.flush(), which will flush the stdout stream which forces it to print.
Buffering. It's not really Python, but rather your operating system/terminal. Output from any program is sent to a buffer, a holding area of memory. When a whole line is collected, it's sent to the screen. There is usually a hook, a method named something like flush(), to force output of partial lines.
It's because output in Python is buffered by default - ordinarily you won't get the output until the buffer is full or something causes the buffer to be flushed. In this case the \n was sensed and caused an automatic flush.
See this question for ways around the problem: How to flush output of Python print?
Consider that when you type a command into a computer, it doesn't know you're finished until you press ENTER
Similarly, the newline tells Python you've finished that line.