I want to run dialog --gauge from print()
Python file is like this
process1
print(30) #updates the gauge percentage to 30
process2
print(60) #updates the gauge percentage to 60
process3
print(100) # completes and exit the gauge
python run.py | dialog --gauge "Running program" 10 30 0
but the gauge is not updating or we can say python prints them at the last in an instant and not like they way they normally prints in terminal
How can I fix that, please tell
Not easy to answer if you don't provide the code.
But I am pretty sure you are missing a flush=True argument to your print.
import time
for i in range(100):
print(i, flush=True)
time.sleep(1)
works fine.
Some explanation about why it is needed: I/O are buffered, for obvious efficiency reason. That goes for print to. So there are different criteria to decide when to effectively do the print, that is when to empty (flush) the buffer and send the chars in it to whatever process is reading (for example, the terminal, in charge of rendering them, for an interactive process; or a process whose standard input is linked to our standard output through a pipe).
Obviously when output buffer is full. That is the most efficient, buffer size being optimized from different hardware optimums. Unless there are some applicative reasons not to do that
For interactive process (just python ran in a terminal), whenever a newline is printed. No need to bother the terminal char by char. Things are printed lines by lines. For most "line oriented" programs that is perfectly sufficient (oldie rant: don't forget that "terminals", which we should call "terminal emulators" are just an imitation of physical terminals, which, themselves, where some "paperless printer" replacing former printers. This is really "line oriented". There is a reason why CLI are not called "TCI" for "text command interface". Interaction with shells and terminal are mainly "line oriented").
But for non interactive process, there is no reason to treat newline differently. What is waiting for our output is neither a printer, nor anything imitating it. It is just another process, reading chars and processing them. There is no reason, a priori, to worry about interactivity.
Sometimes, tho, you need other rules that those default one. For example, because you are printing lines very slowly, and you want the terminal not to wait a newline before updating it. Or because you are feeding another process through a pipe, quite slowly too, and you don't want that other process to have to wait for your buffer to be full. Or, in your case, because you know that even tho your print are not directly sent to a terminal, but to another process, that is still an interactive application.
In those cases, you can force the flushing of the buffer after the print, with keyword argument flush=True
Related
I have a long-running script at work (windows unfortunately) where I programmed it to print the current analysis results if I ctrl-c. However, I was curious if after doing ctrl-c, I could start the script running again where it left off?
This is actually 3 questions:
-is it possible to do this without any programming changes? - e.g. I accidentally hit ctrl-c and want to retroactively start it where it left off
-can I use a command like ctrl-z (only on Mac I believe) on windows and program the script to print results when I issue it?
-what is the best programmatic way of automatically finishing the execution of the line I am on (massive .txt file of data) when I use an interrupt command, store that line number (in a file maybe), and restart the program on the next line with the next execution?
Thanks!
(FYI: I'm a novice Pythoner and my script currently takes about 10 min to perform 1 million lines. Files I will use in the future will often have 100+ million lines)
The short answer to your first question is No. Ctrl-C signals the interpreter, which unwinds the stack, presents you with a stack trace, and halts. You can't recover from ctrl-C for the same reason that you can't recover from any other untrapped exception. What you are asking for is a quick way to put Humpty Dumpty back together again.
You can restart a chess game from any point simply by laying out the pieces according to a picture you made before abandoning the game. But you can't easily do that with a program. The problem is that knowing the line number where the program stopped is not nearly enough information to recreate the state of the program at the time: the values of all the variables, the state of the stack, how much of the input it had read, and so forth. In other words, the picture is complicated, and laying out the pieces accurately is hard.
If your program is writing to the Windows console, you can suspend output by pressing ctrl-S and restart it by pressing ctrl-Q. These control characters are holdovers from the days of Teletype machines, but modern terminal emulators still obey them. This is a quick way to do what you want without program changes. Unsophisticated, but maybe good enough to begin with.
And your program will probably run a lot faster if it writes its output to file, for later examination in a text editor, rather than writing directly to the Windows console.
A full-on solution to your problem is something that I hesitate to recommend to a novice. The idea is to split calculation and display into two processes. The calculation process does its thing and feeds its results line by line to the display process. The display process listens to the calculation process and puts the results that it gets on the screen, but can also accept pause and resume commands. What happens while it is in the paused state is a design decision. You can decide either that the calculation process should block (easier option) or that it should buffer its results until the display process is ready to accept them again (harder option).
I'm asking the same question as this post, but in Python under Linux. I have a thread running in the background which output things constantly, but I would like to keep the input line at the bottom of the terminal without been flushed away. Can someone give me a code example? Suppose my code is like this:
from time import sleep
from multiprocessing import Process
def subproc():
while True:
print "something"
sleep(1)
def main():
p = Process(target=subproc)
p.start()
c = raw_input()
EDIT: by the way, I would like to keep the use of the raw_input function, if possible, because I want to have input history available.
unfortunately, there isn't any way of doing this that doesn't involve some serious or irritating consequences.
The way terminals scroll and handle stdin/stdout has not varied much in decades.
Take a look at terminal escape sequences. As long as the user terminal is based on the VT100 or ansi spec, these should work. You can use them to set the write location, clear lines, etc. But if a user scrolls, the illusion will break.
The problem of specifying a specific location on screen to output a character was addressed for early games like angband, rogue and nethack. The solution was the Curses library, and there happens to be a curses python module
By using curses, you can achieve what your after. But I know of no way of doing this in standard standard shells
I'm running an external process and I need to get the stdout immediately so I can push it to a textview, on GNU/Linux I can use "usePTY=True" to get the stdout by line, unfortunately usePTY is not available on windows.
I'm fairly new to twisted, is there a way to achieve the same result on Windows with some twisted (or python maybe) magic stuff?
on GNU/Linux I can use "usePTY=True" to get the stdout by line
Sort of! What usePTY=True actually does is create a PTY (a "pseudo-terminal" - the thing you always get when you log in to a shell on GNU/Linux unless you have a real terminal which no one does anymore :) instead of a boring old pipe. A PTY is a lot like a pipe but it has some extra features - but more importantly for you, a PTY is strongly associated with interactive sessions (ie, a user) whereas a pipe is pretty strongly associated with programmatic uses (think foo | bar - no user ever sees the output of foo).
This means that people tend to use existence of a PTY as stdout as a signal that they should produce output in a timely manner - because a human is waiting to see it. On the flip side, the existence of a regular old pipe as stdout is taken as a signal that another program is consuming the output and they should instead produce output in the most efficient way possible.
What this tends to mean in practice is that if a program has a PTY then it will line buffer its output and if it has a pipe then it will "block" buffer its output (usually gather up about 4kB of data before writing any of it) - because line buffering is less efficient.
The thing to note here is that it is the program you are running that does this buffering. Whether you pass usePTY=True or usePTY=False makes no direct difference to that buffering: it is just a hint to the program you are running what kind of output buffering it should do.
This means that you might run programs that block buffer even if you pass usePTY=True and vice versa.
However... Windows doesn't have PTYs. So programs on Windows can't consider PTYs as a hint for how to buffer their output.
I don't actually know if there is another hint that it is conventional for programs to respect on Windows. I've never come across one, at least.
If you're lucky, then the program you're running will have some way for you to request line-buffered output. If you're running Python, then it does - the PYTHONUNBUFFERED environment variable controls this, as does the -u command line option (and I think they both work on Windows).
Incidentally, if you plan to pass binary data between the two processes, then you probably also want to put stdio into binary mode in the child process as well:
import os, sys, mscvrt
msvcrt.setmode(sys.stdin.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
msvcrt.setmode(sys.stderr.fileno(), os.O_BINARY)
I am trying to communicate with a command-line chat bot with Python using the subprocess module. (http://howie.sourceforge.net/ using the compiled win32 binary, I have my reasons!)
This works:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
output = proc.communicate()
But Popen.communicate waits for the process to terminate (and sends it EOF?), I want to be able to interact with it. The apparent solution for this was to read stdout / write stdin like so:
This doesn't work:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
while True: print proc.stdout.readline()
(Note that I am actually using more complex code based on http://code.activestate.com/recipes/440554/ but the issue is the same.)
The problem is, the second approach works perfectly for communicating to cmd, but when I run the chatbot, nothing. So my question is, how is this different in capturing output to using Popen.communicate()?
i.e. I can use the second approach to use the command line as per normal, until I run the chatbot, at which point I stop receiving output. Using the first approach correctly displays the first few lines of output from the bot, but leaves me unable to interact with it.
One major difference between the two is that communicate() closes stdin after sending the data. I don't know about your particular case, but in many cases this means that if a process is awaiting the end of the user input, he will get it when communicate() is used, and will never get it when the code blocks on read() or readline().
Try adding Popen.stdin.close() first and see if it affects your case.
If you want to interact with the program after sending the EOF, rather than using Popen.stdin.close(), you can manually send the command-line End Of File character, which has the same effect but leaves stdin open.
In Python this character's escape sequence is '\x1a'.
This question already has answers here:
read subprocess stdout line by line
(10 answers)
Closed 21 days ago.
How can I receive input from the terminal in Python?
I am using Python to interface with another program which generates output from user input.
I am using subprocess.Popen() to input to the program, but I can't set stdout to subprocess.PIPE because the program does not seem to flush ever, so everything gets stuck in the buffer.
The program's standard output seems to be to print to terminal, and I see output when I do not redirect stdout. However, I need Python to read and interpret the output which is now in the terminal.
Sorry if this is a stupid question, but I can't seem to get this to work.
Buffering in child processes is a common problem. Here are four possible approaches.
First, and easiest, you could read one byte at a time from your pipe. This is what I would call a "dirty hack" and it carries a performance penalty, but it's easy and it guarantees that your read() calls will only block until the first byte comes in, rather than wait for a buffer to fill up that's never going to fill up. However, this does not force the other process to flush its write buffer, so if that is the issue this approach will not help you anyway.
Second, and I think next-easiest, consider using the Twisted framework which has a facility for using a virtual terminal, or pty ("pseudo-teletype" I think) to talk to your child process. However, this can affect the design of your application (possibly for the better, but this may not be in the cards for you regardless). http://twistedmatrix.com/documents/current/core/howto/process.html
If neither of the above options works for you, you're reduced to solving gritty I/O concurrency issues yourself.
Third, try setting your pipes (all of them, before fork()) to non-blocking mode using fcntl() with O_NONBLOCK. Then you can use select() to test for read/write readiness before trying the read/write; but you still have to catch IOError and test for EAGAIN because it can happen even in this case. This may, depending on the behavior of the child process, allow you to wait until the data really shows up before trying to read it in.
The last resort is to implement the PTY logic yourself. If you've seen references to stuff like termio options, ioctl() calls, etc. then that's what you're up against. I have not done this before, because it's complicated and I have never really needed to. If this is your destiny, good luck.
Have you tried setting the bufsize in your Popen object to 0? I'm not sure if you can force the buffer to be unbuffered from the receiving size, but I'd try it.
http://docs.python.org/library/subprocess.html#using-the-subprocess-module