This question already has answers here:
read subprocess stdout line by line
(10 answers)
Closed 21 days ago.
How can I receive input from the terminal in Python?
I am using Python to interface with another program which generates output from user input.
I am using subprocess.Popen() to input to the program, but I can't set stdout to subprocess.PIPE because the program does not seem to flush ever, so everything gets stuck in the buffer.
The program's standard output seems to be to print to terminal, and I see output when I do not redirect stdout. However, I need Python to read and interpret the output which is now in the terminal.
Sorry if this is a stupid question, but I can't seem to get this to work.
Buffering in child processes is a common problem. Here are four possible approaches.
First, and easiest, you could read one byte at a time from your pipe. This is what I would call a "dirty hack" and it carries a performance penalty, but it's easy and it guarantees that your read() calls will only block until the first byte comes in, rather than wait for a buffer to fill up that's never going to fill up. However, this does not force the other process to flush its write buffer, so if that is the issue this approach will not help you anyway.
Second, and I think next-easiest, consider using the Twisted framework which has a facility for using a virtual terminal, or pty ("pseudo-teletype" I think) to talk to your child process. However, this can affect the design of your application (possibly for the better, but this may not be in the cards for you regardless). http://twistedmatrix.com/documents/current/core/howto/process.html
If neither of the above options works for you, you're reduced to solving gritty I/O concurrency issues yourself.
Third, try setting your pipes (all of them, before fork()) to non-blocking mode using fcntl() with O_NONBLOCK. Then you can use select() to test for read/write readiness before trying the read/write; but you still have to catch IOError and test for EAGAIN because it can happen even in this case. This may, depending on the behavior of the child process, allow you to wait until the data really shows up before trying to read it in.
The last resort is to implement the PTY logic yourself. If you've seen references to stuff like termio options, ioctl() calls, etc. then that's what you're up against. I have not done this before, because it's complicated and I have never really needed to. If this is your destiny, good luck.
Have you tried setting the bufsize in your Popen object to 0? I'm not sure if you can force the buffer to be unbuffered from the receiving size, but I'd try it.
http://docs.python.org/library/subprocess.html#using-the-subprocess-module
Related
I am trying to simulate a communication protocol where I am following a pattern, so I constantly loop though looking for the same set of characters to reply information. I'm using an RS-232 adapter and the protocol I am simulating is asynchronous and half-duplex where the rx/tx lines are tied together by design and that causes a sort of echo when reading after writing.
That said, I need to be able to clear the input buffer after every write I send out in order to avoid reading what I just wrote. So whenever I use reset_input_buffer() it does not clear the last message I sent out. I have tried to fix this using a couple of methods, such as: using reset_output_buffer() together with reset_input_buffer(), using reset_input_buffer() twice, and using flush(). None of these methods make any difference, the only other method that works to clear the buffer is closing and immediately opening the port but this causes a delay that messes with the timing as it is critical at certain times.
I'm open to any suggestions, please help!
I'm new to python paramiko. I know there are two ways to execute a command in a remote server invokde_shell and exec_command. In few cases the output is read using stdout.readlines() whereas in other cases using channel.recv with exit_status as loop condition. It is very difficult to understand the difference between both and which one to use for my script. Can anyone please explain ?
This is rather broad question, so only briefly:
readlines vs recv – This is nothing Paramiko-specific. You have the same set of functions, when reading local files or local program input. Use whatever fits your needs. If you need to read by bytes (e.g. when processing a binary input), you probably want to use recv (or read). If you want to process a textual input by lines, use readlines (or readline).
You also mix in shell vs. exec into your question, what is a separate stuff, covered here:
What is the difference between exec_command and send with invoke_shell() on Paramiko?
Overall, you better ask a specific question about implementing your specific problem.
I noticed that in python when I use a print statement, that it doesn't print immediately. I found you can use sys.stdout.flush() to make it show the print in the console.
Is this the proper way of getting immediate feedback from a script, or is there a better way?
I mainly want this for debugging. I had a hang and was trying to find where the code stalled, but since my print statements didn't show up I was searching in the wrong place, thinking my code didn't get to the print statement. (I finally found it with breakpoints, which is perhaps a better way of debugging, but print immediate-prints are just more convenient sometimes.)
Exactly how and when stdout will flush itself depends on the implementation of the print function you're using, the string you're trying to print, possibly the OS, etc.
Anecdotally, stdout seems to flush when it hits a newline character, but I don't always trust it, or I won't know until runtime if my string will contain a newline, so I use flush() pretty regularly in C, Java, and Python when I want to be sure that something prints immediately and I am not overly concerned about performance. Places where I've found this especially useful (ie where timing of the messages really matters) is: near code that is likely to throw exceptions, right before blocking IO calls, in multi-threaded sections of code.
As a practical example, something like this seems like a natural usage for flush() (sorry it's in C++ rather than Python)
printf("Connecing to IP Camera at %s...", ipCamAddr);
cout.flush();
// open a network connection to the camera
printf("Done!\n");
Because without the flush, it would hold the first string until it gets a \n, which is after the connection has succeeded / failed.
I do not want to lose my sets if windows is about to shutdown/restart/log off/sleep, Is it possible to save it before shutdown? Or is there an alternative to save information without worring it will get lost on windows shutdown? JSON, CSV, DB? Anything?
s = {1,2,3,4}
with open("s.pick","wb") as f: # pickle it to file when PC about to shutdown to save information
pickle.dump(s,f)
I do not want to lose my sets if windows is about to shutdown/restart/log off/sleep, Is it possible to save it before shutdown?
Yes, if you've built an app with a message loop, you can receive the WM_QUERYENDSESSION message. If you want to have a GUI, most GUI libraries will probably wrap this up in their own way. If you don't need a GUI, your simplest solution is probably to use PyWin32. Somewhere in the docs there's a tutorial on creating a hidden window and writing a simple message loop. Just do that on the main thread, and do your real work on a background thread, and signal your background thread when a WM_QUERYENDSESSION message comes in.
Or, much more simply, as Evgeny Prokurat suggests, just use SetConsoleCtrlHandler (again through PyWin32). This can also catch ^C, ^BREAK, and the user closing your console, as well as the logoff and shutdown messages that WM_QUERYENDSESSION catches. More importantly, it doesn't require a message loop, so if you don't have any other need for one, it's a lot simpler.
Or is there an alternative to save information without worring it will get lost on windows shutdown? JSON, CSV, DB? Anything?
The file format isn't going to magically solve anything. However, a database could have two advantages.
First, you can reduce the problem by writing as often as possible. But with most file formats, that means rewriting the whole file as often as possible, which will be very slow. The solution is to streaming to a simpler "journal" file, packing that into the real file less often, and looking for a leftover journal at every launch. You can do that manually, but a database will usually do that for you automatically.
Second, if you get killed in the middle of a write, you end up with half a file. You can solve that by the atomic writing trick—write a temporary file, then replace the old file with the temporary—but this is hard to get right on Windows (especially with Python 2.x) (see Getting atomic writes right), and again, a database will usually do it for you.
The "right" way to do this is to create a new window class with a msgproc that dispatches to your handler on WM_QUERYENDSESSION. Just as MFC makes this easier than raw Win32 API code, win32ui (which wraps MFC) makes this easier than win32api/win32gui (which wraps raw Win32 API). And you can find lots of samples for that (e.g., a quick search for "pywin32 msgproc example" turned up examples like this, and searches for "python win32ui" and similar terms worked just as well).
However, in this case, you don't have a window that you want to act like a normal window, so it may be easier to go right to the low level and write a quick&dirty message loop. Unfortunately, that's a lot harder to find sample code for—you basically have to search the native APIs for C sample code (like Creating a Message Loop at MSDN), then figure out how to translate that to Python with the pywin32 documentation. Less than ideal, especially if you don't know C, but not that hard. Here's an example to get you started:
def msgloop():
while True:
msg = win32gui.GetMessage(None, 0, 0)
if msg and msg.message == win32con.WM_QUERYENDSESSION:
handle_shutdown()
win32api.TranslateMessage(msg)
win32api.DispatchMessage(msg)
if msg and msg.message == win32con.WM_QUIT:
return msg.wparam
worker = threading.Thread(real_program)
worker.start()
exitcode = msgloop()
worker.join()
sys.exit(exitcode)
I haven't shown the "how to create a minimal hidden window" part, or how to signal the worker to stop with, e.g., a threading.Condition, because there are a lot more (and easier-to-find) good samples for those parts; this is the tricky part to find.
you can detect windows shutdown/log off with win32api.setConsoleCtrlHandler
there is a good example How To Catch “Kill” Events with Python
In this discussion about the easiest way to run a process and discard its output, I suggested the following code:
with open('/dev/null', 'w') as dev_null:
subprocess.call(['command'], stdout=dev_null, stderr=dev_null)
Another developer suggested this version:
subprocess.call(['command'], stdout=open('/dev/null', 'w'), stderr=STDOUT)
The C++ programmer in me wants to say that when objects are released is an implementation detail, so to avoid leaving a filehandle open for an indeterminate period of time, I should use with. But a couple of resources suggest that Python always or almost always uses reference counting for code like this, in which case the filehandle should be reclaimed as soon as subprocess.call is done and using with is unnecessary.
(I guess that leaving a filehandle open to /dev/null in particular is unlikely to matter, so pretend it's an important file.)
Which approach is best?
You are correct, refcouting is not guaranteed. In fact, only CPython (which is the main implementation, yes, but not even remotely the only one) provdies refcounting. In case CPython ever changes that implementation detail (unlikely, yes, but possible), or your code is ever run on an alternate implementation, or you lose refcouting because of any other reason, the file won't be closed. Therefore, and given that the with statement makes cleanup very easy, I would suggest you always use a context manager when you open files.
When the pipe to the null device closes is irrelevant - it won't lead to data loss in the output or some such. While you maybe want to use the with variant always to ensure that your output files are always properly flushed and closed, etc., this isn't an example where this matters.
The entire point of the with statement is to have a controlled cleanup process. You're doing it right, don't let anyone convince you otherwise.