We Keep Coding

python based shell embedded in c++ + select

I have an event based application where I select() on various things and depending on what gets selected, the application handles it. To add to this, I want to add an interactive menu implemented in python. In a typical update() (called from within while(!done) loop):
- while(!done) {
select() with timeout
give control to python menu with timeout
}
Also, I don't want to run the python menu on a separate thread since it brings alongwith it the issue of race conditions and due to the nature of code/application/customers, I cannot have locks in the part of the code where the menu interacts. So, it has to be done in a single threaded environment.
I have been reading online but I couldn't find a way to do this in single threaded environment with a timeout. Can anyone please point me in the right direction or any pointers on how to do this ?

Possible to run a delayed code execution?

Will it is possible to run a small set of code automatically after a script was run?
I am asking this because for some reasons, if I added this set of code into the main script, though it works, it will displays a list of tab errors (its already there, but it is stating that it cannot find it some sort).
I realized that after running my script, Maya seems to 'load' its own setup of refreshing, along with some plugins done by my company. As such, if I am running the small set of code after my main script execution and the Maya/ plugins 'refresher', it works with no problem. I had like to make the process as automated as possible, all within a script if that is possible...
Thus is it possible to do so? Like a delayed sort of coding method?
FYI, the main script execution time depends on the number of elements in the scene. The more there are, it will takes longer...

Maya has a command Maya.cmds.evalDeferred that is meant for this purpose. It waits till no more Maya processing is pending and then evaluates itself.
You can also use Maya.cmds.scriptJob for the same purpose.
Note: While eval is considered dangerous and insecure in Maya context its really normal. Mainly because everything in Maya is inherently insecure as nearly all GUI items are just eval commands that the user may modify. So the second you let anybody use your Maya shell your security is breached.

Writing an active program with wxPython. Where to start?

I spent the last hours trying to get to know wxPython, because I want to write a GUI program. I found some tutorials on that (not too many), but all of them just explain how to add yet another kind of widget, down to fancy things like LED number outputs and mouse gestures (this one e.g. takes it quite far: Another Tutorial). But everything I could find so far does nothing more than create a static GUI, waiting for the user to do something, then execute some handlers and wait again. It took me a while to even find out that wx.App takes a part in all of that, and that you can subclass it.
I want to write a program, that does things without input! The GUI is supposed to be a client that logs in on a server, and when the server sends something, I want the GUI to show what happened. I could not find a single tutorial even mentioning, that such programs exist. How can I write such a thing? How do they integrate with wxpython?
Do I need to span another thread? Is there a way to hook into the MainLoop and have some code executed periodically, that checks for change and then updates some of those fancy GUI things? And is there any page that teaches you, how to do this?

First of all, you should figure out how to do what you want WITHOUT a GUI. In this case, you'll need to figure out how to login to a server. You'll probably need to use something like paramiko for that. See http://www.lag.net/paramiko/
Once you've got that figured out, then you can add it to your GUI. Probably in a button handler so when the user presses a button, it pops up a dialog asking for a user name and password to pass to paramiko to login to the server.
If the server query takes a long time to execute (like say you're querying a database for a huge set of data), then you'll want to run the query in a separate thread. Why? Because that query will block the GUI's main loop and make your app freeze until it finishes. See the following articles for information on wxPython and threads:
http://wiki.wxpython.org/LongRunningTasks
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
I wrote up a tutorial on making wxPython talk to a socket server, so you might find that useful: http://www.blog.pythonlibrary.org/2013/06/27/wxpython-how-to-communicate-with-your-gui-via-sockets/
I also have an article on how to make an image viewer, and do CRUD ops to a database on there.

Obscure, repeatable crashes in multi-threaded Python console application using tk

Using tk in my multi-threaded(*) console application causes it to crash without stacktrace, giving the message "Tcl_WaitForEvent: Notifier not initialized Abort trap".
The symptoms were that all my program's functions worked fine, until I brought up the tk window - then the very next operation would cause the crash.
Immediate searching found that Tkinter is not safe with respect to Python threads, so I made sure that I was not calling any Tk functions anywhere other than my main thread. The crashes continued.
I lost several hours because I believed that it was the specific command I was using that crashed the program - but eventually I realized that any keyboard input would crash the program.
After a lot of debugging, I finally boiled it down to a small program that demonstrates the issue, exposing what I believe is a bug or certainly a feature that needs documentation in the Tkinter library.
I was working on this posting I was debugging. I'm going to post it and answer my own question in the hopes that it will prevent the next person from wasting a day on it.
--
(* - Yes, it certainly needs to be multi-threaded. I have a thread for my socket connection, a thread that listens to a mic and finds levels, a thread to drive my serial port and more. In each case, the thing I'm reading on the thread naturally blocks most of the time.)

The solution!
The issue is that tk crashes if you read from Python's raw_input in a different thread from the tk thread!
A small program demonstrating the issue is here. If you run it, it gets keyboard input perfectly happily from the second thread - until you enter the command "tk" when it brings up an empty tk window. You can do whatever you like with that window - until you type in the console window and press return, when the whole program crashes.
Why am I reading from raw_input in a thread that isn't the main thread?
My program is a console application, but I'm controlling a lot of different parts, one of which is the pi3d OpenGL ES 2.0 graphics library which must be updated at or near the frame rate from the main Python thread.
How to work around it?
"Simple" enough - register for tk menu events and just get the keys directly! Except that's a crappy solution, because you have to emulate all those things that the console does for you already - deleting, left and right arrows and that sort of thing. But that's what I'll have to do.
Should the program become a fully-fledged tk application?
But I can't do that - the whole point of this program is that you can run it through a terminal window - often sshing into headless machines. I'm frankly more likely to make it a curses program!
The tk window is a minor part of the whole thing - just a window to show emulated lights when developing if you don't have the hardware hooked up (or don't want to keep flashing yourself in the face). I won't try to bring it up on headless machines, and that's fine.
Is this a bug?
I'm always loathe to attach such a label to software not my own, but I'm hard-pressed to come up with any other description. It causes a crash, and that crash produces no useful information of any type. I consider that Tkinter is somewhat lame for simply crashing when called from different threads, but at least this behavior is documented (if you dig down a little) - in this case, I'm calling a Python built-in, so I have no basis to expect that it will interact with this library at all, and there's no documentation of this problem.
Could there be a workaround?
I'm sort of hoping there will be a work-around - this one-page program was a single item on a long list of features has turned into a full-day head-scratching debugging session and I don't want to have to throw another day at least after this when none of this time is actually producing new features.
The best thing is if the tk team admitted this was a bug and came up with a fix. But I wouldn't expect that at my desktop before a year from now...
So perhaps the real best thing would be if there were some way to get tk to simply ignore the keyboard, and not crash. I did a tiny experiment with tk's "busy" but that didn't work and just doesn't seem to be the right sort of thing.
A little later, I'm now thinking about running the lighting as an independent program, a separate subprocess using Python's Subprocess library, and sending it text commands through stdin. This would be overkill if this were the only problem that was being solved, but in fact
Got it.
Replacing raw_input() with sys.stdin.readline() did the trick - at least in the demo (which I updated). Feel free to download this and experiment with it yourself!
I hope this saves someone else the time.

In my case (as mentioned in the comments under #Tom Swirly's answer) the solution was to switch to a non-interactive backend:
import matplotlib
matplotlib.use('Agg')

Simplest way to have Python output, from a compiled package?

Prior info: I'm on a Mac.
Q: How can I get terminal-like text output from the program execution, if I compile it with py2app for redistribution?
My case is a program that copies a lot of big files and takes a while to process so I would like to at least have an output notification everytime each file is copied.
This is easy if I run it on the command line, I can just print a new line.
But when I make a self-sufficient package, it simply opens on the bottom dock, with no window, and closes upon completion.
A simple text window would be fine.
Thanks in advance.

If you want to create a simple text window, you need to pick a GUI framework to do that with. For something this simple, there's no reason not to use Tkinter (which comes with any Python) or PyObjC (which is pre-installed with Apple's Python 2.7), unless you happen to be more familiar with wx, gobject, Qt, etc.
At any rate, however you do it, you'll need to write a function that takes a message and appends it to the text window (maybe creating it lazily, if necessary), and call that function wherever you would normally print. You may also want to write and install a logging handler that does the same thing, so you can just log.info stuff. (You could instead create a file-like object that does this and redirect stdout and/or stderr, but unless you have no control over the printing code, that's going to be a lot more work.)
The only real problem here is that a GUI needs an event loop, and you probably just wrote your code as a sequential script.
One way around that is to turn your whole current script into a background thread. If you're using a GUI library that allows you to access the widgets from background threads, everything is easy; your printfunc just does textwidget.append(msg). If not, it may at least have a call_on_main_thread type function, so your printfunc does call_on_main_thread(textwidget.append, msg). If worst comes to worst (and I believe with Tkinter, it does), you have to create an explicit queue to push messages through, and write a queue handler in the event loop. This recipe should give you an idea. Replace the body of workerThread with your code, and end it with self.endApplication(). (There are probably better examples out there; this was just what I found first in a quick search.)
The other way around that is to have your code cooperatively operate with the event loop. Some libraries, like wx, have functions like SafeYield that make things work if you just call it after every chunk of processing. Others don't have that, but have a way to explicitly drive the event loop from your code. Others have neither—but every event loop framework has to have a way to schedule new events, so you can break your code up into a sequence of functions that each finish quickly and then do something like root.after_idle(nextfunc).
However… are you sure you need to do this?
First, any app, including one created by py2app, will send its stdout to the terminal if you run it with Foo.app/Contents/MacOS/Foo. And you can even set things up so that open Foo.app works that way, if you want. Obviously this doesn't help for people who just double-click the app in Finder (because then there is no terminal), but sometimes it's sufficient to just have to output available when people need it and know how to follow instructions.
And you can take this farther: Create a Foo.command file that just does something like $(dirname $0)/Foo.app/Contents/MacOS/Foo, and when you double-click that file, it launches Terminal.app and runs your script.
Or you can get even simpler: Just use logging to syslog the output, and if you want to see when each file is done, just watch the log messages go by in Console.app.
Finally, do you even need py2app in the first place? If you don't have any external dependencies, just rename you script to Foo.command, and double-clicking it will run it in Terminal.app. If you do have external dependencies, you might still be able to get away with bundling it all together as a folder with a .command in it instead of as a .app.
Obviously none of these ideas are exactly a professional or newbie-friendly way to build an interface, so if that matters, you will have to create a GUI.