We Keep Coding

Replacement for execfile in IPython3?

There are various ways of re-enabling an execfile-like behaviour for Python 3.x environments - in the documentation and here on stackoverflow, but I did not find an exact replacement for my use case.
I am using IPython, and in Python 2.7.x execfile used to run a script file just as if I typed the exact same lines straight into IPython. This includes:
useful exception-tracebacks are given
local variables of my environment are available in the scripted code
variables defined locally in the script are available in the environment (after the execfile call, of course)
import X as Y statements in the script also make Y available in the environment
the execfile call works in interactive modes and also directly in python scripts
execution of the entire script code is guaranteed for each call (except when hitting an exception)
execfile is readily available wherever Python is - no lengthy definition or import of obscure package
Common solutions that have not entirely worked so far:
from scriptfile import * does not satisfy #2 and #4. For function definitions, it also fails #6, as re-issuing the import does not update a function - this can be remedied with a reload(scriptfile) call.
The exec(scriptfilehandle.read()) construction satisfies #5-7. With some amendments also #2-4 can be dealt with - but this evolves to a lengthy definition, which I can't recall right now, and tracebacks are still a mess.
IPython's %run scriptfile is nice, but falls short at least on requirement #2, #4 and #5.
Copying the script code from file and using IPython's %paste leaves out on #5 and #7 - and is quite cumbersome to do for each call.
Do you have any solutions I have not yet heard of?
I am using IPython+execfile while playing around with data, producing (lots of) matplotlib figures, trying stuff,...
And if I like some lines I wrote, I put the code snippet in a script. Some examples of what I am doing:
writing a script that prepares the environment for a specific dataset: doing imports, loading some data, defining some useful functions to work on this dataset,...
semiautomatic plotting: elaborate script for beautifully plotting ten figures of data held in a local variable, then revising the plot-script, and re-executing it, then filtering the data, re-executing plot-script,...
writing a script that utilizes several of my smaller snippets, to be run overnight on a large dataset
apart from data exploration and plotting, sometimes I need to write small scripts on various systems: a RasPi, a router with OpenWRT, a machine without internet access, a Windows machine (without admin rights) - all of which may have their restrictions on what libraries are available
On the other hand I have to admit, that I'm not a professional programmer - my insight on Python's inner workings with local/global variables, and what really happens in an import statement, are very limited.
Any help - may it be a solution to my problems or a helpful explaination - would be greatly appreciated!

Embed Python code in Verilog file in Emacs

I am working on a project whereby I need to embed Python within a Verilog file. The Python isn't really intended for execution in the normal sense as it will be read by a secondary tool. The Python will be written blocks that have some fixed demarcation (such as #+BEGIN_SRC, in org-babel).
module name ();
#+BEGIN_SRC python
def my_function ():
...
#+END_SRC
always #(posedge clk)
...
endmodule
Within Emacs this causes havoc, although Python-mode and Verilog-mode work fine, when combining both in the same file things quickly break-down as one would expect. Indentation is hopelessly broken as is syntax-highlighting. I understand this is a very weird thing to do, and I understand that there will almost certainly never been any real need to do this under normal circumstances, however for this particular case it is necessary.
My question: is there anyway within Emacs to specify multiple major modes within the same file. For example, is there some way that I can write a file using Verilog-mode as my major mode, but use Python-mode within the predefined blocks that are then ignored in the reset of the file.

There are a number of possibilities listed here:
http://emacswiki.org/emacs/MultipleModes
I've used multi-mode with latex and haskell, and it works OK.

Debugging the python VM

Is there a debugger that can debug the Python virtual machine while it is running Python code, similar to the way that GDB works with C/C++? I have searched online and have come across pdb, but this steps through the code executed by the Python interpreter, not the Python interpreter as its running the program.

The reference implementation of Python, CPython, is written in C. You can use GDB to debug it as you would debug any other program written in C.
That said, Python does have a few little helpers for use in GDB buried under Misc/gdbinit. It's got comments to describe what each command does, but I'll repeat them here for convenience:
pyo: Dump a PyObject *.
pyg: Dump a PyGC_Head *.
pylocals: Print the local variables of the current Python stack frame.
lineno: Get the current Python line number.
pyframe: Print the source file name, line, and function.
pyframev: pyframe + pylocals
printframe: pyframe if within PyEval_EvalFrameEx; built-in frame otherwise
pystack: Print the Python stack trace.
pystackv: Print the Python stack trace with local variables.
pu: Print a Unicode string.
It looks like the Fedora project has also assembled their own collection of commands to assist with debugging which you may want to look at, too.

If you're looking to debug Python at the bytecode level, that's exactly what pdb does.
If you're looking to debug the CPython reference interpreter… as icktoofay's answer says, it's just a C program like any other, so you can debug it the same way as any other C program. (And you can get the source, compile it with extra debugging info, etc. if you want, too.)
You almost certainly want to look at EasierPythonDebugging, which shows how to set up a bunch of GDB helpers (which are Python scripts, of course) to make your life easier. Most importantly: The Python stack is tightly bound to the C stack, but it's a big mess to try to map things manually. With the right helpers, you can get stack traces, frame dumps, etc. in Python terms instead of or in parallel with the C terms with no effort. Another big benefit is the py-print command, which can look up a Python name (in nearly the same way a live interpreter would), call its __repr__, and print out the result (with proper error handling and everything so you don't end up crashing your gdb session trying to walk the PyObject* stuff manually).
If you're looking for some level in between… well, there is no level in between. (Conceptually, there are multiple layers to the interpreter, but it's all just C code, and it all looks alike to gdb.)
If you're looking to debug any Python interpreter, rather than specifically CPython, you might want to look at PyPy. It's written in a Python-like language called RPython, and there are various ways to use pdb to debug the (R)Python interpreter code, although it's not as easy as it could be (unless you use a flat-translated PyPy, which will probably run about 100x too slow to be tolerable). There are also GDB debug hooks and scripts for PyPy just like the ones for CPython, but they're not as complete.

python.el shell and execute buffer

I am using the python.el from fgallina python.el github There is a long list of features but I am unaware of how to use them.
EDIT: Found this mode was working, however didn't automatically open the interpreter went the information was sent there. How can I send/load my current python buffer to be evaluated? I am trying to use C-c C-c but I don't get any output.
Python shell integration and shell completion? Do I need to add something beside the standard load-path and require statements for this?
Imenu support this isn't showing for me either.
Edit I tried adding this to my emacs but that hasn't worked.
add-hook 'python-mode-hook 'imenu-add-menubar-index)
I am using emacs 23 on ubuntu

For your first question: Usually you can load a buffer/file using C-c C-l. (Just tested this for python.el, and it works for this one as well) This will send the file to the inferior python process -- so you should open the *Python* buffer to see the results. The *Python* buffer is also your regular python interpreter/shell/REPL. C-M-x is bound to python-send-defun which sends a method/function definition to the interpreting process. For other bindings just hit M-x describe-bindings and then look for "python-" (under "Major Mode Bindings") in the opened *Help* buffer window.
There is some more documentation at the beginning of python.el if you want to know more about completion:
Quote:
Shell completion: hitting tab will try to complete the current
word. Shell completion is implemented in a manner that if you
change the python-shell-interpreter to any other (for example
IPython) it should be easy to integrate another way to calculate
completions. You just need to specify your custom
python-shell-completion-setup-code and
python-shell-completion-string-code.
Quote:
Symbol completion: you can complete the symbol at point. It uses
the shell completion in background so you should run
python-shell-send-buffer from time to time to get better results.
Skeletons: 6 skeletons are provided for simple inserting of class,
def, for, if, try and while. These skeletons are integrated with
dabbrev. If you have dabbrev-mode activated and
python-skeleton-autoinsert is set to t, then whenever you type
the name of any of those defined and hit SPC, they will be
automatically expanded.
The imenu entry shows up for me without adding the hook, just (require 'python) in a configuration file.

Can You Embed an TCL Script in Bash Script or Python Script That's Callable by External Programs?

I'm writing a script to extract some useful data about a series of chemical simulations I've been running.
To get this data I need (1) a C-program that calculates the density from a file type called *.pdb. I already have (1). And (2) I need to use a program called vmd to get that pdb. In order to accomplish (2) from the command line, I can submit a tcl script, as vmd has a build in tcl interpreter.
These functions -- calling the vmd to run the tcl script, then running the compiled c-program -- will be the key activities of my wrapper data extraction script.
I would like to eliminate the superfluous TCL script, reducing my count from 2 scripts (wrapper script + tcl script for vmd) down to 1. But I'm not sure quite how to do this. One potentially solution seems to be to embed my TCL script within my wrapper script, if there's a way to make such an embedded script callable from external programs.
Most of my data collection scripts so far have been in BASH, so ideally I would like to stick to a BASH script as I'm very familiar with bash scripting versus having only beginning knowledge of Python/Perl.
Here are my questions:
1. Can you embed a TCL script inside a Bash script?
2. Can you make this script callable by an external program?
e.g. in pseudocode:
#!/bin/bash
....
tclembed extract {
#tcl script
...
}
...
vmd -dispdev text -e extract.tcl >& extract_results.log #where vmd is
#an external program
3. If the answer to #2 is no, can you do this in Python, perhaps with the Minotaur library? I would consider the switch to python, if so...
http://markmail.org/message/6kogjphzqtn4ilch
4. If not, how would you suggest trying to merge these two scripts (a tcl routine and a bash script that calls it) into a single file?
5. If anybody HAS gotten external calls of this nature to work using Minotaur, can you post some explanatory code?
I've thought of one non-embedding solution which to #4, which would be to write a function in my Bash script that writes a file with the entire tcl script. That way I would have a single script, but could dump the subscript for use with external programs, later deleting it. I have a feeling this solution is kinda kludgy though I know for sure that it works, vs. embedded solutions.

There have been several Tcl-Python alloys. As Rafe Kettler's comment above sketches, the place to start is with a standard Python installation. This includes Tkinter, which builds in a full Tcl interpreter, accessible as described in the Wiki page mentioned. So, yes, it is feasible to "do this in Python".
I really don't get what this has to do with vmd, though. vmd builds in a Tcl interpreter already. While I entirely support the aim of "reduction of moving parts", so that you have, for example, one script, rather than two, coding something in Python, when vmd already exposes Tcl, doesn't seem like a step in the direction Jason R. Mick wants to go.
SOMEWHAT LATER: after an exchange of comments with Jason R. Mick, it occurred to me he might find
#!/bin/bash
echo "Here's a bit of bash-iness."
MYSCRIPT='
puts "Here I am, inside Tcl."
puts "See? I can do calculations: [expr 3 + 5]."
exit 0
'
tclsh << HERE
$MYSCRIPT
HERE
suggestive. Its output, of course, is
Here's a bit of bash-iness.
Here I am, inside Tcl.
See? I can do calculations: 8.
I wrote this in terms of tclsh, but, if I'm keeping up, Jason R. Mick will actually want to use vmd. The appropriate homologue for *vmd is something like
...
vmd -dispdev text -eofexit << HERE > output.log
$MYSCRIPT
HERE
While I can think of several other ways to meld bash and Tcl, I believe this one is most in the spirit of the original question.
I want to note, too, that, from the little I know of vmd, it would be entirely appropriate to do the same with Python in place of Tcl: vmd is equally adept with either.
Finally, both Python and Tcl are general-purpose languages, with approximately the same power as bash, so yet another direction to take this project would be to write it entirely in Tcl (or Python), rather than bash. Embedding scripts in the way illustrated above is at least as easy in Tcl (or Python) as in bash.

1. Can you embed a TCL script inside a Bash script?
Not easily. The best way is to write the script to a temporary file and pass the name of that file to tclsh (or wish if it is a Tcl/Tk program). That should be a "simple matter of programming", i.e., some awkward coding but not fundamentally hard.
2. Can you make this script callable by an external program?
I don't quite understand what you want to do here. You can put a #! line at the start of a Tcl script and mark the file executable. That works well. The best way of all to do that is this:
#!/usr/bin/env tclsh8.5
your tcl script here...
3. If the answer to #2 is no, can you do this in Python?
This wiki page mentions something called Typcl, which is reported to allow doing Tcl from inside Python. I have never tried it.
(I think questions 4 and 5 are largely irrelevant based on my answers above.)