We Keep Coding

Embed Python in Python?

I wrote a "compiler" PypTeX that converts an input file a.tex containing Hello #{3+4} to an ouput file a.pyptex containing Hello 7. I evaluate arbitrary Python fragments like #{3+4} using something like eval(compile('3+4','a.tex',mode='eval'),myglobals), where myglobals is some (initially empty) dict. This creates a thin illusion of an embedded interpreter for running code in a.tex, however the call stack when running '3+4' looks pretty weird, because it backs up all the way into the PypTeX interpreter, instead of topping out at the user code '3+4' in a.tex.
Is there a way of doing something like eval but chopping off the top of the call stack?
Motivation: debugging
Imagine an exception is raised by the Python fragment deep inside numpy, and pdb is launched. The user types up until they reach the scope of their user code and then they type list. The way I've done it, this displays the a.tex file, which is the right context to be showing to the user and is the reason why I've done it this way. However, if the user types up again, the user ends up in the bowels of the PypTeX compiler.
An analogy would be if the g++ compiler had an error deep in a template, displayed a template "call stack" in its error message, but that template call stack backed all the way out into the bowels of the actual g++ call stack and exposed internal g++ details that would only serve to confuse the user.
Embedding Python in Python
Maybe the problem is that the illusion of the "embedded interpreter" created by eval is slightly too thin. eval allows to specify globals, but it inherits whatever call stack the caller has, so if one could somehow supply eval with a truncated call stack, that would resolve my problem. Alternatively, if pdb could be told "you shall go no further up" past a certain stack frame, that would help too. For example, if I could chop off a part of the stack in the traceback object and then pass it to pdb.post_mortem().
Or if one could do from sys import Interpreter; foo = Interpreter(); foo.eval(...), meaning that foo is a clean embedded interpreter with a distinct call stack, global variables, etc..., that would also be good.
Is there a way of doing this?
A rejected alternative
One way that is not good is to extract all Python fragments from a.tex by regular expression, dump them into a temporary file a.py and then run them by invoking a fresh new Python interpreter at the command line. This causes pdb to eventually top out into a.py. I've tried this and it's a very bad user experience. a.py should be an implementation detail; it is automatically generated and will look very unfamiliar to the user. It is hard for the user to figure out what bits of a.py came from what bits of a.tex. For large documents, I found this to be much too hard to use. See also pythontex.

I think I found a sufficient solution:
import pdb, traceback
def exec_and_catch(cmd,globals):
try:
exec(cmd,globals) # execute a user program
except Exception as e:
tb = e.__traceback__.tb_next # if user program raises an exception, remove the
f = e.with_traceback(tb) # top stack frame and return the exception.
return f
return None # otherwise, return None signifying success.
foo = exec_and_catch("import module_that_does_not_exist",{})
if foo is not None:
traceback.print_exception(value=foo, tb=foo.__traceback__, etype=type(foo))
pdb.post_mortem(foo.__traceback__)

For what uses do we need `sys` module in python?

I'm a bit experienced without other languages but, novice with Python. I have come across made codes in jupyter notebooks where sys is imported.
I can't see the further use of the sys module in the code. Can someone help me to understand what is the purpose of importing sys?
I do know about the module and it's uses though but can't find a concise reason of why is it used in many code blocks without any further use.

If nothing declared within sys is actually used, then there's no benefit to importing it. There's not a significant amount of cost either.

Sys module is a rather useful module as it allows you to work with your System and those things. Eg:
You can access any command line arguments using sys.argv[1:]
You can see the Path to files.
Version of your Python Interpreter using sys.version
Exit the running code with sys.exit
Mostly you will use it for accessing the Command Line arguments.

I'm a new pythonista bro, I learned to import it whenever I want to exit the program with a nice exit text in red
import sys
name = input("What's your name? ")
if name == "Vedant":
print(f"Hello There {name}.")
else:
sys.exit(f"You're not {name}!")

The sys includes "functions + variable " to help you control and change the python environment #runtime.
Some examples of this control includes:
1- using other sources data as input via using:
sys.stdin
2- using data in the other resources via using:
sys.stdout
3- writing errors when an exception happens, automatically in :
sys.stderr
4- exit from the program by printing a message like:
sys.exit("Finish with the calculations.")
5- The built-in variable to list the directories which the interpreter will looking for functions in them:
sys.pasth
6- Use a function to realize the number of bytes in anonymous datatype via:
sys.getsizeof(1)
sys.getsizeof(3.8)

Python - import local variables of a function

I'm trying to make a debugger function, which is called when an error is raised, and let me access a console so I can check what happened in my program.
Here is the basic function:
def DEBUGGER(error):
print(error)
print("[DEBUGGER] Your program has failed, here is the debugger. Enter EXIT to end program.")
while True:
line = input(">>> ").lower()
if line == 'exit':
sys.exit(0)
else:
try:
exec(line)
except Exception as e:
print(str(e))
The problem is that I can't enter something like print(var) because it's referenced in another function.
Globals functions don't help me since I want to be able to call any variable in my program, and I can't globalize them all.. I know I can resolve it by putting all my functions in classes but I can't for many reasons.
Is there a way to get local variables of the running functions ? (When I call DEBUGGER(), the mother function is still running)
If no, can I export the local variables of the current function and pass it as an argument to DEBUGGER() ?
Thanks for your answers.

You are basically re-implementing the Python debugger pdb. If you want to go this route, you probably want to study the source code. pdb itself is a user-interface around the lower-level bdb (basic debugger) module, and the source code for that is also available.
To answer your direct question: when you catch an exception you have access to a traceback object (either via exception.__traceback__ or via sys.exc_info()), and tracebacks have access to both the local and global namespace of each frame in the stack, via the tb_frame attribute. That attribute is set to a frame object, which has f_locals and f_globals attributes.
The bdb.Bdb.get_stack() method could be an interesting example on how to treat a traceback, and the internal pdb.Pdb._select_frame() method then is used to pick a frame from the stack to use the locals and globals from.
If you don't want to re-implement the full debugger, you can use the pdb.pm() or pdb.port_mortem() functions. These take the last traceback raised and let you inspect the stack frame in an interactive environment:
try:
exec(line)
except Exception as e:
pdb.post_mortem(e.__traceback__)

The correct way to "write" your "DEBUGGER" function is:
import pdb
DEBUGGER = pdb.set_trace
Now you can call DEBUGGER() wherever you want, you will be in an interactive environment with access not only to local vars but also to whole call stack, and the ability to execute the remaining code step by step (including stepping into other functions etc), change the control flow to continue executing from another line etc etc etc.
Oh and yes: you can of course just write import pdb; pdb.set_trace() instead ;-)

Python - peewee - Debugging statements - where are the errors logged

I have just started to use peewee in python. But as i am saving table data using .save() function. There is an error in the line. and control does not go to the next line.
Just wanted to know how can know what the error is. Although i have narrowed down to the line as below
try:
with database.transaction():
driver = Driver()
driver.person = person
driver.qualification = form.getvalue('qualification')
driver.number = form.getvalue('phone')
driver.license = form.getvalue('issu')
driver.audited_by = 0
print "this line prints"
driver.save()
print "this one does not print"
print "Success"
except:
print "Error"
I have used print statements i was able to figure out the error in in the line driver.save(). But how to check what exactly is the error?

Peewee logs queries at the DEBUG level to the peewee namespace, so you just have to configure logging as desired. Per the docs:
import logging
logger = logging.getLogger('peewee')
logger.addHandler(logging.StreamHandler())
logger.setLevel(logging.DEBUG)

This is specified in the peewee documentation here.

In the future, you should also include the traceback when you're asking for help debugging an error. The traceback tells you, as best as it can, exactly what went wrong.
If you want to do some debugging, you can check out pdb (or ipdb if you use iPython):
https://docs.python.org/2/library/pdb.html

In Django/Python, how can I find out which method called my method?

I would the first line of my method to be:
print "this method was called from "+filename_and_linenumber_of_code_that_called_it.
Is it possible to throw an exception, catch it immediately, and print a stack trace when a method is called?

When i just want to make the code crash at some point to see the traceback, i just put "crash" in the code. Because it's not defined, it will crash, and i will see the traceback in django's exception page. If in addition, i use runserver_plus command provided by django-extensions (requires package werkzeug) then i get an AJAX shell at each frame of the stacktrace.
I understand you problem and I'm going to propose a professional method for dealing with this kind of problem. What you are trying to do is called "debugging" and there are tools for that.
Quickstart:
run pip install ipython ipdb
replace the print statement in your code by import ipdb; ipdb.set_trace()
execute your code in runserver, it will pause and spawn a python shell where you can send command "up" to go to the previous stack frame (code that called your code). Type l if you want to see more lines of code.
Longer start: well actually i wrote a an overview of tools which help debugging python and django.
I disagree with other answers which propose to add more elaborate print statement. You want to be a good developer: you want to use a debugger. Be it werkzeug, pdb/ipdb, or GUIs, it doesn't matter as long as you can use it.

No need to throw an exception to view the stack. I have this nice function (it's not perfect, but I think it works) that may help you:
import inspect
def log(error):
frame, filename, ln, fn, lolc, idx = inspect.getouterframes(inspect.currentframe())[1]
print "Error: " + error + " " + filename, ln, fn
It prints the message followed by the name of the file that the parent function is in, then the line number of the call in this file, and then the name of the function. I hope it'll help you :)

This is CPython specific:
import sys
def func():
frm = sys._getframe(1)
print 'called from %s, line %s' % (frm.f_code.co_filename, frm.f_lineno)
def test():
func() # line 8
test()
Prints:
called from /path/to/script.py, line 8

A debugger like pdb can be helpful. Refer below snippet.
def f4():
print "in f4"
def f3():
import pdb
pdb.set_trace()
f4()
print "in f3"
def f2():
f3()
print "in f2"
def f1():
f2()
print "in f1"
f1()
Once entered in pdb console, the up command can be entered to jump to the caller function.
Refer below screenshot.