We Keep Coding

Interpret Python bytecode in C# (with fine control)

For a project idea of mine, I have the following need, which is quite precise:
I would like to be able to execute Python code (pre-compiled before hand if necessary) on a per-bytecode-instruction basis. I also need to access what's inside the Python VM (frame stack, data stacks, etc.). Ideally, I would also like to remove a lot of Python built-in features and reimplement a few of them my own way (such as file writing).
All of this must be coded in C# (I'm using Unity).
I'm okay with loosing a few of Python's actual features, especially concerning complicated stuff with imports, etc. However, I would like most of it to stay intact.
I looked a little bit into IronPython's code but it remains very obscure to me and it seems quite enormous too. I began translating Byterun (a Python bytecode interpreter written in Python) but I face a lot of difficulties as Byterun leverages a lot of Python's features to... interpret Python.
Today, I don't ask for a pre-made solution (except if you have one in mind?), but rather for some advice, places to look at, etc. Do you have any ideas about the things I should research first?

I've tried to do my own implementation of the Python VM in the distant past and learned a lot but never came even close to a fully working implementation. I used the C implementation as a starting point, specifically everything in https://github.com/python/cpython/tree/main/Objects and
https://github.com/python/cpython/blob/main/Python/ceval.c (look for switch(opcode))
Here are some pointers:
Come to grips with the Python object model. Implement an abstract PyObject class with the necessary methods for instancing, attribute access, indexing and slicing, calling, comparisons, aritmetic operations and representation. Provide concrete implemetations for None, booleans, ints, floats, strings, tuples, lists and dictionaries.
Implement the core of your VM: a Frame object that loops over the opcodes and dispatches, using a giant switch statment (following the C implementation here), to the corresponding methods of the PyObject. The frame should maintains a stack of PyObjects for the operants of the opcodes. Depending on the opcode, arguments are popped from and pushed on this stack. A dict can be used to store and retrieve local variables. Use the Frame object to create a PyObject for function objects.
Get familiar with the idea of a namespace and the way Python builds on the concept of namespaces. Implement a module, a class and an instance object, using the dict to map (attribute)names to objects.
Finally, add as many builtin functions as you think you need to get a usefull implementation.
I think it is easy to underestimate the amount of work you're getting yourself into, but ... have fun!

Implementing only part of a Python class in C

I have a python class that has a couple performance-sensitive methods that justify being implemented in C. But it also has some methods that don't need to be fast and that would be a giant pain to write in C.
Is there a standard way to have the best of both worlds, where a few core methods are defined in C but some convenience methods are defined in python?
(Ideally it should work for special methods like __str__.)
For example, maybe I could use inheritance. Is that the right way to do it? Are there performance costs?

Try Cython. It really does a fantastic job blending the best features of both languages. No longer do you have to decide between control and performance, and efficiency and ease of development.

If the C code doesn't need to interact with the object itself, possibly you could use the ctypes module to call C functions from your python code.
Put your C code in into a shared library or DLL and then call it from your method.

Python lambda in C++

I have a Python program, in which there's a function that should be able to accept a Python lambda, and then pass to a running C++ program.
I was thinking of using pickle.dumps to serialize the lambda into a string. Then I just pass the string to the C++ program, after which I do deserialization using boost.python and get the lambda in C++. However, I just realized that functions cannot be pickled.
I believe there should be existing solutions that I'm not aware of. Thanks in advance for any advice!

Have a look at the C API tutorial page. The link goes to the section about calling Python functions from C. It sounds like you're new to the C API, so start at the top.
Python does not distinguish between the various types of callable objects. Functions, lambdas, objects that have a __call__ method are interchangeable as far as the C API is concerned.
I won't copy/paste everything here as it's very well written in the link, but basically you make a Python module in C (or C++, the API works in both). The first example shows you how. Then you can call a C function from Python, and pass it whatever you need to pass it. All Python objects are exposed to C via PyObject* pointers, and the C API provides many functions to manipulate the objects, convert to/from C datatypes, and do things like call callable objects.
Wrappers like boost.python and SWIG use this API internally.

Python C API: PyEval_CallFunction?

I've discovered a function in the Python C API named PyEval_CallFunction which seems to be useful. It allows you to invoke a Python callable by saying something like:
PyEval_CallFunction(obj, "OOO", a, b, c);
However, I can't find any official documentation on this function. A google search brings up various unofficial tutorials which discuss this function, but:
The function isn't
documented in the official
Python docs, so I don't know if it's
even something that is supposed to
be part of the public API.
Searching the web turns up
inconsistent usage policies. Some
tutorials indicate the
format string needs parenthesis
around the type list, like
"(OiiO)", whereas other times I
see it used without the parenthesis.
When I actually try the function in
a real program, it seems to require
the parenthesis, otherwise it
segfaults.
I'd like to use this function because it's convenient. Does anyone know anything about this, or know why it isn't documented? Is it part of the public API?

I couldn't find many references to it either, but the tutorial you linked to mentions this:
The string format and the following
arguments are as for Py_BuildValue
(XXX so i really should have described
that by now!). A call such as
PyEval_CallFunction(obj, "iii", a, b, c);
is equivalent to
PyEval_CallObject(obj, Py_BuildValue("iii", a, b, c));
I suppose PyEval_CallFunction is not public API, as its value seems rather limited. There is not much of a difference between these two. But then again, I'm not really involved in python extensions, so this is just my view on this.
PyEval_CallObject itself is just a macro around PyEval_CallObjectWithKeywords.
#define PyEval_CallObject(func,arg) \
PyEval_CallObjectWithKeywords(func, arg, (PyObject *)NULL)
On the matter of "What is public API?" here is a recent message from Martin v. Löwis:
Just to stress and support Georg's
explanation: the API is not defined
through the documentation, but instead
primarily through the header files.
All functions declared as PyAPI_FUNC
and not starting with _Py are public
API. There used to be a lot of undocumented API (up to 1.4, there was no API documentation at all, only the extension module tutorial); these days, more and more API gets documented.
http://mail.python.org/pipermail/python-dev/2011-February/107973.html

The reason it isn't documented is because you should be using PyObject_CallFunction instead.
The PyEval_* function family are the raw internal calls for the interpreter evaluation loop. The corresponding documented PyObject_* calls include all the additional interpreter state integrity checks, argument validation and stack protection.

Passing C function pointers between two python modules

I'm writing an application working with plugins. There are two types of plugins: Engine and Model. Engine objects have an update() method that call the Model.velocity() method.
For performance reasons these methods are allowed to be written in C. This means that sometimes they will be written in Python and sometimes written in C.
The problem is that this forces to do an expensive Python function call of Model.velocity() in Engine.update() (and also reacquiring the GIL). I thought about adding something like Model.get_velocity_c_func() to the API, that would allow Model implementations to return a pointer to the C version of their velocity() method if available, making possible for Engine to do a faster C function call.
What data type should I use to pass the function pointer ? And is this a good design at all, maybe there is an easier way ?

The CObject (PyCOBject) data type exists for this purpose. It holds a void*, but you can store any data you wish. You do have to be careful not to pass the wrong CObject to the wrong functions, as some other library's CObjects will look just like your own.
If you want more type security, you could easily roll your own PyType for this; all it has to do, after all, is contain a pointer of the right type.