I'm designing a Unity3D game where the user can adjust gameplay by modifying a .py file.
I'm using Python Interpreter asset, which wraps IronPython.
I need to pass control between C# and the running Python instance. Here is a pseudocode example:
// From Unity, C# script...
bool MakeNoise( float vol ) {
//...
}
void Awake() {
m_pythonEngine = Python.CreateEngine();
m_scriptScope = m_pythonEngine.CreateScope();
}
bool NewRound( int n ) {
myPythonRuntime.InvokeFunction( "NewRoundPy", param: roundNumber = n );
return retVal;
}
And then, in the game while it is running, the user could enter in a textbox:
# Python
vol = 0.5
def NewRoundPy( roundNumber ):
theCSharpObject.MakeNoise( vol )
return True
Which would in turn invoke the MakeNoise function every time a new round starts.
Apologies for the clumsy hypothetical situation. The important thing is that I need to be able to transfer execution from one environment to the other, passsing data as I do so.
I'm aware of the code needed to run a Python script. But I want to invoke individual functions from the script. And the script has to stay running between such calls, so that variables set in one function/call are retaining their values the next time some function is called.
Is this possible? If so, how?
PS going Python -> C# seems to be fairly simple; IronPython .NET Integration documentation
PPS I will tidy up the question once I have some understanding of what's going on
Related
I want to execute the same Python function several times and I want each execution to be completely isolated from the previous one.
But the problem is that the first execution of the method changes a global variable and when I execute the method the second time the global variable has the value left by the first execution.
I want that from one call to another the environment is reset and is like the first time.
var1=1
def Met(n) :
global var1
if n!=1 : var1=n
print(f"Py: {var1}")
return var1
if __name__ == "__main__":
args = sys.argv
globals()[args[1]](*args[2:])
If I execute it from command line:
python .\Test1.py Met 2
python .\Test1.py Met 1
The output is (correct):
Py: 2
Py: 1
What I want, OK.
But if I run it from Pythonet I get:
Py: 2
Py: 2
Incorrect!
That is, the second call, the value of var1 is the value of first call.
The code (simplifying):
public string Met2(int n) {
dynamic txtResul="X";
using ( Py.GIL() ) {
using (var scope=gbl.NewScope() ) { //Witout this, it doesn't work too
dynamic f=Py.Import(NOM_FICH_Py);
txtResul=f.Met(n);
}
}
return txtResul.ToString();
}
I have tried initialising with and without this:
PythonEngine.Initialize();
gbl = Py.CreateScope(nombre);
pyPtr=PythonEngine.BeginAllowThreads();
I have also tried to finalize everything and run it, with and without this:
public void Fin() {
PythonEngine.EndAllowThreads(pyPtr);
PythonEngine.Shutdown();
}
I have also tried running each function call as a separate script:
using (Py.GIL()) {
dynamic sys = Py.Import("sys");
string script ="import sys\n"+ "from io import StringIO\n"+
$"from {NOM_FICH_Py} import {funcion.Split('(')[0]}\n"+
"sys.stdout=StringIO()\n"+ "sys.stdout.flush()\n"+
"sys.stderr=StringIO()\n"+ "sys.stderr.flush()\n"+
funcion;
var scope = Py.CreateScope();
var vars=scope.Variables();
scope.Exec(script, new PyDict());
salida=sys.stdout.getvalue();
scope.Dispose();
}
I use Python 3.10 and Python .Net 3.0.0-rc 4 and Python.Net 3.0.0-preview2022-03-03 (in other computer) and I cann't get the 'reset' as I executed it like command line.
Thanks in advance.
Extra question, is there any wiki/documentacion to read the detail of functions like CreateScope, NewScope, Exec...? In https://github.com/pythonnet/pythonnet/wiki I haven't found the function's detail.
The only supported way to fully reset Python environment with Python.NET is to restart the process.
At the end, the only posible solution it's to exectue like a script in cmd.
It's not the best solution (it's not posible to debug) but it's the only one I know.
public string ExecCmd(string funcion, string args) {
Process p = new Process();
p.StartInfo=new ProcessStartInfo(PYTHON_EXE, $"{NombreFichApp} {funcion} {args}") {
RedirectStandardOutput=true,
UseShellExecute=false,
CreateNoWindow=true
};
p.Start();
string salida = p.StandardOutput.ReadToEnd();
p.WaitForExit();
return salida;
}
I've written a large-ish program in Python, which I need to talk to a smaller C# script. (I realise that getting Python and C# to talk to each other is not an ideal state of affairs, but I'm forced to do this by a piece of hardware, which demands a C# script.) What I want to achieve in particular - the motivation behind this question - is that I want to know when a specific caught exception occurs in the C# script.
I've been trying to achieve the above by getting my Python program to look at the C# script's return code. The problem I've been having is that, if I tell C# to give a return code x, my OS will receive a return code y and Python will receive a return code z. While a given x always seems to correspond to a specific y and a specific z, I'm having difficulty deciphering the relationship between the three; they should be the same.
Here are the specifics of my setup:
My version of Python is Python 3.
My OS is Ubuntu 20.04.
I'm using Mono to compile and run my C# script.
And here's a minimal working example of the sort of thing I'm talking about:
This is a tiny C# script:
namespace ConsoleApplication1
{
class Script
{
const int ramanujansNumber = 1729;
bool Run()
{
return false;
}
static int Main(string[] args)
{
Script program = new Script();
if(program.Run()) return 0;
else return ramanujansNumber;
}
}
}
If I compile this using mcs Script.cs, run it using mono Script.exe and then run echo $?, it prints 193. If, on the other hand, I run this Python script:
import os
result = os.system("mono Script.exe")
print(result)
it prints 49408. What is the relationship between these three numbers: 1729, 193, 49408? Can I predict the return code that Python will receive if I know what the C# script will return?
Note: I've tried using Environment.Exit(code) in the C# script instead of having Main return an integer. I ran into exactly the same problem.
With os.system the documentation explicitly states that the result is in the same format as for the os.wait, i.e.:
a 16-bit number, whose low byte is the signal number that killed the process, and whose high byte is the exit status (if the signal number is zero); the high bit of the low byte is set if a core file was produced.
So in your case it looks like:
>>> 193<<8
49408
You might want to change that part to using subprocess, e.g. as in the answer to this question
UPD: As for the mono return code, it looks like only the lower byte of it is used (i.e. it is expected to be between 0 and 255). At least:
>>> 1729 & 255
193
I am currently trying to wrap a c-dll to controll a camera connected via USB. To grab image data off the camera, the SDK-Provider (Thorlabs) describes two possibilities:
Poll the camera in a loop
Use callbacks
I wrapped the DLL and got the polling method to work, which basically consists of calling a method which returns either NULL or the pending frame.
The DLL also provides methods to assign callback functions to certain events, such as camera connected/disconnected and frame available. The C-code from the example looks like this:
void camera_connect_callback(char* cameraSerialNumber, enum USB_PORT_TYPE usb_bus_speed, void* context)
{
printf("camera %s connected with bus speed = %d!\n", cameraSerialNumber, usb_bus_speed);
}
int main(void)
{
[...]
// Set the camera connect event callback. This is used to register for run time camera connect events.
if (tl_camera_set_camera_connect_callback(camera_connect_callback, 0))
{
printf("Failed to set camera connect callback!\n");
close_sdk_dll();
return 1;
}
[...]
}
My goal is now to get the callback working with python, so that python methods are called from the dll. Therefore, I used CFFI in API-mode to compile the wrapper, then tried to pass a callback method to the DLL in the same fashion:
#ffi.callback("void(char*, void*)")
def disconnect_callback(serialNumber, context):
print("Disconnect callback was called!")
[...]
print("Set disconnect callback: {}".format(lib.tl_camera_set_camera_disconnect_callback(disconnect_callback, ffi.NULL)))
[...]
For testing purposes I only included the simplest callback in the working polling example. The setter method returns 0 but the method is never executed.
Is this even the right way of accomplishing my goal? Do I need to introduce some kind of threading so that the callback can interrupt the sequential execution of the remaining program?
There is not much documentation or examples on this topic so Im really hoping you guys can help me out.
EDIT
So I tried a very basic example which works and technically answeres my question, but does not solve my problem.
if I define a C callback method as follows:
#include <math.h>
int quadcalc(int input, int (*getval)(int)){
return (int) pow(getval(input), 2);
}
I can assign a handler as expected: (After compiling of course)
#ffi.callback("int(int)")
def callback_func(value):
return int(rnd.random() * value)
if __name__ == '__main__':
print(lib.quadcalc(10, callback_func))
And everything works as expected. Unfortunately, this doesn't work on the actual Problem. So callback works in general, but not in the specific case, which is why the question is still open.
Do I need to introduce some kind of threading so that the callback can interrupt the sequential execution of the remaining program?
Invoking the Callback method takes care of setting up alternate program flows to capture the event, so it can be routed to a handler to be processed. When the defined event occurs, it does need handler code. For example, here is a very generic, but complete example of code that uses a callback in C (including the handler function.):
void populate_array(int *array, size_t arraySize, int (*getNextValue)(void))
{
for (size_t i=0; i<arraySize; i++)
array[i] = getNextValue();
}
// handler function
int getNextRandomValue(void)
{
return rand();
}
int main(void)
{
int myarray[10];
populate_array(myarray, 10, getNextRandomValue);
...
}
There is an equally good example just below the first one, here.
Beyond these, there is a good tutorial in the CFFI documentation, with the first section dedicated to addressing your title question How to make a C-DLL wrapped with CFFI to callback python:
The first section presents a simple working example of using CFFI to call a C function in a compiled shared object (DLL) from Python.
Steps are: (see link above for details of each step.)
- Create the file piapprox_build.py:
- Execute this script:
- At runtime, you use the extension module like this:
...In the rest of this page, we describe some more
advanced examples and other CFFI modes...
I have successfully created a Python module that appears to work in isolation, but doesn't affect the program that is running it.
I have the following module:
BOOST_PYTHON_MODULE(mandala)
{
class_<state_mgr_t, state_mgr_t*, noncopyable>("states", no_init)
.def("push", &state_mgr_t::push)
.def("pop", &state_mgr_t::pop)
.def("count", &state_mgr_t::count);
scope().attr("states") = boost::python::object(boost::python::ptr(&states));
}
The states object is referencing a global value, states:
extern state_mgr_t states;
I can run the following script lines from within my program:
from mandala import states
states.count()
> 0
All of that is fine and whatnot, but I was expecting that running this python script would affect the actual state of the program that is running it. It appears as though Python is actually just dealing with it's own instance of states and not affecting the parent program.
Now I'm wondering if I've completely misunderstood what Boost.Python is capable of; I was expecting something similar to Lua, where I could modify the C++ program via scripts.
Is this not possible? Or am I doing something very wrong?
Thank you in advance!
If you are embedding Python into your C++ program, it should be possible to access the instance from your script. Obviously, I don't have your full code, but have you tried something like this?
PyImport_AppendInittab("mandala", &initmandala);
Py_Initialize();
try {
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
main_namespace.attr("states") = ptr(&states);
object ignored = exec("print states.count()", main_namespace);
} catch(const error_already_set&) {
PyErr_Print();
}
I'm intending to create a web service which performs a large number of manually-specified calculations as fast as possible, and have been exploring the use of DLR.
Sorry if this is long but feel free to skim over and get the general gist.
I've been using the IronPython library as it makes the calculations very easy to specify. My works laptop gives a performance of about 400,000 calculations per second doing the following:
ScriptEngine py = Python.CreateEngine();
ScriptScope pys = py.CreateScope();
ScriptSource src = py.CreateScriptSourceFromString(#"
def result():
res = [None]*1000000
for i in range(0, 1000000):
res[i] = b.GetValue() + 1
return res
result()
");
CompiledCode compiled = src.Compile();
pys.SetVariable("b", new DynamicValue());
long start = DateTime.Now.Ticks;
var res = compiled.Execute(pys);
long end = DateTime.Now.Ticks;
Console.WriteLine("...Finished. Sample data:");
for (int i = 0; i < 10; i++)
{
Console.WriteLine(res[i]);
}
Console.WriteLine("Took " + (end - start) / 10000 + "ms to run 1000000 times.");
Where DynamicValue is a class that returns random numbers from a pre-built array (seeded and built at run time).
When I create a DLR class to do the same thing, I get much higher performance (~10,000,000 calculations per second). The class is as follows:
class DynamicCalc : IDynamicMetaObjectProvider
{
DynamicMetaObject IDynamicMetaObjectProvider.GetMetaObject(Expression parameter)
{
return new DynamicCalcMetaObject(parameter, this);
}
private class DynamicCalcMetaObject : DynamicMetaObject
{
internal DynamicCalcMetaObject(Expression parameter, DynamicCalc value) : base(parameter, BindingRestrictions.Empty, value) { }
public override DynamicMetaObject BindInvokeMember(InvokeMemberBinder binder, DynamicMetaObject[] args)
{
Expression Add = Expression.Convert(Expression.Add(args[0].Expression, args[1].Expression), typeof(System.Object));
DynamicMetaObject methodInfo = new DynamicMetaObject(Expression.Block(Add), BindingRestrictions.GetTypeRestriction(Expression, LimitType));
return methodInfo;
}
}
}
and is called/tested in the same way by doing the following:
dynamic obj = new DynamicCalc();
long t1 = DateTime.Now.Ticks;
for (int i = 0; i < 10000000; i++)
{
results[i] = obj.Add(ar1[i], ar2[i]);
}
long t2 = DateTime.Now.Ticks;
Where ar1 and ar2 are pre-built, runtime seeded arrays of random numbers.
The speed is great this way, but it's not easy to specify the calculation. I'd basically be looking at creating my own lexer & parser, whereas IronPython has everything I need already there.
I'd have thought I could get much better performance from IronPython since it is implemented on top of the DLR, and I could do with better than what I'm getting.
Is my example making best use of the IronPython engine? Is it possible to get significantly better performance out of it?
(Edit) Same as first example but with the loop in C#, setting variables and calling the python function:
ScriptSource src = py.CreateScriptSourceFromString(#"b + 1");
CompiledCode compiled = src.Compile();
double[] res = new double[1000000];
for(int i=0; i<1000000; i++)
{
pys.SetVariable("b", args1[i]);
res[i] = compiled.Execute(pys);
}
where pys is a ScriptScope from py, and args1 is a pre-built array of random doubles. This example executes slower than running the loop in the Python code and passing in the entire arrays.
delnan's comment leads you to some of the problems here. But I'll just get specific about what the differences are here. In the C# version you've cut out a significant amount of the dynamic calls that you have in the Python version. For starters your loop is typed to int and it sounds like ar1 and ar2 are strongly typed arrays. So in the C# version the only dynamic operations you have are the call to obj.Add (which is 1 operation in C#) and potentially the assignment to results if it's not typed to object which seems unlikely. Also note all of this code is lock free.
In the Python version you first have the allocation of the list - this also appears to be during your timer where as in C# it doesn't look like it is. Then you have the dynamic call to range, luckily that only happens once. But that again creates a gigantic list in memory - delnan's suggestion of xrange is an improvement here. Then you have the loop counter i which is getting boxed to an object for every iteration through the loop. Then you have the call to b.GetValue() which is actually 2 dynamic invocatiosn - first a get member to get the "GetValue" method and then an invoke on that bound method object. This is again creating one new object for every iteration of the loop. Then you have the result of b.GetValue() which may be yet another value that's boxed on every iteration. Then you add 1 to that result and you have another boxing operation on every iteration. Finally you store this into your list which is yet another dynamic operation - I think this final operation needs to lock to ensure the list remains consistent (again, delnan's suggestion of using a list comprehension improves this).
So in summary during the loop we have:
C# IronPython
Dynamic Operations 1 4
Allocations 1 4
Locks Acquired 0 1
So basically Python's dynamic behavior does come at a cost vs C#. If you want the best of both worlds you can try and balance what you do in C# vs what you do in Python. For example you could write the loop in C# and have it call a delegate which is a Python function (you can do scope.GetVariable> to get a function out of the scope as a delegate). You could also consider allocating a .NET array for the results if you really need to get every last bit of performance as it may reduce working set and GC copying by not keeping around a bunch of boxed values.
To do the delegate you could have the user write:
def computeValue(value):
return value + 1
Then in the C# code you'd do:
CompiledCode compiled = src.Compile();
compiled.Execute(pys);
var computer = pys.GetVariable<Func<object,object>>("computeValue");
Now you can do:
for (int i = 0; i < 10000000; i++)
{
results[i] = computer(i);
}
If you concerned about computation speed, is it better to look at lowlevel computation specification? Python and C# are high-level languages, and its implementation runtime can spend a lot of time for undercover work.
Look on this LLVM wrapper library: http://www.llvmpy.org
Install it using: pip install llvmpy ply
or on Debian Linux: apt install python-llvmpy python-ply
You still need to write some tiny compiler (you can use PLY library), and bind it with LLVM JIT calls (see LLVM Execution Engine), but this approach can be more effective (generated code much closer to real CPU code), and multiplatform comparing to .NET jail.
LLVM has ready to use optimizing compiler infrastructure, including a lot of optimizer stage modules, and big user and developer community.
Also look here: http://gmarkall.github.io/tutorials/llvm-cauldron-2016
PS: If you interested in it, I can help you with a compiler, contributing to my project's manual in parallel. But it will not be jumpstart, this theme is new to me too.