I've been working on a project where I'm trying to use an old CodeBase library, written in C++, in Python. What I want is to use CodeBase to reindex a .dbf-file that has a .cdx-index. But currently, Python is crashing during runtime. A more detailed explanation will follow further down.
As for Python, I'm using ctypes to load the dll and then execute a function I added myself which should cause no problems, since it doesn't use a single line of code that CodeBase itself isn't using.
Python Code:
import ctypes
cb_interface = ctypes.CDLL("C4DLL.DLL")
cb_interface.reindex_file("C:\\temp\\list.dbf")
Here's the CodeBase function I added, but it requires some amount of knowledge that I can't provide right now, without blowing this question up quite a bit. If neccessary, I will provide further insight, as much as I can:
S4EXPORT int reindex_file(const char* file){
CODE4 S4PTR *code;
DATA4 S4PTR *data;
code4initLow(&code, 0, S4VERSION, sizeof(CODE4));
data = d4open(code, file);
d4reindex(data);
return 1;
}
According to my own debugging, my problem happens in code4initLow. Python crashes with a window saying "python.exe has stopped working", when the dll reaches the following line of code:
memset( (void *)c4, 0, sizeof( CODE4 ) ) ;
c4 here is the same object as code in the previous code-block.
Is there some problem with a dll trying to alter memory during runtime? Could it be a python problem that would go away if I were to create a .exe-file from my python script?
If someone could answer me these questions and/or provide a solution for my python-crashing-problem, I would greatly appreciate it.
And last but not least, this is my first question here. If I have accidently managed to violate a written or unwritten rule here, I apologize and promise to fix that as soon as possible.
First of all the pointer code doesn't point anywhere as it is uninitialized. Secondly you don't actually try to fill the structure, since you pass memset a pointer to the pointer.
What you should probably do is declare code as a normal structure instance (and not a pointer), and then use &code when passing it to d4open.
Like Joachim Pileborg saying the problem is, to pass a Nullpointer to code4initLow. ("Alternative") Solution is to allocate Memory for the struct CODE4 S4PTR *code = (CODE4*)malloc(sizeof(CODE4));, then pass the Pointer like code4initLow(code, 0, S4VERSION, sizeof(CODE4));.
Related
I have this part of a C# DLL from a larger project. It has no dependencies, runs in either .NET 5.0 or .NET 6.0 C# without anything else in it. It looks something like this:
namespace Some.Namespace {
public static class Rename{
public static string RenameString(string stringToRename){
//Do some stuff to it
return stringToRename;
}
}
}
However, no matter what way we attempt to add the reference to Python 3.6 (ArcGIS Pro Distribution) using Python.NET...
sys.path.append(os.path.join(os.getcwd(),"Project_Folder","bin"))
clr.AddReference("SomeDLL")
from Some.Namespace import Rename
It throws this error on the 3rd line.
Exception has occurred: ModuleNotFoundError No module named Some
We've tried just about every possible means to load the DLL at this point and none of them have worked (Yes I know that 2.5.x Python.NET doesn't support .NET 6.0 - we switched to 5.0 and it didn't work either) Exposing the function through Robert's DllExport using ctypes throws a CLR Exception that we can't debug because it's cross-environment whenever the function is run.
Here's that attempt.
//The DLL With Robert's DLLExport
namespace Some.Namespace {
public static class Rename{
//We've also tried CallingConvention.Cdecl
[DllExport("Rename", System.Runtime.InteropServices.CallingConvention.StdCall)]
public static string RenameString(string stringToRename){
//Do some stuff to it
return stringToRename;
}
}
}
#In Python
#We've tried WINDLL
dll_utils = CDLL(os.path.join(os.getcwd(),"project_folder","bin","SomeDLL.dll"))
#We've tried using CFUNCTTYPE here too
encrypt_proto = ctypes.WINFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)
encrypt_args = (1, "p1", 0),(1, "p2", 0),
#It fails here when we add the encrypt_args flags to it, saying they don't match up,
#but there's no way to figure out how many it wants, we've tried many variations of
#the above
encrypt_call = encrypt_proto(("Rename",dll_utils),encrypt_args)'
#So because of that on one attempt we removed these two lines
p2 = ctypes.c_char_p(text_to_obfuscate)
p1 = ctypes.c_char_p(text_to_obfuscate)
#And the second argument here
return_value = encrypt_call(p1,p2)
#And...
OSError: [WinError -532462766] Windows Error 0xe0434352
Attempting to write a second function in the DLL to convert the C# string to a byte array and back and using a ctypes pointer doesn't work - it throws the same error just posted when it's called. Someone suggested in another question here IronPython next - which we want to avoid, or to try to use comtypes which I've never used before. At one point we even tried to do a COM decoration and it claimed something I've seen on an answer here is obsolete. I've researched this for 2-3 days and haven't been able to solve this.
At this point I'm at a loss. Is there any effective, simple way, to get an external .NET C# function in a DLL to call a function and pass a string in Python without installing any major dependencies? Just out of the box? There's gotta be something simple I'm missing.
I'm testing a very simple NASM dll (64-bit) called from ctypes. I pass a single int_64 and the function is expected to return a different int_64.
I get the same error every time:
OSError: exception: access violation writing 0x000000000000092A
where hex value translates to the value I am returning (in this case, 2346). If I change that value, the hex value changes to that value, so the problem is in the value I am returning in rax. I get the same error if I assign mov rax,2346.
I have tested this repeatedly, trying different things, and I've done a lot of research, but this seemingly simple problem is still not solved.
Here is the Python code:
def lcm_ctypes():
input_value = ctypes.c_int64(235)
hDLL = ctypes.WinDLL(r"C:/NASM_Test_Projects/While_Loop_01/While_loops-01.dll")
CallTest = hDLL.lcm
CallTest.argtypes = [ctypes.c_int64]
CallTest.restype = ctypes.c_int64
retvar = CallTest (input_value)
Here is the NASM code:
[BITS 64]
export lcm
section .data
return_val: dq 2346
section .text
finit
lcm:
push rdi
push rbp
mov rax,qword[return_val]
pop rbp
pop rdi
Thanks for any information to help solve this problem.
Your function correctly loads 2346 (0x92a) into RAX. Then execution continues into some following bytes because you didn't jmp or ret.
In this case, we can deduce that the following bytes are probably 00 00, which decodes as add byte [rax], al, hence the access violation writing 0x000000000000092A error message. (i.e. it's not a coincidence that the address it's complaining about is your constant).
As Michael Petch said, using a debugger would have found the problem.
You also don't need to save/restore rdi because you're not touching it. The Windows x86-64 calling convention has many call-clobbered registers, so for a least-common-multiple function you shouldn't need to save/restore anything, just use rax, rcx, rdx, r8, r9, and whatever else Windows lets you clobber (I forget, check the calling convention docs in the x86 tag wiki, especially Agner Fog's guide).
You should definitely use default rel at the top of your file, so the [return_val] load will use a RIP-relative addressing mode instead of absolute.
Also, finit never executes because it's before your function label. But you don't need it either. You had the same finit in your previous asm question: Passing arrays to NASM DLL, pointer value gets reset to zero, where it was also not needed and not executed. The calling convention requires that on function entry (and return), the x87 FPU is already in the state that finit puts it in, more or less. So you don't need it before executing x87 instructions like fmulp and fidivr. But you weren't doing that anyway, you were using SSE FP instructions (which is recommended, especially in 64-bit mode), which don't touch the x87 state at all.
Go read a good tutorial and some docs (some links in the x86 tag wiki) so you understand what's going on well enough to debug a problem like this on your own, or you will have a bad time writing anything more complicated. Guessing what might work doesn't work very well for asm.
From a deleted non-answer: https://www.cs.uaf.edu/2017/fall/cs301/reference/nasm_vs/ shows how to set up Visual Studio to build an executable out of C++ and NASM sources, so you can debug it.
I am wrapping a C library into Python via ctypes. At the moment I stuck on one line or more on one parameter. Here the C Code:
void* gVimbaHandleFake = (void*)1;
err = VmbFeatureBoolGet(gVimbaHandleFake, "GeVTLIsPresent", &isGigE );
The problem is this strange void pointer. In general I know what a void pointer is but this one seems to be "special". If I change the 1 in (void*)1 the program is not working anymore (it is about finding network cameras). It is not crashing but doesnt find the cameras anymore.
I tried many different things, the last tries in Python:
gVimbaHandle = cast(1, c_void_p)
err = self.dll.VmbFeatureBoolGet(byref(gVimbaHandle), "GeVTLIsPresent", byref(isGigE))
also tried the "normal" way:
gVimbaHandle = c_void_p(1)
My program isnt crashing but it tells me that the handle is invalid ...
When I looked into the pointer with gVimbaHandle.value I get 1L as output. Could this be the problem, the L for the long datatype?
Does anybody knows how to fix this or can explain the "special" (void*)1 pointer in C to me?
Thank you very much!
So the solution/answer is:
gVimbaHandle = c_void_p(1)
err = self.dll.VmbFeatureBoolGet(gVimbaHandle, "GeVTLIsPresent", byref(isGigE))
thanks #eryksun
My python code has been crashing with error 'GC Object already Tracked' . Trying to figure out the best approach to debug this crashes.
OS : Linux.
Is there a proper way to debug this issue.
There were couple of suggestions in the following article.
Python memory debugging with GDB
Not sure which approach worked for the author.
Is there a way to generate memory dumps in such scenario which could be analyzed. Like in Windows world.
Found some article on this. But not entirely answers my question:
http://pfigue.github.io/blog/2012/12/28/where-is-my-core-dump-archlinux/
Found out the reason for this issue in my scenario (not necessarily the only reason for the GC object crash).
I used the GDB and Core dumps to debug this issue.
I have Python and C Extension Code (in shared object).
Python code registers a Callback routine with C Extension code.
In a certain workflow a thread from C Extension code was calling the registered Call back routine in Python code.
This usually worked fine but when multiple threads did the same action concurrently it resulted in the Crash with 'GC Object already tracked'.
Synchronizing the access to python objects for multiple thread does resolve this issue.
Thanks to any responded to this.
I ran into this problem using boost::python when our C++ code would trigger a python callback. I would occasionally get "GC object already tracked" and the program would terminate.
I was able to attach GDB to the process prior to triggering the error. One interesting thing, in the python code we were wrapping the callback with
a functools partial, which was actually masking where the real error was occuring. After replacing the partial with a simple callable wrapper class. The "GC object already tracked error" no longer popped up, instead I was now just getting a segfault.
In our boost::python wrapper, we had lambda functions to handle a C++ callback and the lambda function captured the boost::python::object callback function. It turned out, for whatever reason, in the destructor for the lambda, it wasn't always properly acquiring the GIL when destroying the boost::python::object which was causing the segfault.
The fix was to not use a lambda function, but instead create a functor that makes sure to acquire the GIL in the destructor prior to calling PyDECREF() on the boost::python::object.
class callback_wrapper
{
public:
callback_wrapper(object cb): _cb(cb), _destroyed(false) {
}
callback_wrapper(const callback_wrapper& other) {
_destroyed = other._destroyed;
Py_INCREF(other._cb.ptr());
_cb = other._cb;
}
~callback_wrapper() {
std::lock_guard<std::recursive_mutex> guard(_mutex);
PyGILState_STATE state = PyGILState_Ensure();
Py_DECREF(_cb.ptr());
PyGILState_Release(state);
_destroyed = true;
}
void operator ()(topic_ptr topic) {
std::lock_guard<std::recursive_mutex> guard(_mutex);
if(_destroyed) {
return;
}
PyGILState_STATE state = PyGILState_Ensure();
try {
_cb(topic);
}
catch(error_already_set) { PyErr_Print(); }
PyGILState_Release(state);
}
object _cb;
std::recursive_mutex _mutex;
bool _destroyed;
};
The problem is that you try to add an object to Python's cyclic garbage collector tracking twice.
Check out this bug, specifically:
The documentation for supporting cyclic garbage collection in Python
My documentation patch for the issue and
My explanation in the bug report itself
Long story short: if you set Py_TPFLAGS_HAVE_GC and you are using Python's built-in memory allocation (standard tp_alloc/tp_free), you don't ever have to manually call PyObject_GC_Track() or PyObject_GC_UnTrack(). Python handles it all behind your back.
Unfortunately, this is not documented very well, at the moment. Once you've fixed the issue, feel free to chime in on the bug report (linked above) about better documentation of this behavior.
This is a real project written in VBA before.
I want to move it to Python and use 'ActiveX Automation scripts for AutoCAD with Python' method. This is my code:
# -*- coding: utf-8 -*-
from pyautocad import Autocad, APoint, aDouble
acad = Autocad(False, True)
acad.prompt("Hello, Autocad from Python\n")
print acad.doc.Name
xx = acad.model.AddCircle(APoint(0, 0), 10)
print(xx)
yy = acad.model.Add3Dpoly(aDouble([0, 0, 0, 10, 10, 10, 30, 20, 30, 0, 0, 0]))
print(yy.ObjectName)
print(yy.PlotStyleName)
# How to contruct an objectlist for AddRegion?
#regions = acad.model.AddRegion([yy])
#acad.model.AddExtrudedSolid(regions[0], 20, 0)
My question is, how to construct an object list for AddRegion? Maybe comtypes have some topic about VARINT. I really have no experience about COM and so on...
Getting it all to work right can be more work than it seems like it should be. Reading data using python; not so bad. Writing data, bit trickier. Casual user / Beginners; be warned of what your getting into.
What you will probably need
It helps tremendously if you are familiar with autolisp, because it just works better (in this case), is documented better, and integrates better,... and you will likely need it to milk 'unknown/hidden/undocumented' information that python isn't telling you. (see the vlax- and vla- series of lisp functions).
Next you need win32com make_py and gen_py scripts from the command line, or you can use win32com.client.gencode while staying mostly in python.
Be prepared to visually parse very ugly text (and I wasn't even talking about lisp =] ). Be prepared to fail, and get excited to find out why.
Most of it has to do with COM-Variants. And then you get wierd stuff like Variant-Variant-Arrays. If you check out win32com.client.pythoncom, you will notice all of the datatypes are mapped to integers. (VT_BOOL is 11 for example).
The Nitty Gritty
Next time you attempt a ModelSpace.AddCircle, pay attention to the debug output you get;
All of the parameters passed to InvokeTypes are what you need to watch for... (this is taken from my make-py output for the Autocad Registered Interfaces
def AddLine(self, StartPoint=defaultNamedNotOptArg, EndPoint=defaultNamedNotOptArg):
ret = self._oleobj_.InvokeTypes(
1581, LCID, 1, (9, 0), ((12, 1), (12, 1)),StartPoint, EndPoint)
if ret is not None:
ret = Dispatch(ret, u'AddLine', '{DF524ECB-D59E-464B-89B6-D32822282778}'
This tells you exactly which COM types win32com THINKS it wants, so make sure you are at least matching that.
I have found that many input functions are actually documented and invoked wrong (I learned this after much back and forth with AutoLisp). What we see above has a value of 1581 on the outside (which is something like a class name, not really a datatype), and then tuple that basically says (DISPATCH, EMPTY):(9,0), and then an array of VT_VARIANTS:((12,1),(12,1)).
There is usually a missing outer wrapper that COM was expecting, and for some reason make-py does not realize this. if you go through extensive AutoLisp vlax- nonsense, you will notice their is an additional wrapper around that one. I believe it is either a VARIANT_ARRAY, or quite literally, a VARIANT-VARIANT-ARRAY (quadruple pointer or something). The codes for this are (vt_array=8192, vt_variant=12).
Im sorry I don't remember specifics, but I believe the portion reading ((12,1),(12,1)), should become (8192, 12, ((12,1),(12,1))), or something like that. Even once you do figure out what it should be, Im not sure if their is a quick fix. As of AutoCAD 2010, for me, this meant going through the ungodly large gen_py output, finding the functions I really wanted, and manually changing the InvokeTypes() call to match what COM was expecting.
Everything worked simply as expected after that.
Possible Workarounds
COM is ugly. If you are new to python, but semi-experienced in AutoCAD (meaning you want to do some fairly hefty automation), stay away from the python->win32com->AutoCAD pipeline. Use LISP. As much as that pains me to say, your gonna end up writing so many LISP test cases and debuggers to accompany your python pains, you might as well just commit.
Ironpython and .NET
I believe this interface is much more supported than COM in general
Visual Studio Professional (2008+)
I never used official VS-Pro tools (I used PythonWIN and MINGW), Im not sure if there is any extra magic provided that would change how win32com handles AutoCAD. I know the official AutoCAD ARX extensions provide their source in a Studio project. Worst case you would have actual documentation close at hand, which is where this entire topic of python-AutoCAD becomes tainted.