I am trying to load a .so in cppyy, but getting below error.
Is there any way to see what exact error is there, due to which Load() is failing
load_my_lib.py:57: in <module>
cppyy.load_library("mylib.so")
.venv/lib/python3.6/site-packages/cppyy/__init__.py:219: in load_library
sc = gSystem.Load(name)
E cppyy.gbl.std.exception: int CppyyLegacy::TSystem::Load(const char* module, const char* entry = "", CppyyLegacy::Bool_t system = kFALSE) =>
E exception: std::exception
It looks like the loading of the library results in a std::exception being thrown that has an empty result out of it's what().
Both the use of std::exception rather than one of its derived classes, as well as having no message returned from what() are a bit of an uncommon use, and I'm not aware of anything in the load library call itself that can cause that. Thus, my best guess would be that it gets thrown during the creation of a global or static variable that lives in mylib.so.
Do you know whether there are any static or global variables in that library? These may also live in a library that mylib.so is linked with, and which gets pulled in when loading.
Another way that sometimes gets a better diagnostic (although I doubt it in this case, as it doesn't handle C++ exceptions) is to load the library with ctypes instead:
import ctypes
d = ctypes.CDLL("mylib.so", ctypes.RTLD_GLOBAL)
(ctypes.CDLL does not have portable behavior, which is why load_library exists, but on most, not all, Linux distros it's a perfectly fine alternative.)
Related
I am having issues using ctypes to connect with a gaussmeter (F.W. Bell 5180). The manufacturer provides DLLs for the interface and a header file called FWB5180.h:
FWB5180.h:
// The following ifdef block is the standard way of creating macros which make exporting
// from a DLL simpler. All files within this DLL are compiled with the USB5100_EXPORTS
// symbol defined on the command line. this symbol should not be defined on any project
// that uses this DLL. This way any other project whose source files include this file see
// USB5100_API functions as being imported from a DLL, whereas this DLL sees symbols
// defined with this macro as being exported.
#ifdef USB5100_EXPORTS
#define USB5100_API __declspec(dllexport)
#else
#define USB5100_API __declspec(dllimport)
#endif
extern "C" USB5100_API unsigned int openUSB5100(void);
extern "C" USB5100_API void closeUSB5100(unsigned int fwb5000ID);
extern "C" USB5100_API int scpiCommand(unsigned int usbID, char* cmd, char* result, int len);
My ctypes Python code is the following, where all I try to do is initialize the device then close it:
import ctypes,time
# open DLL
path = "C:\\Users\\Roger\\fw_bell_magnetic_field_probe\\usb5100-x64\\x64-dist\\usb5100.dll"
fwbell = ctypes.WinDLL(path)
# define open and close functions with argument and return types
openUSB5100 = fwbell.openUSB5100
openUSB5100.argtypes = None
openUSB5100.restype = ctypes.c_int
closeUSB5100 = fwbell.closeUSB5100
closeUSB5100.argtypes = [ctypes.c_int]
closeUSB5100.restype = None
# open device
idn = openUSB5100()
print(idn, type(idn))
# close device
time.sleep(0.1)
closeUSB5100(idn)
Expected behavior: it says elsewhere in the documentation that idn is a four bit unsigned integer, so it should return a number like 10203045. So I would expect an idn like that, and no errors while closing the connection.
Actual behavior: In my code, openUSB5100 always returns 0, whether the device is plugged in or not. The print statement always outputs 0 <class 'int'>. Then, the closeUSB5100 function errors out with something like OSError: exception: access violation reading 0x0000000000000028. I have also tried using different types for idn like c_ulong, but that didn't seem to help.
I'm on Windows 10 using python 3.9. Both are 64-bit, as are the DLLs.
I will note that I can use the gaussmeter using their provided dummy programs so its not strictly a connectivity issue. Though, I think they are using a 32-bit application with 32 bit drivers because when I try to use the DLLs that they seem to use, I get the following error: OSError: [WinError 193] %1 is not a valid Win32 application. When I use the DLLs marked as 64 bit I don't get this error.
Any tips or things to try with my code? This is my first foray into ctypes, so I accept there is likely egregious errors.
EDIT:
Here is the exact error message that I am getting.
PS C:\Users\Roger\fw_bell_magnetic_field_probe\usb5100-x64\x64-dist> python .\fw_bell_py.py
Traceback (most recent call last):
File "C:\Users\Roger\fw_bell_magnetic_field_probe\usb5100-x64\x64-dist\fw_bell_py.py", line 30, in <module>
idn = openUSB5100()
OSError: exception: access violation reading 0x00000000B56B1D68
The last value, 0x ... , typically changes a little bit if I run the code more than once.
Additionally, I have discovered that apparently the gauss meter might be really slow at being ready to use after being detected by the PC. So if I wait a long time between plugging in the device (or monitoring with usb.core.list_devices until something shows up) and running this code, the code always errors out on the open command, it doesn't make it to the close command.
There were some issues with the code (see the discussion), but the larger problem is that there is some problem with the DLL files themselves.
I have this part of a C# DLL from a larger project. It has no dependencies, runs in either .NET 5.0 or .NET 6.0 C# without anything else in it. It looks something like this:
namespace Some.Namespace {
public static class Rename{
public static string RenameString(string stringToRename){
//Do some stuff to it
return stringToRename;
}
}
}
However, no matter what way we attempt to add the reference to Python 3.6 (ArcGIS Pro Distribution) using Python.NET...
sys.path.append(os.path.join(os.getcwd(),"Project_Folder","bin"))
clr.AddReference("SomeDLL")
from Some.Namespace import Rename
It throws this error on the 3rd line.
Exception has occurred: ModuleNotFoundError No module named Some
We've tried just about every possible means to load the DLL at this point and none of them have worked (Yes I know that 2.5.x Python.NET doesn't support .NET 6.0 - we switched to 5.0 and it didn't work either) Exposing the function through Robert's DllExport using ctypes throws a CLR Exception that we can't debug because it's cross-environment whenever the function is run.
Here's that attempt.
//The DLL With Robert's DLLExport
namespace Some.Namespace {
public static class Rename{
//We've also tried CallingConvention.Cdecl
[DllExport("Rename", System.Runtime.InteropServices.CallingConvention.StdCall)]
public static string RenameString(string stringToRename){
//Do some stuff to it
return stringToRename;
}
}
}
#In Python
#We've tried WINDLL
dll_utils = CDLL(os.path.join(os.getcwd(),"project_folder","bin","SomeDLL.dll"))
#We've tried using CFUNCTTYPE here too
encrypt_proto = ctypes.WINFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)
encrypt_args = (1, "p1", 0),(1, "p2", 0),
#It fails here when we add the encrypt_args flags to it, saying they don't match up,
#but there's no way to figure out how many it wants, we've tried many variations of
#the above
encrypt_call = encrypt_proto(("Rename",dll_utils),encrypt_args)'
#So because of that on one attempt we removed these two lines
p2 = ctypes.c_char_p(text_to_obfuscate)
p1 = ctypes.c_char_p(text_to_obfuscate)
#And the second argument here
return_value = encrypt_call(p1,p2)
#And...
OSError: [WinError -532462766] Windows Error 0xe0434352
Attempting to write a second function in the DLL to convert the C# string to a byte array and back and using a ctypes pointer doesn't work - it throws the same error just posted when it's called. Someone suggested in another question here IronPython next - which we want to avoid, or to try to use comtypes which I've never used before. At one point we even tried to do a COM decoration and it claimed something I've seen on an answer here is obsolete. I've researched this for 2-3 days and haven't been able to solve this.
At this point I'm at a loss. Is there any effective, simple way, to get an external .NET C# function in a DLL to call a function and pass a string in Python without installing any major dependencies? Just out of the box? There's gotta be something simple I'm missing.
I'm trying to use a SDK with python.
I have multiple dll files in the SPK.
My script:
import ctypes
import os
malib = ctypes.WinDLL(os.path.join('D:\Downloads\Aura API\sdk\AURA_SDK.dll'))
print(malib.GetClaymoreKeyboardLedCount(1)) #function in the dll
I get the error :
WindowsError: exception: access violation reading 0x00000005
I can use some of the functions normaly but for others I get this issue. Also there are different dll's in the SDK and I think the problem could come from the communication between these dll (I only open one of these dll in the script) also because function not working seem to be using the other dll or/and communication with the computer.
Thanks if you got advices
You haven't set the argtypes and restype for the functions you're calling.
This means that, instead of knowing what C types to convert your arguments to, ctypes has to guess based on the Python types you pass it. If it guesses wrong, you will either pass garbage, or, worse, corrupt the stack, leading to an access violation.
For example, imagine this C function:
void func(int64_t n, char *s);
If you do this:
lib = # however you load the library
lib.func(2, 'abc')
… then ctypes, it will convert that 2 to a 32-bit int, not a 64-bit one. If you're using a 32-bit Python and DLL, that means n will get the 2 and the pointer to 'abc' crammed into one meaningless number, and s will be an uninitialized pointer to some arbitrary location in memory that, if your lucky, won't be mapped to anything and will raise an access violation.
But if you first do this:
lib = # however you load the library
lib.func.argtypes = [ctypes.c_int64, ctypes.c_char_p]
lib.func.restype = None
lib.func(2, 'abc')
… then ctypes will convert the 2 to a 64-bit int, so n will get 2 and s will get 'abc' and everyone will be happy.
My python code has been crashing with error 'GC Object already Tracked' . Trying to figure out the best approach to debug this crashes.
OS : Linux.
Is there a proper way to debug this issue.
There were couple of suggestions in the following article.
Python memory debugging with GDB
Not sure which approach worked for the author.
Is there a way to generate memory dumps in such scenario which could be analyzed. Like in Windows world.
Found some article on this. But not entirely answers my question:
http://pfigue.github.io/blog/2012/12/28/where-is-my-core-dump-archlinux/
Found out the reason for this issue in my scenario (not necessarily the only reason for the GC object crash).
I used the GDB and Core dumps to debug this issue.
I have Python and C Extension Code (in shared object).
Python code registers a Callback routine with C Extension code.
In a certain workflow a thread from C Extension code was calling the registered Call back routine in Python code.
This usually worked fine but when multiple threads did the same action concurrently it resulted in the Crash with 'GC Object already tracked'.
Synchronizing the access to python objects for multiple thread does resolve this issue.
Thanks to any responded to this.
I ran into this problem using boost::python when our C++ code would trigger a python callback. I would occasionally get "GC object already tracked" and the program would terminate.
I was able to attach GDB to the process prior to triggering the error. One interesting thing, in the python code we were wrapping the callback with
a functools partial, which was actually masking where the real error was occuring. After replacing the partial with a simple callable wrapper class. The "GC object already tracked error" no longer popped up, instead I was now just getting a segfault.
In our boost::python wrapper, we had lambda functions to handle a C++ callback and the lambda function captured the boost::python::object callback function. It turned out, for whatever reason, in the destructor for the lambda, it wasn't always properly acquiring the GIL when destroying the boost::python::object which was causing the segfault.
The fix was to not use a lambda function, but instead create a functor that makes sure to acquire the GIL in the destructor prior to calling PyDECREF() on the boost::python::object.
class callback_wrapper
{
public:
callback_wrapper(object cb): _cb(cb), _destroyed(false) {
}
callback_wrapper(const callback_wrapper& other) {
_destroyed = other._destroyed;
Py_INCREF(other._cb.ptr());
_cb = other._cb;
}
~callback_wrapper() {
std::lock_guard<std::recursive_mutex> guard(_mutex);
PyGILState_STATE state = PyGILState_Ensure();
Py_DECREF(_cb.ptr());
PyGILState_Release(state);
_destroyed = true;
}
void operator ()(topic_ptr topic) {
std::lock_guard<std::recursive_mutex> guard(_mutex);
if(_destroyed) {
return;
}
PyGILState_STATE state = PyGILState_Ensure();
try {
_cb(topic);
}
catch(error_already_set) { PyErr_Print(); }
PyGILState_Release(state);
}
object _cb;
std::recursive_mutex _mutex;
bool _destroyed;
};
The problem is that you try to add an object to Python's cyclic garbage collector tracking twice.
Check out this bug, specifically:
The documentation for supporting cyclic garbage collection in Python
My documentation patch for the issue and
My explanation in the bug report itself
Long story short: if you set Py_TPFLAGS_HAVE_GC and you are using Python's built-in memory allocation (standard tp_alloc/tp_free), you don't ever have to manually call PyObject_GC_Track() or PyObject_GC_UnTrack(). Python handles it all behind your back.
Unfortunately, this is not documented very well, at the moment. Once you've fixed the issue, feel free to chime in on the bug report (linked above) about better documentation of this behavior.
I need to run a series of python scripts calculating various scripts, that are working fine, but one of them runs very slowly and has to be done in C++.
The C++ code is ready, but I need to find a way to call the C++ function from the python and get the return value.
I found information about SWIG, but didn't get it to work on Windows with Visual Studio (I have to do it in VS on Windows, because of other constraints). I found ctypes much easier for my very simple function with standart input and output values.
So I did many tests with ctypes, tried all the examples I could find online, and each and every time, what happens is that I can load the dll (built with visual studio 2012 compiler) but when I try calling the function in python, it just goes :
Traceback (most recent call last):
File "test.py", line 3, in <module>
print maLib.plop()
File "C:\Python27\lib\ctypes\__init__.py", line 378, in __getattr__
func = self.__getitem__(name)
File "C:\Python27\lib\ctypes\__init__.py", line 383, in __getitem__
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'plop' not found
The python code is (test.py) :
from ctypes import cdll
lib = cdll.LoadLibrary('test')
print lib.plop()
The C++ code is (test.cpp, compiled as test.dll, Visual Studio set to build as dll) :
extern "C"
{
int plop();
}
int plop() { return 4; }
As you can see, I tried to make it as simple as possible, to avoid details making it fail. I have read the python ctypes help, and tutorials on how to use ctypes, trying exactly the same codes as them, but I had to adapt a bit because I am using Visual Studio / windows and most of the other users are using linux.
All of the files are in the same folder.
I have tried multiple ways of loading the library : LoadLibrary('./name.dll'), WinDLL('name.dll'), giving the function a different name, void return type, etc...
Do you think I should use SWIG ? Is my problem an obvious beginner mistake ? I am a student, and new to most of what I'm using, but I put a lot of effort in this, even if I just need it for a particular single task.
When using SWIG, I had to make a wrapper with function pointers, or references, so I thought this was the problem, which made me want to try a simpler solution.
I am using : Python 2.7.7, Visual Studio 2012, Windows 8.1
Many thanks in advance
The error reported by your Python is very clear.
function 'plop' not found
This means that the DLL does not export a function of that name. So, you need to export the function. Either with a .def file, or using __declspec(dllexport):
extern "C"
{
__declspec(dllexport) int plop();
}
To inspect your DLL to debug issues like this, use dumpbin or Dependency Walker.
Note that since you do not specify calling convention, the default of __cdecl is used. That means that cdll is correct on the ctypes side.
You need to give your function "C" linkage to avoid name mangling. Declare it like this:
extern "C"
{
int plop();
}
and it will then properly be called "plop" rather than being name-mangled.