I have a virtual object that I subclass and implement in python using pybind11, and that object then is used in C++ code but the python object goes out of scope when the python script finished and python garbage collector destroys my object before I can use it in C++ side.
To avoid the destruction I have added a "keep" method to the binding code that increases the ref_count, so that python garbage collector doesn't destroy it.
C++ code for the bindings
VNCS::Real VNCS::py::BlendingField::blending(const VNCS::Point_3 &p) const
{
std::array<double, 3> pyP{p[0], p[1], p[2]};
PYBIND11_OVERLOAD_PURE(VNCS::Real, /* Return type */
VNCS::BlendingField, /* Parent class */
blending, /* Name of function in C++ (must match Python name) */
pyP /* Argument(s) */
);
}
void VNCS::py::module::blendingField(pybind11::module &m)
{
pybind11::class_<VNCS::BlendingField, VNCS::py::BlendingField, std::shared_ptr<VNCS::BlendingField>>(
m, "BlendingField")
.def(pybind11::init(
[](pybind11::args &args, pybind11::kwargs &kwargs) { return std::make_shared<VNCS::py::BlendingField>(); }))
.def("keep", [](pybind11::handle handle) { handle.inc_ref(); });
}
Python code
class BlendingField(PyVNCS.BlendingField):
def __init__(self, *args, **kwargs):
PyVNCS.BlendingField.__init__(self, *args, **kwargs)
def __del__(self):
print("dtor from python")
def blending(self, point):
return point[0]
blendingField = BlendingField()
blendingField.keep()
simCreator = initObject.addObject("SimCreator",
name="simCreator",
coarseMesh="#lowResTopology",
detailMesh="#highResTopology")
simCreator.setBlendingField(blendingField)
This doesn't look very nice, as the keep method there feels like a terrible hack. What would be the correct way to implement this?
I don't think messing with the ref count is the best solution. If you want to use an object in C++-land, use C++ lifetime management tools.
Pybind integrates lifetime management into the language bindings, you just need to change it's behavior. To quote the docs:
The binding generator for classes, class_, can be passed a template type that denotes a special holder type that is used to manage references to the object. If no such holder type template argument is given, the default for a type named Type is std::unique_ptr<Type>, which means that the object is deallocated when Python’s reference count goes to zero.
It is possible to switch to other types of reference counting wrappers or smart pointers, which is useful in codebases that rely on them. For instance, the following snippet causes std::shared_ptr to be used instead.
Here's an example lifted from the tests:
// Object managed by a std::shared_ptr<>
class MyObject2 {
public:
MyObject2(const MyObject2 &) = default;
MyObject2(int value) : value(value) { print_created(this, toString()); }
std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; }
virtual ~MyObject2() { print_destroyed(this); }
private:
int value;
};
py::class_<MyObject2, std::shared_ptr<MyObject2>>(m, "MyObject2")
.def(py::init<int>());
m.def("make_myobject2_1", []() { return new MyObject2(6); });
m.def("make_myobject2_2", []() { return std::make_shared<MyObject2>(7); });
m.def("print_myobject2_1", [](const MyObject2 *obj) { py::print(obj->toString()); });
m.def("print_myobject2_2", [](std::shared_ptr<MyObject2> obj) { py::print(obj->toString()); });
m.def("print_myobject2_3", [](const std::shared_ptr<MyObject2> &obj) { py::print(obj->toString()); });
m.def("print_myobject2_4", [](const std::shared_ptr<MyObject2> *obj) { py::print((*obj)->toString()); });
If you do go that route, which I would recommend, this question would be a duplicate of this one.
Note that there are some minor pitfalls here especially if youre codebase uses raw pointers around python code. The docs point to a possible double-free if a C++ method returning a raw pointer to an shared object is called from python. Again stealing from the docs:
class Child { };
class Parent {
public:
Parent() : child(std::make_shared<Child>()) { }
Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */
private:
std::shared_ptr<Child> child;
};
PYBIND11_MODULE(example, m) {
py::class_<Child, std::shared_ptr<Child>>(m, "Child");
py::class_<Parent, std::shared_ptr<Parent>>(m, "Parent")
.def(py::init<>())
.def("get_child", &Parent::get_child);
}
from example import Parent
print(Parent().get_child()) // UB or Segfault
Though these issue don't seem to me to be unique to shared_ptr over unique_ptr.
This appears to be a known limitation in pybind11. I read through all the docs, whatever bug reports seemed applicable, and everything I could find in the pybind11 gitter. I have a custom exception class in c++ that contains custom constructors and fields. A very basic example of such a class, trimmed for space is here:
class BadData : public std::exception
{
public:
// Constructors
BadData()
: msg(),
stack(),
_name("BadData")
{}
BadData(std::string _msg, std::string _stack)
: msg(_msg),
stack(_stack),
_name("BadData")
{}
const std::string&
getMsg() const
{
return msg;
}
void
setMsg(const std::string& arg)
{
msg = arg;
}
// Member stack
const std::string&
getStack() const
{
return stack;
}
void
setStack(const std::string& arg)
{
stack = arg;
}
private:
std::string msg;
std::string stack;
std::string _name;
I currently have python binding code that binds this into python, but it is custom generated and we'd much rather use pybind11 due to its simplicity and compile speed.
The default mechanism for binding an exception into pybind11 would look like
py::register_exception<BadData>(module, "BadData");
That will create an automatic translation between the C++ exception and the python exception, with the what() value of the c++ exception translating into the message of the python exception. However, all the extra data from the c++ exception is lost and if you're trying to throw the exception in python and catch it in c++, you cannot throw it with any of the extra data.
You can bind extra data onto the python object using the attr and I even went somewhat down the path of trying to extend the pybind11:exception class to make it easier to add custom fields to exceptions.
template <typename type>
class exception11 : public ::py::exception<type>
{
public:
exception11(::py::handle scope, const char *name, PyObject *base = PyExc_Exception)
: ::py::exception<type>(scope, name, base)
{}
template <typename Func, typename... Extra>
exception11 &def(const char *name_, Func&& f, const Extra&... extra) {
::py::cpp_function cf(::py::method_adaptor<type>(std::forward<Func>(f)),
::py::name(name_),
::py::is_method(*this),
::py::sibling(getattr(*this, name_, ::py::none())),
extra...);
this->attr(cf.name()) = cf;
return *this;
}
};
This adds a def function to exceptions similar to what is done with class_. The naive approach to using this doesn't work
exception11< ::example::data::BadData>(module, "BadData")
.def("getStack", &::example::data::BadData::getStack);
Because there is no automatic translation between BadData in c++ and in python. You can try to work around this by binding in a lambda:
.def("getStack", [](py::object& obj) {
::example::data::BadData *cls = obj.cast< ::example::data::BadData* >();
return cls->getStack();
});
The obj.cast there also fails because there is no automatic conversion. Basically, with no place to store the c++ instance, there isn't really a workable solution for this approach that I could find. In addition I couldn't find a way to bind in custom constructors at all, which made usability on python very weak.
The next attempt was based on a suggestion in the pybind11 that you could use the python exception type as a metaclass a normal class_ and have python recognize it as a valid exception. I tried a plethora of variations on this approach.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::reinterpret_borrow<py::object>(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception))
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), py::cast(PyExc_Exception->ob_type))
py::class_< ::example::data::BadData>(module, "BadData", py::metaclass((PyObject *) &PyExc_Exception->ob_type))
There were more that I don't have saved. But the overall results was either 1) It was ignored completely or 2) it failed to compile or 3) It compiled and then immediately segfaulted or ImportError'd when trying to make an instance. There might have been one that segfaulted on module import too. It all blurs together. Maybe there is some magic formula that would make such a thing work, but I was unable to find it. From my reading of the pybind11 internals, I do not believe that such a thing is actually possible. Inheriting from a raw python type does not seem to be something it is setup to let you do.
The last thing I tried seemed really clever. I made a python exception type
static py::exception<::example::data::BadData> exc_BadData(module, "BadDataBase");
and then had my pybind11 class_ inherit from that.
py::class_< ::example::data::BadData >(module, "BadData", py::dynamic_attr(), exc_BadData)
But that also segfaulted on import too. So I'm basically back to square one with this.
So I figured out a way to actually do this but it involves 1) doing some hacking of the pybind11 code itself and 2) introducing some size inefficiencies to the bound python types. From my point of view, the size issues are fairly immaterial. Yes it would be better to have everything perfectly sized but I'll take some extra bytes of memory for ease of use. Given this inefficiency, though, I'm not submitting this as a PR to the pybind11 project. While I think the trade-off is worth it, I doubt that making this the default for most people would be desired. It would be possible, I guess to hide this functionality behind a #define in c++ but that seems like it would be super messy long-term. There is probably a better long-term answer that would involve a degree of template meta-programming (parameterizing on the python container type for class_) that I'm just not up to.
I'm providing my changes here as diffs against the current master branch in git when this was written (hash a54eab92d265337996b8e4b4149d9176c2d428a6).
The basic approach was
Modify pybind11 to allow the specification of an exception base class for a class_ instance.
Modify pybind11's internal container to have the extra fields needed for a python exception type
Write a small amount of custom binding code to handle setting the error correctly in python.
For the first part, I added a new attribute to type_record to specify if a class is an exception and added the associated process_attribute call for parsing it.
diff --git a/src/pybind11/include/pybind11/attr.h b/src/pybind11/include/pybind11/attr.h
index 58390239..b5535558 100644
--- a/src/pybind11/include/pybind11/attr.h
+++ b/src/pybind11/include/pybind11/attr.h
## -73,6 +73,9 ## struct module_local { const bool value; constexpr module_local(bool v = true) :
/// Annotation to mark enums as an arithmetic type
struct arithmetic { };
+// Annotation that marks a class as needing an exception base type.
+struct is_except {};
+
/** \rst
A call policy which places one or more guard variables (``Ts...``) around the function call.
## -211,7 +214,8 ## struct function_record {
struct type_record {
PYBIND11_NOINLINE type_record()
: multiple_inheritance(false), dynamic_attr(false), buffer_protocol(false),
- default_holder(true), module_local(false), is_final(false) { }
- default_holder(true), module_local(false), is_final(false),
- is_except(false) { }
/// Handle to the parent scope
handle scope;
## -267,6 +271,9 ## struct type_record {
/// Is the class inheritable from python classes?
bool is_final : 1;
- // Does the class need an exception base type?
- bool is_except : 1;
- PYBIND11_NOINLINE void add_base(const std::type_info &base, void *(*caster)(void *)) {
auto base_info = detail::get_type_info(base, false);
if (!base_info) {
## -451,6 +458,11 ## struct process_attribute<is_final> : process_attribute_default<is_final> {
static void init(const is_final &, type_record *r) { r->is_final = true; }
};
+template <>
+struct process_attribute<is_except> : process_attribute_default<is_except> {
- static void init(const is_except &, type_record *r) { r->is_except = true; }
+};
I modified the internals.h file to add a separate base class for exception types. I also added an extra bool argument to make_object_base_type.
diff --git a/src/pybind11/include/pybind11/detail/internals.h b/src/pybind11/include/pybind11/detail/internals.h
index 6224dfb2..d84df4f5 100644
--- a/src/pybind11/include/pybind11/detail/internals.h
+++ b/src/pybind11/include/pybind11/detail/internals.h
## -16,7 +16,7 ## NAMESPACE_BEGIN(detail)
// Forward declarations
inline PyTypeObject *make_static_property_type();
inline PyTypeObject *make_default_metaclass();
-inline PyObject *make_object_base_type(PyTypeObject *metaclass);
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except);
// The old Python Thread Local Storage (TLS) API is deprecated in Python 3.7 in favor of the new
// Thread Specific Storage (TSS) API.
## -107,6 +107,7 ## struct internals {
PyTypeObject *static_property_type;
PyTypeObject *default_metaclass;
PyObject *instance_base;
+ PyObject *exception_base;
#if defined(WITH_THREAD)
PYBIND11_TLS_KEY_INIT(tstate);
PyInterpreterState *istate = nullptr;
## -292,7 +293,8 ## PYBIND11_NOINLINE inline internals &get_internals() {
internals_ptr->registered_exception_translators.push_front(&translate_exception);
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
- internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass);
+ internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass, false);
+ internals_ptr->exception_base = make_object_base_type(internals_ptr->default_metaclass, true);
And then in class.h I added the necessary code to generate the exception base type. The first caveat is here. Since PyExc_Exception is a garbage collected type, I had to scope the assert call that checked the GC flag on the type. I have not currently seen any bad behavior from this change, but this is definitely voiding the warranty right here. I would highly, highly recommend always passing the py:dynamic_attr() flag to any classes you are using py:except on, since that turns on all the necessary bells and whistles to handle GC correctly (I think). A better solution might be to turn all those things on in make_object_base_type without having to invoke py::dynamic_attr.
diff --git a/src/pybind11/include/pybind11/detail/class.h b/src/pybind11/include/pybind11/detail/class.h
index a05edeb4..bbb9e772 100644
--- a/src/pybind11/include/pybind11/detail/class.h
+++ b/src/pybind11/include/pybind11/detail/class.h
## -368,7 +368,7 ## extern "C" inline void pybind11_object_dealloc(PyObject *self) {
/** Create the type which can be used as a common base for all classes. This is
needed in order to satisfy Python's requirements for multiple inheritance.
Return value: New reference. */
-inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
+inline PyObject *make_object_base_type(PyTypeObject *metaclass, bool is_except=false) {
constexpr auto *name = "pybind11_object";
auto name_obj = reinterpret_steal<object>(PYBIND11_FROM_STRING(name));
## -387,7 +387,12 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
auto type = &heap_type->ht_type;
type->tp_name = name;
- type->tp_base = type_incref(&PyBaseObject_Type);
+ if (is_except) {
+ type->tp_base = type_incref(reinterpret_cast<PyTypeObject*>(PyExc_Exception));
+ }
+ else {
+ type->tp_base = type_incref(&PyBaseObject_Type);
+ }
type->tp_basicsize = static_cast<ssize_t>(sizeof(instance));
type->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_HEAPTYPE;
## -404,7 +409,9 ## inline PyObject *make_object_base_type(PyTypeObject *metaclass) {
setattr((PyObject *) type, "__module__", str("pybind11_builtins"));
PYBIND11_SET_OLDPY_QUALNAME(type, name_obj);
- assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ if (!is_except) {
+ assert(!PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC));
+ }
return (PyObject *) heap_type;
}
## -565,7 +572,8 ## inline PyObject* make_new_python_type(const type_record &rec) {
auto &internals = get_internals();
auto bases = tuple(rec.bases);
- auto base = (bases.size() == 0) ? internals.instance_base
+ auto base = (bases.size() == 0) ? (rec.is_except ? internals.exception_base
+ : internals.instance_base)
And then the final change, which is the inefficiency part. In Python, everything is a PyObject, but that is really only two fields (setup with the PyObject_HEAD macro) and the actual object struct may have a lot of extra fields. And having a very precise layout is important because python uses offsetof to seek into these things some times. From the Python 2.7 source code (Include/pyerrord.h) you can see the struct that is used for base exceptions
typedef struct {
PyObject_HEAD
PyObject *dict;
PyObject *args;
PyObject *message;
} PyBaseExceptionObject;
Any pybind11 type that extends PyExc_Exception has to have a instance struct that contains the same initial layout. And in pybind11 currently, the instance struct just has PyObject_HEAD. That means if you don't change the instance struct, this will all compile, but when python seeks into this object, it will do with the assumption that hose extra fields exist and then it will seek right off the end of viable memory and you'll get all sorts of fun segfaults. So this change adds those extra fields to every class_ in pybind11. It does not seem to break normal classes to have these extra fields and it definitely seems to make exceptions work correctly. If we broke the warranty before, we just tore it up and lit it on fire.
diff --git a/src/pybind11/include/pybind11/detail/common.h b/src/pybind11/include/pybind11/detail/common.h
index dd626793..b32e0c70 100644
--- a/src/pybind11/include/pybind11/detail/common.h
+++ b/src/pybind11/include/pybind11/detail/common.h
## -392,6 +392,10 ## struct nonsimple_values_and_holders {
/// The 'instance' type which needs to be standard layout (need to be able to use 'offsetof')
struct instance {
PyObject_HEAD
+ // Necessary to support exceptions.
+ PyObject *dict;
+ PyObject *args;
+ PyObject *message;
/// Storage for pointers and holder; see simple_layout, below, for a description
However, once these changes are all made, here is what you can do.
Bind in the class
auto PyBadData = py::class_< ::example::data::BadData>(module, "BadData", py::is_except(), py::dynamic_attr())
.def(py::init<>())
.def(py::init< std::string, std::string >())
.def("__str__", &::example::data::BadData::toString)
.def("getStack", &::example::data::BadData::getStack)
.def_property("message", &::example::data::BadData::getMsg, &::example::data::BadData::setMsg)
.def("getMsg", &::example::data::BadData::getMsg);
And take a function in c++ that throws the exception
void raiseMe()
{
throw ::example::data::BadData("this is an error", "");
}
and bind that in too
module.def("raiseMe", &raiseMe, "A function throws");
Add an exception translator to put the entire python type into the exception
py::register_exception_translator([](std::exception_ptr p) {
try {
if (p) {
std::rethrow_exception(p);
}
} catch (const ::example::data::BadData &e) {
auto err = py::cast(e);
auto errType = err.get_type().ptr();
PyErr_SetObject(errType, err.ptr());
}
});
And then you get all the things you could want!
>>> import example
>>> example.raiseMe()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is an error, stack=)
You can, of course, also instantiate and raise the exception from python as well
>>> import example
>>> raise example.BadData("this is my error", "no stack")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
example.BadData: BadData(msg=this is my error, stack=no stack)
I have some code in C++ and I want to call it from Python. The C++ code returns lists of dicts to the python code. I use boost.python to achieve this, in the following manner:
struct MyStruct {
int foo;
};
class MyClass
{
public:
std::vector<MyStruct> get_stuff();
};
struct MyStructConverter
{
static PyObject* convert(const std::vector<MyStruct>& v)
{
boost::python::list ret;
for (auto c : v) {
boost::python::dict *r = new boost::python::dict();
(*r)["foo"] = c.foo;
ret.append(boost::python::object(*r));
}
return boost::python::incref(ret.ptr());
}
};
/** Interface between C++ and Python
*/
BOOST_PYTHON_MODULE(glue)
{
boost::python::to_python_converter<std::vector<MyStruct>, MyStructConverter>();
class_<MyClass, boost::noncopyable>("MyClass")
.def("get_stuff", &MyClass::get_stuff);
}
This works to the extent that I can pass data from C++ to python. However, when used as follows:
# test.py
x = MyClass()
while True:
u = x.get_stuff()
# do something with u
it leaks memory (i.e. the VSZ of the python process keeps growing until the whole system grinds to a halt).
What am I doing wrong? How can I avoid this?
I have a C++ class that is able to output strings in normal ASCII or wide format. I want to get the output in Python as a string. I am using SWIG (version 3.0.4) and have read the SWIG documentation. I'm using the following typemap to convert from a standard c string to my C++ class:
%typemap(out) myNamespace::MyString &
{
$result = PyString_FromString(const char *v);
}
This works fine in Windows with the VS2010 compiler, but it is not working completely in Linux. When I compile the wrap file under Linux, I get the following error:
error: cannot convert ‘std::string*’ to ‘myNamespace::MyString*’ in assignment
So I tried adding an extra typemap to the Linux interface file as so:
%typemap(in) myNamespace::MyString*
{
$result = PyString_FromString(std::string*);
}
But I still get the same error. If I manually go into the wrap code and fix the assignment like so:
arg2 = (myNamespace::MyString*) ptr;
then the code compiles just fine. I don't see why my additional typemap isn't working. Any ideas or solutions would be greatly appreciated. Thanks in advance.
It doesn't look like your typemap is using the arguments quite correctly. You should have something like this instead:
%typemap(out) myNamespace::MyString &
{
$result = PyString_FromString($1);
}
Where the '$1' is the first argument. See the SWIG special variables for more information [http://www.swig.org/Doc3.0/Typemaps.html#Typemaps_special_variables]
EDIT:
To handle the input typemap, you will need something like this:
%typemap(in) myNamespace::MyString*
{
const char* pChars = "";
if(PyString_Check($input))
{
pChars = PyString_AsString($input);
}
$1 = new myNamespace::MyString(pChars);
}
You can do more error checking and handle Unicode with the following code:
%typemap(in) myNamespace::MyString*
{
const char* pChars = "";
PyObject* pyobj = $input;
if(PyString_Check(pyobj))
{
pChars = PyString_AsString(pyobj);
$1 = new myNamespace::MyString(pChars);
}
else if(PyUnicode_Check(pyobj))
{
PyObject* tmp = PyUnicode_AsUTF8String(pyobj);
pChars = PyString_AsString(tmp);
$1 = new myNamespace::MyString(pChars);
}
else
{
std::string strTemp;
int rrr = SWIG_ConvertPtr(pyobj, (void **) &strTemp, $descriptor(String), 0);
if(!SWIG_IsOK(rrr))
SWIG_exception_fail(SWIG_ArgError(rrr), "Expected a String "
"in method '$symname', argument $argnum of type '$type'");
$1 = new myNamespace::MyString(strTemp);
}
}
I'm using sqlite-amalgamation-3080500 within a Python3/C module.
My python module creates some tables and then returns the sqlite3's handle to the python environment using PyCapsule.
So, in a second module, I try to create more tables using this same sqlite3's handle. But my program is breaking. I get an "access violation error" into sqlite3_mutex_enter() - which has been called by sqlite3_prepare_v2().
First-chance exception at 0x00000000 in python.exe: 0xC0000005: Access
violation executing location 0x00000000. Unhandled exception at
0x7531C9F1 in python.exe: 0xC0000005: Access violation executing
location 0x00000000.
Is it really thread-safe? I think I can do it this way. I've already did it in the past, but I was using XCode on Mac. Now I'm trying to do the same on MSVC 2013.
Bellow is my code to run queries:
bool register_run(register_db_t *pReg, const char *query)
{
int ret, len;
sqlite3_stmt *stmt;
const char *err;
stmt = NULL;
len = (int)strlen(query);
ret = sqlite3_prepare_v2(pReg->pDb, query, len, &stmt, NULL);
if (ret != SQLITE_OK) {
err = sqlite3_errmsg(pReg->pDb);
fprintf(stderr, "sqlite3_prepare_v2 error: %s\n%s\n",
err, query);
return false;
}
ret = register_run_stmt(pReg, query, stmt);
sqlite3_finalize(stmt);
return ret;
}
And this is how I export the handle to use it in my 2nd C/module:
// Register's getattro
static PyObject* Register_getattro(RegisterObject *self, PyObject *name)
{
// ...
} else if (PyUnicode_CompareWithASCIIString(name, "handle") == 0) {
register_db_t *handle = self->db;
return PyCapsule_New(handle, NULL, NULL);
}
return PyObject_GenericGetAttr((PyObject *)self, name);
}
This is the python code gluing pieces:
import registermodule, loggermodule
reg = registermodule.Register("mydata.db")
loggermodule.set_register(reg.handle)
And how I use the the handle on my second module:
static PyObject* loggerm_set_register(PyObject *self, PyObject *args)
{
register_db_t *pReg;
PyObject *capsule;
if (!PyArg_ParseTuple(args, "O:set_register", &capsule)) {
return NULL;
}
if (!PyCapsule_CheckExact(capsule)) {
PyErr_SetString(PyExc_ValueError,
"The object isn't a valid pointer.");
return NULL;
}
pReg = PyCapsule_GetPointer(capsule, NULL);
if (!logger_set_register(pReg)) {
PyErr_SetString(PyExc_SystemError,
"Could not set the pointer as register.");
return NULL;
}
Py_RETURN_NONE;
}
And finally the routine that is breaking:
bool logger_set_register(register_db_t *pReg)
{
char *query = "CREATE TABLE IF NOT EXISTS tab_logger ("
"date NUMERIC,"
"level TEXT,"
"file TEXT,"
"function TEXT,"
"line INTEGER,"
"message TEXT)";
g_pReg = pReg;
return register_run(g_pReg, query);
}
And the sqlite3's routine that is breaking all:
SQLITE_API void sqlite3_mutex_enter(sqlite3_mutex *p){
if( p ){
sqlite3GlobalConfig.mutex.xMutexEnter(p);
}
}
Sorry about lots of snippets, but I've no clue about the problem.
Thanks in advance.
I don't know why, but globals are not the same between Python C modules on Windows. It wasn't on Mac OS, in despite of my previous experience doing so.
In Windows, python modules are DLLs, so they don't share the same global stack.
I've discovered that sqlite3Config.mutex was NULL to my second Python C module. It was causing the Access Violation error. But sqlite3Config.mutex is a global variable, this thing should be started by the previous module.
Now, knowing this point, I solved the problem calling this function:
sqlite3_initialize();
And all is working properly!
Not sure your problem is directly related to sqlite3_initialize();
Because sqlite3_initialize(); is automatically called, at least once, during sqlite3_open_v2.
I suggest to dig in the lack of the option SQLITE_OPEN_FULLMUTEX during sqlite3_open_v2.
Recommendation set this option always. Penalty of using Mutex is extremely low especially in view of all the overhead Python add.
Negligeable in single thread and mandatory (nearly) in multithread. I don't even understand why it stay an option. should be there always
So better safe than sorry. I don't know of real downside to use.
SQLite is "lite"
BTW it is legit for sqlite3Config.mutex to be NULL.
This should used only be used by SQLite that check this condition,
search sqlite3.c for "
sqlite3_mutex_enter(db->mutex);
" to understand what I mean.