`dict_values` is hashable but `dict_keys` is not? [duplicate] - python

In python 3, the keys(), values(), and items() methods provide dynamic views of their respective elements. These were backported to python 2.7 and are available there as viewkeys, viewvalues, and viewitems. I'm referring to them interchangeably here.
Is there any reasonable explanation for this:
#!/usr/bin/python3.4
In [1]: hash({}.keys())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-3727b260127e> in <module>()
----> 1 hash({}.keys())
TypeError: unhashable type: 'dict_keys'
In [2]: hash({}.items())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-decac720f012> in <module>()
----> 1 hash({}.items())
TypeError: unhashable type: 'dict_items'
In [3]: hash({}.values())
Out[3]: -9223363248553358775
I found this rather surprising.
The python docs glossary on "hashable" says:
An object is hashable if it has a hash value which never changes
during its lifetime (it needs a __hash__() method), and can be
compared to other objects (it needs an __eq__() method). Hashable
objects which compare equal must have the same hash value.
Okay, the first part actually checks out; it doesn't appear that the hash of a dict_values object will change over its lifetime - even though its underlying values certainly can.
In [11]: d = {}
In [12]: vals = d.values()
In [13]: vals.__hash__()
Out[13]: -9223363248553358718
In [14]: d['a'] = 'b'
In [15]: vals
Out[15]: dict_values(['b'])
In [16]: vals.__hash__()
Out[16]: -9223363248553358718
But the part about __eq__()... well, it doesn't have one of those, actually.
In [17]: {'a':'a'}.values().__eq__('something else')
Out[17]: NotImplemented
So... yeah. Can someone make any sense of this? Is there a reason for this asymmetry that of the three viewfoo methods, only dict_values objects are hashable?

I believe this occurs because viewitems and viewkeys provide custom rich comparison functions, but viewvalues does not. Here are the definitions of each view type:
PyTypeObject PyDictKeys_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_keys", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
&dictviews_as_number, /* tp_as_number */
&dictkeys_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
dictview_richcompare, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictkeys_iter, /* tp_iter */
0, /* tp_iternext */
dictkeys_methods, /* tp_methods */
0,
};
PyTypeObject PyDictItems_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_items", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
&dictviews_as_number, /* tp_as_number */
&dictitems_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
dictview_richcompare, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictitems_iter, /* tp_iter */
0, /* tp_iternext */
dictitems_methods, /* tp_methods */
0,
};
PyTypeObject PyDictValues_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_values", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
0, /* tp_as_number */
&dictvalues_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictvalues_iter, /* tp_iter */
0, /* tp_iternext */
dictvalues_methods, /* tp_methods */
0,
};
Notice that tp_richcompare is defined as dictview_richcompare for items and keys, but not values. Now, the documentation for __hash__ says this:
A class that overrides __eq__() and does not define __hash__() will have its __hash__() implicitly set to None.
...
If a class that overrides __eq__() needs to retain the implementation
of __hash__() from a parent class, the interpreter must be told this
explicitly by setting __hash__ = <ParentClass>.__hash__.
If a class that does not override __eq__() wishes to suppress hash
support, it should include __hash__ = None in the class definition.`
So, because items/keys are overriding __eq__() (by providing a tp_richcompare function), they would need to explicitly define __hash__ as being equal to the parent's to retain an implementation for it. Because values doesn't override __eq__(), it inherits the __hash__ from object, because tp_hash and tp_richcompare get inherited from the parent if they're both NULL:
This field is inherited by subtypes together with tp_richcompare: a
subtype inherits both of tp_richcompare and tp_hash, when the
subtype’s tp_richcompare and tp_hash are both NULL.
The fact that the impelmentation for dict_values isn't preventing this automatic inheritence would probably be considered a bug.

Related

Python object not fully initialized using C API

In the following scenario, the object is not meant to be instantiated within Python (hence no tp_new or tp_init). Calling the ThingType as a function using PyObject_CallObject results in a segfault in _PyObject_FastCallDict, I believe because there are no constructors.
After creating the object using CreatePythonThing, the function get_height is not set unless a workaround is applied - to call dir(thing), which as a side effect initializes the object's properties. The workaround is already present in CreatePythonThing.
#include <Python.h>
typedef struct {
PyObject_HEAD
/* Type-specific fields go here. */
Eval *eval;
} Thing;
static PyObject* ThingGetHeight(PyObject* self, PyObject* args)
{
return PyLong_FromLong(1);
}
static PyMethodDef ThingMethods[] = {
{"get_height", ThingGetHeight, METH_NOARGS, "Get thing height"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
static PyTypeObject ThingType = {
PyVarObject_HEAD_INIT(NULL, 0)
"thing.Thing", /* tp_name */
sizeof(Thing), /* tp_basicsize */
0, /* tp_itemsize */
0, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"Thing", /* tp_doc */
0, 0, 0, 0, 0, 0,
ThingMethods, /* tp_methods */
};
Thing* CreatePythonThing(Eval *eval)
{
Thing* obj = PyObject_New(Thing, &ThingType);
obj->eval = eval;
obj = (Thing*) PyObject_Init((PyObject*) obj, &ThingType);
// I don't understand why, but the above does not fully initialize the object. The method
// table is not set. Calling `dir` on the object causes it to be initialized, so this is
// a workaround.
PyObject_Dir((PyObject*) obj);
return obj;
}
As in DavidW's comment, I was missing PyType_Ready(&ThingType); in my global initializer.

How to store a pointer in a custom embedded Python object

I have created a custom Python type in C as per the tutorial https://docs.python.org/2.7/extending/newtypes.html#the-basics. In my C I receive a pointer to a struct, I want to be able to get and set the values in the struct from Python without taking a copy of it. I.e.
a = myObject.x() # gets the x value in the struct.
or
myObject.x(255) # sets the x value in the struct.
However I cannot see how to store the pointer in the python object.
My current object definition is currently just the basic object implementation from the python website.
typedef struct {
PyObject_HEAD
myStruct *s;
} KeyObject;
static PyTypeObject KeyType = {
PyVarObject_HEAD_INIT(NULL, 0)
"ckb.Key", /* tp_name */
sizeof(KeyObject), /* tp_basicsize */
0, /* tp_itemsize */
0, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_compare */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"Key objects", /* tp_doc */
};
static PyMethodDef key_methods[] = {
{NULL} /* Sentinel */
};
Here's an example (cbk.c), that can act as a backbone for this task:
#include "external.h"
#include "Python.h"
#define MOD_NAME "ckb"
#define KEY_CLASS_NAME "Key"
/*
typedef struct InnerStruct_tag {
int x;
} InnerStruct;
//*/
typedef struct KeyObject_tag {
PyObject_HEAD
InnerStruct *inner;
} KeyObject;
static PyObject *Key_new(PyTypeObject *type, PyObject *args, PyObject *kwds) {
KeyObject *self;
self = (KeyObject*)type->tp_alloc(type, 0);
if (self != NULL) {
//self->inner = (InnerStruct*)calloc(1, sizeof(Key));
self->inner = getExternalPtr(1234); // Don't allocate here, get the pointer from external lib
if (self->inner == NULL) {
Py_DECREF(self);
return NULL;
}
}
return (PyObject*)self;
}
static void Key_dealloc(KeyObject *self) {
//free(self->inner);
delExternalPtr(self->inner); // Use the external dellocation function (optional)
Py_TYPE(self)->tp_free((PyObject*)self);
}
static PyObject *Key_getX(KeyObject *self, void *closure) {
return PyInt_FromLong(self->inner->x);
}
static int Key_setX(KeyObject *self, PyObject *value, void *closure) {
if (value == NULL) {
PyErr_SetString(PyExc_TypeError, "Cannot delete 'x'");
return -1;
}
if (!PyInt_Check(value)) {
PyErr_SetString(PyExc_TypeError, "'x' value must be an int");
return -1;
}
self->inner->x = ((PyIntObject*)value)->ob_ival;
return 0;
}
static PyGetSetDef Key_getsets[] = {
{"x", (getter)Key_getX, (setter)Key_setX, "x", NULL},
{NULL} // Sentinel
};
static PyTypeObject Key_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
MOD_NAME"."KEY_CLASS_NAME, /* tp_name */
sizeof(KeyObject), /* tp_basicsize */
0, /* tp_itemsize */
(destructor)Key_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_compare */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT |
Py_TPFLAGS_BASETYPE, /* tp_flags */
KEY_CLASS_NAME" objects", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
0, /* tp_methods */
0, /* tp_members */
Key_getsets, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
0, /* tp_init */
0, /* tp_alloc */
Key_new, /* tp_new */
};
#define Key_CheckExact(op) ((op)->ob_type == &Key_Type)
static PyMethodDef module_methods[] = {
{NULL} // Sentinel
};
PyMODINIT_FUNC initckb(void) {
PyObject* m;
if (PyType_Ready(&Key_Type) < 0)
return;
m = Py_InitModule3(MOD_NAME, module_methods,
MOD_NAME": Example module that creates an extension type ("KEY_CLASS_NAME").");
Py_INCREF(&Key_Type);
PyModule_AddObject(m, KEY_CLASS_NAME, (PyObject*)&Key_Type);
}
Notes:
Existing structure names / members was renamed (reorganized), for clarity
The inner structure only has one member (x), which is enough for making a point
Everything relies solely on the [Python 2.7.Docs]: Defining New Types page (that was mentioned in the question as well)
Since the wrapper object (Key) contains pointers (inner), in order to avoid checking them for NULL every time they are accessed:
A constructor (Key_new - equivalent to __new__ in Python) - that initializes them - was added
A destructor (Key_dealloc - equivalent __del__ in Python) - that does the exact opposite - was added as well, to avoid memory leaks (this is just a previous bullet consequence)
Access to InnerStruct's x member is done via Key_getX, Key_setX functions (note they are referenced in Key_getsets):
From Python, the inner x member, will be accessed by key_instance.x, as it was a Key instance attribute
This way makes more sense than using a getter (get_x()) and a setter (set_x(value)), and is more Pythonic
If however the getter / setter way is preferred, Key_getX, Key_setX signatures should be slightly modified (I think removing the last argument would do), and they should be referenced in Key_methods - that should be specified to KeyType as tp_methods (also described in the above web page)
When adding new members to InnerStruct, only the stuff done for x needs to be replicated and adapted (of course if there will be functions that look too similar, code should be refactored - but this is outside current scope)
The last part is pretty standard Python extension module code
Update #0
After the 1st comment, it seems like the question is trickier than it seems. Not sure if I still get it wrong, because it doesn't seem such a big deal. The change is (as I understood), that the inner pointer should come from somewhere else (another library (.dll)), instead of being created in the constructor. Changed to the example to mimic the new (and hopefully expected) behavior:
Since a InnerStruct pointer is returned by the external library (called it external.dll), the structure definition was moved in a header file belonging to that library - called it external.h (below), which is included by cbk.c
Would make sense that a library exports some data via a function (also exported by the library): getExternalPtr which may take arguments - currently it only has (a dummy) one: dummyArg0
Since getExternalPtr allocates memory inside, would make sense to have a corresponding function that deallocates it (delExternalPtr), in order to avoid memory leaks and Undefined Behavior (e.g. if memory is allocated in one place, deallocated in another, and the 2 places are deserved by different C runtimes). Any pointer returned by getExternalPtr should be passed to delExternalPtr exactly once
The 2 above functions will now be called from Key_new and Key_dealloc. If this is still not OK, and the object needs to be modified after creation (although it may be possible that some race issues would arise), setting the member could be done like: ((KeyObject*)keyInstancePyObjectPtr)->inner = getExternalPtr(0); with just one catch:
keyInstancePyObjectPtr (which is a generic PyObject*) should be of type Key_Type. Key_CheckExact macro does exactly that check
Now, the module depends (is linked to) on the external lib (not sure how things actually are), but that can be changed to Dynamic (DLL (SO)) loading (via [Man7]: DLOPEN(3) / [Man7]: DLSYM(3) or [MSDN]: LoadLibrary function) / [MSDN]: GetProcAddress function
external library code:
external.h:
#if defined (WIN32)
# if defined (EXTERNAL_DYNAMIC)
# if defined EXTERNAL_EXPORTS
# define EXTERNAL_EXPORT __declspec(dllexport)
# else
# define EXTERNAL_EXPORT __declspec(dllimport)
# endif
# else
# define EXTERNAL_EXPORT
# endif
#else
# define EXTERNAL_EXPORT
#endif
typedef struct InnerStruct_tag {
int x;
} InnerStruct;
#if defined (__cplusplus)
extern "C" {
#endif
EXTERNAL_EXPORT InnerStruct *getExternalPtr(int dummyArg0);
EXTERNAL_EXPORT void delExternalPtr(InnerStruct *ptr);
#if defined (__cplusplus)
}
#endif
external.c:
#include "external.h"
#include <stdlib.h>
InnerStruct *getExternalPtr(int dummyArg0) {
InnerStruct *ret = (InnerStruct*)malloc(sizeof(InnerStruct));
if (ret != NULL)
ret->x = 1618;
return ret;
}
void delExternalPtr(InnerStruct *ptr) {
free(ptr);
}
Test program (ckb_test.py):
import traceback
import ckb
print "\nModule:", ckb
print "Dir:", dir(ckb)
print "\nClass:", ckb.Key
print "Dir:", dir(ckb.Key)
key = ckb.Key()
print "\nInstance:", key
print "Dir:", dir(key)
print "\nKey.x (initial):", key.x
key.x = 123
print "Key.x (modified):", key.x
try:
key.x = 1.0
except:
traceback.print_exc()
del(key)
print "\nEnd"
Output:
c:\Work\Dev\StackOverflow\q46833364>set PATH=%PATH%;.\external\Win32-Release
c:\Work\Dev\StackOverflow\q46833364>set PYTHONPATH=%PYTHONPATH%;.\ckb\Win32-Release
c:\Work\Dev\StackOverflow\q46833364\>"c:\Install\x86\HPE\OPSWpython\2.7.10__00\python.exe" ckb_test.py
Module: <module 'ckb' from 'c:\Work\Dev\StackOverflow\q46833364\ckb\Win32-Release\ckb.pyd'>
Dir: ['Key', '__doc__', '__file__', '__name__', '__package__']
Class: <type 'ckb.Key'>
Dir: ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'x']
Instance: <ckb.Key object at 0x027A7050>
Dir: ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'x']
Key.x (initial): 1618
Key.x (modified): 123
Traceback (most recent call last):
File "..\ckb_test.py", line 20, in <module>
key.x = 1.0
TypeError: 'x' value must be an int
End

Python embedded in C++ try_rich_compare error on types

I have an error in my program, which appears to be something to do with comparing two object types in python. Here is the error from gdb
Program received signal SIGSEGV, Segmentation fault.
0x00007fffc3acd35c in try_rich_compare (v=0x7fffcc433ec0 <UTOPIA::PyNodeType>, w=0x7fffc3a06ec0 <UTOPIA::PyNodeType>, op=3) at ../Objects/object.c:621
621 ../Objects/object.c: No such file or directory.
(gdb) bt
#0 0x00007fffc3acd35c in try_rich_compare (v=0x7fffcc433ec0 <UTOPIA::PyNodeType>, w=0x7fffc3a06ec0 <UTOPIA::PyNodeType>, op=3) at ../Objects/object.c:621
#1 0x00007fffc3acded7 in do_richcmp (v=0x7fffcc433ec0 <UTOPIA::PyNodeType>, w=0x7fffc3a06ec0 <UTOPIA::PyNodeType>, op=3) at ../Objects/object.c:930
#2 0x00007fffc3ace164 in PyObject_RichCompare (v=0x7fffcc433ec0 <UTOPIA::PyNodeType>, w=0x7fffc3a06ec0 <UTOPIA::PyNodeType>, op=3) at ../Objects/object.c:982
#3 0x00007fffc3b74a24 in cmp_outcome (op=3, v=0x7fffcc433ec0 <UTOPIA::PyNodeType>, w=0x7fffc3a06ec0 <UTOPIA::PyNodeType>) at ../Python/ceval.c:4525
#4 0x00007fffc3b6bbbe in PyEval_EvalFrameEx (f=0x157e990, throwflag=0) at ../Python/ceval.c:2287
#5 0x00007fffc3b6ff3e in PyEval_EvalCodeEx (co=0x7fffc2553510, globals=0x7fffc254b1a8, locals=0x0, args=0x7fffc257c6a0, argcount=2, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0)
at ../Python/ceval.c:3252
#6 0x00007fffc3aa643b in function_call (func=0x7fffc2552300, arg=0x7fffc257c678, kw=0x0) at ../Objects/funcobject.c:526
#7 0x00007fffc3a64c79 in PyObject_Call (func=0x7fffc2552300, arg=0x7fffc257c678, kw=0x0) at ../Objects/abstract.c:2529
#8 0x00007fffc3a810a1 in instancemethod_call (func=0x7fffc2552300, arg=0x7fffc257c678, kw=0x0) at ../Objects/classobject.c:2602
#9 0x00007fffc3a64c79 in PyObject_Call (func=0x7fffd80df5e0, arg=0x7fffc2550370, kw=0x0) at ../Objects/abstract.c:2529
#10 0x00007fffc3a64de1 in call_function_tail (callable=0x7fffd80df5e0, args=0x7fffc2550370) at ../Objects/abstract.c:2561
#11 0x00007fffc3a651f3 in PyObject_CallMethod (o=0x7fffc255ced0, name=0x7fffcc4659a4 "invoke", format=0x7fffcc4659a2 "O") at ../Objects/abstract.c:2638
#12 0x00007fffcc45d556 in UTOPIA::Python_service_interface::invoke (this=0x14ed610, invocation_=0x16aa6f0, input_=...)
at /home/oni/Projects/utopia/components/libutopia/plugins/python/service_interface.cpp:134
My program contains a library of objects. Some of these objects are wrapped inside a python object wrapper. My main C++ program loads this python library in order to get the definitions of these types. In this case, the offending type looks like this:
// Node class
static PyTypeObject PyNodeType =
{
PyObject_HEAD_INIT(0)
0, /* ob_size */
"utopia.Node", /* tp_name */
sizeof(PyNode), /* tp_basicsize */
0, /* tp_itemsize */
(destructor) PyNode_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_compare */
(reprfunc) PyNode_repr, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
&PyNode_as_mapping, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT, /* tp_flags */
"UTOPIA::GenericNode class", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
PyNode_methods, /* tp_methods */
0, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc) PyNode_init, /* tp_init */
0, /* tp_alloc */
PyNode_new, /* tp_new */
0, /* tp_free */
0, /* tp_is_gc */
0, /* tp_bases */
0, /* tp_mro */
0, /* tp_cache */
0, /* tp_subclasses */
0, /* tp_weaklist */
0, /* tp_del */
};
Now, later on, the C++ program loads the python2.7 library and launches python inside itself. It then imports this wrapper python library.
This means this PyNodeType appears in the C++ program and also inside the python instance running inside the c++ program. At some point, these two things are compared and the program blows up! :S
Not really sure how to get around this one as the definition is needed in both places.
Further inspection reveals that, while the type is somehow deduced, one of these parameters is full of null pointers
(gdb) print v
$4 = (PyObject *) 0x7fffcc433ec0 <UTOPIA::PyNodeType>
(gdb) print *v
$5 = {_ob_next = 0x0, _ob_prev = 0x0, ob_refcnt = 2, ob_type = 0x0}
(gdb) print *w
$6 = {_ob_next = 0x7fffd80b3310, _ob_prev = 0x7fffe00aca70, ob_refcnt = 43, ob_type = 0x7fffc3efc0c0 <PyType_Type>}
***UPDATE***
So When I create an object I take a look at the memory locations
PyNode* newPyNode = PyObject_New(PyNode, &PyNodeType);
(gdb) print &PyNodeType
$4 = (PyTypeObject *) 0x7fffe353cea0 <UTOPIA::PyNodeType>
But then, if I look at the ob_type field in my newPyNode object
(gdb) print *newPyNode
$7 = {_ob_next = 0x7fffe12fd1c8, _ob_prev = 0x7fffe32024e0 <refchain>, ob_refcnt = 1, ob_type = 0x7fffe2a10ec0 <UTOPIA::PyNodeType>, node = 0xef2cc0}
ob_type does not match. What gives? Looking at the comparison functions like
PyObject_TypeCheck
... these memory locations should be the same.
Looks like its fixed. Ive joined together the CPP .so part of the project with the python library.so which seems to have fixed the problem. The former .so was so the main project could load python scripts whilst the second was to allow a python program access to all the utopia functionality (the opposite in effect). Joining the two libs together appears to have fixed it.

Why are (some) dict views hashable?

In python 3, the keys(), values(), and items() methods provide dynamic views of their respective elements. These were backported to python 2.7 and are available there as viewkeys, viewvalues, and viewitems. I'm referring to them interchangeably here.
Is there any reasonable explanation for this:
#!/usr/bin/python3.4
In [1]: hash({}.keys())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-1-3727b260127e> in <module>()
----> 1 hash({}.keys())
TypeError: unhashable type: 'dict_keys'
In [2]: hash({}.items())
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-decac720f012> in <module>()
----> 1 hash({}.items())
TypeError: unhashable type: 'dict_items'
In [3]: hash({}.values())
Out[3]: -9223363248553358775
I found this rather surprising.
The python docs glossary on "hashable" says:
An object is hashable if it has a hash value which never changes
during its lifetime (it needs a __hash__() method), and can be
compared to other objects (it needs an __eq__() method). Hashable
objects which compare equal must have the same hash value.
Okay, the first part actually checks out; it doesn't appear that the hash of a dict_values object will change over its lifetime - even though its underlying values certainly can.
In [11]: d = {}
In [12]: vals = d.values()
In [13]: vals.__hash__()
Out[13]: -9223363248553358718
In [14]: d['a'] = 'b'
In [15]: vals
Out[15]: dict_values(['b'])
In [16]: vals.__hash__()
Out[16]: -9223363248553358718
But the part about __eq__()... well, it doesn't have one of those, actually.
In [17]: {'a':'a'}.values().__eq__('something else')
Out[17]: NotImplemented
So... yeah. Can someone make any sense of this? Is there a reason for this asymmetry that of the three viewfoo methods, only dict_values objects are hashable?
I believe this occurs because viewitems and viewkeys provide custom rich comparison functions, but viewvalues does not. Here are the definitions of each view type:
PyTypeObject PyDictKeys_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_keys", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
&dictviews_as_number, /* tp_as_number */
&dictkeys_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
dictview_richcompare, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictkeys_iter, /* tp_iter */
0, /* tp_iternext */
dictkeys_methods, /* tp_methods */
0,
};
PyTypeObject PyDictItems_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_items", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
&dictviews_as_number, /* tp_as_number */
&dictitems_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
dictview_richcompare, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictitems_iter, /* tp_iter */
0, /* tp_iternext */
dictitems_methods, /* tp_methods */
0,
};
PyTypeObject PyDictValues_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"dict_values", /* tp_name */
sizeof(dictviewobject), /* tp_basicsize */
0, /* tp_itemsize */
/* methods */
(destructor)dictview_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_reserved */
(reprfunc)dictview_repr, /* tp_repr */
0, /* tp_as_number */
&dictvalues_as_sequence, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
PyObject_GenericGetAttr, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,/* tp_flags */
0, /* tp_doc */
(traverseproc)dictview_traverse, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
(getiterfunc)dictvalues_iter, /* tp_iter */
0, /* tp_iternext */
dictvalues_methods, /* tp_methods */
0,
};
Notice that tp_richcompare is defined as dictview_richcompare for items and keys, but not values. Now, the documentation for __hash__ says this:
A class that overrides __eq__() and does not define __hash__() will have its __hash__() implicitly set to None.
...
If a class that overrides __eq__() needs to retain the implementation
of __hash__() from a parent class, the interpreter must be told this
explicitly by setting __hash__ = <ParentClass>.__hash__.
If a class that does not override __eq__() wishes to suppress hash
support, it should include __hash__ = None in the class definition.`
So, because items/keys are overriding __eq__() (by providing a tp_richcompare function), they would need to explicitly define __hash__ as being equal to the parent's to retain an implementation for it. Because values doesn't override __eq__(), it inherits the __hash__ from object, because tp_hash and tp_richcompare get inherited from the parent if they're both NULL:
This field is inherited by subtypes together with tp_richcompare: a
subtype inherits both of tp_richcompare and tp_hash, when the
subtype’s tp_richcompare and tp_hash are both NULL.
The fact that the impelmentation for dict_values isn't preventing this automatic inheritence would probably be considered a bug.

Python instance method in C

Consider the following Python (3.x) code:
class Foo(object):
def bar(self):
pass
foo = Foo()
How to write the same functionality in C?
I mean, how do I create an object with a method in C? And then create an instance from it?
Edit:
Oh, sorry! I meant the same functionality via Python C API. How to create a Python method via its C API?
Something like:
PyObject *Foo = ?????;
PyMethod??? *bar = ????;
Here's a simple class (adapted from http://nedbatchelder.com/text/whirlext.html for 3.x):
#include "Python.h"
#include "structmember.h"
// The CountDict type.
typedef struct {
PyObject_HEAD
PyObject * dict;
int count;
} CountDict;
static int
CountDict_init(CountDict *self, PyObject *args, PyObject *kwds)
{
self->dict = PyDict_New();
self->count = 0;
return 0;
}
static void
CountDict_dealloc(CountDict *self)
{
Py_XDECREF(self->dict);
self->ob_type->tp_free((PyObject*)self);
}
static PyObject *
CountDict_set(CountDict *self, PyObject *args)
{
const char *key;
PyObject *value;
if (!PyArg_ParseTuple(args, "sO:set", &key, &value)) {
return NULL;
}
if (PyDict_SetItemString(self->dict, key, value) < 0) {
return NULL;
}
self->count++;
return Py_BuildValue("i", self->count);
}
static PyMemberDef
CountDict_members[] = {
{ "dict", T_OBJECT, offsetof(CountDict, dict), 0,
"The dictionary of values collected so far." },
{ "count", T_INT, offsetof(CountDict, count), 0,
"The number of times set() has been called." },
{ NULL }
};
static PyMethodDef
CountDict_methods[] = {
{ "set", (PyCFunction) CountDict_set, METH_VARARGS,
"Set a key and increment the count." },
// typically there would be more here...
{ NULL }
};
static PyTypeObject
CountDictType = {
PyObject_HEAD_INIT(NULL)
0, /* ob_size */
"CountDict", /* tp_name */
sizeof(CountDict), /* tp_basicsize */
0, /* tp_itemsize */
(destructor)CountDict_dealloc, /* tp_dealloc */
0, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
0, /* tp_compare */
0, /* tp_repr */
0, /* tp_as_number */
0, /* tp_as_sequence */
0, /* tp_as_mapping */
0, /* tp_hash */
0, /* tp_call */
0, /* tp_str */
0, /* tp_getattro */
0, /* tp_setattro */
0, /* tp_as_buffer */
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags*/
"CountDict object", /* tp_doc */
0, /* tp_traverse */
0, /* tp_clear */
0, /* tp_richcompare */
0, /* tp_weaklistoffset */
0, /* tp_iter */
0, /* tp_iternext */
CountDict_methods, /* tp_methods */
CountDict_members, /* tp_members */
0, /* tp_getset */
0, /* tp_base */
0, /* tp_dict */
0, /* tp_descr_get */
0, /* tp_descr_set */
0, /* tp_dictoffset */
(initproc)CountDict_init, /* tp_init */
0, /* tp_alloc */
0, /* tp_new */
};
// Module definition
static PyModuleDef
moduledef = {
PyModuleDef_HEAD_INIT,
"countdict",
MODULE_DOC,
-1,
NULL, /* methods */
NULL,
NULL, /* traverse */
NULL, /* clear */
NULL
};
PyObject *
PyInit_countdict(void)
{
PyObject * mod = PyModule_Create(&moduledef);
if (mod == NULL) {
return NULL;
}
CountDictType.tp_new = PyType_GenericNew;
if (PyType_Ready(&CountDictType) < 0) {
Py_DECREF(mod);
return NULL;
}
Py_INCREF(&CountDictType);
PyModule_AddObject(mod, "CountDict", (PyObject *)&CountDictType);
return mod;
}
You can't! C does not have "classes", it only has structs. And a struct cannot have code (methods or functions).
You can, however, fake it with function pointers:
/* struct object has 1 member, namely a pointer to a function */
struct object {
int (*class)(void);
};
/* create a variable of type `struct object` and call it `new` */
struct object new;
/* make its `class` member point to the `rand()` function */
new.class = rand;
/* now call the "object method" */
new.class();
I suggest you start from the example source code here -- it's part of Python 3's sources, and it exists specifically to show you, by example, how to perform what you require (and a few other things besides) -- use the C API to create a module, make a new type in that module, endow that type with methods and attributes. That's basically the first part of the source, culminating in the definition of Xxo_Type -- then you get examples of how to define various kinds of functions, some other types you may not care about, and finally the module object proper and its initialization (you can skip most of that of course, though not the module object and the parts of its initialization that lead up to the definition of the type of interest;-).
Most of the questions you might have while studying and adapting that source to your specific needs have good answers in the docs, especially in the section on "Object Implementation Support" -- but of course you can always open a new question here (one per issue would be best -- a "question" with many actual questions is always a bother!-) showing exactly what you're doing, what you were expecting as a result, and what you are seeing instead -- and you'll get answers which tend to include some pretty useful ones;-).

Categories