I'm writing a Python3 script with some computationally heavy sections in C, using the Python C API. When dealing with int64s, I can't figure out how to ensure that an input number is an unsigned int64; that is , if it's smaller than 0. As the official documentation suggests, I'm using PyArg_ParseTuple() with the formatter K - which does not check for overflow. Here is my C code:
static PyObject* from_uint64(PyObject* self, PyObject*){
uint64_t input;
PyObject* output;
if (!PyArg_ParseTuple(args, "K", &input)){
return PyErr_Format(PyExc_ValueError, "Wrong input: expected unsigned 64-bit integer.");
}
return NULL;
}
However, calling the function with a negative argument throws no error, and the input number is casted to unsigned. E.g., from_uint64(-1) will result in input=2^64-2. As expected, since there's no overflow check.
What would be the correct way of determining whether the input number is negative, possibly before parsing it?
You should use
unsigned long long input = PyLong_AsUnsignedLongLong(args);
You can then check with
if (PyErr_Occurred()) {
// handle out of range here
}
if the number was unsuitable for an unsigned long long.
See also the Python 3 API documentation on Integer Objects
With a little modification from #Ctx 's answer:
The solution is to first parse the input as an object (so, not directly from args), then check its type:
static PyObject* from_uint64(PyObject* self, PyObject* args){
PyObject* output;
PyObject* input_obj;
if (!PyArg_ParseTuple(args, "O", &input_obj)){
return PyErr_Format(PyExc_TypeError, "Wrong input: expected py object.");
}
unsigned long long input = PyLong_AsUnsignedLongLong(input_obj);
if(input == -1 && PyErr_Occurred()) {
PyErr_Clear();
return PyErr_Format(PyExc_TypeError, "Parameter must be an unsigned integer type, but got %s", Py_TYPE
(input_obj)->tp_name);
}
This code, as expected, works on any input in [0, 2^64-1] and throws error on integers outside the boundaries as well as illegal types like float, string, etc.
Related
I'm trying to create a python module using my C++ code and I want to declare a function with multiple arguments. (3 in this case) I've read the docs and it says that I must declare METH_VARARGS which I did, but I think I also must change something inside my function to actually receive the arguments. Otherwise it gives me "too many arguments" error when I use my function in python.
Here is the code snippet I'm using:
...
// This function can be called inside a python file.
static PyObject *
call_opencl(PyObject *self, PyObject *args)
{
const char *command;
int sts;
// We except at least one argument to this function
// Not sure how to accept more than one.
if (!PyArg_ParseTuple(args, "s", &command))
return NULL;
OpenCL kernel = OpenCL();
kernel.init();
std::cout << "This message is called from our C code: " << std::string(command) << std::endl;
sts = 21;
return PyLong_FromLong(sts);
}
static PyMethodDef NervebloxMethods[] = {
{"call_kernel", call_opencl, METH_VARARGS, "Creates an opencv instance."},
{NULL, NULL, 0, NULL} /* Sentinel */
};
...
You are still expecting one argument.
if (!PyArg_ParseTuple(args, "s", &command))
the documentation defines how you can expect optional or additional arguments, for example "s|dd" will expect a string and two optional numbers, you still have to pass two doubles to the function for when the numbers are available.
double a = 0; // initial value
double b = 0;
if (!PyArg_ParseTuple(args, "s|dd", &command, &a, &b))
Context:
I have written a Red Black tree implementation in C language. To allow it to use variable types, it only handles const void * elements, and initialisation of a tree must be given a comparison function with a signature int (*comp)(const void *, const void *). So far, so good, but I now try to use that C code to build an extension module for Python. It looks simple as first sight, because Python languages always pass references to objects which are received as pointers by C routines.
Problem:
Python objects come with rich comparison operators. That means that from a C extension module, comparing 2 arbitrary objects is trivial: just a matter of using int PyObject_RichCompareBool(PyObject *o1, PyObject *o2, int opid).
But the comparison may return -1 to indicate that the objects are not comparable. In Python or C++ it would be simple enough to throw an exception to signal an abnormal condition. Unfortunately C has no notion of exception, and I could not find a way using setjmp-longjmp because:
the environment buffer has do be known to both the englobing function and the internal one
I should free any allocated memory at longjmp time, when the internal function does not know what has been allocated
First idea:
A simple solution is to give a third parameter to the comparison function for it to signal an abnormal condition. But when the library is used in a plain C environment, that third parameter just does not make sense. I then remembered that in the 80', I had learned that in C language, parameters were passed in the stack in reversed order and unstacked by the caller to allow functions with a variable number of parameters. That means that provided the first 2 parameters are correct passing a third parameter to a function expecting 2 should be harmless.
Demo code:
#include <stdio.h>
// declares a type for the comparison functions
typedef int (*func)();
// A simple function for comparing integers - only 2 params
int f1(int a, int b) {
return a - b;
}
/* Inserts a value into an increasing array
* By convention 0 denotes the end of the array
* No size control implemented for brievety
* The comp function recieves a pointer to an int
* to be able to signal abnormal conditions
* */
int insert(int* arr, int val, func comp) {
int err = 0;
while ((0 != *arr) && (comp(*arr, val, &err) < 0)) { // 1
if (err) return 0;
++arr;
}
do {
int tmp = *arr;
*arr = val;
val = tmp;
} while (0 != *arr++);
return 1;
}
int main() {
func f = &f1;
// a simple test with 3 parameters
int cr = f(3, 1, 5); // 2
printf("%d\n", cr);
// demo usage of the insert function
int arr[10] = {0};
int data[] = { 1,5,3,2,4 };
for (int i = 0; i < sizeof(data) / sizeof(*data); i++) {
insert(arr, data[i], f1);
}
for (int i = 0; i < sizeof(data) / sizeof(*data); i++) {
printf("%d ", arr[i]);
}
return 0;
}
At (1) and (2) the 2 parameter function is called with 3 parameters. Of course, this code compiles without even a warning in Clang or MSVC, and runs fine giving the expected result.
Question:
While this code works fine on common implementations, I wonder whether actually passing a third parameter to a function expecting only two is really legit or does it invokes Undefined Behaviour?
Current research
Is it safe to invoke a C function with more parameters than it expects? : the accepted answer suggests that it should be safe when the C calling convention is used (which is my use case) while other answers show that the MSVC stdcall calling convention would not allow it
6.7.6.3 Function declarators (including prototypes) and 6.5.2.2 Function calls in draft n1570 for C11, but as English is not my first language, I could not understand where it was or not allowed
Remark:
The originality of this question is that it uses function pointers conversions.
I think it invokes Undefined Behavior.
From 6.5.2.2p6:
If the expression that denotes the called function has a type that does not include a prototype, the integer promotions are performed on each argument, and arguments that have type float are promoted to double. These are called the default argument promotions. If the number of arguments does not equal the number of parameters, the behavior is undefined.
The proper solution is redesign the Red Black tree implementation to allow passing a context as a third parameter.
int (*comp)(const void *, const void *, void *);
It is highly recommended to add a context argument to any function pointer type to allow emulate closures.
As a workaround, you could use a global variable.
static int err;
int f1(int a, int b) {
err = 0;
return a - b;
}
int insert(int* arr, int val, int comp(int,int)) {
err = 0;
while ((0 != *arr) && (comp(*arr, val) < 0)) { // 1
if (err) return 0;
++arr;
}
...
}
It is not the best solution because it is not re-entrant. Only a single instance of insert()/f1() can run at a time.
This is a complement to the accepted answer. The shown code uses function pointers to solve the compilation errors that would arise when calling a prototyped function with a wrong number of parameters.
But the draft n1570 for C11 says at 6.3.2.3 [Language/Conversions/Other operands/] Pointers ยง8:
... If a converted
pointer is used to call a function whose type is not compatible with the referenced type,
the behavior is undefined.
And it fully applies here because the referenced type is a function taking 2 parameters and the converted pointer type is a function taking 3 parameters. Per the accepted answer and 6.5.2.2p6 those two function type are not compatible, hence the code does invoke UB.
After finding that, I haved decided to give up with that way, and instead choosed to use wrapper functions that call the function passed to the library with their expected number of arguments to avoid UB.
I have a number of C functions that accept different arguments, e.g.
foo_i(int a)
foo_c(char c)
Is it possible to overload these functions in python C api?
I tried to use the following methods table:
static PyMethodDef test_methods[] = {
{"foo", (PyCFunction)foo_i, METH_VARARGS, "int"},
{"foo", (PyCFunction)foo_c, METH_VARARGS, "char"},
{NULL, NULL, 0, NULL}
};
But when I invoke foo from python I always end up using the function at the bottom of the table.
Any ideas on how to invoke both foo_i() and foo_c() using foo() in python C-api?
Thanks!
Either give them different Python level names, or write a single wrapper function that type checks the argument provided and dispatches to the correct "real" function. Python itself has no direct support for overloading functions based on argument types.
If you want the wrapper written for you, you might take a look at pybind11, which does allow overloading in the sense you're attempting (it does so via a type checking wrapper under the hood, so it's just syntactic sugar, not a change in behavior).
Untested example code:
static PyObject*
foo_wrapper(PyObject *self, PyObject *arg)
{
Py_buffer view;
Py_ssize_t ival;
// Check for/handle length 1 bytes-like object (bytes, bytearray, small mmap, etc.)
if (PyObject_GetBuffer(arg, &view, PyBUF_SIMPLE) == 0) {
if (view.len != 1) {
PyErr_Format(PyExc_ValueError, "Must receive exactly one byte, got %zd", view.len);
PyBuffer_Release(&view);
return NULL;
}
foo_c(((char*)view.buf)[0]);
Py_RETURN_NONE; // Or convert return from foo_c if it exists
}
// Check for/handle integer-like object that fits in C int
PyErr_Clear(); // Ignore error for objects not supporting buffer protocol
ival = PyNumber_AsSsize_t(arg, PyExc_ValueError);
if (PyErr_Occurred()) {
if (PyErr_ExceptionMatches(PyExc_TypeError)) {
// Replace with general error message about both accepted argument types,
// since only reporting error from int conversion might confuse folks
PyErr_Format(PyExc_TypeError, "Argument must be length 1 bytes-like object or integer; received %R", Py_TYPE(arg));
}
return NULL;
}
// Check valid range (Py_ssize_t often larger than int)
if (ival < INT_MIN or ival > INT_MAX) {
return PyErr_Format(PyExc_ValueError, "Integer must be in range [%d-%d]; received %zd", INT_MIN, INT_MAX, ival);
}
foo_i((int)ival);
Py_RETURN_NONE; // Or convert return from foo_i if it exists
}
static PyMethodDef test_methods[] = {
{"foo", (PyCFunction)foo_wrapper, METH_O, "Wrapper for foo_c and foo_i"},
{NULL, NULL, 0, NULL}
};
python code
for b in range(4):
for c in range(4):
print myfunc(b/0x100000000, c*8)
c code
unsigned int b,c;
for(b=0;b<4;b++)
for(c=0;c<4; c++)
printf("%L\n", b/0x100000000);
printf("%L\n" , myfunc(b/0x100000000, c*8));
I am getting an error saying:
error: integer constant is too large for "long" type at both printf statement in c code.
'myfunc' function returns a long.
This can be solved by defining 'b' a different type. I tried defining 'b' as 'long' and 'unsigned long' but no help.
Any pointers?
My bad...This is short version of problem
unsigned int b;
b = 1;
printf("%L", b/0x100000000L);
I am getting error and warnings:
error: integer constant is too large for "long" type
warning: conversion lacks type at end of format
warning: too many arguments for format
Your C code needs braces to create the scope that Python does by indentation, so it should look like this:
unsigned int b,c;
for(b=0;b<4;b++)
{
for(c=0;c<4; c++)
{
printf("%L\n", b/0x100000000);
printf("%L\n" , myfunc(b/0x100000000, c*8));
}
}
Try long long. Python automatically uses number representation which fits your constants, but C does not. 0x100000000L simply does not fit in 32-bit unsigned int, unsigned long and so on. Also, read your C textbook on long long data type and working with it.
unsigned int b,c;
const unsigned long d = 0x100000000L; /* 33 bits may be too big for int */
for(b=0;b<4;b++) {
for(c=0;c<4; c++) { /* use braces and indent consistently */
printf("%ud\n", b/d); /* "ud" to print Unsigned int in Decimal */
printf("%ld\n", myfunc(b/d, c*8)); /* "l" is another modifier for "d" */
}
}
While attempting to read a Python list filled with float numbers and to populate real channels[7] with their values (I'm using F2C, so real is just a typedef for float), all I am able to retrieve from it are zero values. Can you point out the error in the code below?
static PyObject *orbital_spectra(PyObject *self, PyObject *args) {
PyListObject *input = (PyListObject*)PyList_New(0);
real channels[7], coefficients[7], values[240];
int i;
if (!PyArg_ParseTuple(args, "O!", &PyList_Type, &input)) {
return NULL;
}
for (i = 0; i < PyList_Size(input); i++) {
printf("%f\n", PyList_GetItem(input, (Py_ssize_t)i)); // <--- Prints zeros
}
//....
}
PyList_GetItem will return a PyObject*. You need to convert that to a number C understands. Try changing your code to this:
printf("%f\n", PyFloat_AsDouble(PyList_GetItem(input, (Py_ssize_t)i)));
Few things I see in this code.
You leak a reference, don't create that empty list at the beginning, it's not needed.
You don't need to cast to PyListObject.
PyList_GetItem returns a PyObject, not a float. Use PyFloat_AsDouble to extract the value.
If PyList_GetItem returns NULL, then an exception has been thrown, and you should check for it.