We Keep Coding

Pointer to gpu memory to Python from CUDA C [duplicate]

I would like to call my C functions within a shared library from Python scripts. Problem arrises when passing pointers, the 64bit addresses seem to be truncated to 32bit addresses within the called function. Both Python and my library are 64bit.
The example codes below demonstrate the problem. The Python script prints the address of the data being passed to the C function. Then, the address received is printed from within the called C function. Additionally, the C function proves that it is 64bit by printing the size and address of locally creating memory. If the pointer is used in any other way, the result is a segfault.
CMakeLists.txt
cmake_minimum_required (VERSION 2.6)
add_library(plate MODULE plate.c)
plate.c
#include <stdio.h>
#include <stdlib.h>
void plate(float *in, float *out, int cnt)
{
void *ptr = malloc(1024);
fprintf(stderr, "passed address: %p\n", in);
fprintf(stderr, "local pointer size: %lu\n local pointer address: %p\n", sizeof(void *), ptr);
free(ptr);
}
test_plate.py
import numpy
import scipy
import ctypes
N = 3
x = numpy.ones(N, dtype=numpy.float32)
y = numpy.ones(N, dtype=numpy.float32)
plate = ctypes.cdll.LoadLibrary('libplate.so')
print 'passing address: %0x' % x.ctypes.data
plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))
Output from python-2.7
In [1]: run ../test_plate.py
passing address: 7f9a09b02320
passed address: 0x9b02320
local pointer size: 8
local pointer address: 0x7f9a0949a400

The problem is that the ctypes module doesn't check the function signature of the function you're trying to call. Instead, it bases the C types on the Python types, so the line...
plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))
...is passing the the first two params as integers. See eryksun's answer for the reason why they're being truncated to 32 bits.
To avoid the truncation, you'll need to tell ctypes that those params are actually pointers with something like...
plate.plate(ctypes.c_void_p(x.ctypes.data),
ctypes.c_void_p(y.ctypes.data),
ctypes.c_int(N))
...although what they're actually pointers to is another matter - they may not be pointers to float as your C code assumes.
Update
eryksun has since posted a much more complete answer for the numpy-specific example in this question, but I'll leave this here, since it might be useful in the general case of pointer truncation for programmers using something other than numpy.

Python's PyIntObject uses a C long internally, which is 64-bit on most 64-bit platforms (excluding 64-bit Windows). However, ctypes assigns the converted result to pa->value.i, where value is a union and the i field is a 32-bit int. For the details, see ConvParam in Modules/_ctypes/callproc.c, lines 588-607 and 645-664. ctypes was developed on Windows, where a long is always 32-bit, but I don't know why this hasn't been changed to use the long field instead, i.e. pa->value.l. Probably, it's just more convenient most of the time to default to creating a C int instead of using the full range of the long.
Anyway, this means you can't simply pass a Python int to create a 64-bit pointer. You have to explicitly create a ctypes pointer. You have a number of options for this. If you're not concerned about type safety, the simplest option for a NumPy array is to use its ctypes attribute. This defines the hook _as_parameter_ that lets Python objects set how they're converted in ctypes function calls (see lines 707-719 in the previous link). In this case it creates a void *. For example, you'd call plate like this:
plate.plate(x.ctypes, y.ctypes, N)
However, this doesn't offer any type safety to prevent the function from being called with an array of the wrong type, which will result in either nonsense, bugs, or a segmentation fault. np.ctypeslib.ndpointer solves this problem. This creates a custom type that you can use in setting the argtypes and restype of a ctypes function pointer. This type can verify the array's data type, number of dimensions, shape, and flags. For example:
import numpy as np
import ctypes
c_npfloat32_1 = np.ctypeslib.ndpointer(
dtype=np.float32,
ndim=1,
flags=['C', 'W'])
plate = ctypes.CDLL('libplate.so')
plate.plate.argtypes = [
c_npfloat32_1,
c_npfloat32_1,
ctypes.c_int,
]
N = 3
x = np.ones(N, dtype=np.float32)
y = np.ones(N, dtype=np.float32)
plate.plate(x, y, N) # the parameter is the array itself

If you don't tell ctypes what type the parameters are, it attempts to infer it from the values that you pass to the function. And this inference will not always work as you need.
The recommended way to deal with this is to set the argtypes attribute of the function and so explicitly tell ctypes what the parameter types are.
plate.plate.argtypes = [
ctypes.POINTER(ctypes.c_float),
ctypes.POINTER(ctypes.c_float),
ctypes.c_int
]
Then you can call the function like this:
plate.plate(x.ctypes.data, y.ctypes.data, N)

Actually, You should set plate.argstype = [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int], and then it will be ok to accept the address in c func from python.
I met the problem and I solved it as what I say.

How do I use ctypes.create_string_buffer to pass it as a char * argument for a C function?

I have a C/C++ dll with a function:
int get_camera_info(char * description, char * serial_number, char * manufacturer)
This function takes the arguments and modifies them, so that you could print them. I'm trying to use this DLL from Python by using ctypes. I guess I would need to use ctypes.create_string_buffer to create an empty buffer so that I have a mutable object. But then, how do I do so that I can pass it as an argument and then retrieve the data?
Do I need to use cast(buffer_object, c_char_p_object)? How would this be any different from creating and passing a c_char_p object in the first place? I don't know much about C/C++, but maybe I should be using ctypes.POINTER.
My problem was that after passing a c_char_p type object as an argument I'm getting the error: OSError: exception: access violation reading 0x00000000.
If you need any help trying to understand what my question is, just ask. Or if you think I'm better off using something like cppyy, then I would need help with that too. :)

Here's the basics. Although not strictly required in this case, setting .argtypes and .restype correctly helps ctypes marshal parameters correctly and detect incorrectly passed parameters.
Similar to passing arrays to C functions, create_string_buffer returns an c_char_Array_array_size object, but it is marshaled as a pointer to it's first element (c_char_p) so it agrees with the .argtypes assignment.
Without the .argtypes assignment, passing something incorrect such as an int or a create_unicode_buffer() array would crash or have the wrong result. But defining it correctly would catch those errors and raise an exception.
test.c
#include <string.h>
#ifdef _WIN32
# define API __declspec(dllexport)
#else
# define API
#endif
API int get_camera_info(char * description, char * serial_number, char * manufacturer)
{
strcpy(description, "Description");
strcpy(serial_number, "12345678");
strcpy(manufacturer, "Manufacturer");
return 1;
}
test.py
from ctypes import *
MAX_STR = 256 # docs better say how big the buffers are required to be.
# int get_camera_info(char * description, char * serial_number, char * manufacturer)
dll = CDLL('./test')
dll.get_camera_info.argtypes = c_char_p,c_char_p,c_char_p
dll.get_camera_info.restype = c_int
desc = create_string_buffer(MAX_STR)
sn = create_string_buffer(MAX_STR)
mfg = create_string_buffer(MAX_STR)
ret = dll.get_camera_info(desc,sn,mfg)
print(desc.value.decode(),sn.value.decode(),mfg.value.decode())
Output:
Description 12345678 Manufacturer
Note that .value returns a byte string at the beginning of the buffer up to the first null byte exclusive. .raw will dump a byte string with every byte of the buffer. .decode() converts a byte string into a Unicode string using UTF-8 as the default encoding.

Python ctypes: c_char_p not returning the correct value

I'm new to Ctypes and trying to make a wrapper to use some dll functions but i'm facing a problem
first here is the code
C function structure
long func(char *pID);
Python code
lib = WinDLL("some.dll")
lib.func.restype = c_long
lib.func.argtypes = [c_char_p]
ID = c_char()
lib.func(byref(ID))
print(ID)
this outputs 8 which is correct but only the first character
the problem is I need the complete output not just the first char. I replaced it with c_char_p it give this c_char_p(925904440) which is a pointer but when I print its value (ID.value) it print an empty string while c_char was giving the correct value why ?
also tried create_string_buffer(10) but gives an error
expected LP_c_char_p instance instead of pointer to c_char_Array_10
Notice : I'm writing a J2534 wrapper since all the libraries that I had found have some kind of error tried python libs and C# libs
if you know or have a working J2534 library then send me its link

You could use create_string_buffer. Documentation says:
ctypes.create_string_buffer(init_or_size, size=None)
This function creates a mutable character buffer. The returned object is a ctypes array of c_char.
init_or_size must be an integer which specifies the size of the array, or a bytes object which will be used to initialize the array items.
see https://docs.python.org/3/library/ctypes.html#ctypes.create_string_buffer
An example could look like this:
from ctypes import *
lib = cdll.LoadLibrary("some.dll")
lib.func.restype = c_long
lib.func.argtype = c_char_p
buf = create_string_buffer(9)
lib.func(buf)
print(buf.value.decode("utf-8"))
A simple test function on the C-side could look like this:
#include <string.h>
static const char some_data[] = "8.07.696";
long func(char *p) {
strcpy(p, some_data);
return 1L;
}
This prints the expected result to the debug console:
8.07.696

Getting error when sending a char array to a DLL call "Don't know how to convert parameter"

I am trying to recreate a DLL function call in Python. The code uses a DLL file to set the laser parameters for various material setting. Here is the original C# code:
public bool ApplyLasFile(string strLasFile)
{
if (!File.Exists(strLasFile))
{
//MessageBox.Show("File not found", "Error", MessageBoxButtons.OK);
return false;
}
Char[] arr = strLasFile.ToCharArray();
ApplyLAS(arr, strLasFile.Length);
return true;
}
Here is my python code:
def setLaserSettings(input):
#laserSettings(power (0-1000), speed (0-1000), height (0-4000), ppi (0-1000))
input=input.upper()
if(input=="ABS"):
settings=laserSettings(825, 1000)
elif(input=="STAINLESS"):
settings=laserSettings(1000, 84)
elif(input=="TITANIUM"):
settings=laserSettings(1000, 84)
else:
return False
charList=[]
for x in range (0, len(settings)): #convert string into c_char[]
charList.append(c_char(settings[x]))
#print charList
print ULSLib.ApplyLAS(charList, len(settings))
The DLL call ULSLib.ApplyLAS(charList, len(settings)) is returning the error ArgumentError: argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1. I started out with just using list(settings) to stand in for ToCharArray(), and when that didn't work I built a c_char array per the Python ctypes man page. However, I am still getting that error. Does anybody see what I am missing? Thanks for any help you can offer.
Edit: I have also tried list(string) and list(charList) and both return the same error.

So this is weird. The function call requires a Char array, but putting the string strait into the function call within python seems to be getting the job done for some reason. I am new to python, so hopefully someday this will make sense to me.

I suspect the reason that your code works unexpectedly is that the C function you are calling in the DLL takes a char * for its first parameter. I'm guessing its declaration looks something like:
bool ApplyLAS(char *filename, int length);
I'm guessing the return type here, and it's also quite possible that the first parameter is actually a wchar_t * or the second is an unsigned int. I'm also assuming your DLL has been compiled from C code. It's possible it may have been compiled from another language such as Fortran instead.
A char * parameter, such as the one I suspect is in your C function, is typically how you pass a string to C. Python's ctypes module will pass a Python string to the C function as a char *, thereby making it easy to pass a string to C.

Python is passing 32bit pointer address to C functions

I would like to call my C functions within a shared library from Python scripts. Problem arrises when passing pointers, the 64bit addresses seem to be truncated to 32bit addresses within the called function. Both Python and my library are 64bit.
The example codes below demonstrate the problem. The Python script prints the address of the data being passed to the C function. Then, the address received is printed from within the called C function. Additionally, the C function proves that it is 64bit by printing the size and address of locally creating memory. If the pointer is used in any other way, the result is a segfault.
CMakeLists.txt
cmake_minimum_required (VERSION 2.6)
add_library(plate MODULE plate.c)
plate.c
#include <stdio.h>
#include <stdlib.h>
void plate(float *in, float *out, int cnt)
{
void *ptr = malloc(1024);
fprintf(stderr, "passed address: %p\n", in);
fprintf(stderr, "local pointer size: %lu\n local pointer address: %p\n", sizeof(void *), ptr);
free(ptr);
}
test_plate.py
import numpy
import scipy
import ctypes
N = 3
x = numpy.ones(N, dtype=numpy.float32)
y = numpy.ones(N, dtype=numpy.float32)
plate = ctypes.cdll.LoadLibrary('libplate.so')
print 'passing address: %0x' % x.ctypes.data
plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))
Output from python-2.7
In [1]: run ../test_plate.py
passing address: 7f9a09b02320
passed address: 0x9b02320
local pointer size: 8
local pointer address: 0x7f9a0949a400

The problem is that the ctypes module doesn't check the function signature of the function you're trying to call. Instead, it bases the C types on the Python types, so the line...
plate.plate(x.ctypes.data, y.ctypes.data, ctypes.c_int(N))
...is passing the the first two params as integers. See eryksun's answer for the reason why they're being truncated to 32 bits.
To avoid the truncation, you'll need to tell ctypes that those params are actually pointers with something like...
plate.plate(ctypes.c_void_p(x.ctypes.data),
ctypes.c_void_p(y.ctypes.data),
ctypes.c_int(N))
...although what they're actually pointers to is another matter - they may not be pointers to float as your C code assumes.
Update
eryksun has since posted a much more complete answer for the numpy-specific example in this question, but I'll leave this here, since it might be useful in the general case of pointer truncation for programmers using something other than numpy.

Python's PyIntObject uses a C long internally, which is 64-bit on most 64-bit platforms (excluding 64-bit Windows). However, ctypes assigns the converted result to pa->value.i, where value is a union and the i field is a 32-bit int. For the details, see ConvParam in Modules/_ctypes/callproc.c, lines 588-607 and 645-664. ctypes was developed on Windows, where a long is always 32-bit, but I don't know why this hasn't been changed to use the long field instead, i.e. pa->value.l. Probably, it's just more convenient most of the time to default to creating a C int instead of using the full range of the long.
Anyway, this means you can't simply pass a Python int to create a 64-bit pointer. You have to explicitly create a ctypes pointer. You have a number of options for this. If you're not concerned about type safety, the simplest option for a NumPy array is to use its ctypes attribute. This defines the hook _as_parameter_ that lets Python objects set how they're converted in ctypes function calls (see lines 707-719 in the previous link). In this case it creates a void *. For example, you'd call plate like this:
plate.plate(x.ctypes, y.ctypes, N)
However, this doesn't offer any type safety to prevent the function from being called with an array of the wrong type, which will result in either nonsense, bugs, or a segmentation fault. np.ctypeslib.ndpointer solves this problem. This creates a custom type that you can use in setting the argtypes and restype of a ctypes function pointer. This type can verify the array's data type, number of dimensions, shape, and flags. For example:
import numpy as np
import ctypes
c_npfloat32_1 = np.ctypeslib.ndpointer(
dtype=np.float32,
ndim=1,
flags=['C', 'W'])
plate = ctypes.CDLL('libplate.so')
plate.plate.argtypes = [
c_npfloat32_1,
c_npfloat32_1,
ctypes.c_int,
]
N = 3
x = np.ones(N, dtype=np.float32)
y = np.ones(N, dtype=np.float32)
plate.plate(x, y, N) # the parameter is the array itself

If you don't tell ctypes what type the parameters are, it attempts to infer it from the values that you pass to the function. And this inference will not always work as you need.
The recommended way to deal with this is to set the argtypes attribute of the function and so explicitly tell ctypes what the parameter types are.
plate.plate.argtypes = [
ctypes.POINTER(ctypes.c_float),
ctypes.POINTER(ctypes.c_float),
ctypes.c_int
]
Then you can call the function like this:
plate.plate(x.ctypes.data, y.ctypes.data, N)

Actually, You should set plate.argstype = [ctypes.c_void_p, ctypes.c_void_p, ctypes.c_int], and then it will be ok to accept the address in c func from python.
I met the problem and I solved it as what I say.