I'm trying to initialize two vectors in memory using gsl_vector_set(). In the main code it is initialized to zero on default, but I wanted to initialize them to some non-zero value. I made a test code based on a working function that uses the gsl_vector_set() function.
from ctypes import *;
gsl = cdll.LoadLibrary('libgsl-0.dll');
gsl.gsl_vector_get.restype = c_double;
gsl.gsl_matrix_get.restype = c_double;
gsl.gsl_vector_set.restype = c_double;
foo = dict(
x_ht = [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0],
x_ht_m = [0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,
0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0]
);
for f in range(0,18):
gsl.gsl_vector_set(foo['x_ht_m'],f,c_double(1.0));
gsl.gsl_vector_set(foo['x_ht'],f,c_double(1.0));
When I run the code I get this error.
ArgumentError: argument 1: <type 'exceptions.TypeError'>: Don't know how to convert parameter 1
I'm new to using ctypes and gsl functions so I'm not sure what the issue is or what the error message means. I an also not sure if there is a better way that I should be trying to save a vector to memory
Thank you #CristiFati for pointing out that I needed gsl_vector_calloc in my test code. I noticed that in the main code I was working in that the vector I needed to set was
NAV.KF_dictnry['x_hat_m']
instead of
NAV.KF_dictnry['x_ht_m']
So I fixed the test code to mirror the real code a bit better by creating a class holding the dictionary, and added the ability to change each value in the vector to an arbitrary value.
from ctypes import *;
gsl = cdll.LoadLibrary('libgsl-0.dll');
gsl.gsl_vector_get.restype = c_double;
gsl.gsl_matrix_get.restype = c_double;
gsl.gsl_vector_set.restype = c_double;
class foo(object):
fu = dict(
x_hat = gsl.gsl_vector_calloc(c_size_t(18)),
x_hat_m = gsl.gsl_vector_calloc(c_size_t(18)),
);
x_ht = [1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]
x_ht_m = [1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,
1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]
for f in range(0,18):
gsl.gsl_vector_set(foo.fu['x_hat_m'],f,c_double(x_ht_m[f]));
gsl.gsl_vector_set(foo.fu['x_hat'],f,c_double(x_ht[f]));
After running I checked with:
gsl.gsl_vector_get(foo.fu['x_hat_m'],0)
and got out a 1.0 (worked for the entire vector).
Turned out to just be some stupid mistakes on my end.
Thanks again!
Related
I meet a problem about using numba jit decorator (#nb.jit)! Here is the warning from jupyter notebook,
NumbaWarning:
Compilation is falling back to object mode WITH looplifting enabled because Function "get_nb_freq" failed type inference due to: No implementation of function Function(<function dotenter image description here at 0x00000190AC399B80>) found for signature:
This is complete information:
Here is my code:
#numba.jit
def get_nb_freq( nb_count = None, onehot_ct = None):
# nb_freq = onehot_ct.T # nb_count
nb_freq = np.dot(onehot_ct.T, nb_count)
res = nb_freq/nb_freq.sum(axis = 1).reshape(Num_celltype,-1)
return res
## onehot_ct is array, and its shape is (921600,4)
## nb_count is array, and its shape is the same as onehot_ct
## Num_celltype equals 4
Based on your mentioned shapes we can create the arrays as:
onehot_ct = np.random.rand(921600, 4)
nb_count = np.random.rand(921600, 4)
Your prepared code will be work correctly and get an answer like:
[[0.25013102754197963 0.25021461207825463 0.2496806287276126 0.24997373165215303]
[0.2501574139037384 0.25018726649940737 0.24975108864220968 0.24990423095464467]
[0.25020550587624757 0.2501303498983212 0.24978335463279314 0.24988078959263807]
[0.2501855533482036 0.2500913419625523 0.24979681404573967 0.24992629064350436]]
So, it shows the code is working and the problem seems to be related to type of the arrays, that numba can not recognize them. So, signature may be helpful here, which by we can recognize the types manually for the function. So, based on the error I think the following signature will pass your issue:
#nb.jit("float64[:, ::1](float64[:, ::1], float32[:, ::1])")
def get_nb_freq( nb_count = None, onehot_ct = None):
nb_freq = np.dot(onehot_ct.T, nb_count)
res = nb_freq/nb_freq.sum(axis=1).reshape(4, -1)
return res
But it will stuck again if you test by get_nb_freq(nb_count.astype(np.float64), onehot_ct.astype(np.float32)), So another cause could be related to unequal types in np.dot. So, use the onehot_ct array as array type np.float64, could pass the issue:
#nb.jit("float64[:, ::1](float64[:, ::1], float32[:, ::1])")
def get_nb_freq( nb_count, onehot_ct):
nb_freq = np.dot(onehot_ct.astype(np.float64).T, nb_count)
res = nb_freq/nb_freq.sum(axis=1).reshape(4, -1)
return res
It ran on my machine with this correction. I recommend write numba equivalent codes (like this for np.dot) instead using np.dot or …, which can be much faster.
I'm in the process of developing a PDE package (similar to fenics) and am trying to come up with the best way for a user to define their functions. The current method I am using is fine. However, as the package grows the run-time compilation of the functions will take much longer.
The current method takes in a dictionary of function specs and redefines and recompiles the global default. As the package gets larger, it would be ideal if I could have everything compiled ahead of time; the problem is for numba's pycc AOT compilation, you can't pass in a function because there is no recognizable signature type (that I could find in the documentaion). I have also tried using cython but the function stays as a weak reference to a pyobject which ruins the performance.
The code in the module where the function exists looks like
import numpy as np
from numba import njit
#njit
def _user_defined_function(z, t):
return -z
def define_vector_func(func_specs, **numba_kwargs):
# func_specs is a dict, numba_kwargs are for jit decorator
# create a string to be compiled into a new function
exprs = []
for var in func_specs:
s = func_specs[var]
for i, (variable, expression) in enumerate(func_specs.items()):
s = s.replace(variable, f'z[{i}]')
exprs.append(s)
fp = '<ipython-cache-safe>' # needed to allow for caching
src = ', '.join(exprs)
src = f'def local_func(z, t): return np.array([{src}])\n'
lines = [src]
linecache.cache[fp] = (len(src), None, lines, fp)
code = compile(src, fp, 'exec')
exec(code)
# redefine global function and return a reference if user wants to use it
local_func_ptr = vars()['local_func']
local_func_ptr(np.ones(len(func_specs)), 0.0)
global _user_defined_function
_user_defined_function = njit(**numba_kwargs)(local_func_ptr)
_rec() # recompile other functions to recognize new function
return _user_defined_function
# rest of module
# ...
To use it from a separate module, you would do something like
from module_above import define_vector_func, other_funcs_from_module
if __name__ == '__main__':
func_spec = {'v': 'v - (v * v * v) / 3.0 - w + 0.08',
'w': '0.08 * (v - 0.8 * w + 0.7)'}
f = define_vector_func(func_spec, fastmath=True, cache=True)
# do what user wants
What would be the best route for compiling the module ahead of time and maintain performance?
I am trying to load a DLL in Python 2.7 using ctypes. The DLL was written using Fortran and has multiple subroutines in it. I was able to successfully set up couple of the exported functions that that long and double pointers as arguments.
import ctypes as C
import numpy as np
dll = C.windll.LoadLibrary('C:\\Temp\\program.dll')
_cp_from_t = getattr(dll, "CP_FROM_T")
_cp_from_t.restype = C.c_double
_cp_from_t.argtypes = [C.POINTER(C.c_longdouble),
np.ctypeslib.ndpointer(C.c_longdouble)]
# Mixture Rgas function
_mix_r = getattr(dll, "MIX_R")
_mix_r.restype = C.c_double
_mix_r.argtypes = [np.ctypeslib.ndpointer(dtype=C.c_longdouble)]
def cp_from_t(composition, temp):
""" Calculates Cp in BTU/lb/R given a fuel composition and temperature.
:param composition: numpy array containing fuel composition
:param temp: temperature of fuel
:return: Cp
:rtype : float
"""
return _cp_from_t(C.byref(C.c_double(temp)), composition)
def mix_r(composition):
"""Return the gas constant for a given composition.
:rtype : float
:param composition: numpy array containing fuel composition
"""
return _mix_r(composition)
# At this point, I can just pass a numpy array as the composition and I can get the
# calculated values without a problem
comps = np.array([0, 0, 12.0, 23.0, 33.0, 10, 5.0])
temp = 900.0
cp = cp_from_t(comps, temp)
rgas = mix_r(comps)
So far, so good.
The problem arises when I try another subroutine called Function2 which needs some strings as input. The strings are all fixed length (255) and they also ask for the length of each of the string parameters.
The function is implemented in Fortran as follows:
Subroutine FUNCTION2(localBasePath,localTempPath,InputFileName,Model,DataArray,ErrCode)
!DEC$ ATTRIBUTES STDCALL,REFERENCE, ALIAS:'FUNCTION2',DLLEXPORT :: FUNCTION2
Implicit None
Character *255 localBasePath,localTempPath,InputFileName
Integer *4 Model(20), ErrCode(20)
Real *8 DataArray(900)
The function prototype in Python is set up as follows
function2 = getattr(dll, 'FUNCTION2')
function2.argtypes = [C.POINTER(C.c_char_p), C.c_long,
C.POINTER(C.c_char_p), C.c_long,
C.POINTER(C.c_char_p), C.c_long,
np.ctypeslib.ndpointer(C.c_long , flags='F_CONTIGUOUS'),
np.ctypeslib.ndpointer(C.c_double, flags='F_CONTIGUOUS'),
np.ctypeslib.ndpointer(C.c_long, flags='F_CONTIGUOUS')]
And I call it using:
base_path = "D:\\Users\\xxxxxxx\\Documents\\xxxxx\\".ljust(255)
temp_path = "D:\\Users\\xxxxxxx\\Documents\\xxxxx\\temp".ljust(255)
inp_file = "inp.txt".ljust(255)
function2(C.byref(C.c_char_p(base_path)),
C.c_long(len(base_path)),
C.byref(C.c_char_p(temp_dir)),
C.c_long(len(temp_dir))),
C.byref(C.c_char_p(inp_file)),
C.c_long(len(inp_file)),
model_array,
data_array,
error_array)
The strings are essentially paths. The function Function2 does not recognize the paths and proves an error message with some non-readable characters at the end, such as:
forrtl: severe (43): file name specification error, unit 16, D:\Users\xxxxxxx\Documents\xxxxx\ωa.
What I wanted the function to receive was D:\Users\xxxxxxx\Documents\xxxxx\. Obviously, the strings are not passed correctly.
I have read that Python uses NULL terminated strings. Can that be a problem while passing strings to a Fortran dll? If so, how do I get around it?
Any recommendations?
Following comment from #eryksun, I made the following changes to make it work.
Changed the argtypes to:
function2 = getattr(dll, 'FUNCTION2')
function2.argtypes = [C.c_char_p, C.c_long,
C.c_char_p, C.c_long,
C.c_char_p, C.c_long,
np.ctypeslib.ndpointer(C.c_long , flags='F_CONTIGUOUS'),
np.ctypeslib.ndpointer(C.c_double, flags='F_CONTIGUOUS'),
np.ctypeslib.ndpointer(C.c_long, flags='F_CONTIGUOUS')]
And instead of passing the string as byref, I changed it to the following.
base_path = "D:\\Users\\xxxxxxx\\Documents\\xxxxx\\".ljust(255)
temp_path = "D:\\Users\\xxxxxxx\\Documents\\xxxxx\\temp".ljust(255)
inp_file = "inp.txt".ljust(255)
function2(base_path, len(base_path), temp_dir, len(temp_dir), inp_file, len(inp_file),
model_array, data_array, error_array)
It was sufficient to pass the values directly.
I want to calculate the sumproduct of two arrays in Theano. Both arrays are declared as shared variables and are the result of prior computations. Reading the tutorial, I found out how to use scan to compute what I want using 'normal' tensor arrays, but when I tried to adapt the code to shared arrays I got the error message TypeError: function() takes at least 1 argument (1 given). (See minimal running code example below)
Where is the mistake in my code? Where is my misconception? I am also open to a different approach for solving my problem.
Generally I would prefer a version which takes the shared variables directly, because in my understanding, converting the arrays first back to Numpy arrays and than again passing them to Theano, would be wasteful.
Error message producing sumproduct code using shared variables:
import theano
import theano.tensor as T
import numpy
a1 = [1,2,4]
a2 = [3,4,5]
Ta1_shared = theano.shared(numpy.array(a1))
Ta2_shared = theano.shared(numpy.array(a2))
outputs_info = T.as_tensor_variable(numpy.asarray(0, 'float64'))
Tsumprod_result, updates = theano.scan(fn=lambda Ta1_shared, Ta2_shared, prior_value:
prior_value + Ta1_shared * Ta2_shared,
outputs_info=outputs_info,
sequences=[Ta1_shared, Ta2_shared])
Tsumprod_result = Tsumprod_result[-1]
Tsumprod = theano.function(outputs=Tsumprod_result)
print Tsumprod()
Error message:
TypeError: function() takes at least 1 argument (1 given)
Working sumproduct code using non-shared variables:
import theano
import theano.tensor as T
import numpy
a1 = [1, 2, 4]
a2 = [3, 4, 5]
Ta1 = theano.tensor.vector("a1")
Ta2 = theano.tensor.vector("coefficients")
outputs_info = T.as_tensor_variable(numpy.asarray(0, 'float64'))
Tsumprod_result, updates = theano.scan(fn=lambda Ta1, Ta2, prior_value:
prior_value + Ta1 * Ta2,
outputs_info=outputs_info,
sequences=[Ta1, Ta2])
Tsumprod_result = Tsumprod_result[-1]
Tsumprod = theano.function(inputs=[Ta1, Ta2], outputs=Tsumprod_result)
print Tsumprod(a1, a2)
You need to change the compilation line to this one:
Tsumprod = theano.function([], outputs=Tsumprod_result)
theano.function() always need a list of inputs. If the function take 0 input, like in this case, you need to give an empty list for the inputs.
I am writing swig bindings for some c functions. One of these functions takes a float**. I am already using cpointer.i for the normal pointers and looked into carrays.i, but I did not find a way to declare a float**. What do you recommend?
interface file:
extern int read_data(const char
*file,int *n_,int *m_,float **data_,int **classes_);
This answer is a repost of one to a related question Framester posted about using ctypes instead of swig. I've included it here in case any web-searches turn up a link to his original question.
I've used ctypes for several projects
now and have been quite happy with the
results. I don't think I've personally
needed a pointer-to-pointer wrapper
yet but, in theory, you should be able
to do the following:
from ctypes import *
your_dll = cdll.LoadLibrary("your_dll.dll")
PFloat = POINTER(c_float)
PInt = POINTER(c_int)
p_data = PFloat()
p_classes = PInt()
buff = create_string_buffer(1024)
n1 = c_int( 0 )
n2 = c_int( 0 )
ret = your_dll.read_data( buff, byref(n1), byref(n2), byref(p_data), byref(p_classes) )
print('Data: ', p_data.contents)
print('Classes: ', p_classes.contents)