I am implementing a shooting method type problem and i used scipy.optimize.bisect from the scipy module.To achieve higher precision i wanted to go to large iteration numbers, but frequently got the error "unable to parse arguments".
It appears that the scipy function is unable to parse 2147483648=2^31 .
This has to be due to the fact that those large integers are stored as 64 bit instead of 32 bit numbers, but there must be a reason to circumvent this, right? is there anything i can do to have scipy accept large integers?
It seems unlikely that scipy would just straight up breaks down at those iteration numbers.
Help is appreciated!
code example:
#maxN=int(2**31)
maxN=int(2**31-1)
A=0
B=1
scipy.optimize.bisect(lambda x: x**2, a, b,maxiter=maxN)
If I set maxN to a number smaller than 2^31 everything works, but anything bigger than that leads to the error described above.
Under the hood, scipy.optimize.bisect calls the this C function with signature:
double
bisect(callback_type f, double xa, double xb, double xtol, double rtol,
int iter, void *func_data, scipy_zeros_info *solver_stats)
and typedefs
typedef struct {
int funcalls;
int iterations;
int error_num;
} scipy_zeros_info;
typedef double (*callback_type)(double, void*);
typedef double (*solver_type)(callback_type, double, double, double, double,
int, void *, scipy_zeros_info*);
where iter is the maximal number of allowed iterations. Since a int is exactly 32 bits (4 bytes) in size on most platforms, you can't pass a value larger than 2^31-1 as you already observed.
However, you can easily write your own bisect function with Cython. You only need to change the function signatures, i.e. the type of iter, iterations and the loop variable i to long long.
Related
I am fairly new to cython and I am wondering why the following takes very long:
cpdef test(a):
cdef np.ndarray[dtype=int] b
for i in range(10):
b=a
a=np.array([1,2,3],dtype=int)
t = timeit.Timer(functools.partial(test.test, a))
print(t.timeit(1000000))
-> 0.5446977 Seconds
If i comment out the cdef declaration this is done in no-time. If i declare "a" as np.ndarray in the function header nothing changes. Also, id(a) == id(b) so no new objects are created.
Similar behaviour can be observed when calling a function that takes many ndarray as args, e.g.
cpdef foo(np.ndarray a, np.ndarray b,np.ndarray c, ..... )
Can anybody help me? What am i missing here?
Edit:
I noticed the following:
This is slow:
cpdef foo(np.ndarray[dtype=int,ndim=1] a,np.ndarray[dtype=int,ndim=1] b,np.ndarray[dtype=int,ndim=1] c ) :
return
This is faster:
def foo(np.ndarray[dtype=int,ndim=1] a,np.ndarray[dtype=int,ndim=1] b,np.ndarray[dtype=int,ndim=1] c ) :
return
This is the fastest
cpdef foo( a,b,c ) :
return
The function foo() is called very frequently (many million times) in my project from many different locations and does some calculus with the three numpy arrays (however, it doesnt change their content).
I basically need the speed of knowing the data-type inside of the arrays while also having a very low function-call overead. What would be the most adequate solution for this?
b = a generates a bunch of type checking that needs to identify whether the type of a is actually an ndarray and makes sure it exports the buffer protocol with an appropriate element type. In exchange for this one-off cost you get fast indexing of single elements.
If you're not doing indexing of single elements then typing as np.ndarray is literally pointless and you're pessimizing your code. If you are doing this indexing then you can get significant optimizations.
If i comment out the cdef declaration this is done in no-time.
This is often a sign that the C compiler has realized the entire function does nothing and optimized it out completely. And therefore your measurement may be meaningless.
cpdef foo(np.ndarray a, np.ndarray b,np.ndarray c, ..... )
just specifying the type as np.ndarray without specifying the element dtype usually gains you very little, and is probably not worthwhile.
If you have a function that you're calling millions of times then it is likely that the input arrays come from somewhere, and can be pre-typed, probably with less frequency. For example they might come by taking slices from a larger array?
The newer memoryview syntax (int[:]) is quick to slice, so for example if you already have a 2D memoryview (int[:,:] x) it's very quick to generate a 1D memoryview from it with (e.g. x[:,0]), and it's quick to pass existing memoryviews into a cdef function with memoryview arguments. (Note that (a) I'm just unsure if all of this applies to np.ndarray too, and (b) seeing up a fresh memoryview is likely to be about the same cost an an np.ndarray so I'm only suggesting using them because I know slicing is quick).
Therefore my main suggestion is to move the typing outwards to try to reduce the number of fresh initializations of these typed arrays. If that isn't possible then I think you may be stuck.
In coding up a simple Fibonacci script, I found some 'odd' behaviour in how Python treats numpy.int32 vs how it treats regular int numbers.
Can anyone help me understand what causes this behaviour?
Using the Fibonacci code as follows, leveraging caching to significantly speed things up;
from functools import lru_cache
import numpy as np
#lru_cache(maxsize=None)
def fibo(n):
if n <= 1:
return n
else:
return fibo(n-1)+fibo(n-2)
If I define a Numpy array of numbers to calculate over (with np.arange), it all works well until n = 47, then things start going haywire. If, on the other hand, I use a regular python list, then the values are all correctly calculated
You should be able to see the difference with the following;
fibo(np.int32(47)), fibo(47)
Which should return (at least it does for me);
(-1323752223, 2971215073)
Obviously, something very wrong has occured with the calculations against the numpy.int32 input. Now, I can get around the issue by simply inserting a 'n = int(n)' line in the fibo function before anything else is evaluated, but I dont understand why this is necessary.
I've also tried np.int(47) instead of np.int32(47), and found that the former works just fine. However, using np.arange to create the array seems to default to np.int32 data type.
I've tried removing the caching (I wouldn't recommend you try - it takes around 2 hours to calculate to n = 47) - and I get the same behaviour, so that is not the cause.
Can anyone shed some insight into this for me?
Thanks
Python's "integers have unlimited precision". This was built into the language so that new users have "one less thing to learn".
Though maybe not in your case, or for anyone using NumPy. That library is designed to make computations as fast as possible. It therefore uses data types that are well supported by the CPU architecture, such as 32-bit and 64-bit integers that neatly fit into a CPU register and have an invariable memory footprint.
But then we're back to dealing with overflow problems like in any other programming language. NumPy does warn about that though:
>>> print(fibo(np.int32(47)))
fib.py:9: RuntimeWarning: overflow encountered in long_scalars
return fibo(n-1)+fibo(n-2)
-1323752223
Here we are using a signed 32-bit integer. The largest positive number it can hold is 231 - 1 = 2147483647. But the 47th Fibonacci number is even larger than that, it's 2971215073 as you calculated. In that case, the 32-bit integer overflows and we end up with -1323752223, which is its two's complement:
>>> 2971215073 + 1323752223 == 2**32
True
It worked with np.int because that's just an alias of the built-in int, so it returns a Python integer:
>>> np.int is int
True
For more on this, see: What is the difference between native int type and the numpy.int types?
Also note that np.arange for integer arguments returns an integer array of type np.int_ (with a trailing underscore, unlike np.int). That data type is platform-dependent and maps to 32-bit integers on Windows, but 64-bit on Linux.
from math import log,sqrt
import sys
n = 760 ** 890
print(log(n))
I get a valid result.
Now change log by sqrt and you get (as expected):
OverflowError: int too large to convert to float
So I suppose there's a trick for integer arguments in log function, using integer logarithms but I didn't find that in the documentation. There's just this:
math.log(x[, base])
With one argument, return the natural logarithm of x (to base e).
With two arguments, return the logarithm of x to the given base, calculated as log(x)/log(base).
Where is that documented?
I finally dug into python math lib source code and found this:
/* A decent logarithm is easy to compute even for huge ints, but libm can't
do that by itself -- loghelper can. func is log or log10, and name is
"log" or "log10". Note that overflow of the result isn't possible: an int
can contain no more than INT_MAX * SHIFT bits, so has value certainly less
than 2**(2**64 * 2**16) == 2**2**80, and log2 of that is 2**80, which is
small enough to fit in an IEEE single. log and log10 are even smaller.
However, intermediate overflow is possible for an int if the number of bits
in that int is larger than PY_SSIZE_T_MAX. */
static PyObject*
loghelper(PyObject* arg, double (*func)(double), const char *funcname)
{
/* If it is int, do it ourselves. */
if (PyLong_Check(arg)) {
double x, result;
Py_ssize_t e;
...
I'll spare you the rest of the source (check the link), but what I understand from it is that Python checks if the passed argument is integer, and if it is, don't use math lib (If it is int, do it ourselves.) comment. Also: A decent logarithm is easy to compute even for huge ints, but libm can't do that by itself -- loghelper can
If it's a double, then call native math library.
From the source comments, we see that Python tries the hardest to provide the result even in case of the overflow (Here the conversion to double overflowed, but it's possible to compute the log anyway. Clear the exception and continue)
So thanks to the python wrapping of the log function, Python is able to compute logarithm of huge integers (which is specific to some functions, since some others like sqrt cannot do it), and it's documented, but only in the source code, probably making it an implementation detail as Jon hinted.
I think this thread is useful since python now uses long ints, the trick to avoid overflow is the use of _PyLong_Frexp function see here and an alternative formula to compute the log function even after an OverflowError is raised when trying to convert a long int to a Double, check loghelper at this module.
_PyLong_Frexp returns an approximation to the initial long int arg, given inside loghelper with the help of a double x and an exponent e (arg~x*2**e) and the log is calculated as log~log(x*2**e)=log(x)+log(2)*e. I am missing the specifics of the approximation using x,e but you can find it in the implementation of _PyLong_Frexp in the link provided.
I would like to wrap a C function with SWIG.
The function takes a couple arrays (of the same length) as input and returns three more arrays.
It is however not possible to predict the length of the return arrays beforehand and these are dynamically allocated in the function.
Is it possible to wrap such a function with SWIG (using numpy.i) and if so how?
A simplified function declaration looks like:
int func(double **a, double **b, long int *N, double *x, double *y, long int *Nx, long int *Ny);
Where Nx and Ny are known beforehand but N (the length of a and b) is not and a and b are allocated (with malloc) in the function.
It seems that SWIG (or any other Python wrapper generator for that matter) cannot do this.
I ended up writing the Python wrapper by hand, which is actually quite easy, using PyArray_SimpleNew or PyArray_SimpleNewFromData to create the output arrays.
With the latter one has to be extra careful so as to not generate memory leaks.
After playing with it a bit I found the former combined with a simple memcpy to be safer.
I have the function prototype here:
extern "C" void __stdcall__declspec(dllexport) ReturnPulse(double*,double*,double*,double*,double*);
I need to write some python to access this function that is in a DLL.
I have loaded the DLL, but
each of the double* is actually pointing to a variable number of doubles (an array), and
I'm having trouble getting it to function properly.
Thanks all!
To make an array with, say, n doubles:
arr7 = ctypes.c_double * `n`
x = arr7()
and pass x to your function where it wants a double*. Or if you need to initialize x as you make it:
x = arr7(i*0.1 for i in xrange(7))
and the like. You can loop over x, index it, and so on.
I haven't looked at ctypes too much, but try using a numpy array of the right type. If that doesn't just automatically work, they also have a ctypes attribute that should contain a pointer to the data.