I would like to have an array of variable length arrays in ctypes. I know the size of the outer array and all of the inner arrays, too.
I found an interesting thread here:
How do I emulate a dynamically sized C structure in Python using ctypes
But the problem with this is how to create an array of Var classes (see the comment How do I emulate a dynamically sized C structure in Python using ctypes )
Maybe it's something that i cannot do with ctypes at all, i don't really know, i'm getting to know the module only for some hours, any pointers are appreciated.
Thank you!
Dynamically sized data structures are handled in ctypes the same way they are in C; you use pointers to dynamic data. Unfortunately, there's no shortcut on this one. The ctypes documentation includes fairly detailed descriptions on how to handle pointers to dynamic data (such as using a pointer as an arbitrarily sized array). It can be a little hard to grasp at first though. Personally, I've found that creating a few small test applications is helpful in verifing my use of ctypes and dynamic data. It may take some time and a little head-scratching but the interface is pretty flexible so I would expect that you'll be able to accomplish your goal using ctypes.
Related
I can't seem to find the code for numpy argmax.
The source link in the docs lead me to here, which doesn't have any actual code.
I went through every function that mentions argmax using the github search tool and still no luck. I'm sure I'm missing something.
Can someone lead me in the right direction?
Thanks
Numpy is written in C. It uses a template engine that parsing some comments to generate many versions of the same generic function (typically for many different types). This tool is very helpful to generate fast code since the C language does not provide (proper) templates unlike C++ for example. However, it also make the code more cryptic than necessary since the name of the function is often generated. For example, generic functions names can look like #TYPE#_#OP# where #TYPE# and #OP# are two macros that can take different values each. On top of all of this, the CPython binding also make the code more complex since C functions have to be wrapped to be called from a CPython code with complex arrays (possibly with a high amount of dimensions and custom user types) and CPython arguments to decode.
_PyArray_ArgMinMaxCommon is a quite good entry point but it is only a wrapping function and not the main computing one. It is only useful if you plan to change the prototype of the Numpy function from Python.
The main computational function can be found here. The comment just above the function is the one used to generate the variants of the functions (eg. CDOUBLE_argmax). Note that there are some alternative specific implementation for alternative type below the main one like OBJECT_argmax since CPython objects and strings must be computed a bit differently. Thank you for contributing to Numpy.
As mentioned in the comments, you'll likely find what you are searching in the C code implementation (here under _PyArray_ArgMinMaxCommon). The code itself can be very convoluted, so if your intent was to open an issue on numpy with a broad idea, I would do it on the page you linked anyway.
I have a function that takes in a couple of large multi-dimension float arrays as input. As a result, I crash the stack.
I'm a beginner to C/C++ so I will apologize in advance if this question is dumb. Anyways, after looking around the net, this problem is not particularly new and the general solution is either make it as a global variable or use vector (instead of array).
However, this piece of C++ code was intended to be used as a shared library and takes input from a Python script and returns (or modifies) the value (also in arrays). Therefore I don't think it would be possible to declare as global (correct me if I am wrong).
So, apart from using vector, is there any other way to do so? The reason that put me off from using C++ vector is there is no equivalent data type in Python. I'm using ctypes for communication between Python and C++ if that matters.
There is no way in C/C++ to pass an array as a function argument by value. You can pass a pointer to the array and the size of the array. In C++, you also can pass a reference to the array and the size of the array (in template parameters). In either case, the array is not copied and the called function accesses the array in-place, where the caller allocated it.
Note that the array can be wrapped in a structure, in which case it can be passed by value. In C++, std::array is an example of such. But the caller has to initialize that structure instead of the raw array. std::vector is similar in that it is a structure and it automatically copies elements when the vector is copied, but unlike C arrays and std::array it dynamically allocates memory for the elements.
However, if you're integrating with C API, you are most likely limited to a pointer+size or pointer+size wrapped in a C structure solutions. The rules of working with the array (e.g. who allocates and frees the array, are writes to the array allowed, etc.) are specific to the particular API you're working with. Often, there is a dedicated set of functions for working with arrays provided by the library. You should read the API documentation about conventions taken in Python.
I am facing a problem, yesterday I made a post asking how to import a C++ funtion in Python: Post. In this post they suggested me to use Boost Python. So I started to learn it. However all the tutorials are really complicated for me. I know Python language but I am learing C++ so I found it difficult to understand. The other point is that in all the posts that I have found here they talk about 1D vectors in C++, but my function takes 2D vectors.
Indeed all the posts usually employ a C++ class instead of a function. And I don't know anything about classes in C++. But I found it useless in my case since I only whant to evaluate a function and return the result (double) to python. So the first question is. Is It completaley necessary to employ classes instead of functions for Boost python?
As you can see in the other post my function have the following structure:
double many_body_pot(
std::vector< std::vector<double> > &par,
std::vector< std::vector<double> > &geometry,
double x, double y, double z)
{
// ...
}
So it takes 2 2D vectors and 3 doubles as parameters. So what I have learned until now is I have to use #include <boost/python.hpp> in my C++ script and I have to include something like this:
BOOST_PYTHON_MODULE(many_body_pot) {
using namespace boost::python;
def("many_body_pot", many_body_pot);
}
I Python have to send rather 2D ndarrays or 2D lists to the function to be converted to 2D vectors. And if I use 2D ndarrays I will have to use numpy Boost. In my case I don't mind to use one or the other. But I don't understand how to do the conversion to 2D vectors. Could you please give me an easy-to-understand solution for this? It would be really appreciated.
Thank you
C++ has a complicated learning curve for people who only know scripting. C++ has much more freedom than you could ever imagine. This freedom can be a curse for new learners. So unless you dedicate some time to understand how C++ works, you're not only gonna do the job wrong, but you may also do it inefficiently.
I know this might not directly be the answer to your question, but if you want to avoid using classes, then consider using Python's ctypes. You can create a shared library, and then import it in Python.
I, personally, am a big opponent of adding unnecessary libraries unless you have to. And I think the freedom provided by ctypes and the backward/forward compatibility with almost all Python versions and the possibility to decouple your C++ work from Python is priceless. So, consider using ctypes, and then all you have to learn is how to create functions and how to compile them to shared libraries. And since you're a Python expert, you won't have a problem importing that to Python and utilizing it.
I'm using python to set up a computationally intense simulation, then running it in a custom built C-extension and finally processing the results in python. During the simulation, I want to store a fixed-length number of floats (C doubles converted to PyFloatObjects) representing my variables at every time step, but I don't know how many time steps there will be in advance. Once the simulation is done, I need to pass back the results to python in a form where the data logged for each individual variable is available as a list-like object (for example a (wrapper around a) continuous array, piece-wise continuous array or column in a matrix with a fixed stride).
At the moment I'm creating a dictionary mapping the name of each variable to a list containing PyFloatObject objects. This format is perfect for working with in the post-processing stage but I have a feeling the creation stage could be a lot faster.
Time is quite crucial since the simulation is a computationally heavy task already. I expect that a combination of A. buying lots of memory and B. setting up your experiment wisely will allow the entire log to fit in the RAM. However, with my current dict-of-lists solution keeping every variable's log in a continuous section of memory would require a lot of copying and overhead.
My question is: What is a clever, low-level way of quickly logging gigabytes of doubles in memory with minimal space/time overhead, that still translates to a neat python data structure?
Clarification: when I say "logging", I mean storing until after the simulation. Once that's done a post-processing phase begins and in most cases I'll only store the resulting graphs. So I don't actually need to store the numbers on disk.
Update: In the end, I changed my approach a little and added the log (as a dict mapping variable names to sequence types) to the function parameters. This allows you to pass in objects such as lists or array.arrays or anything that has an append method. This adds a little time overhead because I'm using the PyObject_CallMethodObjArgs function to call the Append method instead of PyList_Append or similar. Using arrays allows you to reduce the memory load, which appears to be the best I can do short of writing my own expanding storage type. Thanks everyone!
You might want to consider doing this in Cython, instead of as a C extension module. Cython is smart, and lets you do things in a pretty pythonic way, even though it at the same time lets you use C datatypes and python datatypes.
Have you checked out the array module? It allows you to store lots of scalar, homogeneous types in a single collection.
If you're truly "logging" these, and not just returning them to CPython, you might try opening a file and fprintf'ing them.
BTW, realloc might be your friend here, whether you go with a C extension module or Cython.
This is going to be more a huge dump of ideas rather than a consistent answer, because it sounds like that's what you're looking for. If not, I apologize.
The main thing you're trying to avoid here is storing billions of PyFloatObjects in memory. There are a few ways around that, but they all revolve on storing billions of plain C doubles instead, and finding some way to expose them to Python as if they were sequences of PyFloatObjects.
To make Python (or someone else's module) do the work, you can use a numpy array, a standard library array, a simple hand-made wrapper on top of the struct module, or ctypes. (It's a bit odd to use ctypes to deal with an extension module, but there's nothing stopping you from doing it.) If you're using struct or ctypes, you can even go beyond the limits of your memory by creating a huge file and mmapping in windows into it as needed.
To make your C module do the work, instead of actually returning a list, return a custom object that meets the sequence protocol, so when someone calls, say, foo.getitem(i) you convert _array[i] to a PyFloatObject on the fly.
Another advantage of mmap is that, if you're creating the arrays iteratively, you can create them by just streaming to a file, and then use them by mmapping the resulting file back as a block of memory.
Otherwise, you need to handle the allocations. If you're using the standard array, it takes care of auto-expanding as needed, but otherwise, you're doing it yourself. The code to do a realloc and copy if necessary isn't that difficult, and there's lots of sample code online, but you do have to write it. Or you may want to consider building a strided container that you can expose to Python as if it were contiguous even though it isn't. (You can do this directly via the complex buffer protocol, but personally I've always found that harder than writing my own sequence implementation.) If you can use C++, vector is an auto-expanding array, and deque is a strided container (and if you've got the SGI STL rope, it may be an even better strided container for the kind of thing you're doing).
As the other answer pointed out, Cython can help for some of this. Not so much for the "exposing lots of floats to Python" part; you can just move pieces of the Python part into Cython, where they'll get compiled into C. If you're lucky, all of the code that needs to deal with the lots of floats will work within the subset of Python that Cython implements, and the only things you'll need to expose to actual interpreted code are higher-level drivers (if even that).
i want to implement 1024x1024 monochromatic grid , i need read data from any cell and insert rectangles with various dimensions, i have tried to make list in list ( and use it like 2d array ), what i have found is that list of booleans is slower than list of integers.... i have tried 1d list, and it was slower than 2d one, numpy is slower about 10 times that standard python list, fastest way that i have found is PIL and monochromatic bitmap used with "load" method, but i want it to run a lot faster, so i have tried to compile it with shedskin, but unfortunately there is no pil support there, do you know any way of implementing such grid faster without rewriting it to c or c++ ?
Raph's suggestin of using array is good, but it won't help on CPython, in fact I'd expect it to be 10-15% slower, however if you use it on PyPy (http://pypy.org/) I'd expect excellent results.
One thing I might suggest is using Python's built-in array class (http://docs.python.org/library/array.html), with a type of 'B'. Coding will be simplest if you use one byte per pixel, but if you want to save memory, you can pack 8 to a byte, and access using your own bit manipulation.
I would look into Cython which translates the Python into C that is readily compiled (or compiled for you if you use distutils). Just compiling your code in Cython will make it faster for something like this, but you can get much greater speed-ups by adding a few cdef statements. If you use it with Numpy, then you can quickly access Numpy arrays. The speed-up can be quite large by using Cython in this manner. However, it would easier to help you if you provided some example code.