I've been looking for a simple answer to this question, but it seems that I can't find one. I would prefer to stay away from any external libraries that aren't already included in Python 2.6/2.7.
I have 2 c header files that resemble the following:
//constants_a.h
const double constant1 = 2.25;
const double constant2 = -0.173;
const int constant3 = 13;
...
//constants_b.h
const double constant1 = 123.25;
const double constant2 = -0.12373;
const int constant3 = 14;
...
And I have a python class that I want to import these constants into:
#pythonclass.py
class MyObject(object):
def __init(self, mode):
if mode is "a":
# import from constants_a.h, like:
# self.constant1 = constant1
# self.constant2 = constant2
elif mode is "b":
# import from constants_b.h, like:
# self.constant1 = constant1
# self.constant2 = constant2
...
I have c code which uses the constants as well, and resembles this:
//computations.c
#include <stdio.h>
#include <math.h>
#include "constants_a.h"
// do some calculations, blah blah blah
How can I import the constants from the header file into the Python class?
The reason for the header files constants_a.h and constants_b.h is that I am using python to do most of the calculations using the constants, but at one point I need to use C to do more optimized calculations. At this point I am using ctypes to wrap the c code into Python. I want to keep the constants away from the code just in case I need to update or change them, and make my code much cleaner as well. I don't know if it helps to note I am also using NumPy, but other than that, no other non-standard Python extensions. I am also open to any suggestions regarding the design or architecture of this program.
In general, defining variables in C header file is poor style. The header file should only declare objects, leaving their definition for the appropriate ".c" source code file.
One thing you may want to do is to declare the library-global constants like extern const whatever_type_t foo; and define (or "implement") them (i.e. assigning values to them) somewhere in your C code (make sure you do this only once).
Anyway, let's ignore how you do it. Just suppose you've already defined the constants and made their symbols visible in your shared object file "libfoo.so". Let us suppose you want to access the symbol pi, defined as extern const double pi = 3.1415926; in libfoo, from your Python code.
Now you typically load your object file in Python using ctypes like this:
>>> import ctypes
>>> libfoo = ctypes.CDLL("path/to/libfoo.so")
But then you'll see ctypes thinks libfoo.pi is a function, not a symbol for constant data!
>>> libfoo.pi
<_FuncPtr object at 0x1c9c6d0>
To access its value, you have to do something rather awkward -- casting what ctypes thinks is a function back to a number.
>>> pi = ctypes.cast(foo.pi, ctypes.POINTER(ctypes.c_double))
>>> pi.contents.value
3.1415926
In C jargon, this vaguely corresponds to the following thing happening: You have a const double pi, but someone forces you to use it only via a function pointer:
typedef int (*view_anything_as_a_function_t)(void);
view_anyting_as_a_function_t pi_view = π
What do you do with the pointer pi_view in order to use the value of pi? You cast it back as a const double * and dereference it: *(const double *)(pi_view).
So this is all very awkward. Maybe I'm missing something but this I believe is by design of the ctypes module -- it's there chiefly for making foreign function calls, not for accessing "foreign" data. And exporting pure data symbol in a loadable library is arguably rare.
And this will not work if the constants are only C macro definitions. There's in general no way you can access macro-defined data externally. They're macro-expanded at compile time, leaving no visible symbol in the generated library file, unless you also export their macro values in your C code.
I recommend using regular expressions (re module) to parse the information you want out of the files.
Building a full C parser would be huge, but if you only use the variables and the file is reasonably simple/predictable/under control, then what you need to write is straightforward.
Just watch out for 'gotcha' artifacts such as commented-out code!
I would recommend using some kind of configuration file readable by both Python and C program, rather than storing constant values in headers. E.g. a simple csv, ini-file, or even your own simple format of 'key:value' pairs. And there will be no need to recompile the C program every time you'd like to change one of the values :)
I'd up-vote emilio, but I'm lacking rep!
Although you have requested to avoid other non-standard libraries, you may wish to take a look at Cython (Cython: C-Extensions for Python www.cython.org/), which offers the flexibility of Python coding and the raw speed of execution of C/C++-compiled code.
This way you can use regular Python for everything, but handle the expensive elements of code using its built-in C-types. You can then convert your Python code into .c files too (or just wrap external C-libraries themselves. ), which can then be compiled into a binary. I've achieved up to 10x speed-ups doing so for numerical routines. I also believe NumPy uses it.
Related
I work with Python most of the time, for some reasons now I also need to use C++.
I find Python's import XXX as X very neat in the following way, for example:
import numpy as np
a = np.array([1,2,3])
where I'm very clear by looking at my code that the array() function is provided by the numpy module.
However, when working with C++, if I do:
#include<cstdio>
std::remove(filename);
It's not clear to me at first sight that remove() function under the std namespace is provided by <cstdio>.
So I'm wondering if there is a way to do it in C++ as the import XXX as X way in Python?
Nope.
It'll be slightly clearer if you write std::remove (which you should be doing anyway; there's no guarantee that the symbol is available in the global namespace) because then at least you'll know it comes from a standard header.
Beyond that, it's up to your memory. 😊
Some people try to introduce hacks like:
namespace SomeThing {
#include <cstdio>
}
// Now it's SomeThing::std::remove
That might work for your own headers (though I'd still discourage it even then). But it'll cause all manner of chaos with standard headers for sure and is not permitted:
[using.headers]/1: The entities in the C++ standard library are defined in headers, whose contents are made available to a translation unit when it contains the appropriate #include preprocessing directive.
[using.headers]/3: A translation unit shall include a header only outside of any declaration or definition, and shall include the header lexically before the first reference in that translation unit to any of the entities declared in that header. No diagnostic is required.
Recall that #include and import are fundamentally different things. C++ modules may go some way towards this sort of functionality, perhaps, but by including source code you are not even touching namespaces of symbols created by that code.
No there is no way to force this syntax. The person who developped the code that you include is free. Generally people split their code into namespaces, which can result to this syntax:
#include <MyLibrary.h>
int main()
{
MyLibrary::SayHello();
return 0;
}
But you have no guarentee on how the code in the header is written.
C++ #include<XXX.h> equivalent of Python's import XXX as X
There is no equivalent in C++.
When you include a file into another, you get every single declaration from the included file, and you have no option of changing their names.
You can add aliases for types and namespaces though, and references to objects, as well as write wrapper functions to do some of what the as X part does in Python.
It's not clear to me at first sight that remove() is provided by <cstdio>.
The std namespace at least tells you that it is provided by the standard library.
What I like to do, is document which header provides the used declarations:
#include<cstdio> // std::remove
std::remove(filename);
That said, most IDE's can show you where an identifier is declared by ctrl-clicking or hovering over it (although this doesn't always work well when there are overloads in different headers). My primary use for inclusion comments is checking which includes can be removed after refactoring.
I'm trying to talk to this DLL using python's ctypes. Many of the functions take or return an HGRABBER type:
typedef struct HGRABBER_t__ { int unused; } HGRABBER_t;
#define HGRABBER HGRABBER_t*
(the full header file can be viewed here). Here's an example of a function prototype that returns an HGRABBER type:
HGRABBER __stdcall IC_CreateGrabber();
Here's my attempt at implementing this struct in python, and using it to call that function from the DLL:
import ctypes as C
class GrabberHandle(C.Structure):
_fields_ = [('unused', C.c_int)]
dll = C.windll.LoadLibrary('tisgrabber_x64.dll')
dll.create_grabber = dll.IC_CreateGrabber
dll.create_grabber.argtypes = []
dll.create_grabber.restype = GrabberHandle
my_handle = dll.create_grabber()
This seems to work, but I'm worried that I'm doing this wrong. I'm not experienced with C, and I don't think I understand the typedef and #define statements which define the HGRABBER type. Am I calling IC_CreateGrabber correctly? Should I have defined GrabberHandle to be a pointer to a struct, instead of a struct?
Thanks for reading, please let me know if I can clarify my question somehow.
You're right that you actually want a POINTER to the Structure, not the Structure itself.
Translating the C into English, being very loose (in a way that would be dangerous if you were trying to learn C but is good enough for using ctypes):
The struct defines a type named struct HGRABBER_t__, as a structure with one int in it.
The typedef defines a type named HGRABBER_t, as a synonym for struct HGRABBER_t__.
The #define defines a type named HGRABBER as a pointer to HGRABBER_t.
So, your GrabberHandle is the equivalent of HGRABBER_t; the equivalent of HGRABBER is:
GrabberHandlePtr = C.POINTER(GrabberHandle)
So you want this:
dll.create_grabber.restype = GrabberHandlePtr
It may be hard to debug the difference. A C struct with nothing but an int in it looks identical to an int in memory. And on Win32, an int and a pointer are both 32-bit values. And an int named unused is likely to be filled with meaningless garbage, making it hard to distinguish it from a pointer you've accidentally treated as an int. So everything will look fine, until you segfault 30 lines later in your code and have no idea what's wrong. :)
This library does what you are trying to do: https://github.com/morefigs/py-ic-imaging-control :)
But to answer your question, the library uses the code:
from ctypes import *
import os
class GrabberHandle(Structure):
pass
GrabberHandle._fields_ = [('unused', c_int)]
# set and check path
dll_path = os.path.join(os.path.expanduser('~'),
'Documents\\The Imaging Source Europe GmbH\\TIS Grabber DLL\\bin\\win32\\tisgrabber.dll')
with open(dll_path) as thefile:
pass
# open DLL
_ic_grabber_dll = windll.LoadLibrary(dll_path)
# create grabber
create_grabber = _ic_grabber_dll.IC_CreateGrabber
create_grabber.restype = POINTER(GrabberHandle)
create_grabber.argtypes = None
# get handle
handle = create_grabber()
Edit: changed code to use a pointer to GrabberHandle as per abarnert's answer as this is correct. However, in this particular case I have found no practical difference (with the 32-bit DLL), probably because the GrabberHandle structure is so simple.
I started on my first Python extension today and was only creating a very small wrapper around a C library as an exercise. As is typical with C libraries, you start of with an initialization function that yields a handler. You can pass that handler to functions and later you pass it to the cleanup function that frees memory.
When I started writing the wrapper I basically wanted to have a way to call each native C function from python. Quickly I hit the problem that I need to return an arbitrary pointer from C to Python only to give it from there to C again in another function. I doesn't matter how it looks as I don't use it in Python, I just store it and pass it around.
So how do you pass around a void pointer between Python and C?
Please note: I know it is not recommended to write such small wrappers using the extension system but rather ctypes and friends. This is just for practice right now.
PyLong_FromVoidPtr() and PyLong_AsVoidPtr() can be abused to inject malicious data into your program. I recommend against them.
Python has PyCapsule for exactly that job. Capsules provide a safe way to exchange void ptr between modules or Python space and C space. The capsules are type-safe, too. If you need some example, the socket / ssl modules and pyexpat / _elementtree modules use capsules to exchange CAPI structs.
http://docs.python.org/3/c-api/capsule.html
After some searching I found the functions PyLong_AsVoidPtr and PyLong_FromVoidPtr. This yields a nice way to convert between a void * and a PyObject:
# in init function
return PyLong_FromVoidPtr(handle);
# in function using handle
handle = PyLong_AsVoidPtr(python_handle);
The one problem now might be how to retrieve python_handle from the typical *args given to a function:
PyObject *python_handle;
PyArg_ParseTuple(args, "O", &python_handle);
Careful here: The argument given for the "O" object must be a pointer to a PyObject pointer: PyObject **. The "O" itself only denotes to pass this PyObject through without any handling and converting. And with this, you can pass around any pointers any way you like.
Note: I think this solution is not really pretty, because you now have to variables, one that is only needed for a short time.
I have dll, that builded in Delphi, and I try to call function from it. Declaration of function looks like this:
function GetUid(UID:Pointer):Integer; stdcall;
This is equivalent to this C function signature:
int GetUID(void *pointer);
Library handled using ctypes:
from ctypes import *
lib = cdll.LoadLibrary("mylib.dll")
But i stuck here:
res = lib.GetUid(?)
What I need to pass in this function?
Pointer is void *, but how make this rightly in python?
Python is a high level language. You do not typically import a DLL from a C or Pascal native library and invoke it and pass variables from Python into a C or Pascal function taking a void * type raw pointer and then manipulate raw memory this way.
In short if you knew what you were doing you would know better than to try to do what you're doing here.
Let's suppose that your implementation is like this:
function GetUid(UID:Pointer):Integer; stdcall;
var
P2:^Integer;
begin
P2 := UID;
P2^ := 0;
end;
Then, what you would want to do is pass in an address to a 32 bit integer. Of course my example above is absurd, because what would have made sense above is to just declare the parameter as an "int *pointer" (in C terms) rather than as a "void *pointer".
Whatever it is you're doing, the next thing that will likely happen is that you will corrupt your python interpreter's heap, and cause lots of fun crashes and errors.
A far more sensible approach is to read the Python documentation on writing C extensions that can manipulate native Python types (PyObject), and doing the same thing but in pascal, if you like.
p4d appears to be a workable way of writing extension DLLs in delphi:
https://code.google.com/p/python4delphi/source/list
I have statically declared a large structure in C, but I need to use this same data to do some analysis in Python. I'd rather not re-copy this data in to Python to avoid errors, is there a way to access (read only) this data directly in Python? I have looked at "ctypes" and SWIG, and neither one of them seems to provide what I'm looking for....
For example I have:
/* .h file */
typedef struct
{
double data[10];
} NestedStruct;
typedef struct
{
NestedStruct array[10];
} MyStruct;
/* .c file */
MyStruct the_data_i_want =
{
{0},
{
{1,2,3,4}
},
{0},
};
Ideally, I'd like something that would allow me to get this into python and access it via the_data_i_want.array[1].data[2] or something similar. Any thoughts? I got swig to "work" in the sense that I was able to compile/import a .so created from my .c file, but I couldn't access any of it through cvars. Maybe there's another way? It seems like this should't be that hard....
Actually, I figured it out. I'm adding this because my reputation does not allow me to answer my own question within 8 hours, and since I don't want to have to remember in 8 hours I will add it now. I'm sure there's a good reason for this that I don't understand.
Figured it out.
1st I compiled my .c file into an library:
Then, I used types to define a python class that would hold the data:
from ctypes import *
class NestedStruct(Structure):
_fields_ = [("data", c_double*10)]
class MyStruct(Structure):
_fields_ = [("array", NestedStruct*10)]
Then, I loaded the shared library into python:
my_lib = cdll.LoadLibrary("my_lib.so")
Then, I used the "in_dll" method to get the data:
the_data_i_want = MyStruct.in_dll(my_lib, "the_data_i_want")
Then, I could access it as if it were C. the_data_i_want.array[1].data[2]
Note I may have messed up the syntax slightly here because my actual data structure is nested 3 levels and I wanted to simplify for illustration purposes here.
You could've also in C read the data and written to a JSON-File, which you could then easily parse (usually there's a library which will even do that for you; python import json) and access form any different platform with almost every language setup you could think of. And at the same time you could've accessed you're data very similar compared to how you accessed it within you're original C code.
Just as a suggestion. This would make you're data also more portable and versatile I think, but you'll spend more time on writing and parsing the JSON as if you just read the stream of data directly from you're C code into python.