I have a header file with struct definitions that I'd like to be able to parse in python. In order to do this I turned to Swig.
Lets say the header file is named "a.h". I first renamed it to "a.c" and added an empty "a.h" file in the same folder.
Next, I added in an "a_wrap.i" file with the following contents.
%module a
%{
/* the resulting C file should be built as a python extension */
#define SWIG_FILE_WITH_INIT
/* Includes the header in the wrapper code */
#include "a.h"
%}
/* Parse the header file to generate wrappers */
%include "a.h"
extern struct a_a;
extern struct a_b;
extern struct a_c;
Next, I wrote a setup.py file as follows :
from distutils.core import setup, Extension
setup(ext_modules=[Extension("_a",
sources=["a.c", "a_wrap.i"])])
Next, I did the build as
python setup.py build_ext --inplace
I finally tried to import it in python
>>> import a # it works, yaay
>>> dir(a)
...
...
I was hoping for a way to access the structs defined in "a.c"(originally a.h). However, I don't seem to be able to find a way to do that. How can I solve this? I'm looking for a way to access the struct's defined in the header file from python.
The global variables a_a, a_b anda_cshould be accessible from [within your SWIG Python module viacvar`]1:
import a
print a.cvar.a_a
print a.cvar.a_b
# etc.
Related
Using SWIG to generate a Python binding for my C++ project has not been easy but I have finally been able to do so. The only issue is that the generated .py file that houses essentially the class and method definitions of my wrapped C++ code (but callable for Python) is quite large. I basically want to modularize the generated .py file into submodules of relevant classes.
Here is a basic and stripped down sample of what my swig interface file looks like:
%module example
%{
/* these two headers should belong to ModuleOne */
#include "header1.hpp"
#include "header2.hpp"
/* these two headers should belong to ModuleTwo */
#include "header3.hpp"
#include "header4.hpp"
}
%include "header1.hpp"
%include "header2.hpp"
%include "header3.hpp"
%include "header4.hpp"
And from Python importing the package would be done like so:
from example import *
I find this messy as I either need to import each class individually with from example import ClassOne or import the entirety of the module.
How could I go about creating "submodules" of the swig generated .py file allowing me to modularize my project a bit cleaner and import those without necessarily importing the entire package. For example something like:
import example.ModuleOne
import example.ModuleTwo
I think you just need to add an __init__.py file that imports both modules, something like this:
from example.ModuleOne import *
from example.ModuleTwo import *
__all__ = [x for x in dir() if x[0] != '_']
The last line allows your program to use from example import * to import everything.
Edit: I've just read your question a bit more closely and realise you want to import just one of your submodules. You still need an __init__.py file to make your two modules into a package, but it can be empty. Your interface files should include a package declaration, e.g. %module(package="example") ModuleOne.
i have a SWIG C-python interface. I want to make individual functions from test.c available in my python interface. I feel like i am wasting hrs on the documentation without finding the simple way to do it. The only options i know are to either:
%module test
%{
#include "test.h"
//#include "test.c"
%}
%include "stdint.i" # Add support for uint8_t, int32_t, etc...
/** Keep test immutable for now because we have const struct members in the
* header file which can not be reassigned.
*/
%immutable;
%include test.h // Test header for interface debugging
//%include test.c
//%import test.h
%mutable;
I tried either #including 'test.c' in the %{} section of my interface (This section just copies the file content into the target interface), %include 'test.c' or %import 'test.c'
But i have a whole bunch of other functions in 'test.c' that have dependencies to other files and i cannot import the generated interface in python (ImportError: Undefined symbol:)
#include "test.h"
#include "someotherfunctions.h"
#include "specialdefines.h"
int testFunc(int n, int m) // The function i want to use
{
int res = n + m;
return res;
}
int otherFunc(void)
{
otherDifferentFunc(); // Stuff from other header/c files which my interface doesnt know
}
SPECIALMACRO(X) // More stuff from other headers that i dont want in my interface
int foo = someOtherOperation(500);
The header file test.h:
/* Functions ----------------------------------------------------------------*/
int testFunc(int n, int m);
I feel like i am not having the right approach to make single functions available in my Python interface without including all the unwanted stuff from all the headers in my source file. In reality, my 'test.c' is quite large and includes a whole bunch of header for all sorts of symbols and external functions used. If i dont #include all oft those in my %{} interface section, i get the import error. But if i include them, they again require other headers (its a large project) and i am starting a messy chain of includes for just a tiny function, unnecessary bloating my interface. This can't be the SWIG way to go guys?
If someone knows a good way to solve this, please let me know.
%module test
%{
// Whatever goes here is directly copied to the generated wrapper code.
// Put what you need here for the wrapper to build, typically just
// your public headers.
#include "test.h"
%}
// Anything you declare or %import is processed to make the SWIG interface.
// You can %import "test.h" and all the functions *directly* in
// the header (not other #include in the header), are exposed to Python.
//
// If you only want one function, just put its prototype, like:
int testFunc(int n, int m);
When you build the wrapper, swig the .i file to generate test_wrap.c, then compile test.c and test_wrap.c and link them together at the link step.
An example:
test.i
%module test
%{
#include "test.h"
%}
// uncomment to bring in func1 and func2 (not func3, but it's still used by func2)
//%include "test.h"
// only exposes func1
int func1();
test.h
int func1();
int func2();
test.c
#include "test.h"
int func3() { return 3; }
int func2() { return func3() - 1; }
int func1() { return func2() - 1; }
makefile (Windows, Microsoft compiler)
all: _test.pyd
test_wrap.c test.py: test.i test.h
echo swigging...
swig -python test.i
_test.pyd: test_wrap.c test.c test.h
cl /nologo /LD /MD /W3 /Fe_test.pyd /Id:\dev\python310\include test_wrap.c test.c -link /libpath:d:\dev\python310\libs
Output (built as above, only func1() exposed):
>>> import test
>>> test.func1
<function func1 at 0x000001CF6DEF7F40>
>>> test.func1()
1
>>> test.func2()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'test' has no attribute 'func2'. Did you mean: 'func1'?
Output (built uncommenting the %include and commenting the prototype):
>>> import test
>>> test.func1()
1
>>> test.func2()
2
>>> test.func3()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'test' has no attribute 'func3'. Did you mean: 'func1'?
I am currently programming a program which searches song according to diffrent
parameters.
In my system there are 2 types of songs: lyric and instrumetal.
Since i need to put both of them in 1 vector, I have a song class and
a LyricsSong & InstrumentalSong subclasses.
So I there is a Song.h file:
#include <stdio.h>
#include <iostream>
#include <string>
class Song
{
public:
std::string title;
virtual void print();
virtual void printSong(std::string query);
};
and there are the instrumental and lyrics subclasses, which are defined this way:
class LyricsSong : public Song
class InstrumentalSong : public Song
both of the include Song.h, and in both of them the class is defines
only in the header file.
when I try to run another file which use the both subclasses,
and includes:
#include "LyricsSong.h"
#include "InstrumentalSong.h"
(and obviously more cpp libraries), i get the following compilation error:
In file included from /cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/InstrumentalSong.h:16:0,
from /cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/songsParser.cpp:26:
/cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/Song.h:6:7: error: redefinition of 'class Song'
class Song
^
In file included from /cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/LyricsSong.h:15:0,
from /cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/songsParser.cpp:25:
/cygdrive/c/Users/Username/Documents/C++ Workshop/ex2/ex2_code/Song.h:6:7: error: previous definition of 'class Song'
class Song
^
when:
lines InstrumentalSong.h:16:0 and LyricsSong.h:15:0 are where i
include "Song.h"
lines songsParser.cpp:25 and songsParser.cpp:26 are where i
include InstrumentalSong.h and LyricsSong.h
line Song.h:6:7: is the defination of Song.h (where it's say's
class Song, as showed above).
What should I do?
P.S. I do not import any cpp file ever, only header files.
You have to tell preprocessor to include your header files only once. You can achieve by adding #pragma once on the top of all *.h files:
#pragma once
//Your header file's code
It is also a good practice to always begin header files with this line.
They both include 'Song.h' file and preprocessor takes the file content twice.
You need to write 'LyricsSong.h' and 'InstrumentalSong.h' file contents inside #ifndef #define and #endif directives. Like this
#ifndef LYRICS_SONG_H
#define LYRICS_SONG_H
your code goes here.
...
#endif
As already answered, I would also use #pragma once, it is more convenient and clean. But be aware that it is not a C++ standard, so it can be a problem if you have to use different compilers (although it is a wide-spread extension).
This is very related to this question
Regardless of whether or not this is coding practice, I have come across code that looks like this
test.hh
#include <vector>
using std::vector;
class Test
{
public:
vector<double> data;
};
I am trying to swig this using swig3.0 using the following interface file
test.i
%module test_swig
%include "std_vector.i"
namespace std {
%template(VectorDouble) vector<double>;
};
%{
#include "test.hh"
%}
%naturalvar Test::data;
%include "test.hh"
And the following test code
test.py
t = test.Test()
jprint(t)
a = [1, 2, 3]
t.data = a # fails
doing so gives me the following error
in method 'Test_data_set', argument 2 of type 'vector< double >'
This can be fixed by either changing the using std::vector in test.hh to using namespace std or by removing using std::vector and changing vector<double> to std::vector<double>. This is not what I want.
The problem is that I was given this code as is. I am not allowed to make changes, but I am supposed to still make everything available in python via SWIG. What's going on here?
Thanks in advance.
To me, this looks like SWIG does not support the using std::vector; statement correctly. I think it's a SWIG bug. I can think of the following workarounds:
Add using namespace std; to the SWIG interface file (this will only affect the way wrappers are created; the using statement will not enter C++ code)
Add #define vector std::vector to the SWIG interface file (this will only work if vector is never used as std::vector)
Copy the declarations from the header file to the SWIG interface file, and change vector to std::vector. This will cause SWIG to generate correct wrappers, and again will not affect the C++ library code.
I am using swig to write a wrapper to a c++ class for use with python.
When I try to do from CSMPy import * (CSMPy is my module) I get this message:
ImportError: dlopen(/Users/MUL_mac2/anaconda/lib/python2.7/site-packages/_CSMPy.so, 2): Symbol not found: __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
Referenced from: /Users/MUL_mac2/anaconda/lib/python2.7/site-packages/_CSMPy.so
Expected in: dynamic lookup
A little bit of background:
I have one interface file that has an include to one header file containing my wrapper class:
This class has an object as a private member.
I then want to pass a number of objects of type std::deque<int> to a member function
of this object like so: this->object.Function(int_deque_a,int_deque_b) where object is a member of the class I am wrapping using swig.
When I comment the above line out everything works like a charm.
All the containers I am passing are valid datatypes to pass to this objects member function and contain the correct number of entries.
Everything compiles and this occurs only on import of the module.
What am I missing here?
I am using distutils to compile using python setup.py install
setup.py:
CSMPy_module = Extension('_CSMPy',
include_dirs = [Bunch of include directories here],
library_dirs = ['MyLibraryPath'],
libraries = ['MyLibrary'],
sources=['CSMPy_wrap.cxx', 'WrapperClass.cpp'],
)
setup (name = 'CSMPy',
version = '0.1',
author = "My name",
description = """Simple Test""",
ext_modules = [CSMPy_module],
py_modules = ["CSMPy"],
)
MyLibrary is a static library.
Edit 1:
I am providing you with a version of the code I can show to everyone
Setup.h
#include <iostream>
#include <vector>
#include <deque>
#include "VSet.h"
class Setup {
public:
Setup();
~Setup();
void InitializeSetup();
private:
std::deque<size_t> npes;
std::deque<size_t> epes;
std::deque<std::vector<size_t> > eni; //plist
std::deque<std::vector<csmp::int32> > enb; //pfverts
std::deque<std::vector<csmp::double64> > ncl; //pelmt
std::map<size_t, csmp::int32> bnf; //bflags
std::deque<csmp::int32> et;
csmp::VSet<2U> v;
};
Setup.cpp
#include "Setup.h"
Setup::Setup() {
std::cout<<"Setup initialized."<<std::endl;
}
Setup::~Setup() {
}
void Setup::InitializeSetup() {
for(size_t i = 0; i < this->eni.size(); i++) {
this->npes.push_back(this->eni[i].size());
}
for(size_t i = 0; i < this->enb.size(); i++) {
this->epes.push_back(this->enb[i].size());
}
this->v.Resize(this->et, npes, epes, this->ncl.size()); //This is the line that does not work
}
CSMPy.i
%module CSMPy
%{
#define SWIG_FILE_WITH_INIT
#include "stdlib.h"
#include <vector>
#include <deque>
#include <map>
#include "VSet.cpp"
#include "Setup.h"
#include "Number_Types.h"
%}
%include "Number_Types.h"
%include "std_map.i"
%include "std_vector.i"
%include "std_deque.i"
// Instantiate templates used by CSMPy
namespace std {
%template() pair<size_t, csmp::int32>;
%template() pair<size_t, csmp::double64>;
%template() pair<size_t, vector<size_t> >;
%template() pair<size_t, vector<csmp::int32> >;
%template() pair<size_t, vector<csmp::double64> >;
%template(Deque_SizeT) deque<size_t>;
%template(Deque_Int) deque<csmp::int32>;
%template(Vector_SizeT) vector<size_t>;
%template(Vector_Int32) vector<csmp::int32>;
%template(Vector_Double64) vector<csmp::double64>;
%template(Deque_Double64) deque<csmp::double64>;
%template(Deque_Vector_Int) deque<vector<csmp::int32> >;
%template(Deque_Vector_SizeT) deque<vector<size_t> >;
%template(Deque_Vector_Double64) deque<vector<csmp::double64> >;
%template(Map_SizeT_Int) map< size_t, csmp::int32>;
%template(Map_SizeT_Double64) map< size_t, csmp::double64>;
%template(Map_SizeT_Vector_SizeT) map< size_t, vector<size_t> >;
%template(Map_SizeT_Vector_Int) map< size_t, vector<csmp::int32> >;
%template(Map_SizeT_Vector_Double64) map< size_t, vector<csmp::double64> >;
}
%include "Setup.h"
Edit 2:
I did nm -gC myLib.so
I found this echo
__ZN4csmp4VSetILm2EE6ResizeERKNSt3__15dequeIiNS2_9allocatorIiEEEERKNS3_ImNS4_ImEEEESC_m
which on c++ tilt tells me:
csmp::VSet<2ul>::Resize(std::__1::deque<int, std::__1::allocator<int> > const&, std::__1::deque<unsigned long, std::__1::allocator<unsigned long> > const&, std::__1::deque<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)
couple of notes on this, I have switched to using clang++ as my compiler and manually compiling. I have also put #include "VSet.cpp" in my .i file. (See edit in previous post)
I am now getting this error on import in python:
Symbol not found: __ZN4csmp5VData6InTextERSt14basic_ifstreamIcSt11char_traitsIcEE
Referenced from: ./_CSMPy.so
Expected in: flat namespace
I have also created a main that will instantiate the object and the call to Initialize() works.
It's not finding the symbol
__ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
in the .so. Thanks to Dave for demangling this, we now know that it refers to
csmp::VSet<2ul>::Resize(
const std::deque<int>&,
const std::deque<unsigned long> &,
const std::deque<unsigned long> &)
So it is a little odd that you have two types of deques, based on what you posted.
Here are some things to try:
Verify that your _CSMP.so links to the STL library that comes with your compiler, you may have to specify an extra switch or field in your setup.py. Your code works when Resize is not there, you say, so that's not likely the problem.
turn on verbose output in your setup.py so that you can see the compilation and link command line parameters
make sure you %include std_deque.i in your SWIG .i file. You're not getting compile error so this is not likely the issue.
verify that you have instantiated your deque<int> with a %template(IntDeque) std::deque<int> in your .i, since Python knows nothing about C++ templates, and a template is not a class, but a recipe so compiler can create a class for you. If you really do use both int and unsigned long, you have to instantiate both. I'm only seeing int and size_t in your code. You can't assume that size_t is same as unsigned long.
Confirm that your DLL contains an intantiation of this Resize method for unsigned int. in your C++ code. I think you defined the size_t version via, or are the unsigned long unexpected?
About #5:
SWIG generates a header and a source file. In the header it puts functions that adhere to the Python C API and registers them in the Python interpreter, and in the body of those functions it figures out what C/C++ functions to call from your library. The fact that the above Resize is not found in DLL is an indication that SWIG thinks this overload of Resize is needed, so it is called from the function it generated, but your C++ lib did not instantiate it.
How is this possible? In your C++ lib, you have a class template with a Resize method. The trick with class templates is that the compiler will only generate code for the methods that are used in the DLL (so if your class defines 5 methods but your DLL only uses 1, it won't generate code for the other 4 methods), except if you explicitly instantiate the template in your library. You would do this by putting a statement
template class VSet<2ul>;
(whatever 2ul stands for) either in your C++ DLL, or the wrapper DLL via the %template directive in your .i file. This will instantiate all methods of VSet<2ul>, so Resize will be there too. IF the Resize thus generated has parameters deque<int> and deque<unsigned long>. Your code indicates that you are assuming that size_t is unsigned int. If size_t is typedefd to unsigned int, SWIG should be able to handle it, but maybe there is a bug. Better not assume. You could add a Resize overload for unsigned int. Or you could create an inline extension method in Setup taking two unsigneld long deques and calling the size_t version. Something like
%template DequeULong std::deque<unsigned long>
%extend Setup {
void Resize(const DequeInt& a, const DequeULong& b)
{
DequeSizet c;
... copy b into a DequeSizet
Resize(a, c);
}
}
The problem most likely isn't a compilation issue. It's much more likely that there's a mismatch between your header file and implementation files. The header promises an interface that you aren't implementing. You won't see the undefined reference in a standalone, C++-only application if you never call that member function in your C++ code.
The mismatch between header and implementation becomes a real problem when you tell SWIG to wrap that C++ header. The generated SWIG code contains a reference to that unimplemented function. The dynamic linking fails because that function is never defined.
So what function is it? Look at the error message:
Symbol not found: __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
This tells you exactly what's missing, but in a very convoluted (name mangled) way. Copy that symbol, open a terminal window, and issue the command echo <paste mangled name here> | c++filt:
echo __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m | c++filt
The c++filt utility is a very useful capability on Macs and Linux boxes. In this case it gives you the unmangled name of the missing symbol. See my comment to Schollii's answer.
I'm starting a new answer because the fact that VSet<2U> is not wrapped makes most of my other answer not relevant to this problem (although everything in there is still correct). And the fact that Setup has a data member of type VSet<2U> is not relevant to SWIG since you aren't accessing Setup::v directly from Python.
Verify that Setup works without Python or SWIG: create a void main() where you instantiate a Setup and call its InitializeSetup() method, build and run. Most likely yo will get same runtime error due to symbol not found.
The Setup object code is looking for
csmp::VSet<2ul>::Resize(
const std::deque<int>&,
const std::deque<unsigned long> &,
const std::deque<unsigned long> &,
unsigned long)
So verify that your DLL has this symbol:
~> nm -gC yourLib.so
It probably doesn't. Are there other Resize overloads that were instantiated? This can give a clue.
There could be a variety of reasons why the compiler failed to instantiate the Resize for VSet<2U>. For example, a template class's method definitions must appear in the .h otherwise how it the compiler going to know what code to generate? If you tell the compiler to compile VSet.cpp it won't generate anything in the .o unless you explicitly instantiate the template for a specific type. Then the .o will contain object code for class of that specific templated class with that type. I like to have my method definitions separate class definition, but then I include the .cpp in the .h since any user of the .h would need to also nclude the .cpp so compiler can generate the right code. For you this would mean you would have at the bottom of VSet.h an #include "VSet.cpp" // templated code.