We Keep Coding

Mocking shared library (.so) functions in Python [duplicate]

New to cpp (Java guy).
I have 3rd party library that has method sendMail(txt).
I don't want to test the library. i want to test my own method, so in order to do this , i need to mock the library calls .
My own method is looking like this:
#include "mailsender.h"
int run(txt){
analysis(txt);
...
...
int status = sendMail(txt);//sendMail is a 3rd party library call. i need to mock it.its not part of the unit test
return status;
}
In Java the mailsender was interface and it was injected to my class, so in case of test i inject mock.
What is a good practice in cpp to mock library calls?
I can wrap the 3rd party library call in a class and inject this class, but i am looking for something simpler and for the common practice (maybe ifndf).
I am familiar with googlemock.
googlemock allow me to mock classes . i am not aware to option how to mock a call in my tested method.

So I assume you have a 'global' function that is implemented in a library that you both include a header file for (to get the definition) and link (to get the implementation).
You obviously need to replace the implementation of the library with your own - one that does "nothing", so you can do this in 2 ways:
you replace the .dll (or .so) with your own implementation that has all the methods the 3rd party library exposes. This is easy once you've written a new version of all the 3rd party lib functions, but writing them all out can be a pain.
you remove the library temporarily, and replace the calls you make to that in a .cpp source file that implements those functions. So you'd create your own sendMail() function in a .cpp file and include this into the program instead of the mailsender.h include.
The latter is easier, but you might also have to modify your program to not link with the 3rd party lib. This can also require changing the #include as well, as some compilers (eg VC++) allow you to embed linker directives in the source. If your does this, then you won't be able to stop the linker from including the 3rd party lib.
The other option is to modify your code to use a different call to the sendMail call, eg test__sendMail() that you implement yourself. Wrap this is a macro to conditionally include your, or the real, function call depending on your build options.
If this was a c++ library then you'd probably be able to use a mocking framework like you're used to, but it sounds like its a C library, and they simply provide a list of functions that you use directly in your code. You could wrap the library in your own class and use that instead of calling the 3rd party lib functions directly.
There is a list of C mocking frameworks.

This is an old question, with an already choosen response, but maybe the following contribution can help someone else.
First solution
You still have to create a custom library to redefine the functions, but you do not need to change Makefiles to link to your "fake-library", just use LD_PRELOAD with the path to the fake-library and that will be the first that the linker will find and then use.
example
Second solution
ld (GNU) linker has an option --wrap that let you wrap only one function with another provided by the user. This way you do not have to create a new library/class just to mock the behavior
Here is the example from the man page
--wrap=symbol
Use a wrapper function for symbol. Any undefined reference to symbol will be resolved to "__wrap_ symbol ". Any undefined reference
to "__real_ symbol " will be resolved to symbol.
This can be used to provide a wrapper for a system function. The wrapper function should be called "__wrap_ symbol ". If it wishes to
call the system function, it should call "__real_ symbol ".
Here is a trivial example:
void *
__wrap_malloc (size_t c)
{
printf ("malloc called with %zu\n", c);
return __real_malloc (c);
}
If you link other code with this file using --wrap malloc, then all calls to "malloc" will call the function "__wrap_malloc" instead.
The call to "__real_malloc" in "__wrap_malloc" will call the real
"malloc" function.
You may wish to provide a "__real_malloc" function as well, so that links without the --wrap option will succeed. If you do this, you
should not put the definition of "__real_malloc" in the same file as
"__wrap_malloc"; if you do, the assembler may resolve the call before
the linker has a chance to wrap it to "malloc".

Disclaimer: I wrote ELFspy.
Using ELFspy, the following code will allow you to fake/mock the sendMail function by replacing it with an alternative implementation.
void yourSendMail(const char* txt) // ensure same signature as sendMail
{
// your mocking code
}
int main(int argc, char** argv)
{
spy::initialise(argc, argv);
auto sendMail_hook = SPY(&sendMail); // grab a hook to sendMail
// use hook to reroute all program calls to sendMail to yourSendMail
auto sendMail_fake = spy::fake(sendMail_hook, &yourSendMail);
// call run here..
}
Your program must be compiled with position independent code (built with shared libraries) to achieve this.
Further examples are here:
https://github.com/mollismerx/elfspy/wiki

Though there is no interface keyword, you can use Abstract Base Classes for similar things in C++.
If the library you are using doesn't come with such abstractions, you can wrap it behind your own "interface". If your code separates construction of objects from usage (e.g. by IoC), you can either use this to inject a fake or use Mocks:
https://stackoverflow.com/questions/38493/are-there-any-good-c-mock-object-frameworks

C++ #include<XXX.h> equivalent of Python's import XXX as X

I work with Python most of the time, for some reasons now I also need to use C++.
I find Python's import XXX as X very neat in the following way, for example:
import numpy as np
a = np.array([1,2,3])
where I'm very clear by looking at my code that the array() function is provided by the numpy module.
However, when working with C++, if I do:
#include<cstdio>
std::remove(filename);
It's not clear to me at first sight that remove() function under the std namespace is provided by <cstdio>.
So I'm wondering if there is a way to do it in C++ as the import XXX as X way in Python?

Nope.
It'll be slightly clearer if you write std::remove (which you should be doing anyway; there's no guarantee that the symbol is available in the global namespace) because then at least you'll know it comes from a standard header.
Beyond that, it's up to your memory. 😊
Some people try to introduce hacks like:
namespace SomeThing {
#include <cstdio>
}
// Now it's SomeThing::std::remove
That might work for your own headers (though I'd still discourage it even then). But it'll cause all manner of chaos with standard headers for sure and is not permitted:
[using.headers]/1: The entities in the C++ standard library are defined in headers, whose contents are made available to a translation unit when it contains the appropriate #include preprocessing directive.
[using.headers]/3: A translation unit shall include a header only outside of any declaration or definition, and shall include the header lexically before the first reference in that translation unit to any of the entities declared in that header. No diagnostic is required.
Recall that #include and import are fundamentally different things. C++ modules may go some way towards this sort of functionality, perhaps, but by including source code you are not even touching namespaces of symbols created by that code.

No there is no way to force this syntax. The person who developped the code that you include is free. Generally people split their code into namespaces, which can result to this syntax:
#include <MyLibrary.h>
int main()
{
MyLibrary::SayHello();
return 0;
}
But you have no guarentee on how the code in the header is written.

C++ #include<XXX.h> equivalent of Python's import XXX as X
There is no equivalent in C++.
When you include a file into another, you get every single declaration from the included file, and you have no option of changing their names.
You can add aliases for types and namespaces though, and references to objects, as well as write wrapper functions to do some of what the as X part does in Python.
It's not clear to me at first sight that remove() is provided by <cstdio>.
The std namespace at least tells you that it is provided by the standard library.
What I like to do, is document which header provides the used declarations:
#include<cstdio> // std::remove
std::remove(filename);
That said, most IDE's can show you where an identifier is declared by ctrl-clicking or hovering over it (although this doesn't always work well when there are overloads in different headers). My primary use for inclusion comments is checking which includes can be removed after refactoring.

How does module loading work in CPython?

How does module loading work in CPython under the hood? Especially, how does the dynamic loading of extensions written in C work? Where can I learn about this?
I find the source code itself rather overwhelming. I can see that trusty ol' dlopen() and friends is used on systems that support it but without any sense of the bigger picture it would take a long time to figure this out from the source code.
An enormous amount could be written on this topic but as far as I can tell, almost nothing has been — the abundance of webpages describing the Python language itself makes this difficult to search for. A great answer would provide a reasonably brief overview and references to resources where I can learn more.
I'm mostly concerned with how this works on Unix-like systems simply because that's what I know but I am interested in if the process is similar elsewhere.
To be more specific (but also risk assuming too much), how does CPython use the module methods table and initialization function to "make sense" of dynamically loaded C?

TLDR short version bolded.
References to the Python source code are based on version 2.7.6.
Python imports most extensions written in C through dynamic loading. Dynamic loading is an esoteric topic that isn't well documented but it's an absolute prerequisite. Before explaining how Python uses it, I must briefly explain what it is and why Python uses it.
Historically C extensions to Python were statically linked against the Python interpreter itself. This required Python users to recompile the interpreter every time they wanted to use a new module written in C. As you can imagine, and as Guido van Rossum describes, this became impractical as the community grew. Today, most Python users never compile the interpreter once. We simply "pip install module" and then "import module" even if that module contains compiled C code.
Linking is what allows us to make function calls across compiled units of code. Dynamic loading solves the problem of linking code when the decision for what to link is made at runtime. That is, it allows a running program to interface with the linker and tell the linker what it wants to link with. For the Python interpreter to import modules with C code, this is what's called for. Writing code that makes this decision at runtime is quite uncommon and most programmers would be surprised that it's possible. Simply put, a C function has an address, it expects you to put certain data in certain places, and it promises to have put certain data in certain places upon return. If you know the secret handshake, you can call it.
The challenge with dynamic loading is that it's incumbent upon the programmer to get the handshake right and there are no safety checks. At least, they're not provided for us. Normally, if we try to call a function name with an incorrect signature we get a compile or linker error. With dynamic loading we ask the linker for a function by name (a "symbol") at runtime. The linker can tell us if that name was found but it can’t tell us how to call that function. It just gives us an address - a void pointer. We can try to cast to a function pointer of some sort but it is solely up to the programmer to get the cast correct. If we get the function signature wrong in our cast, it’s too late for the compiler or linker to warn us. We’ll likely get a segfault after the program careens out of control and ends up accessing memory inappropriately. Programs using dynamic loading must rely on pre-arranged conventions and information gathered at runtime to make proper function calls. Here's a small example before we tackle the Python interpreter.
File 1: main.c
/* gcc-4.8 -o main main -ldl */
#include <dlfcn.h> /* key include, also in Python/dynload_shlib.c */
/* used for cast to pointer to function that takes no args and returns nothing */
typedef void (say_hi_type)(void);
int main(void) {
/* get a handle to the shared library dyload1.so */
void* handle1 = dlopen("./dyload1.so", RTLD_LAZY);
/* acquire function ptr through string with name, cast to function ptr */
say_hi_type* say_hi1_ptr = (say_hi_type*)dlsym(handle1, "say_hi1");
/* dereference pointer and call function */
(*say_hi1_ptr)();
return 0;
}
/* error checking normally follows both dlopen() and dlsym() */
File 2: dyload1.c
/* gcc-4.8 -o dyload1.so dyload1.c -shared -fpic */
/* compile as C, C++ does name mangling -- changes function names */
#include <stdio.h>
void say_hi1() {
puts("dy1: hi");
}
These files are compiled and linked separately but main.c knows to go looking for ./dyload1.so at runtime. The code in main assumes that dyload1.so will have a symbol "say_hi1". It gets a handle to dyload1.so's symbols with dlopen(), gets the address of a symbol using dlsym(), assumes it is a function that takes no arguments and returns nothing, and calls it. It has no way to know for sure what "say_hi1" is -- a prior agreement is all that keeps us from segfaulting.
What I've shown above is the dlopen() family of functions. Python is deployed on many platforms, not all of which provide dlopen() but most have similar dynamic loading mechanisms. Python achieves portable dynamic loading by wrapping the dynamic loading mechanisms of several operating systems in a common interface.
This comment in Python/importdl.c summarizes the strategy.
/* ./configure sets HAVE_DYNAMIC_LOADING if dynamic loading of modules is
supported on this platform. configure will then compile and link in one
of the dynload_*.c files, as appropriate. We will call a function in
those modules to get a function pointer to the module's init function.
*/
As referenced, in Python 2.7.6 we have these dynload*.c files:
Python/dynload_aix.c Python/dynload_beos.c Python/dynload_hpux.c
Python/dynload_os2.c Python/dynload_stub.c Python/dynload_atheos.c
Python/dynload_dl.c Python/dynload_next.c Python/dynload_shlib.c
Python/dynload_win.c
They each define a function with this signature:
dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname,
const char *pathname, FILE *fp)
These functions contain the different dynamic loading mechanisms for different operating systems. The mechanism for dynamic loading on Mac OS newer than 10.2 and most Unix(-like) systems is dlopen(), which is called in Python/dynload_shlib.c.
Skimming over dynload_win.c, the analagous function for Windows is LoadLibraryEx(). Its use looks very similar.
At the bottom of Python/dynload_shlib.c you can see the actual call to dlopen() and to dlsym().
handle = dlopen(pathname, dlopenflags);
/* error handling */
p = (dl_funcptr) dlsym(handle, funcname);
return p;
Right before this, Python composes the string with the function name it will look for. The module name is in the shortname variable.
PyOS_snprintf(funcname, sizeof(funcname),
LEAD_UNDERSCORE "init%.200s", shortname);
Python simply hopes there's a function called init{modulename} and asks the linker for it. Starting here, Python relies on a small set of conventions to make dynamic loading of C code possible and reliable.
Let's look at what C extensions must do to fulfill the contract that makes the above call to dlsym() work. For compiled C Python modules, the first convention that allows Python to access the compiled C code is the init{shared_library_filename}() function. For a module named spam compiled as shared library named “spam.so”, we might provide this initspam() function:
PyMODINIT_FUNC
initspam(void)
{
PyObject *m;
m = Py_InitModule("spam", SpamMethods);
if (m == NULL)
return;
}
If the name of the init function does not match the filename, the Python interpreter cannot know how to find it. For example, renaming spam.so to notspam.so and trying to import gives the following.
>>> import spam
ImportError: No module named spam
>>> import notspam
ImportError: dynamic module does not define init function (initnotspam)
If the naming convention is violated there's simply no telling if the shared library even contains an initialization function.
The second key convention is that once called, the init function is responsible for initializing itself by calling Py_InitModule. This call adds the module to a "dictionary"/hash table kept by the interpreter that maps module name to module data. It also registers the C functions in the method table. After calling Py_InitModule, modules may initialize themselves in other ways such as adding objects. (Ex: the SpamError object in the Python C API tutorial). (Py_InitModule is actually a macro that creates the real init call but with some info baked in like what version of Python our compiled C extension used.)
If the init function has the proper name but does not call Py_InitModule(), we get this:
SystemError: dynamic module not initialized properly
Our methods table happens to be called SpamMethods and looks like this.
static PyMethodDef SpamMethods[] = {
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
{NULL, NULL, 0, NULL}
};
The method table itself and the function signature contracts it entails is the third and final key convention necessary for Python to make sense of dynamically loaded C. The method table is an array of struct PyMethodDef with a final sentinel entry. A PyMethodDef is defined in Include/methodobject.h as follows.
struct PyMethodDef {
const char *ml_name; /* The name of the built-in function/method */
PyCFunction ml_meth; /* The C function that implements it */
int ml_flags; /* Combination of METH_xxx flags, which mostly
describe the args expected by the C func */
const char *ml_doc; /* The __doc__ attribute, or NULL */
};
The crucial part here is that the second member is a PyCFunction. We passed in the address of a function, so what is a PyCFunction? It's a typedef, also in Include/methodobject.h
typedef PyObject *(*PyCFunction)(PyObject *, PyObject *);
PyCFunction is a typedef for a pointer to a function that returns a pointer to a PyObject and that takes for arguments two pointers to PyObjects. As a lemma to convention three, C functions registered with the method table all have the same signature.
Python circumvents much of the difficulty in dynamic loading by using a limited set of C function signatures. One signature in particular is used for most C functions. Pointers to C functions that take additional arguments can be "snuck in" by casting to PyCFunction. (See the keywdarg_parrot example in the Python C API tutorial.) Even C functions that backup Python functions that take no arguments in Python will take two arguments in C (shown below). All functions are also expected to return something (which may just be the None object). Functions that take multiple positional arguments in Python have to unpack those arguments from a single object in C.
That's how the data for interfacing with dynamically loaded C functions is acquired and stored. Finally, here's an example of how that data is used.
The context here is that we're evaluating Python "opcodes", instruction by instruction, and we've hit a function call opcode. (see https://docs.python.org/2/library/dis.html. It's worth a skim.) We've determined that the Python function object is backed by a C function. In the code below we check if the function in Python takes no arguments (in Python) and if so, call it (with two arguments in C).
Python/ceval.c.
if (flags & (METH_NOARGS | METH_O)) {
PyCFunction meth = PyCFunction_GET_FUNCTION(func);
PyObject *self = PyCFunction_GET_SELF(func);
if (flags & METH_NOARGS && na == 0) {
C_TRACE(x, (*meth)(self,NULL));
}
It does of course take arguments in C - exactly two. Since everything is an object in Python, it gets a self argument. At the bottom you can see that meth is assigned a function pointer which is then dereferenced and called. The return value ends up in x.

Access methods not exposed in the method mapping table with ctypes

The method mapping table is an array of PyMethodDef structures,
struct PyMethodDef {
char *ml_name;
PyCFunction ml_meth;
int ml_flags;
char *ml_doc;
};
where the ml_name is the name of the functions accessed in the interpreter (i.e. gc.collect()), ml_meth is the address of the function described in the previous section, ml_flags indicates which of the signatures ml_meth is using and ml_doc is the docstring for the function.
Now, say I want to access from within the gcmodule.c gc_list_append which is not exposed by the gc interface.
I've tried the following:
pythonapi.PyImport_ImportModule("gcmodule")
or
pythonapi.PyImport_ImportModule(ctypes.c_wchar("gcmodule"))
Hoping that the reference returned can be used to access unexposed methods, but in both cases I get errors:
ImportError: No module named 'g'
or
TypeError: one character unicode string expected
Any idea how an unexposed function or data structures can be accessed via ctypes/pythonapi?

PyImport_ImportModule imports a module as named by the const char * argument. It calls PyImport_Import, which in turn calls __builtin__.__import__ in PY2. Calling it with the C string "gc" returns a reference to sys.modules['gc'] if it exists, else the interpreter creates the module by calling initgc , which it finds in _PyImport_Inittab. The gc module's init function calls Py_InitModule4, which loops over the GcMethods table to add methods to the module dict.
Every global definition you see in gcmodule.c that's marked static is only visible within that compilation unit at the system level, i.e. it doesn't have external linkage. Commonly an extension module only exports the name of the PyMODINIT_FUNC, but several of the built-in modules in CPython also export public APIs such as PyObject_GC_Track.
On Windows, exporting a symbol from a DLL by name or ordinal requires the extra step of declaring it in a .def file or with the modifier __declspec(dllexport). In CPython that gets applied via the macros PyAPI_FUNC (pyport.h), PyAPI_DATA, and PyMODINIT_FUNC. For example, objimpl.h declares PyAPI_FUNC(void) PyObject_GC_Track(void *).
Declaring a symbol static is namespace management. However, if you can get the address of a function somehow, you can certainly call it with a ctypes function pointer. Finding the address is the issue. ctypes uses the standard POSIX dlsym and Windows GetProcAddress functions to look up exported symbols, which won't help you here. You may be able to access function pointers referenced directly or indirectly by CPython objects, since id returns the object base address.
The following isn't directly related to your problem, but I'm including it for general interest.
On a POSIX system that loads extension modules with dlopen, you can specify RTLD_GLOBAL to merge the dynamic symbols into the global namespace. For example, the _ctypes extension module is linked from multiple object files and needs several extern symbols, such as PyCData_set. You can tweak the import to have PyCData_set accessible globally (e.g. via ctypes.pythonapi):
>>> import sys, DLFCN
>>> sys.setdlopenflags(DLFCN.RTLD_GLOBAL | DLFCN.RTLD_NOW)
>>> import ctypes
>>> hasattr(ctypes.pythonapi, 'PyCData_set')
True
Of course, that won't work on Windows. The proper way for an extension module to make a C API available is by using a PyCapsule, such as datetime.datetime_CAPI (datetimemodule.c).

Interacting SWIG modules with and without `-builtin`

How can I tell a module compiled without -builtin that an %imported module is compiled with -builtin? Doing this naively gives me segfaults when the non-builtin module assumes that objects from the first module have wrappers.
(I never get segfaults if everything is compiled with -builtin turned off, or when using the second module alone with -builtin turned on; it's just when using them together with different compilation options.)
Details
I have several separate modules that I use SWIG for. Let's say one of them is named A, and contains fundamental objects (quaternions). Because it contains basic objects that are involved in many computations, I prefer to use SWIG's -builtin option for it. I have tested it, and this does make a pretty significant difference in timing.
Now, I also have another module named B which needs to use objects from A. But B contains big fat composite objects that I don't act on very many times, so I don't expect that there's much advantage in using -builtin here. Moreover, I really like to extend the classes in B, and do various things that are not possible with -builtin.
The problem is that I have to %import A.i inside of B.i. But then the code that's generated for B assumes that A objects have the extra wrappers, rather than using -builtin. So when I use B, I get segfaults.
(At least, I assume the segfaults result because B assumes the extra wrappers. I looked through my B_wrap.cpp file enough to see that it was assuming the presence of those wrappers, though I can't really say that I did any test to ensure that's where the problem was coming from. But the segfaults did coincide only with uses of A from B. On its own, A has never given me any trouble. Also, if I compile A and B without -builtin, I never get segfaults.)
In principle, I could use MONK's approach and just subclass any class that I need to add methods to, while compiling everything with -builtin. But this would break the nice correspondence between names in my C++ code and names in my python code, as well as requiring one or other set of users to change the names they use, as well as being a general pain in the butt.
I apologize for not having a MWE, but I think it would be an unreasonably large MWE.

I don't know that it's possible to compile with separate flags, but I have satisfied myself with MONK's solution. In combination with SWIG's %rename functionality, MONK's approach doesn't require renaming anything visible to the user. Plus, it's easy to implement, requiring just five extra lines per class I want to modify. Thus, everything gets to be compiled with -builtin, and no segfaults. Though this doesn't technically answer the question I asked at the top, it suits me.
So, let's suppose that the crucial object inside B is a class named Classy. I'll just tell SWIG to rename this to _Classy (the underscore so that I don't have to look at it every time I use tab completion in ipython). Then, I'll make an empty metaclass that subclasses those objects. Finally, I'll make a new object named Classy that just has the metaclass. So my python users and my C++ users will see it as the same object, but python users will be able to subclass it.
Here's a MWE for this part of it. A simple header:
// Classy.hpp
class Classy {
public:
Classy() { }
};
And the SWIG file
// test.i
%module "test"
%{
#include "Classy.hpp"
%}
%rename(_Classy) Classy;
%include "Classy.hpp"
%insert("python") %{
class _MetaClassy(type(_Classy)):
pass
class Classy(_Classy):
__metaclass__ = _MetaClassy
Classy.myattr = 'anything'
%}
(See how we added an attribute there at the end.) And, finally, the setup file:
# setup.py
from distutils.core import setup, Extension
example_module = Extension('_test',
sources=['test_wrap.cxx'])
setup (name = 'test',
ext_modules = [example_module],
py_modules = ["test"])
Now, just compile and test with
swig -python -builtin -c++ test.i
python setup.py build_ext --inplace
python -c 'import test; x=test.Classy(); print x.myattr'
In that last line, the python object x, which is of type Classy, does indeed have an attribute -- even though the C++ class had nothing at all. So we've succeeded.
Presumably, this subclassing defeats the speed advantage of -builtin for the Classy object, but I had already decided that I don't care about that one class. On the other hand, I get to retain the speed advantage for any objects that I don't explicitly subclass, so there is still a reason to use builtin.