First, pythonanywhere is an amazing remote hosting site, and it would be great if it weren't just for python.
I've read loads of solutions, trying to get this simple example to work, in an attempt to use c++ code, using python code hosted in pythonanywhere.
c++ code
char const* greet()
{
return "hello, world";
}
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(hello)
{
using namespace boost::python;
def("greet", greet);
}
In my account in pythonanywhere I did the following:
Create hello.cpp file with C++ code and upload it.
Run the following two cmds:
gcc -o middle -c -fPIC -I /usr/include/python3.8/ -L /usr/include/python3.8/ hello.cpp
gcc -o hello.so -shared middle -lboost_python -lpython3.8
The second command terminates unexpectedly showing the error:
/usr/bin/ld: cannot find -lboost_python
If I run the second command without the -lboost_python argument, it succeeds, but when I use python an error is thrown.
$ python3.8
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: /home/Ottoman/hello.so: undefined symbol: _ZNK5boost6python7objects21py_function_impl_base9max_arityEv
I need two kinds of answers. First, what am I doing wrong and how can I solve my problem? Second, What I'm trying to do is good practice?, since my ultimate goal is to execute much more complex C++ code. If not, what alternatives do I have? Thank you!!!
Instead of -lboost_python you should use -lboost_python38, since you are using python3.8.
Related
Good day, I have a few problems with simple examples from boost::python - All methods that must return any types except char const* or create Classes doesn't return anything and all what do I see its RESTART: Shell.
I'm using:
CLion 2018.3.3, Build #CL-183.5153.40, built on January 10, 2019;
MinGW-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0;
CMake 3.13.0-rc1-;
Python 3.7 (x86_64);
boost 1.68.0,
which was built with the next arguments:
bootstrap gcc
b2 -j12 toolset=gcc runtime-link=shared architecture=x86 address-model=64 --with-python cxxflags="-D__int64=\"long long int\" -DBOOST_REGEX_MATCH_EXTRA -D__NO_INLINE__ -D_hypot=hypot" --build-type=complete --layout=tagged stage
CMakeList.txt contains:
cmake_minimum_required(VERSION 3.13)
project(pylibtest)
set(CMAKE_CXX_STANDARD 17)
set(GCC_COVERAGE_COMPILE_FLAGS "-march=native -m64 -D_hypot=hypot -static -static-libstdc++ -static-libgcc -Wall")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${GCC_COVERAGE_COMPILE_FLAGS}" )
message(STATUS "start running cmake...")
set(Boost_DEBUG OFF)
#set(Boost_USE_STATIC_LIBS ON)
set(Boost_USE_MULTITHREADED ON)
#set(Boost_USE_STATIC_RUNTIME ON)
find_package(Boost 1.68.0 COMPONENTS python37 REQUIRED)
if(NOT Boost_FOUND)
set(Boost_LIBRARY_DIR "${Boost_Path}/stage/lib")
message(STATUS "Boost_LIBRARIES: ${Boost_LIBRARY_DIR}")
message(STATUS "Boost_VERSION: ${Boost_VERSION}")
include_directories(${Boost_INCLUDE_DIRS})
link_directories(${Boost_LIBRARIES})
endif()
find_package(PythonInterp 3.7 REQUIRED)
find_package(PythonLibs 3.7 REQUIRED)
if(PYTHONLIBS_FOUND)
message(STATUS "PYTHON_LIBRARIES = ${PYTHON_LIBRARIES}")
message(STATUS "PYTHON_EXECUTABLE = ${PYTHON_EXECUTABLE}")
message(STATUS "PYTHON_INCLUDE_DIRS = ${PYTHON_INCLUDE_DIRS}")
include_directories(${PYTHON_INCLUDE_DIRS})
link_directories(${PYTHON_LIBRARIES})
endif()
if(Boost_FOUND AND PYTHONLIBS_FOUND)
add_library(pylibtest SHARED library.cpp library.h wrap.cpp Some.cpp Some.h)
target_include_directories(pylibtest SYSTEM PRIVATE ${Boost_INCLUDE_DIRS} ${PYTHON_INCLUDE_DIRS})
target_link_libraries(pylibtest ${Boost_LIBRARIES} ${PYTHON_LIBRARIES})
endif()
wrap.cpp which are contains all methods and they definitions in python module:
#include <boost/python.hpp>
//#include "Some.h"
using namespace boost::python;
char const* SayHello()
{
return "Hello, from c++ dll!";
}
int GetMeNumber(){
int temp = 255;
return temp;
}
BOOST_PYTHON_MODULE( example )
{
def("SayHello", SayHello);
def("GetMeNumber", GetMeNumber);
}
The Python script in IDLE 3.7 (64 bit):
print("Procedure started:")
import example
print(example.SayHello())
print("Procudure ended")
print("-=-=-=-=-=-=-=-=-=-=-=-")
print("New procedure started")
count = example.GetMeNumber();
print(count)
print("New procudure ended")
And the out in Python 3.7.0 Shell:
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
===== RESTART: C:\..\pi.py =====
Procedure started:
Hello, from c++ dll!
Procudure ended
-=-=-=-=-=-=-=-=-=-=-=-
New procedure started
=============================== RESTART: Shell ===============================
>>>
When I tried to use same script in PyCharm I've got the out with error code message:
Procedure started:
Hello, from c++ dll!
Procudure ended
-=-=-=-=-=-=-=-=-=-=-=-
New procedure started
Process finished with exit code -1073741819 (0xC0000005)
Does any one know what's wrong with my out lib and what is the better way to make its work?
The code 0xC0000005 is Access Violation error. You can try to repeat everything under the administrative account.
I'm writing a SWIG c++ file to generate a python module. I want to let the users import it in both Python2 and Python3 scripts. Since SWIG has different flags for binding Python2 and Python 3, I was wondering is there a way I can write a create a general module for both.
Let's rewind a little and keep SWIG itself out of the question to start with.
Historically it's been necessary to compile a Python module for every (sometimes even minor) version change of the Python interpreter. This led to PEP-384, which defined a stable ABI for a subset of the Python C-API from Python 3.2 onwards. So obviously that doesn't actually work for your request, because you're interested in 2 vs 3. Furthermore SWIG itself doesn't actually generate PEP-384 compatible, code.
To experiment further and see a bit more about what's going on I've made the following SWIG file:
%module test
%inline %{
char *frobinate(const char *in) {
static char buf[1024];
snprintf(buf, 1024, "World of hello: %s", in);
return buf;
}
%}
If we try to compile this with -DPy_LIMITED_API it failed:
swig3.0 -Wall -python test.i && gcc -Wall -Wextra -I/usr/include/python3.2 -shared -o _test.so test_wrap.c -DPy_LIMITED_API 2>&1|head
test_wrap.c: In function ‘SWIG_PyInstanceMethod_New’:
test_wrap.c:1114:3: warning: implicit declaration of function ‘PyInstanceMethod_New’ [-Wimplicit-function-declaration]
return PyInstanceMethod_New(func);
^
test_wrap.c:1114:3: warning: return makes pointer from integer without a cast
test_wrap.c: In function ‘SWIG_Python_UnpackTuple’:
test_wrap.c:1315:5: warning: implicit declaration of function ‘PyTuple_GET_SIZE’ [-Wimplicit-function-declaration]
Py_ssize_t l = PyTuple_GET_SIZE(args);
^
test_wrap.c:1327:2: warning: implicit declaration of function ‘PyTuple_GET_ITEM’ [-Wimplicit-function-declaration]
I.e. nobody has picked up that ticket, at least not on the version of SWIG 3.0.2 that I am testing with.
So where does that leave us? Well the SWIG 3.0 documentation on Python 3 support says something interesting:
SWIG is able to support Python 3.0. The wrapper code generated by SWIG can be compiled with both Python 2.x or 3.0. Further more, by passing the -py3 command line option to SWIG, wrapper code with some Python 3 specific features can be generated (see below subsections for details of these features). The -py3 option also disables some incompatible features for Python 3, such as -classic.
My reading of that statement is that if you want to generate source code that can be compiled with both 2.x and 3.x Python headers all you need to do is omit the -py3 argument when running SWIG. And that seems to work with my testing, the same code generated by SWIG compiles just fine with all of:
$ gcc -Wall -Wextra -I/usr/include/python2.7/ -shared -o _test.so test_wrap.c
$ gcc -Wall -Wextra -I/usr/include/python3.2 -shared -o _test.so test_wrap.c
$ gcc -Wall -Wextra -I/usr/include/python3.4 -shared -o _test.so test_wrap.c
(Note that 3.4 does generate some warnings, but no errors and it does work)
The problem is that there's no interchangeability between the compiled _test.so of any given version. Trying to import a version from one in another will fail with errors from the dynamic linker about missing or undefined symbols. So we're still stuck at the initial problem, although we can compile our module for any Python interpreter we can't compile it for all versions.
In my view the right way to deal with this is use distuitls and simply install your module into the search path for each version of Python you want to support on any given machine.
That said there is a workaround we could use. Essentially we want to build multiple versions of our module for each interpreter and use some generic code to switch between the various implementations. Assuming that you're not building with code generated from SWIG's -builtin option there are two places we could try and do this:
In the shared object itself
In some Python code
The later is substantially simpler to achieve, so let's look at that. I did something by using the following bash script to build a shared object for each version of Python I intended to support:
#!/bin/bash
for v in 2.7 3.2 3.4
do
d=$(echo $v|tr . _)
mkdir -p $d
touch $d/__init__.py
gcc -Wall -Wextra -I/usr/include/python$v -shared -o $d/_test.so test_wrap.c
done
This builds 3 versions of _test.so, in directories named after the Python version they're intended to support. If we tried to import our SWIG generated module now it would complain because there's no indication of where to find the _test module we've compiled. So we need to fix that. There are three ways to do it:
Modify the SWIG generated import code. I didn't manage to make this work with SWIG 3.0.2 - perhaps it's too old?
Modify the Python search path, before the second import. We can do that using %pythonbegin:
%pythonbegin %{
import os.path
import sys
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), '%d_%d' % (sys.version_info.major, sys.version_info.minor)))
%}
Write a _test.py stub that finds the right one and then switches them around:
from sys import version_info,modules
from importlib import import_module
real_mod = import_module('%d_%d.%s' % (version_info.major, version_info.minor, __name__))
modules[__name__] = modules[real_mod.__name__]
Either of the last two options worked and resulted in import test succeeding in all three versions of Python. But it's cheating a bit really and far better just to install from source 3 times over.
The initial problem was solved, but there are more ...
__
I found this neat LTS tool online. Since it takes quite some time for training I'd like to speed things up. While searching for an approach I came across PyPy.
Pypy is now setup and seems to be working, but not with the Sequitur g2p code. When importing a module which comes from a .so file I get a No module named _sequitur_ .
I also tried installing/compiling the code with PyPy instead of Python which crashes without any useful(?) error.
I run PyPy v2.6.1 on a Ubuntu Linux 14.04 with Python 2.7.10.
Calling g2p with the default interpreter works fine.
Does the problematic .so file need to be compiled via a call from PyPy ?
Edit #1:
When trying to install g2p with PyPy instead of CPython it crashes without after this command:
error: Command "cc -O2 -fPIC -Wimplicit -DMULTIGRAM_SIZE=2 -I/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include -I/opt/pypy-2.6.1-linux64/include -c sequitur_wrap.cpp -o build/temp.linux-x86_64-2.7/sequitur_wrap.o -fpermissive" failed with exit status 1
When calling this particular command from the console, without being a part of the setup.py, there are two errors:
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:192:55: error: ‘PyArray_DescrFromType’ was not declared in this scope
PyArray_FromAny(op, PyArray_DescrFromType(type), min_depth, \
&
/opt/pypy-2.6.1-linux64/site-packages/numpy/core/include/numpy/ndarrayobject.h:194:69: error: ‘PyArray_FromAny’ was not declared in this scope
NPY_ARRAY_ENSUREARRAY, NULL)
This either didn't appear or caused an error when installing with CPhyton.
diff also shows that ndarrayobject.h under the mentioned path doesn't differ from the one in /usr/local/lib/python2.7/site-packages/numpy/andsoon .
Since I have no clue how c++ works, I'm lost there...
Edit #2:
Well, somehow it didn't work with the first include (pointing to pypy's numpy). Probably not the clean way to do it, but it helped to get rid of that problem. So I substituted the include_dirs entry in the setup.py and pointed that one to cPython's numpy include dir.
That done, the setup.py runs without any error. When I now try to import sequitur (one of the necessary files) I noticed that one .so was missing, copied it from cPython's numpy to Pypy's numpy. So far so good but now I still get this error:
$ pypy
Python 2.7.10 (f3ad1e1e1d62, Aug 28 2015, 10:45:29)
[PyPy 2.6.1 with GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import sequitur
AttributeError: _ARRAY_API not found
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sequitur.py", line 32, in <module>
import sequitur_, SequenceModel, Minimization, misc
File "sequitur_.py", line 28, in <module>
_sequitur_ = swig_import_helper()
File "sequitur_.py", line 20, in swig_import_helper
import _sequitur_
ImportError: numpy.core.multiarray failed to import
Any suggestions?
Thanks
Simple C++ example class I want to talk to in a file called foo.cpp
#include <iostream>
Since ctypes can only talk to C functions, you need to provide those declaring them as extern "C"
extern "C" {
Foo* Foo_new(){ return new Foo(); }
void Foo_bar(Foo* foo){ foo->bar(); }
}
class Foo{
public:
void bar(){
std::cout << "Hello" << std::endl;
}
};
compile this to a shared library
g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
finally I have wrote python wrapper
from ctypes import cdll
lib = cdll.LoadLibrary('./libfoo.so')
class Foo(object):
def __init__(self):
self.obj = lib.Foo_new()
def bar(self):
lib.Foo_bar(self.obj)
f = Foo()
f.bar() #prints "Hello" on the screen
"My main intension is to compile C++ code in eclipse and call the C++ function from python in Linux". This works fine when I compiled C++ code in Linux and call the C++ method from python in Linux. But it doesn't work if I compile C++ code in eclipse and call the C++ method from python in Linux.
Error message:
symbol not found
I am new to the eclipse tool chain, But I am giving compiler option and linking option in as in this
g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
Snapshot of eclipse compiler option and linking option will be highly appreciated. Please help me in sorting out this issue. Thanks in advance
You need to create two projects in the Eclipse.
Makefile project with existing code. (File->New->Makefile project with existing code). In this project you must point to your foo.cpp file. Then in the project folder you must create file which name is "Makefile". Makefile must contain folowing lines:
all:
g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -W1,-soname,libfoo.so -o libfoo.so foo.o
clean:
rm -f libfoo.so
Then You must create rules ("all" and "clean") for this project in the "Make Target" window. If you don't see this window You must do Window->Show view->Make Target. Thus you can create libfoo.so file using Eclipse when double-clicked on the "all" rule in the "Make target" view.
At this moment You can create PyDev project with foo.py file. If you don't know about PyDev you must go to this site. It is Eclipse plugin for python language. When you will have installed this plugin You will can to work with your python file under the Eclipse.
See some images.
I am compiling a fortran program called prog.f. It contains an include file called test.inc.
This below runs successfully and shows that my include file is found. I have a prog.so file generated.
f2py -c prog.f -m prog
However when I call this module from python I get this error message:
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import prog
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: ./prog.so: undefined symbol: unknown_function_
I suspect something to do with my compilation arguments but I am not too familiar with Fortran. Do I need to include my include file as well? If so how?
Yes, you need the include file. It might be something as simple as:
f2py -c include_file.f prog.f -m prog
although I haven't tested that. Alternatively, you might need to use the fortran include statement to include include_file into prog ... Also, if you're more familiar with C, virtually all fortran compilers that I know use the convention that prog.F is fortran code that should be preprocessed by cpp. So you could probably get your file included that way as well.
I also ran into this problem, and was able to find other discussions that simply stated that current f2py does not support preprocessing of files, such as #include statements and conditional compilation.
https://groups.google.com/forum/#!topic/f2py-dev/aW65sEoSdG8
http://mail.scipy.org/pipermail/numpy-discussion/2009-November/046381.html
Two solutions:
Simplest: If you are using gfortran to compile your code, rename any .f90 or .f file to .F90 or .F. Using a capital letter keys gfortran to automatically use the preprocessor (even if you don't specify).
Second:
Use your Fortran compiler to preprocess the file and save the result as an intermediate Fortran code to be sent to f2py.
gfortran -E -D<Defines-for-compiler> file.f -cpp -o outdir/file.f
where -E is the command to tell gfortran to just preprocess the file (may vary from compiler to compiler), and -D option to define any values to use during the preprocessing.
Saving the file to another directory (or changing the name completely) is necessary to avoid overwriting the original .f file.