We Keep Coding

Running python from command line asks me to import modules

I have a script written in Spyder(Python 3.8) on Linux. While on the Spyder console the script runs fine, while I an calling it from the Linux terminal it seems like it doesn't see the modules I import in the script. Opening the terminal I run: python3 /zhome/c9/f/144817/Desktop/ChargersDaniel.py but, here is the error I take:
Traceback (most recent call last): File "/zhome/c9/f/144817/Desktop/ChargersDaniel.py", line 9, in <module>
import GPyOpt ModuleNotFoundError: No module named 'GPyOpt'
where GPyOpt is the very first library I have at the first lines in my script.
It looks like for some reason, python3 doesn't see the libraries installed. I have checked the solution here but that's not my case cause I am already calling Python3 at my terminal.
Any suggestions ?
Thanks in advance

Maybe check if GPyOpt is installed when running from your terminal:
python3 -c "import GPyOpt"
if [ $? -eq 0 ]
then
echo "GPyOpt is installed!"
fi

C++ Qt: QProcess run Python script path specifying Python version

Qt Creator 4.7.1
Based on Qt 5.11.2 (Clang 8.0 (Apple), 64 bit)
I'm running this in Qt.
QProcess p;
QStringList params;
params << "/Users/johan/Documents/testQt/hello.py";
p.start("python", params);
p.waitForFinished(-1);
qDebug() << "finished";
QString p_stdout = p.readAll();
qDebug() << p_stdout;
QString p_stderr = p.readAllStandardError();
if(!p_stderr.isEmpty())
qDebug()<<"Python error:"<<p_stderr;
I at first had the same error as this: Qt calling python using QProcess
Python error: "ImportError: No module named site\r\n"
And I added:
QProcessEnvironment env = QProcessEnvironment::systemEnvironment();
env.insert("PYTHONPATH", "/Users/johan/anaconda3/lib/python3.7");
env.insert("PYTHONHOME", "/Users/johan/anaconda3/bin/python");
p.setProcessEnvironment(env);
I can directly run the python script from terminal with python hello.py. /Users/johan/anaconda3/bin/python is the output of which python. I suppose I have the correct path for PYTHONHOME, but I'm still getting error.
Python error: " File \"/Users/johan/anaconda3/lib/python3.7/site.py\", line 177\n file=sys.stderr)\n ^\nSyntaxError: invalid syntax\n"
now this is the same error as this: Using multiple versions of Python
But adding what's suggested #!python3 in the script didn't help. I've also tried #!/Users/johan/anaconda3/bin/python.
After searching for hours, now I really don't know how to solve this. How do I specify to run with Python 3? Any help is appreciated.
I guess it's probably still a path problem. Please kindly educate me what I don't understand about PATH in general. I do know PATH is where shell looks for the executable. But why are we inserting PYTHONPATH and PYTHONHOME here instead of just adding it to PATH? What are PYTHONPATH and PYTHONHOME for? (I've read PYTHONHOME documentation but I don't understand.)
EDIT (hello.py for testing package imports):
import time
import sys
import os
import tensorflow as tf
import numpy as np
import time
import inspect
import cv2
def main():
time.sleep(1)
print(os.path)
print(sys.version_info[0])
print("hello")
if __name__ == '__main__':
main()

In PYTHONPATH there must be the paths of the modules(so the minimum is site-packages) so the solution is:
env.insert("PYTHONPATH", "/Users/johan/anaconda3/lib/python3.7/site-packages")
You must also place the path of the python binary that is used:
p.start("/Users/johan/anaconda3/bin/python", params);

gdal_merge.py not working after gdal and it python bindings have been installed

I installed Python 3.4.0 64 bit and gdal file release-1400-x64-gdal-1-11-1-mapserver-6-4-1.zip from http://www.gisinternals.com/release.php. I found the binding from http://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal and the filename is GDAL-1.11.2-cp34-none-win_amd64. I successfully install these files and import gdal. However, when I run the following command within the Python IDE to merge files 1 2 and 3, I got an error
>>> gdal_merge.py -o out.tif 1.tif 2.tif 3.tif
File "<console>", line 1
gdal_merge.py -o out.tif 1.tif 2.tif 3.tif
^
SyntaxError: invalid syntax
I specifically check to see if I can import gdal_merge as below
>>> import gdal_merge
and it was ok.
I appreciate if anybody could help with this issue.

gdal_merge.py is part of the GDAL utilities which are executed from the command line, not from within a Python IDE or another Python script.
Just open a command line (cmd) and type:
python gdal_merge.py -o out.tif 1.tif 2.tif 3.tif
Depending on your environment variables and whether you included GDAL in your Path variable you might need to specificy the full path to gdal_merge.py and/or can leave out python at the beginning of the call.

Undefined Symbol in C++ When Loading a Python Shared Library

I have been trying to get a project of mine to run but I have run into trouble. After much debugging I have narrowed down the problem but have no idea how to proceed.
Some background, I am using a python script inside C++ code. This is somewhat documented on Python, and I managed to get it running very well in my basic executable. #include and a -lpython2.6 and everything was grand.
However, difficulty has arisen when running this python script from a shared library(.so). This shared library is "loaded" as a "module" by a simulation system (OpenRAVE). The system interacts with this module using a virtual method for "modules" called SendCommand. The module then starts a boost::thread, giving python its own thread, and returns to the simulation system. However, when python begins importing its modules and thus loading its dynamic libraries it fails, I assume due to the following error:
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
I have run ldd on my executable and the shared library, there doesn't some to be a difference. I have also run nm -D on the file above, the _Py_ZeroStruct is indeed undefined. If you guys would like print outs of the commands I would be glad to supply them. Any advice would be greatly appreciated, thank you.
Here is the full python error:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/numpy/__init__.py", line 130, in
import add_newdocs
File "/usr/lib/python2.6/dist-packages/numpy/add_newdocs.py", line 9, in
from lib import add_newdoc
File "/usr/lib/python2.6/dist-packages/numpy/lib/__init__.py", line 4, in
from type_check import *
File "/usr/lib/python2.6/dist-packages/numpy/lib/type_check.py", line 8, in
import numpy.core.numeric as _nx
File "/usr/lib/python2.6/dist-packages/numpy/core/__init__.py", line 5, in
import multiarray
ImportError: /usr/lib/python2.6/dist-packages/numpy/core/multiarray.so: undefined symbol: _Py_ZeroStruct
Traceback (most recent call last):
File "/home/constantin/workspace/OpenRAVE/src/grasp_behavior_2.py", line 3, in
from openravepy import *
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 35, in
openravepy_currentversion = loadlatest()
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 16, in loadlatest
return _loadversion('_openravepy_')
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/__init__.py", line 19, in _loadversion
mainpackage = __import__("openravepy", globals(), locals(), [targetname])
File "/home/constantin/workspace/rospackages/openrave/lib/python2.6/site-packages/openravepy/_openravepy_/__init__.py", line 29, in
from openravepy_int import *
ImportError: numpy.core.multiarray failed to import

I experienced the same problem with my application and solved it without linking python to the executable.
The setup is as follows:
Executable --links--> library --dynamically-loads--> plugin --loads--> python interpreter
The solution to avoid the ImportErrors was to change the parameters of dlopen, with which the plugin was loaded to RTLD_GLOBAL.
dlopen("plugin.so", RTLD_NOW | RTLD_GLOBAL)
This makes the symbols available to other things loaded afterwards, i.e. other plugins or the python interpreter.
It can, however, happen that symbol clashes occur, because a plugin later exports the same symbols.

The solution was linking the python2.6 library with my executable as well.
Even though the executable made no python calls, it needed to be linked with the python library. I assume its because my shared library doesn't pass the symbols of python library through to the executable. If anyone could explain why my executable (which loads my dynamic library at runtime, without linking) needs those symbols it would be great.
For clarification, my program model is something like:
[My Executable] -(dynamically loads)-> [My Shared Library] -(calls and links with)-> [Python shared Library]

Check your python-headers and python's runtime. It looks like you have mix of 2.5 and 2.6 versions.

there's an example in openrave that shows how to build C++ shared objects that use boost python without having the application know about it:
http://openrave.org/en/coreapihtml/orpythonbinding_8cpp-example.html
search for "python" in the cmake file here:
https://openrave.svn.sourceforge.net/svnroot/openrave/trunk/src/cppexamples/CMakeLists.txt
the relevant info is:
if( Boost_PYTHON_FOUND AND Boost_THREAD_FOUND )
find_package(PythonLibs)
if( PYTHONLIBS_FOUND OR PYTHON_LIBRARIES )
if( PYTHON_EXECUTABLE )
# get the site-packages directory
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "from distutils.sysconfig import get_python_lib; print get_python_lib(1)"
OUTPUT_VARIABLE _python_sitepackage
RESULT_VARIABLE _python_failed)
if( ${_python_failed} EQUAL 0 )
string(REGEX REPLACE "[\r\n]" "" _python_sitepackage "${_python_sitepackage}")
set(PYTHON_INCLUDE_PATH ${PYTHON_INCLUDE_PATH} ${_python_sitepackage}/numpy/core/include)
else()
message(STATUS "failed to get python site-package directory")
endif()
endif()
include_directories(${PYTHON_INCLUDE_PATH} ${OpenRAVE_INCLUDE_DIRS})
add_library(orpythonbinding SHARED orpythonbinding.cpp)
target_link_libraries(orpythonbinding ${OpenRAVE_LIBRARIES} ${PYTHON_LIBRARIES} ${Boost_PYTHON_LIBRARY} ${Boost_THREAD_LIBRARY})
set_target_properties(orpythonbinding PROPERTIES PREFIX "" COMPILE_FLAGS "${OpenRAVE_CXX_FLAGS}")
if( WIN32 )
set_target_properties(orpythonbinding PROPERTIES SUFFIX ".pyd")
endif()
endif()
endif()

Loading DLL from embedded python code in c

The crux of my problem is this:
I am developing code on Windows XP in C with MS Visual Studio 10.0, and I need to embed Python to do some plotting, file management, and some other things. I had problems with sys.path finding my Pure-Python modules, but I have fixed the problem by modifying PYTHONPATH.
Now, my problem is getting python to find dynamic libraries that are pulled in by some modules. In particular, my problem is to compress a folder into a bzip2 achive of the same name.
From a normal python command prompt, this works just fine:
import tarfile
tar=tarfile.open('Code.tar.bz2','w:bz2')
tar.add('Code',arcname='Code')
tar.close()
But when I call this code from my c-code, it gives me this error:
Traceback (most recent call last):
File "<string>", line 4, in <module>
File "D:\My_Documents\Code\ScrollModel\trunk\PythonCode.py", line 20, in Colle
ctFiles
tar=tarfile.open(os.path.join(runPath,'CODE.tar.bz2'),'w:bz2')
File "c:\Python26\lib\tarfile.py", line 1671, in open
return func(name, filemode, fileobj, **kwargs)
File "c:\Python26\lib\tarfile.py", line 1737, in bz2open
raise CompressionError("bz2 module is not available")
tarfile.CompressionError: bz2 module is not available
I have a suspicion the problem is similar to what is described at section 5.6 of Embedded Python, but it is a bit hard to tell. For what its worth, if I do
Py_Initialize();
PyRun_SimpleString("import ssl\n");
Py_Finalize();
it doesn't work either and I get an ImportError.
Anyone had any problems like this? Am I missing something critical?

Try this, it works on my machine.
Create a simple Windows console application in Visual Studio 2010 (remove precompiled headers option in the wizard). Replace the generated code with this one :
#include <Python.h>
int main(int argc, char *argv[]) {
Py_Initialize();
PyRun_SimpleString("import ssl \n"
"for f in dir(ssl):\n"
" print f \n" );
Py_Finalize();
return 0;
}
With PYTHONHOME set to something like c:\Python...
add C:\Python\Include to the include path
add C:\Python\Libs to the library path
add python26.lib to the linker input (adjust with your Python version)
Build. Run from anywhere and you should see a listing of the content of the ssl module.
I also tried with Mingw. Same file, build with this command line :
gcc -Wall -o test.exe embeed.c -I%PYTHONHOME%\Include -L%PYTHONHOME%\libs -lpython26

Hey, I have asked a similar question, my operation system is Linux.
When i compile c file, option $(python-config --cflags --ldflags) should be added, as
gcc test.c $(python-config --cflags --ldflags) -o test
I think in Windows you may also check python-config option, Hope this helps!

I had a similar problem with Boost C++ DLL. Any external DLL need to be in the DLL search path.
In my experience, PYTHONPATH affects Python module (the import statement in Python will end up a LoadLibrary call), and build options have nothing to do with it.
When you load a DLL, Windows doesn't care what the process is. In other words, Python follows the same DLL loading rules as Notepad. You can confirm that you are facing a Windows path problem by copying any missing DLL in the same directory as your python extension, or to a directory in your path.
To find what DLL are required by any other executable or DLL, simply open the DLL or EXE file with DependencyWalker. There is also a "Profile" menu which will allow you to run your application and watch it search and load DLLs.