We Keep Coding

How can I make my PyO3-based libraries more portable across Python versions?

I have a library that I've created with PyO3 on my system using Python 3.5.2. The .so file links to the corresponding libpython3.5m file:
$ ldd my_library.so
linux-vdso.so.1 => (0x00007ffffc823000)
libpython3.5m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.5m.so.1.0 (0x00007fcac34b0000)
...
But if I try to use my library on another system with a different minor version of Python (eg, 3.6.9 or 3.7.3), that library doesn't exist:
$ ldd my_library.so
linux-vdso.so.1 (0x00007fffefaae000)
libpython3.5m.so.1.0 => not found
...
And therefore, I can't use my library:
$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_library
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: libpython3.5m.so.1.0: cannot open shared object file: No such file or directory
Is there a way that I can link my library to libpython more generically to be resilient to these minor version changes or at least allow backward-compatibility (build on newer version but allow older versions of Python to also use my library)? Or anything I can do at runtime to let the library still function?

Python extension modules shouldn't link to libpython at all. The Python symbols will stay unresolved, and the dynamic loader will look them up in whichever python interpreter loads the module. That way you can even use your module with a statically linked Python!
rust-cpython has an extension-module feature for this and PyO3 seems to have copied that.
You an enable it in your Cargo.toml:
[dependencies]
pyo3 = { version = ..., features = [ "extension-module" ] }

python import cx_Oracle error on cygwin

I tried to install cx_Oracle from pypi source since there is no available port for it in cygwin. I did make some changes as suggested in http://permalink.gmane.org/gmane.comp.python.db.cx-oracle/2492 and modified my setup.py. However, I still get the following error :-
$ python
Python 2.7.3 (default, Dec 18 2012, 13:50:09)
[GCC 4.5.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
/usr/lib/python2.7/site-packages/cx_Oracle-5.1.3-py2.7-cygwin-1.7.24-i686.egg/cx_Oracle.py:3: UserWa
rning: Module cx_Oracle was already imported from /usr/lib/python2.7/site-packages/cx_Oracle-5.1.3-p
y2.7-cygwin-1.7.24-i686.egg/cx_Oracle.pyc, but /home/zerog/cx_Oracle-5.1.3 is being added to sys.pat
h
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.cygwin-1.7.24-i686/egg/cx_Oracle.py", line 7, in <module>
File "build/bdist.cygwin-1.7.24-i686/egg/cx_Oracle.py", line 6, in __bootstrap__
ImportError: Exec format error
>>>
If someone can please help me fix this ?
TIA.

Fixed this by specifying the path to instantclient as below :
$ export PATH=$PATH:/cygdrive/d/Tools/instantclient_11_2
(Other, possibly important stuff) :
$ echo $LD_LIBRARY_PATH
/cygdrive/d/Tools/instantclient_11_2
$ echo $ORACLE_HOME
/cygdrive/d/Tools/instantclient_11_2
Now, I get :-
$ python
Python 2.7.3 (default, Dec 18 2012, 13:50:09)
[GCC 4.5.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>>

It's hard to pin down from the error message alone, but I am guessing that you have two different copies of cx_Oracle in your sys.path. The error message is complaining that a different version of the same module had already been import-ed.
Presumably the pristine upstream version is installed system-wide in /usr/lib/python2.7/site-packages/cx_Oracle-5.1.3-py2.7-cygwin-1.7.24-i686.egg, and your modified version in /home/zerog/cx_Oracle-5.1.3.
Does it work if you pare down sys.path so that only the original, or only your modified version, is included?
(You might want to use virtualenv if you need to switch back and forth between two versions frequently.)

I ran into this error "Exec format error."
For me, this was likely caused by a mismatch between cygwin being installed as 64 bit, but the instant client being installed as 32 bit. Double check that everything (oracle, cygwin) is either 32 bit or 64 bit.
What fixed my issue:
Since my cygwin is 64 bit (see uname -a, and look for x86_64), I downloaded the 64 bit instant client from oracle's website, and unzipped
I set the env vars in .profile, to point where it was unzipped:
export ORACLE_HOME=/cygdrive/c/oracle/instantclient_x64_11_2
export LD_LIBRARY_PATH=$ORACLE_HOME
export DYLD_LIBRARY_PATH=$ORACLE_HOME
export TNS_ADMIN='//optional/path/to/your/oracle/tns/files/'
source ~/.profile
To test, you should now be able to run this python command with no error:
import cx_Oracle
To verify the path is correct, if you run ls, you should see something like
ls $ORACLE_HOME
adrci.exe genezi.exe oci.sym ociw32.dll ojdbc6.jar
oraocci11.dll oraociei11.sym uidrvci.exe vc9
adrci.sym genezi.sym ocijdbc11.dll ociw32.sym orannzsbb11.dll
oraocci11.sym orasql11.dll uidrvci.sym xstreams.jar
BASIC_README oci.dll ocijdbc11.sym ojdbc5.jar orannzsbb11.sym
oraociei11.dll orasql11.sym vc8

Running SL4A facade APIs from host python shell

Objective:
Trying to run SL4A facade APIs from python shell on the host system (windows 7 PC)
My environment:
1. On my windows 7 PC, i have python 2.6.2
2. Android sdk tools rev 21, platform tools rev 16
3. API level 17 supported for JB 4.2
4. I have 2 devices ( one running android 2.3.3 and another android 4.2.2) both running Python for android and SL4A
I'm trying these commands as specified at http://code.google.com/p/android-scripting/wiki/RemoteControl
Here are the commands which i'm trying on the python shell:
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import android
>>> droid=android.Android
>>> droid.makeToast("Hello")
Traceback (most recent call last):
File "", line 1, in
AttributeError: type object 'Android' has no attribute 'makeToast'
Before this i'm performing the port forwarding and starting a private server as shown below
$ adb forward tcp:9999 tcp:4321
$ set AP_PORT=9999
Also set the server on the target listening on port 9999 ( through SL4A->preferences->serverport.
Please help to understand where i'm doing mistake which gives the above error while trying droid.makeToast("Hello") ?

Try
import android
droid=android.Android()
droid.makeToast("Hello")
(you are missing parentheses after android.Android)

Hope you solved the problem. This is what I found, so maybe for somebody else it will be useful.
You need to run all your commands inside the DOS shell...otherwise you will get that error[11001].
If you still want to run you commands outside the DOS shell - add a new environment variable as described here:
http://www.smartphonedaq.com/installation.page

Problem with sh profile in IPython on Mac

I recently heard about profiles in python and quickly discovered the sh profile. I went to invoke it to try it out and got the following error on my Mac running Snow Leopard and the system Python2.6. Anyone else see this before? I was thinking about trying to reinstall the IPython egg to see if it fixed it, but thought I'd ask if any other users had seen something similar (on an Apple computer most likely).
mwoods 5 22:11:48 ~$ ipython -p sh
[IPythonApp] Config file not found, skipping: <ipython_config_sh.py>
Traceback (most recent call last):
File "/Library/Python/2.6/site-packages/ipython-0.11.bzr.r1205-py2.6.egg/IPython/core/application.py", line 223, in load_file_config
self.file_config = loader.load_config()
File "/Library/Python/2.6/site-packages/ipython-0.11.bzr.r1205-py2.6.egg/IPython/config/loader.py", line 227, in load_config
self._find_file()
File "/Library/Python/2.6/site-packages/ipython-0.11.bzr.r1205-py2.6.egg/IPython/config/loader.py", line 234, in _find_file
self.full_filename = filefind(self.filename, self.path)
File "/Library/Python/2.6/site-packages/ipython-0.11.bzr.r1205-py2.6.egg/IPython/utils/genutils.py", line 569, in filefind
(filename, path_dirs))
IOError: File does not exist in any of the search paths: 'ipython_config_sh.py', ('/Users/mwoods', u'/Users/mwoods/.ipython')
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29)
Type "copyright", "credits" or "license" for more information.
IPython 0.11.bzr.r1205 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object'. ?object also works, ?? prints more.
In [1]:

I had success running ipython --profile=sh. However, the features of the sh profile (as advertised here) weren't available.

Django + MySQL on Mac OS 10.6.2 Snow Leopard

There were some excellent answers to this question already, however, they are now outdated.
I've been able to get the module installed, but "python manage.py runserver" fails with
iMac:myproject drhoden$ python manage.py runserver
Validating models...
Unhandled exception in thread started by <function inner_run at 0x10496f0>
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/commands/runserver.py", line 48, in inner_run
self.validate(display_num_errors=True)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/base.py", line 249, in validate
num_errors = get_validation_errors(s, app)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/core/management/validation.py", line 22, in get_validation_errors
from django.db import models, connection
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 41, in <module>
backend = load_backend(settings.DATABASE_ENGINE)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/__init__.py", line 17, in load_backend
return import_module('.base', 'django.db.backends.%s' % backend_name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/utils/importlib.py", line 35, in import_module
__import__(name)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 13, in <module>
raise ImproperlyConfigured("Error loading MySQLdb module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: dynamic module does not define init function (init_mysql)
^CiMac:segisys drhoden$
Likewise, from the python shell:
iMac:myproject drhoden$ python
Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.3-fat/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.3-fat/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.3-fat/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dynamic module does not define init function (init_mysql)
>>>
Using MySQL-python-1.2.3c1 with setuptools-0.6c11-py2.6.egg
Any help would be appreciated.

I have ultimately solved my own problem, with of course, the subconscious and conscious help from the many posts, blogs, and mail logs I've read. I would give links if I could remember.
In a nutshell, I reinstalled EVERYTHING using MacPorts.
After editing ~/.bash_profile and commenting out all the previous modifications to ${PATH}, I downloaded the dmg for Snow Leopard and ran through its installation.
Then opened the terminal and ran the self update.
sudo port selfupdate
sudo port install python26
That second part, installing Python 2.6, took forever. But when it completed it prompted me with the following:
To fully complete your installation and make python 2.6 the default, please run
sudo port install python_select
sudo python_select python26
I did both and they went quick.
I forgot to mention how handy 'port search ' command is. I searched for 'mysql' and similar to find the thing to type after 'install'. But I proceeded with reinstalling both the client and server for MySQL. Perhaps I did this in reverse order, but the end result worked fine.
sudo port install mysql5
...
---> Installing mysql5 #5.1.41_0
The MySQL client has been installed.
If you also want a MySQL server, install the mysql5-server port.
So naturally:
sudo port install mysql5-server
I love how the so many of the macports installations give you feedback as to what to do next. At the end of the server installation, it said the following:
******************************************************
* In order to setup the database, you might want to run
* sudo -u _mysql mysql_install_db5
* if this is a new install
******************************************************
It was a new install for me (didn't have any local schemas). For completeness, and for my own reference, here is the output of running that command:
Installing MySQL system tables...
OK
Filling help tables...
OK
To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:
/opt/local/lib/mysql5/bin/mysqladmin -u root password 'new-password'
/opt/local/lib/mysql5/bin/mysqladmin -u root -h iMac.local password 'new-password'
Alternatively you can run:
/opt/local/lib/mysql5/bin/mysql_secure_installation
which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.
See the manual for more instructions.
You can start the MySQL daemon with:
cd /opt/local ; /opt/local/lib/mysql5/bin/mysqld_safe &
You can test the MySQL daemon with mysql-test-run.pl
cd /opt/local/mysql-test ; perl mysql-test-run.pl
Please report any problems with the /opt/local/lib/mysql5/bin/mysqlbug script!
The latest information about MySQL is available at http://www.mysql.com/
Support MySQL by buying support/licenses from http://shop.mysql.com/
Almost done. Earlier in my 'port search'ing I came across this interesting port:
py26-mysql #1.2.2 (python, devel, databases)
Python interface to mysql
With much, much hope that this would provide me with MySQLdb package, I installed it (and it did).
sudo port install py26-mysql
Afterwords I cranked up the python interpreter attempted to import MySQLdb, the very thing in my way all this time.
iMac:~ drhoden$ python
Python 2.6.4 (r264:75706, Dec 15 2009, 18:00:14)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet
>>>
A warning, but It worked!!
Just one more thing:
sudo port install py26-django
After all of this I was finally able to crank up my Django project and remotely connect to my company's MySQL server!! It may not have been necessary to reinstall Django using MacPorts, but I wasn't going to risk complications.

I wrote a blog post a few months ago following my successful installation of MySQL on Snow Leopard:
http://jboxer.com/2009/09/installing-mysql-on-snow-leopard/
If you follow those steps, it should (theoretically) fix your problem (which sounds like it's caused by a mix of 32-bit and 64-bit software).
By the way, I'm not trying to self-promote here; the text in the blog post is basically what I would've posted here, and I'm trying to apply DRY to more areas of my life :)

This happens when you have mixed 32 and 64bit software.
Basically, for Snow Leopard, you need to install MySQL 64bit package (which still is listed as 10.5, but that is no problem) , after that do an easy install of python-mysql again. All will work.