We Keep Coding

How build pyobjc 7.3 with nix?

I want to build pyobjc-7.3, because it has fix for send2trash.
Classic building on BigSur 20.5.0 is strait forward.
cd pyobjc-7.3/pyobjc-framework-Cocoa
python3 setup.py build
though once I run same build inside nix-shell magic happens.
nix-shell -p pkgs.python39Packages.setuptools
python3 setup.py build
clang-7: error: argument unused during compilation:
'-fno-strict-overflow' [-Werror,-Wunused-command-lin\ e-argument]
ok. no big deal. let's disable warning.
CFLAGS="-Wno-unused-argument" python3 setup.py build
what? now clang is like a blind kitten.
Modules/pyobjc-api.h:19:10: fatal error: 'objc/objc.h' file not found
#include <objc/objc.h>
-isysroot option and -I has no effect.
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk
I noticed lots additions to -I flag in clang such as:
-iwithprefix /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include
it helps clang to find objc header file, though this is not the end of the story.
Modules/pyobjc-api.h:21:9: fatal error: 'Foundation/Foundation.h' file
not found
how come?!
oh there is another header files of special kind - frameworks. Wheel reinvention...
clang, take another argument
-iframeworkwithsysroot /System/Library/Frameworks
Here I get tons of type errors and I run out of ideas what could I try next:
/Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/System/Library/Frameworks/Foundation.framework/Headers/NSString.h:138:1:
error:
function cannot return function type 'NSComparisonResult' (aka 'int (int)')
(NSComparisonResult)compare:(NSString *)string options:(NSStringCompareOptions)mask range:(NSR...

After days of trying I've found solution.
There are a few bugs are causing the problems:
First is nix provides older (10.12) sdk while setup.py thinks is 10.15.
This enables CPP sections for unsupported SDK API therefore type errors.
Following hack makes pyobjc to think that SDK is older that it is.
with pkgs;
with pkgs.lib;
with pkgs.python39Packages;
let
pyobjc-core = buildPythonPackage rec {
pname = "pyobjc-core";
version = "7.3";
name = "${pname}-${version}";
src = pkgs.python39Packages.fetchPypi {
pname = "pyobjc-core";
inherit version;
sha256 = "0x3msrzvcszlmladdpl64s48l52fwk4xlnnri8daq2mliggsx0ah";
};
preBuild=''
export SDKROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX10.12.sdk"
Second problem is with header discovery and overstrict lint from python nix
CFLAGS = "-iwithsysroot /usr/include -Wno-unused-argument";
Third problem big sur linkder is dynamic and ffi libray is not found.
Providing through nix derivation
buildInputs = [ pkgs.libffi ];
Forth problem is tests are broken
doCheck = false;

How to package a linked DLL and a pyd file into one self contained pyd file?

I am building a python module with Cython that links against a DLL file. In order to succesfully import my module I need to have the DLL in the Windows search path. Otherwise, the typical error message is:
ImportError: DLL load failed: The specified module could not be found.
Is there a way to packaged the DLL directly into the produced pyd file to make the distribution easier?
One example of this is with the OpenCV distribution, where a (huge) pyd file is distributed and is the only file needed for the Python bindings to work.

Python's packaging & deployment is still a pain point for many of us. There is just not a silver bullet. Here are several methods:
1. OpenCV build method
The method is decribed here : https://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_bindings/py_bindings_basics/py_bindings_basics.html#bindings-basics
OpenCV generates these wrapper functions automatically from the C++
headers using some Python scripts which are located in
modules/python/src2.
Basically it parse the header files and generate the static PyObject keywords whenever it's needed. Once the header are created appropriately, it just call python setup. Honestly, it might work, but I would not advise this method.
2. Makefiles
If you already use Makefile, just create a rule to place your lib accordinlgy. Example, from my own code :
setup.py
from distutils.core import setup, Extension
setup(name='sha1_hmac', version='1.0', \
ext_modules=[Extension('sha1_hmac',
library_dirs=['C:\MinGW\lib'],
sources= ['../tools/sha1.c','sha1_hmac.c'])])
Makefile
# The hmac generation used by the webserver is done
# using the sha1.c implementation. There is a binding needed to
# glue the C code with the python script
libsha1_hmac:
ifeq ($(OS), Windows_NT)
$(PYTHON) setup.py build --compiler=mingw32
else
$(PYTHON) setup.py install --home=$(CURDIR)
endif
.PHONY: webserver
webserver: libsha1_hmac
ifeq ($(OS), Windows_NT)
mv $(shell find build -type f -name "sha1*.pyd") $(LIB)
else
mv -f $(shell find $(LIB)/python -type f -name "sha1*.so") $(LIB)
endif
$(PYTHON) hmac_server.py
3. Modern deployement tools
There are several new tools to deploy python applications, namely wheels which seem to gain traction. I don't use it, but it look like it can ease up your bundling problem :
How can I make a Python Wheel from an existing native library?
Once it wheeled, you can install it like this : pip install some-package.whl

Cython and fortran - how to compile together without f2py

FINAL UPDATE
This question is about how to write a setup.py that will compile a cython module which accesses FORTRAN code directly, like C would. It was a rather long and arduous journey to the solution, but the full mess is included below for context.
ORIGINAL QUESTION
I have an extension which is a Cython file, which sets up some heap memory and passes it to the fortran code, and a fortran file, which is a venerable old module that I'd like to avoid reimplementing if I can.
The .pyx file compiles fine to C, but the cython compiler chokes on the .f90 file with the following error:
$ python setup.py build_ext --inplace
running build_ext
cythoning delaunay/__init__.pyx to delaunay/__init__.c
building 'delaunay' extension
error: unknown file type '.f90' (from 'delaunay/stripack.f90')
Here's (the top half of) my setup file:
from distutils.core import setup, Extension
from Cython.Distutils import build_ext
ext_modules = [
Extension("delaunay",
sources=["delaunay/__init__.pyx",
"delaunay/stripack.f90"])
]
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
...
)
NOTE: I originally had the fortran file's location incorrectly specified (without the directory prefix) but this breaks in exactly the same way after I fixed that.
Things I have tried:
I found this, and tried passing in the name of the fortran compiler (i.e. gfortran) like this:
$ python setup.py config --fcompiler=gfortran build_ext --inplace
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --fcompiler not recognized
And I've also tried removing --inplace, in case that was the problem (it wasn't, same as the top error message).
So, how do I compile this fortran? Can I hack it into a .o myself and get away with linking it? Or is this a bug in Cython, which will force me to reimplement distutils or hack around with the preprocessor?
UPDATE
So, having checked out the numpy.distutils packages, I understand the problem a bit more. It seems that you have to
Use cython to convert the .pyx files to cpython .c files,
Then use an Extension/setup() combination that supports fortran, like numpy's.
Having tried this, my setup.py now looks like this:
from numpy.distutils.core import setup
from Cython.Build import cythonize
from numpy.distutils.extension import Extension
cy_modules = cythonize('delaunay/sphere.pyx')
e = cy_modules[0]
ext_modules = [
Extension("delaunay.sphere",
sources=e.sources + ['delaunay/stripack.f90'])
]
setup(
ext_modules = ext_modules,
name="delaunay",
...
)
(note that I've also restructured the module a bit, since seemingly an __init__.pyx is disallowed...)
Now is where things become buggy and platform-dependent. I have two testing systems available - one Mac OS X 10.6 (Snow Leopard), using Macports Python 2.7, and one Mac OS X 10.7 (Lion) using the system python 2.7.
On Snow Leopard, the following applies:
This means that the module compiles (hurray!) (although there's no --inplace for numpy, it seems, so I had to system-wide install the testing module :/) but I still get a crash on import as follows:
>>> import delaunay
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "<snip>site-packages/delaunay/__init__.py", line 1, in <module>
from sphere import delaunay_mesh
ImportError: dlopen(<snip>site-packages/delaunay/sphere.so, 2): no suitable image found. Did find:
<snip>site-packages/delaunay/sphere.so: mach-o, but wrong architecture
and on Lion, I get a compile error, following a rather confusing looking compile line:
gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
/usr/local/bin/gfortran -Wall -arch i686 -arch x86_64 -Wall -undefined dynamic_lookup -bundle build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/fortranobject.o build/temp.macosx-10.7-intel-2.7/delaunay/stripack.o build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.o -lgfortran -o build/lib.macosx-10.7-intel-2.7/delaunay/sphere.so
ld: duplicate symbol _initsphere in build/temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o ldand :build /temp.macosx-10.7-intelduplicate- 2.7symbol/ delaunay/sphere.o _initsphere in forbuild architecture /i386
temp.macosx-10.7-intel-2.7/build/src.macosx-10.7-intel-2.7/delaunay/spheremodule.o and build/temp.macosx-10.7-intel-2.7/delaunay/sphere.o for architecture x86_64
Now let's just step back a moment before we pore over the details here. Firstly, I know there are a bunch of headaches over architecture clashes in 64-bit Mac OS X; I had to work very hard to get Macports Python working on the Snow Leopard machine (just to upgrade from system python 2.6). I also know that when you see gfortran -arch i686 -arch x86_64 you are sending mixed messages to your compiler. There are all manner of platform-specific problems buried in there, that we don't need to worry about in the context of this question.
But let's just look at this line:
gfortran:f77: build/src.macosx-10.7-intel-2.7/delaunay/sphere-f2pywrappers.f
what is numpy doing?! I don't need any f2py features in this build! I actually wrote a cython module in order to avoid dealing with f2py's insanity (I need to have 4 or 5 output variables, as well as neither-in-nor-out arguments - neither of which is well supported in f2py.) I just want it to compile .c -> .o, and .f90 -> .o and link them. I could write this compiler line myself if I knew how to include all the relevant headers.
Please tell me I don't need to write my own makefile for this... or that there's a way to translate fortran to (output-compatible) C so I can just avoid python ever seeing the .f90 extension (which fixes the whole problem.) Note that f2c is not suitable for this as it only works on F77 and this is a more modern dialect (hence the .f90 file extension).
UPDATE 2
The following bash script will happily compile and link the code in place:
PYTHON_H_LOCATION="/opt/local/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7/"
cython sphere.pyx
gcc -arch x86_64 -c sphere.c -I$PYTHON_H_LOCATION
gfortran -arch x86_64 -c stripack.f90
gfortran -arch x86_64 -bundle -undefined dynamic_lookup -L/opt/local/lib *.o -o sphere.so
Any advice on how to make this kind of hack compatible with a setup.py? I don't anyone installing this module to have to go find Python.h manually...

UPDATE: I've created a project on github which wraps up this generating of compile lines by hand. it's called complicated_build.
UPDATE 2: in fact, "generating by hand" is a really bad idea as it's platform specific — the project now reads the values from the distutils.sysconfig module, which is the settings used to compile python (i.e. exactly what we want,) the only setting which is guessed is fortran compiler and file extensions (which are user-configurable). I suspect it is reimplementing a fair bit of distutils now!
The way to do this is to write your own compiler lines, and hack them into your setup.py. I show an example below which works for my (very simple) case, which has the following strucutre:
imports
cythonize() any .pyx files, so you only have fortran and C files.
define a build() function which compiles your code:
maybe some easy-to-change constants, like compiler names and architecture
list up the fortran and C files
generate the shell commands that will build the modules
add the linker line
run the shell commands.
if the command was install and the target doesn't exist yet, build it.
run setup (which will build the pure python sections)
if the command was build, run the build now.
my implementation of this is shown below. It's designed for only one extension module, and it recompiles all the files every time, so may require further extension to be of more general use. Also note that I've hard coded various unix /s, so if you're porting this to windows make sure you adapt or replace with os.path.sep.
from distutils.core import setup
from distutils.sysconfig import get_python_inc
from Cython.Build import cythonize
import sys, os, shutil
cythonize('delaunay/sphere.pyx')
target = 'build/lib/delaunay/sphere.so'
def build():
fortran_compiler = 'gfortran'
c_compiler = 'gcc'
architecture = 'x86_64'
python_h_location = get_python_inc()
build_temp = 'build/custom_temp'
global target
try:
shutil.rmtree(build_temp)
except OSError:
pass
os.makedirs(build_temp) # if you get an error here, please ensure the build/ ...
# folder is writable by this user.
c_files = ['delaunay/sphere.c']
fortran_files = ['delaunay/stripack.f90']
c_compile_commands = []
for cf in c_files:
# use the path (sans /s), without the extension, as the object file name:
components = os.path.split(cf)
name = components[0].replace('/', '') + '.'.join(components[1].split('.')[:-1])
c_compile_commands.append(
c_compiler + ' -arch ' + architecture + ' -I' + python_h_location + ' -o ' +
build_temp + '/' + name + '.o -c ' + cf
)
fortran_compile_commands = []
for ff in fortran_files:
# prefix with f in case of name collisions with c files:
components = os.path.split(ff)
name = components[0].replace('/', '') + 'f' + '.'.join(components[1].split('.')[:-1])
fortran_compile_commands.append(
fortran_compiler + ' -arch ' + architecture + ' -o ' + build_temp +
'/' + name + '.o -c ' + ff
)
commands = c_compile_commands + fortran_compile_commands + [
fortran_compiler + ' -arch ' + architecture +
' -bundle -undefined dynamic_lookup ' + build_temp + '/*.o -o ' + target
]
for c in commands:
os.system(c)
if 'install' in sys.argv and not os.path.exists(target):
try:
os.makedirs('build/lib/delaunay')
except OSError:
# we don't care if the containing folder already exists.
pass
build()
setup(
name="delaunay",
version="0.1",
...
packages=["delaunay"]
)
if 'build' in sys.argv:
build()
This could be wrapped up into a new Extension class I guess, with it's own build_ext command - an exercise for the advanced student ;)

Simply build and install your vintage Fortran library outside of Python, then link to it in distutils. Your question indicates that you do not intend to temper with this library, so a once-and-for-all install will probably do (using the library's build and installation instructions). Then link the Python extension to the installed external library:
ext_modules = [
Extension("delaunay",
sources = ["delaunay/__init__.pyx"],
libraries = ["delaunay"])
]
This approach is also safe for the case that you realize that you need wrappers for other languages as well, such as Matlab, Octave, IDL, ...
Update
At some point, if you end up with more than a few such external libraries that you want to wrap, it is advantageous to add a top-level build system that installs all these libraries, and manages the building of all the wrappers as well. I have cmake for this purpose, which is great at handling system-wide builds and installations. However, it cannot build Python stuff out of the box, but it can be taught easily to call "python setup.py install" in each subdirectory python, thus invoking distutils. So the overall build process looks like this:
mkdir build
cd build
cmake ..
make
make install
make python
(make octave)
(make matlab)
It is very important to always separate core library code from wrappers for specific front-end languages (also for your own projects!), since they tend to change rather fast. What happens otherwise can be seen at the example of numpy: Instead of writing one general-purpose C library libndarray.so and creating thin wrappers for Python, there are Python C API calls everywhere in the sources. This is what is now holding back Pypy as a serious alternative to CPython, since in order to get numpy they have to support every last bit of the CPython API, which they can't do, since they have a just-in-time compiler and a different garbage collector. This means we are missing out on a lot of potential improvements.
Bottom line:
Build general purpose Fortran/C libraries separately and install them system-wide.
Have a separate build step for the wrappers, which should be kept as lightweight as possible, so that it's easy to adapt for the next big language X that comes about. If there is one safe assumption, it is that X will support linking with C libraries.

You can build the object file outside of distutils then include it at the linking step using the extra_objects argument to Extension's constructor. In setup.py:
...
e = Extension(..., extra_objects = ['holycode.o'])
...
On the command prompt:
# gfortran -c -fPIC holycode.f
# ./setup.py build_ext ...
With only one external object, this will be the easiest way for many.

How may I override the compiler (GCC) flags that setup.py uses by default?

I understand that setup.py uses the same CFLAGS that were used to build Python. I have a single C extension of ours that is segfaulting. I need to build it without -O2 because -O2 is optimizing out some values and code so that the core files are not sufficient to pin down the problem.
I just need to modify setup.py so that -O2 is not used.
I've read distutils documentation, in particular distutils.ccompiler and distutils.unixccompiler and see how to add flags and libs and includes, but not how to modify the default GCC flags.
Specifically, this is for a legacy product on Python 2.5.1 with a bunch of backports (Fedora 8, yes, I know...). No, I cannot change the OS or Python version and I cannot, without great problems, recompile Python. I just need to build a one off of the C extension for one customer whose environment is the only one segfaulting.

Prepend CFLAGS="-O0" before you run setup.py:
% CFLAGS="-O0" python ./setup.py
The -O0 will be appended to CFLAGS while compiling, therefore will override previous -O2 setting.
Another way is add -O0 to extra_compile_args in setup.py:
moduleA = Extension('moduleA', .....,
include_dirs = ['/usr/include', '/usr/local/include'],
extra_compile_args = ["-O0"],
)
If you want to remove all default flags, use:
% OPT="" python ./setup.py

I ran into this problem when I needed to fully remove a flag (-pipe) so I could compile SciPy on a low-memory system. I found that, as a hack, I could remove unwanted flags by editing /usr/lib/pythonN.N/_sysconfigdata.py to remove every instance of that flag, where N.N is your Python version. There are a lot of duplicates, and I'm not sure which are actually used by setup.py.

distutils/​setuptools allows any compiler/​linker flags to be specified with extra_compile_args/​extra_link_args argument when defining a Python extension in setup.py script. These extra flags will be added after default ones and will override any mutually exclusive flags present earlier.
For regular use, however, this is not much useful as a package you distribute through PyPI can be built by different compilers having an incompatible options.
The following code allows you to specify these options in extension- and compiler-specific way:
from setuptools import setup
from setuptools.command.build_ext import build_ext
class build_ext_ex(build_ext):
extra_compile_args = {
'extension_name': {
'unix': ['-O0'],
'msvc': ['/Od']
}
}
def build_extension(self, ext):
extra_args = self.extra_compile_args.get(ext.name)
if extra_args is not None:
ctype = self.compiler.compiler_type
ext.extra_compile_args = extra_args.get(ctype, [])
build_ext.build_extension(self, ext)
setup(
...
cmdclass = {'build_ext': build_ext_ex},
...
)
Of course you could simplify it if you want all extensions to use the same (but still compiler-specific) options.
Here is a list of supported compiler types (as returned by setup.py build_ext --help-compiler):
--compiler=bcpp Borland C++ Compiler
--compiler=cygwin Cygwin port of GNU C Compiler for Win32
--compiler=mingw32 Mingw32 port of GNU C Compiler for Win32
--compiler=msvc Microsoft Visual C++
--compiler=unix standard UNIX-style compiler

Error setting up thrift modules for python

I'm trying to set up thrift in order to incorporate with Cassandra, so when I ran the
setup.py
it out puts this message in command line
running build
running build_py
running build_ext
building 'thrift.protocol.fastbinary' extension
C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python26\include -IC:\Pytho
n26\PC -c src/protocol/fastbinary.c -o build\temp.win32-2.6\Release\src\protocol
\fastbinary.o
src/protocol/fastbinary.c:24:24: netinet/in.h: No such file or directory
src/protocol/fastbinary.c:85:4: #error "Cannot determine endianness"
src/protocol/fastbinary.c: In function `writeI16':
src/protocol/fastbinary.c:295: warning: implicit declaration of function `htons'
src/protocol/fastbinary.c: In function `writeI32':
src/protocol/fastbinary.c:300: warning: implicit declaration of function `htonl'
src/protocol/fastbinary.c: In function `readI16':
src/protocol/fastbinary.c:688: warning: implicit declaration of function `ntohs'
src/protocol/fastbinary.c: In function `readI32':
src/protocol/fastbinary.c:696: warning: implicit declaration of function `ntohl'
error: command 'gcc' failed with exit status 1
Need some helping on this issue.I have already install the MigW32
Thanks.

With a bit of source file tweaking it is possible to install it with MINGW on Windows. I'm using thrift 0.9.1 and Python 27
The steps I followed were:
If you are using Python 2.7, follow the normal setup steps and workarounds for minGW. In particular you may need to open the file C:\Python27\Lib\distutils\cygwinccompiler.py, and modify the class Mingw32CCompiler to remove all references to the -mno-cygwin option. This option is deprecated and will cause the compiler to halt with an error if it is left in.
Open fastbinary.c and add the following include statement,
#include <stdbool.h>,
this includes definitions for true / false which will otherwise cause the compile to fail. (I assume they're included by default on MSVC?)
3) Modify the setup.py file to tell the linker to link to ws2_32.lib. This is done using a pragma comment on MSVC, but gcc doesn't support this option. So your ext_modules should look like :
ext_modules = [
Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries=['ws2_32'],
include_dirs = include_dirs,
)
],
4) Build as normal using the setup.py
On my setup, I didn't get much of a speed improvement when using the C extension rather than pure python (about a 5% difference), so the effort to do this might not be justified except in extreme cases.

I've only succeeded in installing Thrift with MSVC.
Install MSVC
Get Thrift
Apply thrift-252-python-msvc-1.diff patch (google it)
The fastbinary.c will be patched, but setup.py patch will fail, update manually from hints at setup.py.rej, here's a (seemingly) correct copy:
from distutils.core import setup, Extension
import sys
libraries = []
if sys.platform == 'win32':
libraries.append('ws2_32')
fastbinarymod = Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries = libraries,
)
setup(name = 'Thrift',
version = '0.1',
description = 'Thrift Python Libraries',
author = 'Thrift Developers',
author_email = 'thrift-dev#incubator.apache.org',
url = 'http://incubator.apache.org/thrift/',
license = 'Apache License 2.0',
packages = [
'thrift',
'thrift.protocol',
'thrift.transport',
'thrift.server',
],
package_dir = {'thrift' : 'src'},
ext_modules = [fastbinarymod],
)
Endianness test will fail, modify fastbinary.c (around line 68):
#ifdef _MSC_VER
#define __BYTE_ORDER __LITTLE_ENDIAN
#endif
After that run python setup.py install, hopefully, you'll get what you need.

Install python-dev
You can run:
sudo apt-get install python-dev

I would recommend you to execute the following command:
pip3 install thriftpy2