We Keep Coding

failed to create executable with pyinstaller and cefpython on Linux (Invalid file descriptor to ICU data)

I have some simple cefpython code opening a url and am trying to create a stand alone executable with pyinstaller:
I copied files from https://github.com/cztomczak/cefpython/tree/master/examples/pyinstaller to a a directry named pyinstaller
I made following minor changes to pyinstaller.spec
+SECRET_CIPHER = ""
...
- ["../wxpython.py"],
+ ["../hello.py"],
...
- icon="../resources/wxpython.ico")
+ )
I can successfully compile my application on windows with python
On the same machine with python 3.5.4 64 bit and following virtualenv:
cefpython3==66.0
future==0.18.2
PyInstaller==3.2.1
pypiwin32==223
pywin32==228
I can also compile windows with python 3.6.4 64 and following virtualenv:
altgraph==0.17
cefpython3==66.0
future==0.18.2
macholib==1.14
pefile==2019.4.18
PyInstaller==3.3.1
pyinstaller-hooks-contrib==2020.9
pypiwin32==223
pywin32==228
pywin32-ctypes==0.2.0
On Linux compilation works as well, but the executable is not operational.
I get following output and error:
CEF Python 66.0
Chromium 66.0.3359.181
CEF 3.3359.1774.gd49d25f
Python 3.5.2 64bit
[1013/180954.001980:ERROR:icu_util.cc(133)] Invalid file descriptor to ICU data received.
Trace/breakpoint trap (core dumped)
version is python 3.5.2 64bit and the virtualenv is:
cefpython3==66.0
pkg-resources==0.0.0
PyInstaller==3.2.1
What could be the cause?
The code, that I try to compile is below:
import platform
import sys
from cefpython3 import cefpython as cef
def check_versions():
ver = cef.GetVersion()
print("CEF Python {ver}".format(ver=ver["version"]))
print("Chromium {ver}".format(ver=ver["chrome_version"]))
print("CEF {ver}".format(ver=ver["cef_version"]))
print("Python {ver} {arch}".format(
ver=platform.python_version(),
arch=platform.architecture()[0]))
assert cef.__version__ >= "57.0", "CEF Python v57.0+ required to run this"
def main(url="https://www.stackoverflow.com"):
sys.excepthook = cef.ExceptHook
check_versions()
settings = {}
switches = {}
browser_settings = {}
cef.Initialize(settings=settings, switches=switches)
cef.CreateBrowserSync(
url=url,
window_title="CEF_HELLO: ",
settings=browser_settings,
)
cef.MessageLoop()
cef.Shutdown()
if __name__ == "__main__":
main()
Addendum: 2020-10-14:
same error on linux with other versions:
so far I tried python 3.5 and 3.7
Is there anybody who successfully created an executable?
I could be, that this just an issue with the example project and its configuration?

As alternative, a solution could be found in PyInstaller bug 5400
Here the steps:
1- download the PyInstaller helper in CEFpython named hook-cefpython3.py from:
https://github.com/cztomczak/cefpython/tree/master/examples/pyinstaller and put in the root directory of your project
2- In that file, replace the line:
from PyInstaller.compat import is_win, is_darwin, is_linux, is_py2
with:
from PyInstaller.compat import is_win, is_darwin, is_linux
is_py2 = False
3- in your PyInstaller .spec file, add the '.' to the hookspath, e.g. hookspath=['.']. I think it is also possible to add it as PyInstaller command line option.
These steps should solve the problem, until CEFPython deliver a correct version of the hook file.

This is not really the answer I would like to accept, but it is at least one solution and contains information, that might lead to a better fix, a better answer.
After debugging with strace I found out, that the executable searches many files like for example icudtl.dat, v8_context_snapshot.bin, locales/* were searched in
'dist/cefapp/cefpython3but were copied todist/cefapp/`
An ugly work around is to do following after compilation
cd dist/cefapp/cefpython3
ln -s ../* .
and the executable works.
I'm sure there is also a nicer non-brute-force solution, but for the time being I wanted to answer in case others are stuck as well
Probably this can be fixed in the spec file but would we need one spec file for linux and one for windows then?
Perhaps there's also an option to tell the excutable to search for these files one level higer?

To solve this, you need to set this in your spec file:
hookspath=[r'YOUR_ENV_SITE_PACKAGES\cefpython3\examples\pyinstaller\']
And then rebuild, you will have things in the right place.

The following steps solved the issue for me on Windows 10, Python 3.9.5 32-bit, PyInstaller 4.3, and CEFPython 66.1:
Download the hook-cefpython3.py file from here and put it into your project root directory.
Run the pyinstaller command as usual but add the --additional-hooks-dir . command line option, so the command will look like this:
pyinstaller --additional-hooks-dir . <main-file.py>
As opposed to other answers here, this anser neither requires changes of hookspath directive in pyinstaller's spec file and, as of now, nor any changes to the downloaded hook-cefpython3.py file.

Buildroot Python Dependencies (_sqlite3)

Full Disclaimer: I have been using buildroot for the last 6 weeks. This is my first introduction to embedded Linux, thus I am still very green but have been able to solve 99% of my problems myself. for the most part the process has been straightforward.
So to occupy myself while I am stuck and home and cant work I have been working on an embedded hardware project. I've selected my hardware, built a prototype, learned buildroot basics, brought up the basic system, optimized the kernel config, built a custom device tree for my hardware and I am happy up until this point.
In parallel I have been programming the target application in python on my desktop, its dependent on a few libraries (hardware abstraction, communication, display etc) but is relatively straight forward.
I've got to the point where I have a list of requisite packages to build into my buildroot system. The buildroot tools are great here using the scanpypi tool:
~/buildroot$ utils/scanpypi diskcache -o package
Simply adding all the dependencies into /package/config.in has allowed them to be selected in menuconfig and added to the recipe.
The problem comes at build time where the building of the python module fails for the module above python-diskcache.
It has dependencies on a few things but one is slqite3, this has been added as:
the core python module "sqlite module"
external package "python-pysqlite3"
libraries > database > sqlite
However, it fails at build:
>>> python-diskcache 4.1.0 Building
Traceback (most recent call last):
File "setup.py", line 5, in <module>
import diskcache
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/__init__.py", line 9, in <module>
from .core import Cache, Disk, EmptyDirWarning, JSONDisk, UnknownFileWarning, Timeout
File "/home/buildroot/output/build/python-diskcache-4.1.0/diskcache/core.py", line 14, in <module>
import sqlite3
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/__init__.py", line 23, in <module>
from sqlite3.dbapi2 import *
File "/home/buildroot/output/host/arm-buildroot-linux-uclibcgnueabihf/sysroot/usr/lib/python3.8/sqlite3/dbapi2.py", line 27, in <module>
from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'
make[1]: *** [package/pkg-generic.mk:269: /home/buildroot/output/build/python-diskcache-4.1.0/.stamp_built] Error 1
make: *** [Makefile:84: _all] Error 2
In looking for a solution it seems like _sqlite3 is the C module external to python for communicating with an sqlite database. It should be installed with python (using 3.8) and should be enabled in buildroot with the enabling of the core sqlite module.
discussion 1
discussion 2
There are several fixes for dealing with this issue on a host (e.g. apt get install libsqlite3-dev and reinstall/reconfigure python). Obviously this is not possible in the image, and both the sqlite and python3 installs are latest builds and installed to the image at build time.
I'm really struggling to understand the problem or how I might patch it. I have a few theories based on the discussion, but I am unsure.
1) python is getting installed to the image before sqlite, so the appropriate module is not getting cp or symlink'ed to the python install.
2) there is some other unknown dependency not being met at build time and its failing silently
any ideas or assistance would be hugely appreciated.
Thanks

The problem is that Python on your build machine finds the cross-compiled _sqlite3 module, which it can't load because it's for the wrong architecture.
This usually doesn't occur, because the setup script normally doesn't go and load the package it is trying to build/install.
One workaround could be to install the host version of all the dependencies of diskcache, and set PYTHONPATH=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/:$(PYTHON3_PATH) in DISKCACHE_ENV. However, this is liable to lead to all kinds of other breakage.
A better solution, therefore, is to patch the setup.py script of diskcache so it doesn't try to import diskcache itself. It probably only does that to get a version number or something similar; that can be solved by moving the version number into a separate file and load that one instead.

As suggested by Arnout, I got my diskcache building by creating a following patch
diff --git a/setup.py b/setup.py
index b49d9c8..22cd3c8 100644
--- a/setup.py
+++ b/setup.py
## -3,8 +3,6 ## from io import open
from setuptools import setup
from setuptools.command.test import test as TestCommand
-import diskcache
-
class Tox(TestCommand):
def finalize_options(self):
## -23,8 +21,8 ## with open('README.rst', encoding='utf-8') as reader:
readme = reader.read()
setup(
- name=diskcache.__title__,
- version=diskcache.__version__,
+ name='diskcache',
+ version='5.4.0',
description='Disk Cache -- Disk and file backed persistent cache.',
long_description=readme,
author='Grant Jenks',
--
2.25.1

Change where pyodbc expects libodbc.2.dylib to live (changing default odbc file locations)

When importing pyodbc
❯ python
>>> import pyodbc
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/pcosta/Documents/test/myenv/lib/python3.7/site-packages/pyodbc.cpython-37m-darwin.so, 2): Library not loaded: /usr/local/opt/unixodbc/lib/libodbc.2.dylib
Referenced from: /Users/pcosta/Documents/test/myenv/lib/python3.7/site-packages/pyodbc.cpython-37m-darwin.so
Reason: image not found
I know why this is happening, as I don't have libodbc.2.dylib in the expected location. The reason is I do not have permission to write to /usr/local/, so I have Homebrew installing into ~/.brew. This mostly works fine. I am even able to get both tsql and isql working as expected by following the steps outlined here: https://github.com/mkleehammer/pyodbc/wiki/Connecting-to-SQL-Server-from-Mac-OSX.
So I do have libodbc.2.dylib, it's just that it lives in /Users/pcosta/.brew/lib, not /usr/local/opt/unixodbc/lib.
The main questions is can I get pyodbc to look for libodbc.2.dylib (and other associated files) in another directory?
I have all the files needed and have configured them correctly, I just need to repoint pyodbc somehow.
Thanks!

Thanks in part to guidance from this GitHub issue I was able to come to some solution.
Assuming you have brew install unixodbc:
Add the following paths (to .zshrc, .bashrc, or .bash_profile):
export LDFLAGS="-L/Users/pcosta/homebrew/opt/unixodbc/lib $LDFLAGS"
export CPPFLAGS="-I/Users/pcosta/homebrew/opt/unixodbc/include $CPPFLAGS"
export PKG_CONFIG_PATH="/Users/pcosta/homebrew/opt/unixodbc/lib/pkgconfig $PKG_CONFIG_PATH"
Run pip install --no-binary pyodbc pyodbc to bypass the binary and build yourself

Missing dll's when using Py2Exe, PyGtk, and Glade

I'm trying to build my first application using Py2Exe and I'm following the instructions listed at http://www.py2exe.org/index.cgi/Py2exeAndPyGTK, with the exception that I'm creating the UI with Glade. (Its just a very minimal gui with a close button and a label.)
When I try to run the resulting exe, I get an error dialog asking me to check the log file which contains the following error message:
Traceback (most recent call last):
File "pygtkpy2exetest.py", line 8, in <module>
File "gtk\glade.pyc", line 12, in <module>
File "gtk\glade.pyc", line 10, in __load
ImportError: DLL load failed: The specified module could not be found.
So I started digging around a bit, the first thing that I noticed is that it looks like py2exe is not able to find glib modules:
The following modules appear to be missing
['gdk', 'unix', 'glib.GError', 'glib.IOChannel', 'glib.IO_ERR', 'glib.IO_FLAG_APPEND', 'glib.IO_FLAG_GET_MASK', 'glib.IO_FLAG_IS_READABLE', 'glib.IO_FLAG_IS_SEEKABLE', 'glib.IO_FLAG_IS_WRITEABLE', 'glib.IO_FLAG_MASK', 'glib.IO_FLAG_NONBLOCK', 'glib.IO_FLAG_SET_MASK', 'glib.IO_HUP', 'glib.IO_IN', 'glib.IO_NVAL', 'glib.IO_OUT', 'glib.IO_PRI', 'glib.IO_STATUS_AGAIN', 'glib.IO_STATUS_EOF', 'glib.IO_STATUS_ERROR', 'glib.IO_STATUS_NORMAL', 'glib.Idle', 'glib.MainContext', 'glib.MainLoop', 'glib.OPTION_ERROR', 'glib.OPTION_ERROR_BAD_VALUE', 'glib.OPTION_ERROR_FAILED', 'glib.OPTION_ERROR_UNKNOWN_OPTION', 'glib.OPTION_FLAG_FILENAME', 'glib.OPTION_FLAG_HIDDEN', 'glib.OPTION_FLAG_IN_MAIN', 'glib.OPTION_FLAG_NOALIAS', 'glib.OPTION_FLAG_NO_ARG', 'glib.OPTION_FLAG_OPTIONAL_ARG', 'glib.OPTION_FLAG_REVERSE', 'glib.OPTION_REMAINING', 'glib.OptionContext', 'glib.OptionGroup', 'glib.PRIORITY_DEFAULT', 'glib.PRIORITY_DEFAULT_IDLE', 'glib.PRIORITY_HIGH', 'glib.PRIORITY_HIGH_IDLE', 'glib.PRIORITY_LOW', 'glib.Pid', 'glib.PollFD', 'glib.SPAWN_CHILD_INHERITS_STDIN', 'glib.SPAWN_DO_NOT_REAP_CHILD', 'glib.SPAWN_FILE_AND_ARGV_ZERO', 'glib.SPAWN_LEAVE_DESCRIPTORS_OPEN', 'glib.SPAWN_SEARCH_PATH', 'glib.SPAWN_STDERR_TO_DEV_NULL', 'glib.SPAWN_STDOUT_TO_DEV_NULL', 'glib.Source', 'glib.Timeout', 'glib.child_watch_add', 'glib.filename_display_basename', 'glib.filename_display_name', 'glib.filename_from_utf8', 'glib.get_application_name', 'glib.get_current_time', 'glib.get_prgname', 'glib.glib_version', 'glib.idle_add', 'glib.io_add_watch', 'glib.main_context_default', 'glib.main_depth', 'glib.markup_escape_text', 'glib.set_application_name', 'glib.set_prgname', 'glib.source_remove', 'glib.spawn_async', 'glib.timeout_add', 'glib.timeout_add_seconds', 'glib.uri_list_extract_uris']
If I try to manually add the glib module to the "includes" option in setyp.py it doesn't have any effect.
Digging further, I opened up the resulting exe with Dependency Walker, it shows me that there are two dll's missing, MPR.DLL and SHLWAPI.DLL, but both of these are in my system32 folder.
If I just create the UI without glade, I have no problems what so ever, but given that I've already created another application with glade, I'd like to stick with glade if possible. Any suggestions on where to go from here? I'm using Python 2.7, Py2Exe 0.6.9, PyGtk 2.22.6 on Windows XP.
Thanks,
Brent

i'm also having the same problem, however after spending few hours in google, i manage to find the solution.
use dependencywalker and profiling ur xxx.exe
u can download from http://www.dependencywalker.com/
find out the error file (for my case, they are MSJAVA.DLL and libxml2-2.dll)
search ur pc directory and copy them into 'dist' folder. u also can download the missing DLL from other website.
run the dependencywalker again, u will see the error reduce or become no more.
xxx.exe is executable now.
I hope above would helps those who are facing such kind of problem.
Neoh !!!share is care!!!

This may not help, but I had the same issue & it turned out to be a gtk installation issue. This caused problems with the gtk._gtk.pyd file which py2exe produced.
The setup.py I used was pretty much the same as that which py2exe suggests on http://www.py2exe.org/index.cgi/Py2exeAndPyGTK

I had the same error once and fix it. when you run the setup.py py2exe your files are created in win32. pyo as glade.pyo, and when you do you believe in win64 .pyc, at least that's what I experience, which ran the py2exe my app in window7-64 and then gave me the error, but when you create the executable on win32 I did, I think the glade.pyo, then gave me the same error, go to my project and do everithing with
gtk.Builder()
` not use more
gtk.glade.XML("file.glade")
and remove the import gtk.glade of my .py files and my setup.py just use
import gtk
and everything worked perfectly.
I hope my experiences will be of your help.

How can I use Django with MySQL in MAMP stack?

I have difficulty especially in installing MySQLdb module (MySQL-python-1.2.3c1), to connect to the MySQL in MAMP stack.
I've done a number of things such as copying the mysql include directory and library (including plugin) from a fresh installation of mysql (version 5.1.47) to the one inside MAMP (version 5.1.37).
Now, the MySQLdb module build and install doesnt give me error.
The error happens when I'm calling 'import MySQLdb' from python shell (version 2.6).
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Any idea, what else do I need to do to make it works?
Thanks a bunch,
Robert
=========
Add the system response after using virtualenv as suggested by Hank Gay below...
(MyDjangoProject)MyMacPro:MyDjangoProject rhenru$ which python
/Users/rhenru/Workspace/django/MyDjangoProject/bin/python
After I run python in virtualenv, importing MySQLdb:
>>> import MySQLdb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.macosx-10.6-universal/egg/MySQLdb/__init__.py", line 19, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 7, in <module>
File "build/bdist.macosx-10.6-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dlopen(/Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so, 2): Symbol not found: _mysql_affected_rows
Referenced from: /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
Expected in: flat namespace
in /Users/rhenru/.python-eggs/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg-tmp/_mysql.so
import sys and sys.path
>>> import sys
>>> print sys.path
['', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/distribute-0.6.10-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python26.zip', '/Library/Python/2.6/site-packages/PyXML-0.8.4-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/pydot-1.0.2-py2.6.egg', '/Library/Python/2.6/site-packages/pyparsing-1.5.2-py2.6.egg', '/Library/Python/2.6/site-packages/vobject-0.8.1c-py2.6.egg', '/Library/Python/2.6/site-packages/pytz-2010h-py2.6.egg', '/Library/Python/2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg', '/Library/Python/2.6/site-packages/distribute-0.6.12-py2.6.egg', '/Library/Python/2.6/site-packages/pip-0.7.1-py2.6.egg', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-darwin', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/Extras/lib/python', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-tk', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-old', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/lib-dynload', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Users/rhenru/Workspace/django/MyDjangoProject/lib/python2.6/site-packages', '/Library/Python/2.6/site-packages', '/Library/Python/2.6/site-packages/PIL', '/Library/Python/2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/PyObjC', '/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/wx-2.8-mac-unicode']

How are you installing MySQL-Python? I just tested in a fresh virtualenv and pip install mysql-python seems to have done the trick.
UPDATE:
pip is sort of like a package manager for Python packages.
By default, pip installs to your current site-packages directory, which is on your $PYTHONPATH. This lets other libraries/applications (like Django) access it. pip also works well with virtualenv (it should; Ian Bicking wrote them both), which is a nifty library that lets you sandbox an application. This is nice because it means you can try out new things without polluting (or even needing write access to) the global site-packages directory.
It probably seems like yak-shaving right now, but I'd say it's worth the effort to get up to speed on pip and virtualenv (you may also want to look into virtualenvwrapper, but we'll skip that for now; it's just sugar for virtualenv). It will lead to a slightly more complicated deployment scenario than putting everything in the global site-packages, but for development it's really no harder, and there are lots of good guides to deploying using a virtualenv.
I'd recommend something like the following:
curl -0 http://python-distribute.org/distribute_setup.py
python distribute_setup.py
easy_install pip
pip install virtualenv
virtualenv --distribute MyDjangoProject --no-site-packages
cd MyDjangoProject
source bin/activate (this activates the sandbox that virtualenv created)
pip install django mysql-python
At this point, you should have a totally functional Django+MySQL install (if I missed any steps, just comment and I'll try to add it in). You can start your Django project like this: django-admin.py startproject MyDjangoProject. cd into your project's directory, edit your settings.py file to point to your MySQL database, and run the dev server to test it out like so: ./manage.py runserver (you may need to chmod u+x your manage.py file). Voila! You should be able to access your site on localhost:8000. When you're done working on the project, you can just use deactivate to exit the virtualenv sandbox.
Try not to hold all this against Django: a lot of it is just best practices stuff for working with Python libraries. You could get by with a lot less, but this way it's more reproducible and you're less likely to accidentally mess up one of this project's dependencies when working on a different project.

I had this problem and it turned out to be due to an errant configuration:
export VERSIONER_PYTHON_PREFER_32_BIT=yes
I can't recall what I had this enabled for (some package that required 32-bit), probably related to Google AppEngine. But Setting it to 'no' solved by issues.
Otherwise I just installed everything using homebrew and pip.