I have an application in which I added a module that plots data using vispy and scipy (for Delaunay).
It works fine when I run within the Python (3.4 x64 on Windows) interpreter, but not when frozen using cx_freeze. It does not give me any error message, simply it does not run (quietly).
Here my cx_freeze script:
buildOptions = dict(packages = ['osgeo._gdal', 'scipy.sparse.csgraph._validation'])
import sys
base = 'Win32GUI' if sys.platform=='win32' else None
executables = [
Executable('main.py', base=base, targetName = 'myApp.exe', icon='ico/myApp.ico')
]
setup(name='MyApp',
version = '0.0.1',
description = 'My fancy app',
author = 'xxxx#xxxx.xxx',
options = dict(build_exe = buildOptions),
executables = executables)
I have to add 'scipy.sparse.csgraph._validation' to fix a previous missing inclusion as suggested here: scipy with py2exe and here
Looking for DLL issues, I have already attempted with Dependency Walker but without luck.
If I comment out the module with the vispy plot, everything works fine. Any hint?
I have attempted to freeze just the module with the problem by adding a main with a QApplication that display the QWigdet with the vispy.app.canvas. This helped because I got a very useful backtrace error related to vispy.app.backends._pyside.
After explicitly adding this to my posted cx_freeze script, the frozen application works:
packages = ['osgeo._gdal', 'vispy.app.backends._pyside', 'scipy.sparse.csgraph._validation']
The difference that I found in the build directory is the presence of QtOpenGL4.dll and PySide.QtOpenGL.pyd. They were not there without the explicit package inclusion (my application was already using PySide).
We've not thought about freezing apps with Vispy much yet. The pitfall that I'd expect matches with gmas80's answer; Vispy can use multiple backends, which means that these are dynamically loaded and cx_Freeze is unable to select the backend modules as a dependency. Depending on the backend that you need, you need to add some modules in vispy.backends to the list of includes.
Related
I am building an application which would make use of htmltidy, a library that cleans up bad HTML. There is a python binding which makes use of the soname libtidy.so. However, I intend to distribute the application using cx_Freeze. How should I do this? Also, I would like this to be a multiplatform solution, rather than just for Linux.
I tried to simply copy the soname to the created /lib folder, but that does not seem to work, most likely because of the soname search paths.
I also tried to use the bin_includes option in cx_Freeze setup.py, but that does not work either.
Script: testtidy.py
from tidylib import tidy_document
document, errors = tidy_document('''<p>fõo <img src="bar.jpg">''',
options={'numeric-entities':1})
print(document)
print(errors)
setup.py
import sys
from cx_Freeze import setup, Executable
# Dependencies are automatically detected, but it might need fine tuning.
# "packages": ["os"] is used as example only
build_exe_options = {"packages": ["os"],
"excludes": ["tkinter"],
"bin_includes": ["libtidy.so"]
}
base = None
setup(
name="testtidy",
version="0.1",
description="My application",
options={"build_exe": build_exe_options},
executables=[Executable("testtidy.py", base=base)],
)
The bin_includes option presumably does not work because the given library name is not an existing path.
In your setup.py, try to build an existing path to the tidy library:
import ctypes
from distutils import sysconfig
import os
lib_dir = sysconfig.get_config_var('LIBDIR')
libtidy_name = ctypes.util.find_library('tidy')
libtidy_path = os.path.join(lib_dir, libtidy_name)
libtidy_path should now contain the (symlink) path /usr/lib64/libtidy.so.58, use it in the bin_includes option:
build_exe_options = {...
"bin_includes": [libtidy_path]
}
After freezing, you should now see the symlink libtidy.so.58 and the the library itself libtidy.so.5.8.0 in the lib subdirectory of the build directory.
This still might not be enough to solve your problem. Looking at the source code of the python binding tidylib, starting from line 86, you will notice that it does not only rely on the soname libtidy.so, it rather relies on ctypes.util.find_library('tidy') and a pre-configured LIB_NAMES list as fallback. I'm not sure whether the modification proposed above will suffice to make ctypes.util.find_library('tidy') work in the frozen executable, if not we will still need to teach tidylib to find the library by tweaking the LIB_NAMES in the frozen executable.
EDIT:
Regarding building a multiplatform solution, see the cx_Freeze documentation:
cx_Freeze works on Windows, Mac and Linux, but on each platform it only makes an executable that runs on that platform. So if you want to freeze your programs for Windows, freeze it on Windows; if you want to run it on Macs, freeze it on a Mac.
You will thus need to make your source code run on the target platform, including htmltidy and its python binding pytidylib, and build the running code into an executable using cx_Freeze on that platform. Tweaks will probably be necessary there as well to let cx_Freeze include all dependencies of tidylib correctly, and these tweaks may look different for each platform.
I have a program that helps visualize some data in 3D by plotting a surface and a cloud of points to see how they relate to the surface. For the visualization I am using mayavi since it was easy to set up and does a better job than matplotlib in 3D. The program works great when I run it in my python environment and makes beautiful visualizations. However, I need to distribute this to a few people who don't have Python and would prefer not to have to install python and all the add-ins on each computer, so I have been using pyinstaller to create standalone .exe files that they can run after I develop a program.
For reference, this is all done on Windows 10, Python 3.6, pyqt 4.11.4, pyface 6.0.0, traits 4.6.0, pyinstaller 3.3.1, mayavi 4.5.0+vtk81. Just about every module I use was installed using pip.
The problem is that I can't seem to get a working exe if I use/import the mayavi module. I have been reading so much github documentation on different hook files and hidden-imports and each time I fix one error another pops up. It all started with scipy but I believe I have those worked out. So I have a few questions that could help me solve the problem:
1) Has anyone successfully created a standalone exe using pyinstaller with a mayavi import (specifically from mayavi import mlab)? What is your secret?!?
This seems similar but I haven't been able to figure it out yet... SO_link
I have used the following links (scipy,h5py,pandas,traits/qt4,ETS_TOOLKIT) to add hidden imports or fix other issues but I am stuck now after setting my ETS_TOOLKIT=qt4. Before setting it, I would get a pyface/traits error RuntimeError: No traitsui.toolkits plugin found for toolkit null, but now it says the same thing for qt4 instead of null. I have qt4 installed so I don't understand that... It is in the import_toolkit function in pyface/base_toolkit.py.
2) Is there a better route to go in terms of 3D visualization / exe creation? I need a 3D program that can accurately render if the points are in front of or behind the surface and be able to rotate/zoom/pan interactively, plus it needs to be intuitive. Mayavi had very simple commands similar to matplotlib but others seem very complicated working around how the UI interacts with everything.
3) How do I interpret all of these error codes I get? They are usually pretty cryptic saying a certain line threw an exception nested in 10 other function calls and it seems very difficult to back out where exactly things went wrong, especially when nothing pops up on Google that seems to be related.
EDIT
While I am still very confused, I have been able to change where the error occurs. Following the traceback, I commented out a line setting the marker color in traitsui/editors/code_editor.py (line 49), at which point the exception then started the next time the Color method was called... but I still get the same RuntimeError. So that doesn't tell me much other than I am still looking for what hidden import I need to include for this thing to work.
Also note that I get the exact same error with both PyInstaller and cx_Freeze, in case that helps...
Edit 2
I have now tried it using anaconda for python 2.7, SAME EXACT ISSUE..... I'm starting to believe the universe doesn't want this to happen. Is there somewhere else I should bring this issue up?? I have posted on the traitsui GitHub but that wasn't very helpful. This seems to be bigger than pyinstaller/cx_freeze since it happens in both....
I dealt with the same problem and finally switched to cx_freeze, which now works fine on linux and windows. The problems you are dealing with arise from statements like in the SE answer, you found, i.e. dynamic import statements, where what is imported is only determined at runtime:
be = 'pyface.ui.%s.' % tk
__import__(be + 'init')
I couldn't fix that in pyinstaller, while in cx_freeze it works, when you explicitely add the required packages in the build file. Here is the package list I used:
"packages": ["pyface.ui.qt4", "tvtk.vtk_module", "tvtk.pyface.ui.wx", "matplotlib.backends.backend_qt4",'pkg_resources._vendor','pkg_resources.extern','pygments.lexers',
'tvtk.pyface.ui.qt4','pyface.qt','pyface.qt.QtGui','pyface.qt.QtCore','numpy','matplotlib','mayavi']
Here is a full build script that works with python3.6, cx_freeze 5.0.2, mayavi 4.5.0+vtk71, traits 4.6.0, pyface 5.1.0 and traitsui 5.1.0.
import os
from cx_Freeze import setup, Executable
import cx_Freeze.hooks
def hack(finder, module):
return
cx_Freeze.hooks.load_matplotlib = hack
import scipy
import matplotlib
scipy_path = os.path.dirname(scipy.__file__) #use this if you are also using scipy in your application
build_exe_options = {"packages": ["pyface.ui.qt4", "tvtk.vtk_module", "tvtk.pyface.ui.wx", "matplotlib.backends.backend_qt4",'pygments.lexers',
'tvtk.pyface.ui.qt4','pyface.qt','pyface.qt.QtGui','pyface.qt.QtCore','numpy','matplotlib','mayavi'],
"include_files": [(str(scipy_path), "scipy"), #for scipy
(matplotlib.get_data_path(), "mpl-data"),],
"includes":['PyQt4.QtCore','PyQt4.QtGui','mayavi','PyQt4'],
'excludes':'Tkinter',
"namespace_packages": ['mayavi']
}
executables = [
Executable('main.py', targetName="main.exe",base = 'Win32GUI',)
]
setup(name='main',
version='1.0',
description='',
options = {"build_exe": build_exe_options},
executables=executables,
)
I import pyface in the following way:
os.environ['ETS_TOOLKIT'] = 'qt4'
import imp
try:
imp.find_module('PySide') # test if PySide if available
except ImportError:
os.environ['QT_API'] = 'pyqt' # signal to pyface that PyQt4 should be used
from pyface.qt import QtGui, QtCore
before importing mayavi
So I'm trying to compile a python script named "File.py" using this code:
from cx_Freeze import setup, Executable
# Define packages needed(not builtin)
packages = ['scipy' , 'numpy']
options = {
'build_exe': {
'packages': packages,
},
}
setup(name='Monitor',
version = '0.8',
description = 'Monitors stuff.',
options = options,
executables = [Executable("File.py")])
"File.py" only uses scipy and numpy, which I have included in the packages variable. However, after compiling and running File.exe, I get the error: "ModuleNotFoundError: No module named 'scipy.spatial.ckdtree'". I am using python 3.6 and cx_Freeze 6 (which supports python 3.6). How can I get cx_Freeze to include all of the necessary modules?
Try giving the actual address of the library. So find where the scipy and numpy files are and in the packages, include something like this: r"C:\Users\yourname\AppData\Local\Programs\Python\Python36\DLLs\tcl86t.dll",
r"C:\Users\yourname\AppData\Local\Programs\Python\Python36\DLLs\tk86t.dll",
Probably dead, but I'll put in my two cents having encountered a couple issues like this with scipy over the years in combination with cx_Freeze. I had trouble finding documentation of this issue on SO.
First, when you run File.exe, check the full stack trace in the dialog box when you get the error. ModuleNotFoundError is your symptom, but root cause is in which file is attempting to load the module, usually an __init__.py file somewhere in scipy.
Second, go to that file and compare the module name being loaded against the package name and aliases. You are looking for discrepancies in the case of the module name or path to the module. For example, one ckdtree load issue I have come across was __init__.py was attempting to load 'ckdtree', but the module in the distribution was 'cKDTree' or had a capital letter where there wasn't supposed to be one. Changing the case in the file attempting to load the module fixes the issue.
I'm a python programmer and I'm trying to build an executable binary to distribute my software to my clients, even if it's not fully executable I want to be able to distribute my software in a way so that it is convenient for the end user.
I have already tried PyInstaller as well as Py2Exe and I'm facing the same problem with a particular software.
I used the splinter module for my program (which of course is a new high level framework to interact with other frameworks like Selenium) and every time I try to compile it there seems to be a file called "webdriver.xpi" that is always left out from the final package and therefore when the program attempts to execute the web-driver it fails with an IO Error saying that the file "webdriver.xpi" was not found....but other than that the GUI and everything works perfectly fine.
So is there a way to include it even manually? I tried including it manually by browsing to the specific folder # library.zip file but it didn't work.
I'm not really expert in this matter and I rely on GUI2Exe for building everything...and I would really appreciate some advice if possible on how to fix this.
Thanks.
I was at this all day and found a workaround, it's sneaky but it works. In the error message I was receiving I noticed that there was a space between in library .zip. I could not trace it down in the source code for py2exe or selenium. I too had tried putting the xpi file in the library zip and it did not work. The workaround is:
In your setup file use these options:
setup(
console=['yourFile.py'],
options={
"py2exe":{
"skip_archive": True,
"unbuffered": True,
"optimize": 2
}
}
)
Run the py2exe install
Copy the xpi file into the dist directory
That should do it.
You need an instruction in your setup.py to include any resource files in your distribution. There is a couple of ways of doing this (see distutils, setuptools, distribute - depending on what you are using to build your distribution), but the py2exe wiki has an example.
You may need to use this py2exe tip to find your resources if you're installing them into the same directory as your exe.
See this answer for some additional info on including resource files in your distribution.
Here is a solution of your question:
I have modify a code little and it should be work since I had a same issue and I solved it:
from distutils.core import setup
import py2exe
wd_base = 'C:\\Python27\\Lib\site-packages\\selenium-2.44.0-py2.7.egg \\selenium\\webdriver'
RequiredDataFailes = [
('selenium/webdriver/firefox', ['%s\\firefox\\webdriver.xpi'%(wd_base), '%s\\firefox\\webdriver_prefs.json'%(wd_base)])
]
setup(
windows=[{"script":"gui_final.py"}],options={"py2exe":{"skip_archive": True,"includes":["sip"]}},
data_files=RequiredDataFailes,
)
I know this is old, but I wanted to give an updated answer to avoid suggesting that programmers do something manually.
There is a py2exe option to specify a list of data files as tuples. (pathtocopyto, [list of files and where to get them])
Example:
from disutils.core import setup
import py2exe
wd_base = 'C:\\Python27\\Lib\\site-packages\\selenium\\webdriver'
RequiredDataFailes = [
('selenium/webdriver/firefox', ['%s\\firefox\\webdriver.xpi'%(wd_base), '%s\\firefox\\webdriver_prefs.json'%(wd_base)])
]
setup(
console=['MyScript.py'],
data_files=RequiredDataFiles,
options={
**mypy2exeopts
}
)
The only downside I am aware of currently is that you still need skip_archive = True. There are workarounds to get the data files in the library.zip, but I haven't had much luck with the webdriver's info.
how to create a good plugin engine for standalone executables created with pyInstaller, py2exe or similar tools?
I do not have experience with py2exe, but pyInstaller uses an import hook to import packages from it's compressed repository. Of course I am able to import dynamically another compressed repository created with pyInstaller and execute the code - this may be a simple plugin engine.
Problems appears when the plugin (this what is imported dynamically) uses a library that is not present in original repository (never imported). This is because import hook is for the original application and searches for packages in original repository - not the one imported later (plugin package repository).
Is there an easy way to solve this problem? Maybe there exist such engine?
When compiling to exe, your going to have this issue.
The only option I can think of to allow users access with thier plugins to use any python library is to include all libraries in the exe package.
It's probably a good idea to limit supported libraries to a subset, and list it in your documentation. Up to you.
I've only used py2exe.
In py2exe you can specify libraries that were not found in the search in the setup.py file.
Here's a sample:
from distutils.core import setup
import py2exe
setup (name = "script2compile",
console=['script2compile.pyw'],
version = "1.4",
author = "me",
author_email="somemail#me.com",
url="myurl.com",
windows = [{
"script":"script2compile.pyw",
"icon_resources":[(1,"./ICONS/app.ico")] # Icon file to use for display
}],
# put packages/libraries to include in the "packages" list
options = {"py2exe":{"packages": [ "pickle",
"csv",
"Tkconstants",
"Tkinter",
"tkFileDialog",
"pyexpat",
"xml.dom.minidom",
"win32pdh",
"win32pdhutil",
"win32api",
"win32con",
"subprocess",
]}}
)
import win32pdh
import win32pdhutil
import win32api
PyInstaller does have a plugin system for handling hidden imports, and ships with several of those already in. See the webpage (http://www.pyinstaller.org) which says:
The main goal of PyInstaller is to be compatible with 3rd-party packages out-of-the-box. This means that, with PyInstaller, all the required tricks to make external packages work are already integrated within PyInstaller itself so that there is no user intervention required. You'll never be required to look for tricks in wikis and apply custom modification to your files or your setup scripts. Check our compatibility list of SupportedPackages.