Here is my system info:
123 INFO: PyInstaller: 4.0
123 INFO: Python: 3.5.4
124 INFO: Platform: Windows-10-10.0.18362-SP0
I've been trying to generate a Python (PyQt) executable using Pyinstaller to be used in an app. However, when I package the executable and run it, it will throw this:
pkg_resources.DistributionNotFound: The 'flask-compress' distribution was not found and is required
by the application
[14684] Failed to execute script main
This dependency already exists in my virtual environment and I have tried specifying the path to the site packages directory and the flask_compress import like this:
pyinstaller --paths C:\Users\alan9\PycharmProjects\PracticumProject\venv\Lib\site-packages --hidden-import=flask_compress main.py
Note: I have tried to create the executable for this application using different python versions, with different pyinstaller flags (onefile, windowed, onedir), on different computers with Windows 7/10, on a clean copy of a Windows 10 VM, and with fbs but I always receive the same error message:(
I solved this problem with monkey patching. Just paste this code in a module that you import before dash and you should be good to go. In my case I had flask-compress==1.5.0 so I just hardcoded the version but you could probably do something more clever.
"""
Simple module that monkey patches pkg_resources.get_distribution used by dash
to determine the version of Flask-Compress which is not available with a
flask_compress.__version__ attribute. Known to work with dash==1.16.3 and
PyInstaller==3.6.
"""
import sys
from collections import namedtuple
import pkg_resources
IS_FROZEN = hasattr(sys, '_MEIPASS')
# backup true function
_true_get_distribution = pkg_resources.get_distribution
# create small placeholder for the dash call
# _flask_compress_version = parse_version(get_distribution("flask-compress").version)
_Dist = namedtuple('_Dist', ['version'])
def _get_distribution(dist):
if IS_FROZEN and dist == 'flask-compress':
return _Dist('1.5.0')
else:
return _true_get_distribution(dist)
# monkey patch the function so it can work once frozen and pkg_resources is of
# no help
pkg_resources.get_distribution = _get_distribution
I was experiencing a similar problem and this solution was not working for me in an environment with pyinstaller==4.0, dash==1.17.0 and flask_compress==1.8.0.
However, I found that the solution proposed by Legorooj here works quite well in my case, basically instead of tensorflow one uses flask and flask_compress in the hook file.
For completeness here is how I wrote the hook-flask.py file.
from PyInstaller.utils.hooks import collect_all
def hook(hook_api):
packages = [
'flask',
'flask_compress',
'flask_caching'
]
for package in packages:
datas, binaries, hiddenimports = collect_all(package)
hook_api.add_datas(datas)
hook_api.add_binaries(binaries)
hook_api.add_imports(*hiddenimports)
Of this way no monkey patching is needed.
second update:
Since I cannot put comments yet (not enough contributions on stackoverflow?) I am adding this update to say that I tried the hook-based solution (which, in theory is the best way to deal with this sort of problem), but like Sören reported, this did not seem to get flask-compress into the distribution.
update:
The monkey-patching solution is clearly preferable, both because it leaves the flask code intact (and apparently the original functionality), but also because it (presumably) identifies the true underlying problem -- flask-compress is being brought along for the ride but simply isn't detected by pkg_resources... I wonder what (if anything) can be done to generalize this -- it may be possible to modify pkg_resources to work in a "bundled" Pyinstaller context.
original post:
I got hit with the exact same problem. Now, the "correct" thing to do is to figure out the apparently subtle way to add a proper Pyinstaller hook to take care of this issue. Maybe I will get around to doing that at some point -- certainly, what you tried to do makes sense and should've worked. I did the even more basic move of explicitly importing flask_compress in the main python script, without success.
However, since we are both trying to Pyinstallerize a flask-dependent (desktop) application, chances are very good that neither of us actually needs to gzip the responses that are being generated, so it is possible to eliminate the whole problem by "simplifying the software a bit". In my case, the references to flask_compress came from dash, so I ripped out the gzip functionality (found in Lib/site-packages/dash/dash.py). The key relevant references are on lines 21, 53, 292, 321 and 432.
"Rewire" references like those (in whatever library you use which references flask-compress) and all your problems magically disappear -- worked for me :-)
My bet is that line 53 engages machinery that is too clever/dynamic for Pyinstaller. Don't get me wrong -- Pyinstaller is nothing short of a miracle but as a community we will be forever running around trying to plug these clever-hacks because Python simply was not made to generate nice standalone executables.
Related
I have a big C++ module with Python 3 bindings using Boost.Python, that I compile to a .so file using CMake on macOS.
When I try to import it in the REPL, everything seems to work fine:
>>>import myModule
>>>
However, as soon as I run the import statement, the famous rocket icon of Python shows up in the Dock and stays there jumping for some minutes and stops after. Obviously then, I cannot access any of the functions defined in my module, so the import looks fine but doesn't actually do anything.
I tried looking in the Console and saw that whenever I import myModule, I get two launchservicesd[83]: SecTaskLoadEntitlements failed error=22.
It brought me to this and that related questions but I can't find what the exact problem is.
The C++ module is huge so I just can't look at the code and find the problem, thus I'm asking for any hints about at least how to debug that problem.
I can suggest the following steps:
Try to import that module though local python session. So, run interactive python interpreter, and 'import myModule'.
If bad, try to check:
are python version, with which myMoudle was linked with, is similiar to used interpreter
check that build architectires are the same
check that you can load even simple boost.python example module
If ok, check that you have correctly set up module search path in your python code.
I am trying to package my program using pyinstaller. The code runs fine on windows, and uses SqlAlchemy, OpenCV and pyodbc packages.
I ran pyinstaller to create the executable and tried to run it. I'm getting an error:
ImportError: No module named ConfigParser
now, I reran the same thing and looked at logs from pyinstaller and got a warning:
WARNING: Hidden import "sqlalchemy.sql.functions.func" not found!
along with a few others.
then there was a warning about trying to import ConfigParser in lower and uppercase.
Attempted to add Python module twice with different upper/lowercases: ConfigParser
What might be the issue here?
So, I figured it out. To an extent.
Seems like pyInstaller doesn't deal with SWIG files that well.
In sqlalchemy.utils there's a file called compat.py. It is there to make the module compatible with all versions of python.
for example, in python2.x, there's ConfigParser whereas in py3, it is named configparser
So there is a part in compat.py to deal with it:
if py3:
import configparser
# Some other such import statements
elif py2:
import ConfigParser as configparser
Now, pyinstaller gets stumped here as it just focuses on the import, and hence it tries to import both and fails miserably.
My crude workaround to this involved modifying the compat.py file and retaining only the parts relevant to the python version I have (2.x).
Running pyinstaller again proved to be a success! :)
Although this is all very crude and there's probably something better out there, but I couldn't find anything, so I'm sharing what worked for me.
In my IronPython script, I'm using standard libary modules like ConfigParser, logging and JSON.
Then I use pyc.py to create an executable. At first I ran into problems, namely '...ImportException: no module named ...'
since they weren't being included in the exe and accompanying dlls.
So I ran a solution from here: IronPython: EXE compiled using pyc.py cannot import module "os" and it mostly worked.
For example, importing 'ConfigParser' would work since in the IronPython 'Lib' folder as a module, it's there as 'ConfigParser.py'. However I'm still having trouble using JSON and logging since they're inside of folders with their name (packages?).
I'm feeling that I'm just missing something simple, and probably need to read up more on python modules and how they really work, but I'm not sure what I should be looking for.
Any help would be greatly appreciated.
Thanks!
EDIT:
I can't answer my own question yet, so I'll leave this here.
Somehow got it to work in a really 'hacky' way. There must be another much cleaner solution to this that I'm missing (some option in pyc.py?)
Here's what I did:
1) Made the StdLib.dll file generated from the link above (IronPython: EXE compiled using pyc.py cannot import module "os"). This would be missing the std lib packages.
2) Used SharpDevelop to compile the standard lib packages that weren't included in the above dll following the method here: http://community.sharpdevelop.net/blogs/mattward/archive/2010/03/16/CompilingPythonPackagesWithIronPython.aspx
3) Used SharpDevelop to build my program and tie together all the references.
- Reference to the dlls made in step 2
- Reference to the StdLib.dll made in step 1
Again, there must be a better solution to this.
I've found two ways to compile standard library python packages:
1st way: Individually compile each package into a dll
Using pyc.py, run something like (this example compiles logging package):
ipy pyc.py ".\Lib\logging\__init__.py" ".\Lib\logging\config.py" ".\Lib\logging\handlers.py" /target:dll /out:logging
This creates a logging.dll file, which you can then use like this:
import clr
clr.AddReference('StdLib') #from the compilation of non-package std libraries
clr.AddReference('logging')
import logging
**Note: This is assuming you've run the solution from IronPython: EXE compiled using pyc.py cannot import module "os" to create StdLib.dll
2nd way: Modify the compilation script that generated StdLib.dll
I changed this line:
#Build StdLib.DLL
gb = glob.glob(r".\Lib\*.py")
gb.append("/out:StdLib")
To this:
#Build StdLib.DLL
gb1 = glob.glob(r".\Lib\*.py")
gb2 = glob.glob(r".\Lib\*\*.py")
gb3 = glob.glob(r".\Lib\*\*\*.py")
gb = list(set(gb1 + gb2 + gb3))
gb.append("/out:StdLib")
This includes the subfolders in the Lib directory which get missed in the original regex (only modules get included). Now, packages like xml, json, logging, etc. get included into StdLib.dll
I am having this problem that we compiled our python project into an exe with py2exe and the resulting exe does not work unless it is executed as an administrator.
I wanted to ask that is this supposed to happen ?? As in there are so many applications that we can run as not an administrator so is there any way I can convert my python code to such an application ...
Thanks a lot..
It sounds like your application is attempting to write to a directory for which the basic user doesn't have access; most likely the "Program Files" directory. I believe in Vista/Win7 this is not allowed, and the standard convention is to write to the user's appdata folder, for any user data you might wish to store.
You can reliably obtain the location of this directory using the ctypes module, here's an example:
import ctypes
from ctypes import wintypes
def get_appdata_directory():
CSIDL_APPDATA = 0x001a
dll = ctypes.windll.shell32
app_data_directory = ctypes.create_unicode_buffer(wintypes.MAX_PATH)
found = dll.SHGetFolderPathW(0, CSIDL_APPDATA, 0, 0, app_data_directory)
# FYI: if `found` is False, then it failed to locate the appdata directory
# and app_data_directory.value is empty. So you might want to add some
# code here to verify that a valid path is going to be returned.
# This would probably only happen on older versions of windows,
# but, this is just a guess as I don't have any older OSs available
# for testing. (see my note below)
return app_data_directory.value
appdata = get_appdata_directory()
print(appdata)
# outputs something such as: 'C:\Users\bob\AppData'
Note: I believe the appdata folder was introduced with WinXP/Win2k. Not sure about WinME and prior, however, I don't believe you have to worry about the administrator restrictions on these earlier OSs. If you really care to support them, you could use python's builtin platform module along with some conditionals, then simply write user data to the "Program Files" directory for the archaic versions of Windows.
I've had serious problems with getting py2exe to work. Luckily I found the excellent PyInstaller which not only works, but also creates smaller executables. I haven't ran into the problem you mention with PyInstaller so I recommend trying it.
http://www.pyinstaller.org/
I recently updated from Python 2.5 to 2.7 (I tried 2.6 during my hassles) and while everything works fine from the command line or in the Django runserver, mod_wsgi cannot load any module that contains DLLs (pyd) built with MSVC.
For example, if I build my own versions of pycrypto or lxml then I will get the following error only from mod_wsgi:
ImportError at /
DLL load failed: The specified module could not be found.
Even the official PIL binaries will fail to import the _imaging C module in mod_wsgi but that may be another problem.
However, if I use a version of pycrypto built with MinGW from somewhere like http://www.voidspace.org.uk/python/modules.shtml#pycrypto then it will import fine even in mod_wsgi. I don't find this solution satisfactory though since the whole reason I updated Python was to avoid needing to hunt for prebuilt binaries and I can't build them myself because MinGW fails >50% of the time for me.
EDIT2:
I noticed this in Python27/Lib/distutils/msvc9compiler.py on lines 680-705:
try:
# Remove references to the Visual C runtime, so they will
# fall through to the Visual C dependency of Python.exe.
# This way, when installed for a restricted user (e.g.
# runtimes are not in WinSxS folder, but in Python's own
# folder), the runtimes do not need to be in every folder
# with .pyd's.
manifest_f = open(manifest_file)
try:
manifest_buf = manifest_f.read()
finally:
manifest_f.close()
pattern = re.compile(
r"""<assemblyIdentity.*?name=("|')Microsoft\."""\
r"""VC\d{2}\.CRT("|').*?(/>|</assemblyIdentity>)""",
re.DOTALL)
manifest_buf = re.sub(pattern, "", manifest_buf)
pattern = "<dependentAssembly>\s*</dependentAssembly>"
manifest_buf = re.sub(pattern, "", manifest_buf)
manifest_f = open(manifest_file, 'w')
try:
manifest_f.write(manifest_buf)
finally:
manifest_f.close()
except IOError:
pass
This probably explains why everything works from the command line but not in mod_wsgi. Commenting all this out seems to fix the problem but doesn't feel like the proper fix. The question now is where to put msvcr90.dll so that Apache can use it? I notice that Apache's bin folder contains msvcr70.dll and msvcr80.dll but putting 90 in there doesn't work.
I've had a similar issue and eventually found a solution here: download/update your apache server with one from http://www.apachelounge.com/download/.
While I don't know anything about mod_wsgi, I dare to guess that the most probable reason is missing runtime dependencies. You may want to inspect your MSVC-build with Dependency Walker that ships with MSVC (e.g., in MSVC 2005, it's located at \Common7\Tools\Bin\Depends.Exe). It will show you which DLLs are required by a binary.
As another workaround, it should be possible to build your modules with statically linked runtime (see Project Properties -> C/C++ -> Code Generation -> Runtime -- choose "Multithreaded" (not "Multithreaded DLL"); or, if building from command line, make sure /MT is used instead of /MD). However there may be problems if runtime-dependent things (e.g. FILE* objects) cross module boundary.
UPD If you have correct VC redist installed, the reason may be a problem with SxS configuration (i.e. manifest of .pyd itself is wrong or missing, or conflicts with manifest of the application that loads the .pyd). You can use sxstrace utility to see what exactly is going on. See Diagnosing SideBySide failures.
Also, did you try static linking of the runtime? Or, better yet, check what are requirements of your host process.
I was getting this error with zmq. The solution was to include the python27.dll manifest in the libzmq.pyd file (and it'll most likely work for other pyd/dll's). Make sure you use all 64-bit or all 32-bit.
"C:\Program Files (x86)\Windows Kits\8.0\bin\x64\mt.exe" -inputresource:C:\windows\system32\python27.dll;#2 -outputresource:libzmq.pyd;#2
See https://code.google.com/p/pyodbc/issues/detail?id=214