We Keep Coding

Python shove module not working after py2app build on mac

I am using the shove module for object persistence. It works fine but when I attempt to build the app using py2app, it ceases to work. The following code works normally except after freezing using py2app:
import os
from shove import Shove
if __name__ == '__main__':
home=os.path.expanduser("~")
path = os.path.join(home, 'testdb')
uri = 'file://{0}'.format(path)
print path
print uri
db = Shove(uri)
print db.keys()
db['1'] = 'dog'
db['2'] = 'cat'
db.sync()
db.close()
The setup.py file contains the following:
from setuptools import setup
OPTIONS = dict(
argv_emulation = True,
includes=['future_builtins','concurrent.futures']
)
DATA_FILES = []
setup(
app='highlightdb.py',
data_files=DATA_FILES,
options={'py2app': OPTIONS},
setup_requires=['py2app'],
)
The traceback I am getting is:
highlightdb[68150]: db = Shove(uri)
highlightdb[68150]: File "shove/core.pyc", line 23, in __init__
highlightdb[68150]: File "shove/_imports.pyc", line 46, in store_backend
highlightdb[68150]: KeyError: 'file'
What could the problem be?

I found the problem is that shove functionality is largely implemented using plugins, which are loaded using pkg_resources.iter_entry_points. Unfortunately py2app doesn't support iter_entry_points. Nevertheless I found this, which may be a possible workaround.

Did you restarted Python after installing Shove?
The same KeyError: 'file' happens if you don't restart: you can import the library, but Shove can't load any modules

Cx-Freeze Error - Python 34

I have a Cx_Freeze setup file that I am trying to make work. What is terribly frustrating is that it used to Freeze appropriately. Now, however, I get the following error:
edit. the error that shows up is not a Python exception through the console, but a crash report when attempting to launch the resulting exe file generated through the freeze.
'File 'notetest.py', line 1, in
_find_and_load importlib_bootstrap.py, line 2214
....
AttributeError 'module' object has no attribute '_fix_up_module'
My setup.py file follows:
import sys
import os
from cx_Freeze import setup, Executable
build_exe_options = {'packages': [], 'excludes' : []}
base = 'Win32GUI'
exe = Executable(
script = 'notetest.py',
initScript = None,
base = 'Win32GUI',
targetName = 'MedicaidAid.exe',
compress = True,
appendScriptToExe = True,
appendScriptToLibrary = True,
icon = None
)
setup( name = 'MedicaidAid',
version = '0.85',
description = 'MedicaidAid Software',
options = {'build_exe': build_exe_options},
executables = [Executable('notetest.py', base = base)])

You should install cx_freeze from this site. It contains an important patch that solves the problem (see this discussion for detailed).

Apparently this is a known issue with Cx_Freeze. BitBucket.
Apparently there is a compiling error that occurs. The bitbucket link lists the work-arounds.

I had the same issue and the solution by Northcat worked just perfectly. The issue of missing patch in the other cx_freeze came into picture while installing(some alert messages were thrown) it. It is this missing patch because of which the alerts were given. The new installation from this link had no such alerts while installing and it so worked fine.

How to add a custom nose plugin to the `nosetests` command

So I'm very noob in dealing with nose plugins.
I've been searching a lot but docs regarding nose plugins seem scarce.
I read and tried what's in the following links to try to write a simple nose plugin
and run it with nosetests, without success:
https://nose.readthedocs.org/en/latest/doc_tests/test_init_plugin/init_plugin.html
https://nose.readthedocs.org/en/latest/plugins/writing.html
I don't want to write my own test-runner or run the tests from any other script (via run(argv=argv, suite=suite(), ...)),
like they do in the first link.
I wrote a file myplugin.py with a class like this:
import os
from nose.plugins import Plugin
class MyCustomPlugin(Plugin):
name = 'myplugin'
def options(self, parser, env=os.environ):
parser.add_option('--custom-path', action='store',
dest='custom_path', default=None,
help='Specify path to widget config file')
def configure(self, options, conf):
if options.custom_path:
self.make_some_configs(options.custom_path)
self.enabled = True
def make_some_configs(self, path):
# do some stuff based on the given path
def begin(self):
print 'Maybe print some useful stuff...'
# do some more stuff
and added a setup.py like this:
try:
from setuptools import setup, find_packages
except ImportError:
import distribute_setup
distribute_setup.use_setuptools()
from setuptools import setup, find_packages
setup(
name='mypackage',
...
install_requires=['nose==1.3.0'],
py_modules=['myplugin'],
entry_points={
'nose.plugins.1.3.0': [
'myplugin = myplugin:MyCustomPlugin'
]
}
)
Both files are in the same directory.
Every time I run nosetests --custom-path [path], I get:
nosetests: error: no such option: --custom-path
From the links mentioned above, I thought that's all that was required to register and enable a custom plugin.
But it seems that, either I'm doing something really wrong, or nose's docs are outdated.
Can someone please point me the correct way to register and enable a plugin, that I can use with nosetests?
Thanks a lot!! :)

You don't want the nose version in entry_points in setup.py. Just use nose.plugins.0.10 as the docs say. The dotted version in the entry point name is not so much a nose version as a plugin API version.

py2app picking up .git subdir of a package during build

We use py2app extensively at our facility to produce self contained .app packages for easy internal deployment without dependency issues. Something I noticed recently, and have no idea how it began, is that when building an .app, py2app started including the .git directory of our main library.
commonLib, for instance, is our root python library package, which is a git repo. Under this package are the various subpackages such as database, utility, etc.
commonLib/
|- .git/ # because commonLib is a git repo
|- __init__.py
|- database/
|- __init__.py
|- utility/
|- __init__.py
# ... etc
In a given project, say Foo, we will do imports like from commonLib import xyz to use our common packages. Building via py2app looks something like: python setup.py py2app
So the recent issue I am seeing is that when building an app for project Foo, I will see it include everything in commonLib/.git/ into the app, which is extra bloat. py2app has an excludes option but that only seems to be for python modules. I cant quite figure out what it would take to exclude the .git subdir, or in fact, what is causing it to be included in the first place.
Has anyone experienced this when using a python package import that is a git repo?
Nothing has changed in our setup.py files for each project, and commonLib has always been a git repo. So the only thing I can think of being a variable is the version of py2app and its deps which have obviously been upgraded over time.
Edit
I'm using the latest py2app 0.6.4 as of right now. Also, my setup.py was first generated from py2applet a while back, but has been hand configured since and copied over as a template for every new project. I am using PyQt4/sip for every single one of these projects, so it also makes me wonder if its an issue with one of the recipes?
Update
From the first answer, I tried to fix this using various combinations of exclude_package_data settings. Nothing seems to force the .git directory to become excluded. Here is a sample of what my setup.py files generally look like:
from setuptools import setup
from myApp import VERSION
appname = 'MyApp'
APP = ['myApp.py']
DATA_FILES = []
OPTIONS = {
'includes': 'atexit, sip, PyQt4.QtCore, PyQt4.QtGui',
'strip': True,
'iconfile':'ui/myApp.icns',
'resources':['src/myApp.png'],
'plist':{
'CFBundleIconFile':'ui/myApp.icns',
'CFBundleIdentifier':'com.company.myApp',
'CFBundleGetInfoString': appname,
'CFBundleVersion' : VERSION,
'CFBundleShortVersionString' : VERSION
}
}
setup(
app=APP,
data_files=DATA_FILES,
options={'py2app': OPTIONS},
setup_requires=['py2app'],
)
I have tried things like:
setup(
...
exclude_package_data = { 'commonLib': ['.git'] },
#exclude_package_data = { '': ['.git'] },
#exclude_package_data = { 'commonLib/.git/': ['*'] },
#exclude_package_data = { '.git': ['*'] },
...
)
Update #2
I have posted my own answer which does a monkeypatch on distutils. Its ugly and not preferred, but until someone can offer me a better solution, I guess this is what I have.

I am adding an answer to my own question, to document the only thing I have found to work thus far. My approach was to monkeypatch distutils to ignore certain patterns when creating a directory or copying a file. This is really not what I wanted to do, but like I said, its the only thing that works so far.
## setup.py ##
import re
# file_util has to come first because dir_util uses it
from distutils import file_util, dir_util
def wrapper(fn):
def wrapped(src, *args, **kwargs):
if not re.search(r'/\.git/?', src):
fn(src, *args, **kwargs)
return wrapped
file_util.copy_file = wrapper(file_util.copy_file)
dir_util.mkpath = wrapper(dir_util.mkpath)
# now import setuptools so it uses the monkeypatched methods
from setuptools import setup
Hopefully someone will comment on this and tell me a higher level approach to avoid doing this. But as of now, I will probably wrap this into a utility method like exclude_data_patterns(re_pattern) to be reused in my projects.

I can see two options for excluding the .git directory.
Build the application from a 'clean' checkout of the code. When deploying a new version, we always build from a fresh svn export based on a tag to ensure we don't pick up spurious changes/files. You could try the equivalent here - although the git equivalent seems somewhat more involved.
Modify the setup.py file to massage the files included in the application. This might be done using the exclude_package_data functionality as described in the docs, or build the list of data_files and pass it to setup.
As for why it has suddenly started happening, knowing the version of py2app you are using might help, as will knowing the contents of your setup.py and perhaps how this was made (by hand or using py2applet).

I have a similar experience with Pyinstaller, so I'm not sure it applies directly.
Pyinstaller creates a "manifest" of all files to be included in the distribution, before running the export process. You could "massage" this manifest, as per Mark's second suggestion, to exclude any files you want. Including anything within .git or .git itself.
In the end, I stuck with checking out my code before producing a binary as there was more than just .git being bloat (such as UML documents and raw resource files for Qt). A checkout guaranteed a clean result and I experienced no issues automating that process along with the process of creating the installer for the binary.

There is a good answer to this, but I have a more elaborate answer to solve the problem mentioned here with a white-list approach. To have the monkey patch also work for packages outside site-packages.zip I had to monkey patch also copy_tree (because it imports copy_file inside its function), this helps in making a standalone application.
In addition, I create a white-list recipe to mark certain packages zip-unsafe. The approach makes it easy to add filters other than white-list.
import pkgutil
from os.path import join, dirname, realpath
from distutils import log
# file_util has to come first because dir_util uses it
from distutils import file_util, dir_util
# noinspection PyUnresolvedReferences
from py2app import util
def keep_only_filter(base_mod, sub_mods):
prefix = join(realpath(dirname(base_mod.filename)), '')
all_prefix = [join(prefix, sm) for sm in sub_mods]
log.info("Set filter for prefix %s" % prefix)
def wrapped(mod):
name = getattr(mod, 'filename', None)
if name is None:
# ignore anything that does not have file name
return True
name = join(realpath(dirname(name)), '')
if not name.startswith(prefix):
# ignore those that are not in this prefix
return True
for p in all_prefix:
if name.startswith(p):
return True
# log.info('ignoring %s' % name)
return False
return wrapped
# define all the filters we need
all_filts = {
'mypackage': (keep_only_filter, [
'subpackage1', 'subpackage2',
]),
}
def keep_only_wrapper(fn, is_dir=False):
filts = [(f, k[1]) for (f, k) in all_filts.iteritems()
if k[0] == keep_only_filter]
prefixes = {}
for f, sms in filts:
pkg = pkgutil.get_loader(f)
assert pkg, '{f} package not found'.format(f=f)
p = join(pkg.filename, '')
sp = [join(p, sm, '') for sm in sms]
prefixes[p] = sp
def wrapped(src, *args, **kwargs):
name = src
if not is_dir:
name = dirname(src)
name = join(realpath(name), '')
keep = True
for prefix, sub_prefixes in prefixes.iteritems():
if name == prefix:
# let the root pass
continue
# if it is a package we have a filter for
if name.startswith(prefix):
keep = False
for sub_prefix in sub_prefixes:
if name.startswith(sub_prefix):
keep = True
break
if keep:
return fn(src, *args, **kwargs)
return []
return wrapped
file_util.copy_file = keep_only_wrapper(file_util.copy_file)
dir_util.mkpath = keep_only_wrapper(dir_util.mkpath, is_dir=True)
util.copy_tree = keep_only_wrapper(util.copy_tree, is_dir=True)
class ZipUnsafe(object):
def __init__(self, _module, _filt):
self.module = _module
self.filt = _filt
def check(self, dist, mf):
m = mf.findNode(self.module)
if m is None:
return None
# Do not put this package in site-packages.zip
if self.filt:
return dict(
packages=[self.module],
filters=[self.filt[0](m, self.filt[1])],
)
return dict(
packages=[self.module]
)
# Any package that is zip-unsafe (uses __file__ ,... ) should be added here
# noinspection PyUnresolvedReferences
import py2app.recipes
for module in [
'sklearn', 'mypackage',
]:
filt = all_filts.get(module)
setattr(py2app.recipes, module, ZipUnsafe(module, filt))

issues with cx_freeze and python 3.2.2?

I'm trying to freeze a python 3.2.2 script with cx_freeze 4.2.3. PyQt4 is used by the source script, I'm not sure if that is a potential source of the issue. Python crashes during the build process. Here is the command line output:
C:\Python32\New Folder>python setup.py build
running build
running build_exe
copying C:\Python32\Lib\site-packages\cx_Freeze\bases\Win32GUI.exe -> build\exe.win32-3.2\app.exe
copying C:\WINDOWS\system32\python32.dll -> build\exe.win32-3.2\python32.dll
Python itself crashes in Windows at this point and gives the "send error report" MS dialog:
python.exe has encountered a problem and needs to close. We are sorry
for the inconvenience.
Here is my setup.py file:
from cx_Freeze import setup, Executable
GUI2Exe_Target_1 = Executable(
script = "script.pyw",
initScript = None,
base = 'Win32GUI',
targetName = "app.exe",
compress = True,
copyDependentFiles = True,
appendScriptToExe = False,
appendScriptToLibrary = False,
icon = "icon.png"
)
excludes = ["pywin", "tcl", "pywin.debugger", "pywin.debugger.dbgcon",
"pywin.dialogs", "pywin.dialogs.list", "win32com.server",
"email"]
includes = ["PyQt4.QtCore","PyQt4.QtGui","win32gui","win32com","win32api","html.parser","sys","threading","datetime","time","urllib.request","re","queue","os"]
packages = []
path = []
setup(
version = "1.0",
description = "myapp",
author = "me",
author_email = "email#email.com",
name = "app",
options = {"build_exe": {"includes": includes,
"excludes": excludes,
"packages": packages,
"path": path
}
},
executables = [GUI2Exe_Target_1]
)
Any ideas on where I'm going wrong?
edit: After some experimentation it appears the icon I am trying to use is causing issues. It will build if I leave out the icon setting.

Apparently cx_freeze wants icons to be in .ico format. If you try to use a .png for an icon the build process will crash. Also, simply renaming the file extension from .png to .ico does not work, you have actually convert the file to ico.
This may have been obvious to some people but the online docs don't go into detail about required formats for icons.