We Keep Coding

scons uninstall runs Substfile

I have created the SConstruct to install the systemd user services but when I try to scons uninstall the temporary services files are creates which should not happen.
import os
PATH_WD = os.path.abspath(os.curdir)
env = Environment(
SUBSTFILESUFFIX = '.service',
SUBST_DICT = { '{{PATH_ROOT}}' : os.path.dirname(PATH_WD) },
ENV = {
'DBUS_SESSION_BUS_ADDRESS' : os.environ['DBUS_SESSION_BUS_ADDRESS'],
'XDG_RUNTIME_DIR' : os.environ['XDG_RUNTIME_DIR']
}
)
INSTALLED = env.Install(
target = os.path.expanduser('~/.config/systemd/user/'),
source = [
env.Substfile('service1'),
env.Substfile('service2'),
]
)
env.AddPostAction(INSTALLED, env.Action('systemctl --user daemon-reload'))
Alias('install', INSTALLED)
NoClean(INSTALLED)
Command('uninstall', INSTALLED, Delete(INSTALLED))
Default('install')

Second try..
Here's a trivial example which should work for you..
env=Environment()
prog=env.Program('main.c')
Default(prog)
installed_prog = env.Install('install_dir', prog)
Alias('install', installed_prog)
NoClean(installed_prog)
# You don't have to specify targets to Alias.. so it won't
# try to build those before executing the Action
Alias('uninstall', action=Delete(installed_prog))
AlwaysBuild('uninstall')

SCons builder calls are statements of relationships between nodes. You've associated the target "uninstall" with the source INSTALLED by calling the Command builder. So in order to "build" this target, you need the source, and the source is the list of nodes returned by calling the Install builder. So the Install has to happen before the uninstall can take place. Is there a reason you don't want SCons' clean functionality to be used here? To see this, try: scons --tree=all,linedraw -n uninstall

Compile a Python project Windows

I have the following directory structure to my python project:
eplusplus/
|
|
----__main__.py
----model/
----exception/
----controller/
----view/
The directories: model, exception, controller and view each one has its
__init__.py. When I run the program at my machine I always use this following command: py -m eplusplus. But when I tried to use py2exe or pytinstaller the the points to: permission denied. For what I found, this is because its a directory I trying to compile, but when I compiled the __main__.py it compiled normally, but when I try to execute it says: Error! No eplusplus module founded!
I have no setup.py file and I don't know how they worked.

After some very intensive research and error and try I succeeded by doing this:
I added an empty __init__.py at the eplusplus folder
Out of the eplusplus folder, I had to write a compilation.py file (the file doesn't necessary must have this) to include all libraries I was using (I will post the file at the end of this answer)
Finally, at the PowerShell, all I have to type was py compilation.py py2exe
Thanks for all that tried to help me!
compilation.py file:
#To compile we need to run: python compilation.py py2exe
from distutils.core import setup
from glob import glob
import os
import py2exe
import pyDOE
VERSION=1.0
includes = [
"sip",
"PyQt5",
"PyQt5.QtCore",
"PyQt5.QtGui",
"PyQt5.QtWidgets",
"scipy.linalg.cython_blas",
"scipy.linalg.cython_lapack",
"pyDOE"
]
platforms = ["C:\\Python34\\Lib\\site-packages\\PyQt5\\plugins" +
"\\platforms\\qwindows.dll"]
dll = ["C:\\windows\\syswow64\\MSVCP100.dll",
"C:\\windows\\syswow64\\MSVCR100.dll"]
media = ["C:\\Users\\GUSTAVO\\EPlusPlus\\media\\title.png",
"C:\\Users\\GUSTAVO\\EPlusPlus\\media\\icon.png"]
documents = ["C:\\Users\\GUSTAVO\\EPlusPlus\\docs\\"+
"documentacaoEPlusPlus.pdf"]
examples = ["C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\baseline2A.idf",
"C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\vectors.csv",
"C:\\Users\\GUSTAVO\\EPlusPlus\\files\\"+
"\\examples\\BRA_SC_Florianopolis.838970_INMET.epw"]
datafiles = [("platforms", platforms),
("", dll),
("media", media),
("docs", documents),
("Examples", examples)]
imageformats = glob("C:\\Python34\\Lib\\site-packages\\PyQt5\\"+
"plugins\\imageformats\\*")
datafiles.append(("imageformats", imageformats))
setup(
name="eplusplus",
version=VERSION,
packages=["eplusplus"],
url="",
license="",
windows=[{"script": "eplusplus/__main__.py"}],
scripts=[],
data_files = datafiles,
options={
"py2exe": {
"includes": includes,
}
}
)

What does "Error 309" mean?

In our build we're creating an executable file with unit tests like this:
tests = env.Program(os.path.join(env['testDir'], name + '_test'),
src + createManifest(env),
LIBS = libs,
LIBPATH = buildLibPath(env),
LINKFLAGS = env['LINKFLAGS'],
CPPPATH = cppPath)
This correctly creates an executable, which later is being run by the following builder:
action = tests[0].abspath + '&& echo %DATE% %TIME% > ${TARGET}'
runTests = env.Command(source = tests,
target = 'test_'+name+'.tmp',
action = action)
Up to this point everything works fine: the tests are being run during the build.
I've recently found Visual Leak Detector tool and wanted to include this in the build. So, I've changed the environment for the builders like this:
vldInclude = os.path.join(os.path.normpath(env['vldIncDir']), 'vld.h')
env.Append(CPPFLAGS='/FI' + vldInclude)
env.Append(LIBPATH = env['vldLibDir'])
vldLib = os.path.join(env['vldLibDir'], 'vld.lib')
libs.append(vldLib) # used in the Program call for the LIBS parameter, see above
scons: *** [build\debug\libname\test_libname.dummy] Error 309
This error message isn't very helpful. What does it mean and how to fix it?

It turns out that the magic number 309 is more googleable when written as: 0xC0000135 (no idea why C, but 135HEX == 309DEC), and it is an identifier of the STATUS_DLL_NOT_FOUND error.
So, it's not a SCons error, but Windows error, that leaks through SCons.
This means that some DLLs are missing, needed by VLD. Lurking into the VLD installation directory (usually: C:\Program Files (x86)\Visual Leak Detector) two DLL files and one manifest file can be found in the bin\Win32 subdirectory.
Not to have the build being dependent on the machine's environment, you can either add the directory to env['ENV']['PATH'] or copy the files to the directory where the tests are being run.
To do the latter:
You need another VLD configuration option, beside the library directory, namely the binaries directory. Let's call it vldBinDir. At the build's startup you can copy these files to the build directory:
def setupVld(env):
sourcePath = env['vldBinDir']
targetPath = env['testDir']
toCopy = ['dbghelp.dll',
'vld_x86.dll',
'Microsoft.DTfW.DHL.manifest']
nodes = []
for c in toCopy:
n = env.Command(os.path.join(targetPath, c),
os.path.join(sourcePath, c),
SCons.Defaults.Copy("${TARGET}", "${SOURCE}"))
nodes.append(n)
env['vldDeps'] = nodes
And then, when creating particular tests, make sure to add the dependency:
for n in env['vldDeps']:
env.Depends(tests, n)

pyInstaller changing dll and pyd output location

I'm trying to use pyInstaller to package a wxpython application. I looking for a variation of the "one-folder" mode whereby the dlls and pyds are not stored in the top-level directory but in a subdirectory instead (like "dlls" or "libs").
This is the spec file currently:
# -*- mode: python -*-
import os
a = Analysis\
(
["..\\job_scraper\\load_gui.py"],
pathex = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
hiddenimports = [],
hookspath = None,
runtime_hooks = None
)
a_binaries = []
for (name, path, data_type) in a.binaries:
(non_ext, ext) = os.path.splitext(name)
if(ext in [".pyd", ".dll"]):
a_binaries.append((os.path.join("libs", name), path, data_type))
else:
a_binaries.append((name, path, data_type))
a.binaries = TOC(a_binaries)
pyz = PYZ(a.pure)
exe = EXE\
(
pyz,
a.scripts,
exclude_binaries = True,
name = "load_gui.exe",
debug = False,
strip = None,
upx = True,
console = False
)
coll = COLLECT\
(
exe,
a.binaries,
a.zipfiles,
a.datas,
[("control.csv", "..\\job_scraper\\control.csv", "DATA")],
strip = None,
upx = True,
name = "load_gui"
)
This does to put the dlls (not the pyds) into a lib folder, however it seems to do this after linking and so the program fails to launch because it can't find the expected dlls.

The problem is that the sys.path doesn't include your subdirectories. So when the program runs, it doesn't know where to look for your .dll or .pyd files.
Of course putting the code sys.path.append("relative/path/to/your/subdirectories") on top of your main script would come to mind. But one again, this code is only executed after everything is loaded and in place.
According to this blog, the solution is using the runtime hook of pyinstaller.
Runtime hook tells the bootstrapping code to run any of your arbitrary code before your main script runs - before importing any stuff.
1. Preparation
Create a hooker.py which will add all your custom paths to sys.path. Put it somewhere and do it 1 time only.
import sys
import os
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "lib")) # for pyd
sys.path.append(os.path.join(os.path.dirname(sys.argv[0]), "windll")) # for dll
2. Tell pyinstaller to include the hooker.py
with spec file:
a = Analysis\
(
["..\\job_scraper\\load_gui.py"],
pathex = ["C:\\Users\\Administrator\\Documents\\Projects\\python\\PyInstaller\\load_gui"],
hiddenimports = [],
hookspath = None,
runtime_hooks = "absolute/path/to/hooker.py" # <----- add it here
)
or with commandline:
pyinstaller --runtime-hook="absolute/path/to/hooker.py" the_rest_parameters
3. Run pyinstaller
As usually, it will create dist/your_main_script_name folder which contains the exe file, manifest, library.zip, and a bunch of .dll and .pyd
4. Create custom folders
Now you can create a windll folder and lib or anything that you add to sys.path at step 1. Then move all .pyd files to lib and all .dll files to windll.
Run your exe and it will crash! So move back these below files to parent folder.
pythonXX.dll , where XX is your python version
VCRUNTIME140.dll
pywintypesXX.dll, where XX is your python version and if it is included
These files are needed for the bootstrap so we cannot move them without modifying the bootstrap code.
Run the exe again and it should work normally.
Soon you will get bored of repeating all of the above over and over again.
So here is what I have been doing.
Create compiler.bat with the content similar to:
pyinstaller --runtime-hook="absolute/path/to/hooker.py" --onedir --icon path/to/icon ^
--exclude-module=UnNeeded_module_A ^
--exclude-module=UnNeeded_module_B ^
%1
#echo off
for %%F in (%1) do set pyi_output=%%~nxF
set pyi_output=%pyi_output:~0,-3%
mkdir dist\%pyi_output%\windll
mkdir dist\%pyi_output%\lib
move dist\%pyi_output%\*.dll dist\%pyi_output%\windll
move dist\%pyi_output%\*.pyd dist\%pyi_output%\lib
move dist\%pyi_output%\windll\python36.dll dist\%pyi_output%
move dist\%pyi_output%\windll\VCRUNTIME140.dll dist\%pyi_output%
if exist dist\%pyi_output%\windll\pywintypes36.dll (
move dist\%pyi_output%\windll\pywintypes36.dll dist\%pyi_output%
)
pause
To not mess up your project, create a copy of your code folder and place this compile.bat inside. Then, just drag and drop your main_script.py to compile.bat.
The pause command keeps the console windows open so that you know if the compilation is successful or not.

another way, new loadmyapp.c:
#include<stdlib.h>
main(int argc,char *argv[]) {
execv("yourapp/app.exe", argv);
}
gcc -o loadmyapp loadmyapp.c
./loadmyapp

py2app picking up .git subdir of a package during build

We use py2app extensively at our facility to produce self contained .app packages for easy internal deployment without dependency issues. Something I noticed recently, and have no idea how it began, is that when building an .app, py2app started including the .git directory of our main library.
commonLib, for instance, is our root python library package, which is a git repo. Under this package are the various subpackages such as database, utility, etc.
commonLib/
|- .git/ # because commonLib is a git repo
|- __init__.py
|- database/
|- __init__.py
|- utility/
|- __init__.py
# ... etc
In a given project, say Foo, we will do imports like from commonLib import xyz to use our common packages. Building via py2app looks something like: python setup.py py2app
So the recent issue I am seeing is that when building an app for project Foo, I will see it include everything in commonLib/.git/ into the app, which is extra bloat. py2app has an excludes option but that only seems to be for python modules. I cant quite figure out what it would take to exclude the .git subdir, or in fact, what is causing it to be included in the first place.
Has anyone experienced this when using a python package import that is a git repo?
Nothing has changed in our setup.py files for each project, and commonLib has always been a git repo. So the only thing I can think of being a variable is the version of py2app and its deps which have obviously been upgraded over time.
Edit
I'm using the latest py2app 0.6.4 as of right now. Also, my setup.py was first generated from py2applet a while back, but has been hand configured since and copied over as a template for every new project. I am using PyQt4/sip for every single one of these projects, so it also makes me wonder if its an issue with one of the recipes?
Update
From the first answer, I tried to fix this using various combinations of exclude_package_data settings. Nothing seems to force the .git directory to become excluded. Here is a sample of what my setup.py files generally look like:
from setuptools import setup
from myApp import VERSION
appname = 'MyApp'
APP = ['myApp.py']
DATA_FILES = []
OPTIONS = {
'includes': 'atexit, sip, PyQt4.QtCore, PyQt4.QtGui',
'strip': True,
'iconfile':'ui/myApp.icns',
'resources':['src/myApp.png'],
'plist':{
'CFBundleIconFile':'ui/myApp.icns',
'CFBundleIdentifier':'com.company.myApp',
'CFBundleGetInfoString': appname,
'CFBundleVersion' : VERSION,
'CFBundleShortVersionString' : VERSION
}
}
setup(
app=APP,
data_files=DATA_FILES,
options={'py2app': OPTIONS},
setup_requires=['py2app'],
)
I have tried things like:
setup(
...
exclude_package_data = { 'commonLib': ['.git'] },
#exclude_package_data = { '': ['.git'] },
#exclude_package_data = { 'commonLib/.git/': ['*'] },
#exclude_package_data = { '.git': ['*'] },
...
)
Update #2
I have posted my own answer which does a monkeypatch on distutils. Its ugly and not preferred, but until someone can offer me a better solution, I guess this is what I have.

I am adding an answer to my own question, to document the only thing I have found to work thus far. My approach was to monkeypatch distutils to ignore certain patterns when creating a directory or copying a file. This is really not what I wanted to do, but like I said, its the only thing that works so far.
## setup.py ##
import re
# file_util has to come first because dir_util uses it
from distutils import file_util, dir_util
def wrapper(fn):
def wrapped(src, *args, **kwargs):
if not re.search(r'/\.git/?', src):
fn(src, *args, **kwargs)
return wrapped
file_util.copy_file = wrapper(file_util.copy_file)
dir_util.mkpath = wrapper(dir_util.mkpath)
# now import setuptools so it uses the monkeypatched methods
from setuptools import setup
Hopefully someone will comment on this and tell me a higher level approach to avoid doing this. But as of now, I will probably wrap this into a utility method like exclude_data_patterns(re_pattern) to be reused in my projects.

I can see two options for excluding the .git directory.
Build the application from a 'clean' checkout of the code. When deploying a new version, we always build from a fresh svn export based on a tag to ensure we don't pick up spurious changes/files. You could try the equivalent here - although the git equivalent seems somewhat more involved.
Modify the setup.py file to massage the files included in the application. This might be done using the exclude_package_data functionality as described in the docs, or build the list of data_files and pass it to setup.
As for why it has suddenly started happening, knowing the version of py2app you are using might help, as will knowing the contents of your setup.py and perhaps how this was made (by hand or using py2applet).

I have a similar experience with Pyinstaller, so I'm not sure it applies directly.
Pyinstaller creates a "manifest" of all files to be included in the distribution, before running the export process. You could "massage" this manifest, as per Mark's second suggestion, to exclude any files you want. Including anything within .git or .git itself.
In the end, I stuck with checking out my code before producing a binary as there was more than just .git being bloat (such as UML documents and raw resource files for Qt). A checkout guaranteed a clean result and I experienced no issues automating that process along with the process of creating the installer for the binary.

There is a good answer to this, but I have a more elaborate answer to solve the problem mentioned here with a white-list approach. To have the monkey patch also work for packages outside site-packages.zip I had to monkey patch also copy_tree (because it imports copy_file inside its function), this helps in making a standalone application.
In addition, I create a white-list recipe to mark certain packages zip-unsafe. The approach makes it easy to add filters other than white-list.
import pkgutil
from os.path import join, dirname, realpath
from distutils import log
# file_util has to come first because dir_util uses it
from distutils import file_util, dir_util
# noinspection PyUnresolvedReferences
from py2app import util
def keep_only_filter(base_mod, sub_mods):
prefix = join(realpath(dirname(base_mod.filename)), '')
all_prefix = [join(prefix, sm) for sm in sub_mods]
log.info("Set filter for prefix %s" % prefix)
def wrapped(mod):
name = getattr(mod, 'filename', None)
if name is None:
# ignore anything that does not have file name
return True
name = join(realpath(dirname(name)), '')
if not name.startswith(prefix):
# ignore those that are not in this prefix
return True
for p in all_prefix:
if name.startswith(p):
return True
# log.info('ignoring %s' % name)
return False
return wrapped
# define all the filters we need
all_filts = {
'mypackage': (keep_only_filter, [
'subpackage1', 'subpackage2',
]),
}
def keep_only_wrapper(fn, is_dir=False):
filts = [(f, k[1]) for (f, k) in all_filts.iteritems()
if k[0] == keep_only_filter]
prefixes = {}
for f, sms in filts:
pkg = pkgutil.get_loader(f)
assert pkg, '{f} package not found'.format(f=f)
p = join(pkg.filename, '')
sp = [join(p, sm, '') for sm in sms]
prefixes[p] = sp
def wrapped(src, *args, **kwargs):
name = src
if not is_dir:
name = dirname(src)
name = join(realpath(name), '')
keep = True
for prefix, sub_prefixes in prefixes.iteritems():
if name == prefix:
# let the root pass
continue
# if it is a package we have a filter for
if name.startswith(prefix):
keep = False
for sub_prefix in sub_prefixes:
if name.startswith(sub_prefix):
keep = True
break
if keep:
return fn(src, *args, **kwargs)
return []
return wrapped
file_util.copy_file = keep_only_wrapper(file_util.copy_file)
dir_util.mkpath = keep_only_wrapper(dir_util.mkpath, is_dir=True)
util.copy_tree = keep_only_wrapper(util.copy_tree, is_dir=True)
class ZipUnsafe(object):
def __init__(self, _module, _filt):
self.module = _module
self.filt = _filt
def check(self, dist, mf):
m = mf.findNode(self.module)
if m is None:
return None
# Do not put this package in site-packages.zip
if self.filt:
return dict(
packages=[self.module],
filters=[self.filt[0](m, self.filt[1])],
)
return dict(
packages=[self.module]
)
# Any package that is zip-unsafe (uses __file__ ,... ) should be added here
# noinspection PyUnresolvedReferences
import py2app.recipes
for module in [
'sklearn', 'mypackage',
]:
filt = all_filts.get(module)
setattr(py2app.recipes, module, ZipUnsafe(module, filt))