We Keep Coding

Can I achieve precise control over location of .c files generated by cythonize?

I am using Cython as part of my build setup for a large project, driven by CMake.
I can't seem to get Cython to generate the .c files in a sensible location.
My file layout:
C:\mypath\src\demo.py # Cython source file
C:\mypath\build\bin # I want demo.pyd to end up here
C:\mypath\build\projects\cyt\setup.py # Generated by CMake
My setup.py is generated by CMake (a lot of it depends on configure_file), in the location specified above.
This location conforms to the usual structure of the overarching project (which builds over a hundred libraries and executables) and is not something I want to (or can easily) change.
The generated setup.py looks like this:
from distutils.core import setup, Extension
from Cython.Build import cythonize
import os.path
extension_args = {
'extra_compile_args' : ['/DWIN32','/DWIN64'],
'extra_link_args' : ['/MACHINE:X64'],
}
source = '../../../src/demo.py'
modules = [Extension(
os.path.splitext(os.path.basename(source))[0],
sources = [source],
**extension_args
)]
modules = cythonize(
modules,
build_dir = 'BUILD_DIR',
compiler_directives = {'language_level' : 2}
)
setup(name = 'demo',
version = '0.1',
description = '',
ext_modules = modules)
(Note that this is heavily simplified compared to the real case, which passes many additional arguments in extension_args, and includes many source files, each with its own object in modules.
Nevertheless, I have verified that the minimised version above reproduces my issue).
Cython is run like this:
cd C:\mypath\build\projects\cyt
python setup.py build_ext --build-lib C:/mypath/build/bin --build-temp C:/mypath/build/projects/cyt
Ideally, I would want all intermediary build artefacts from Cython (the generated C files, object files, exp files, etc.) to reside somewhere in or below C:\mypath\build\projects\cyt.
However, I can't seem to achieve that.
Here is where build artefacts actually end up:
demo.pyd ends up in C:\mypath\build\bin, where I want it. No problem here.
The object file demo.obj, along with the linked files demo.exp and demo.lib, end up in C:\mypath\build\projects\src. I want them inside cyt.
The C file demo.c ends up in C:\mypath\build\src. Again, I want this in projects\cyt.
In the setup.py, I am setting the build_dir parameter for cythonize as suggested in this answer, but it doesn't seem to work as I would like.
I also tried using cython_c_in_temp as per another answer on that question, but that has no effect (and judging from my inspection of Cython source code, does not apply to cythonize calls at all).
I tried using an absolute paths for source, but that made things even worse, as the C file ended up generated right next to demo.py, inside the source tree (as C:\src\demo.c).
My question: How can I make sure that all the generated intermediary files (C, obj, and friends) end up in the same directory as the generated setup.py, or below that?
I can think of two workarounds for my situation, but they both feel like hacks which I would like to avoid:
Copy all the Python source files from their locations in C:\mypath\src to alongside the generated setup.py, so that I can refer to them without .. in the path.
That would likely solve the issue, but burdens the (already long) build process with tens of additional file copy operations I'd rather avoid.
Since the path where the files end up seems to be composed by concatenating "the directory of setup.py + the value of build_dir + the value of source", I could count the number of .. in the source path and specify build_dir deep enough so that the evaluation results in the path I actually want.
This is both extremely hacky and very fragile.
I hope a better solution exists.

So it seems like you've run into a bug. Here's the relevant section of code in Cython. Basically, cythonize will try to build the paths to your .c and .o files as so:
C:/mypath/build/projects/cyt/BUILD_DIR/../../../src/demo.c
and so instead of nicely contained temporary files, you end up with insanity. Using an absolute path to demo.py won't help either, since that same code will just pass absolute paths through unchanged.
There doesn't seem to be a way to fix this in user-space short of extensive monkey-patching, so I submitted a pull-request to Cython with an actual fix. Once that's been merged in, you should then be able to run:
cd C:\mypath\build\projects\cyt
python setup.py build_ext -b C:/mypath/build/bin -t .
to get the result you want (-b and -t are the short forms of --build-lib and --build-temp).

Make cx_Freeze main.py permanently being able to use numpy module

I am using the cx_Freeze script located in the Python36/Scripts folder on a regular basis to convert python files into executables and it works fine. However it seems to still not being able to convert numpy so I am trying to make it work by adding an option into the main.py which is used by the cx_Freeze script described above. This main.py is located in the site-packages/cx_Freeze folder.
Thomas K. provided a solution here: Creating cx_Freeze exe with Numpy for Python
by adding this line to the options:
options = {"build_exe": {"packages": ["numpy.lib.format"]}}
Is it possible to add this line to the main.py in the options section? If so how would I do that?
Your help is much appreciated.

If I understand correctly what you like to do, you could try to add the following two lines to the file site-packages/cx_Freeze/freezer.py
## -127,6 +127,8 ## class Freezer(object):
self.includes = list(includes)
self.excludes = list(excludes)
self.packages = list(packages)
+ if 'numpy.lib.format' not in self.packages:
+ self.packages.append('numpy.lib.format')
self.namespacePackages = list(namespacePackages)
self.replacePaths = list(replacePaths)
self.compress = compress

Including sound files in a pygame script using pyinstaller

I am new to programming, so I made myself the challenge to create Pong, and so I did. Now I want to share it with a couple of friends, so I decided to try using pyinstaller (have tried cx_Freeze).
In this Pong game I have 3 sound effects, located in the folder "sfx". So I've looked into including files using pyinstaller, so my .spec file says:
added_files = [
('E:\Game Development Stuff\Python 3\Games\Pong\sfx\hitOutline.ogg', 'sfx'),
('E:\Game Development Stuff\Python 3\Games\Pong\sfx\hitPaddle.ogg', 'sfx'),
('E:\Game Development Stuff\Python 3\Games\Pong\sfx/score.ogg', 'sfx')
]
a = Analysis(['pong.py'],
pathex=['E:\\Game Development Stuff\\Python 3\\Games\\Pong'],
binaries=None,
datas=added_files,
and in the Pong program itself, I use this code to get the path:
def resource_path(relative):
if hasattr(sys, "_MEIPASS"):
return os.path.join(sys._MEIPASS, relative)
return os.path.join(relative)
fileDir = os.path.dirname(os.path.realpath('__file__'))
hitPaddle = resource_path(os.path.join(fileDir, "sfx", "hitPaddle.ogg"))
hitOutline = resource_path(os.path.join(fileDir, "sfx", "hitOutline.ogg"))
score = resource_path(os.path.join(fileDir, "sfx", "score.ogg"))
hitPaddleSound=pygame.mixer.Sound(hitPaddle)
hitOutlineSound=pygame.mixer.Sound(hitOutline)
scoreSound=pygame.mixer.Sound(score)
So I make the exe file using pyinstaller (with the command pyinstaller pong.spec)
but when I open the pong.exe file the command window says:
Traceback "<string>", Unable to open file 'E:\\Game Development Stuff\\Python 3\\Games\\Pong\\dist\\pong\\sfx\\hitPaddle.ogg'
but in that exact same path is hitPaddle.ogg.
It seems to me that pygame isn't able to found it for some weird reason?

I believe the issue is in this line. You are not refrencing the files correctly. You wrote:
hitPaddle = resource_path(os.path.join(fileDir, "sfx", "hitPaddle.ogg"))
Instead you should of just:
hitpaddle = resource_path("sfx\hitPaddle.ogg")
This is because when you added the files in the spec file, you stated that they should be in "root\sfx" folder. When the .exe is run in onefile mode, all files are actually located in a temp folder called MEIXXXX, with XXXX being some integers. When you run the .exe, if you open this folder you should be able to see your files there.

Solved it for me after struggling with the same issue for hours. Conclusions:
The problem is not in not being able to find it, then it would say something with 'could not find'. It is really an issue with opening the file. Somehow, the .ogg format is giving issues. I changed all my .ogg files to .wav files and my game is running without problems now as executable.
I have no idea why though, because two weeks ago for a previous version I did manage to make a working .exe with the exact same .ogg files. And I don't see how the changes I made should have any effect on this. Anyway, it works now, and perhaps this can also solve this problem for others.

Creating single EXE using py2exe for a Tkinter program

I'm trying to use minty's solution provided on this link to generate a single exe file for my Tkinter based program:
py2exe - generate single executable file
Here's what I wrote in the setup.py:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(windows=[{'script': 'filename.py'}], \
options={"py2exe": {"includes": ["decimal", "Tkinter", \
"tkFileDialog", "csv", "xml.dom.minidom", "os"], \
'bundle_files': 1, 'compressed': False}}, \
zipfile = None)
It creates a 'tcl' folder for Tkinter stuff even though I have specified bundle_files = 1. Plus it also generates some other exe w9xpopen.exe. My actual exe, however, does not run, and it doesn't give any errors either. It doesn't even work if I remove all those includes.
Any thoughts on what I could be missing here? I'm working on a 64-bit Windows 7 machine.

Thanks to this link, you have to edit site-packages/py2exe/build_exe.py and add "tcl85.dll" and "tk85.dll" to the dlls_in_exedir list. This will get it to run, although you'll still have the tcl folders, and those two dlls will be there along-side the exe. But it's way better than bundle_files=3.
self.dlls_in_exedir = [python_dll,
"w9xpopen%s.exe" % (is_debug_build and "_d" or ""),
"msvcr71%s.dll" % (is_debug_build and "d" or ""),
"tcl85.dll",
"tk85.dll"]

py2exe - generate single executable file

I thought I heard that py2exe was able to do this, but I never figured it out. Has anyone successfully done this? Can I see your setup.py file, and what command line options you used?
Basically I'm thinking of it giving me a single executable file that does something like unzips itself to maybe /temp and runs.

The way to do this using py2exe is to use the bundle_files option in your setup.py file. For a single file you will want to set bundle_files to 1, compressed to True, and set the zipfile option to None. That way it creates one compressed file for easy distribution.
Here is a more complete description of the bundle_file option quoted directly from the py2exe site*
Using "bundle_files" and "zipfile"
An easier (and better) way to create
single-file executables is to set
bundle_files to 1 or 2, and to set
zipfile to None. This approach does
not require extracting files to a
temporary location, which provides
much faster program startup.
Valid values for bundle_files are:
3 (default) don't bundle
2 bundle everything but the Python interpreter
1 bundle everything, including the Python interpreter
If zipfile is set to None, the files will be bundle
within the executable instead of library.zip.
Here is a sample setup.py:
from distutils.core import setup
import py2exe, sys, os
sys.argv.append('py2exe')
setup(
options = {'py2exe': {'bundle_files': 1, 'compressed': True}},
windows = [{'script': "single.py"}],
zipfile = None,
)

PyInstaller will create a single .exe file with no dependencies; use the --onefile option. It does this by packing all the needed shared libs into the executable, and unpacking them before it runs, just as you describe (EDIT: py2exe also has this feature, see minty's answer)
I use the version of PyInstaller from svn, since the latest release (1.3) is somewhat outdated. It's been working really well for an app which depends on PyQt, PyQwt, numpy, scipy and a few more.

As the other poster mention, py2exe, will generate an executable + some libraries to load. You can also have some data to add to your program.
Next step is to use an installer, to package all this into one easy-to-use installable/unistallable program.
I have used InnoSetup with delight for several years and for commercial programs, so I heartily recommend it.

I've been able to create a single exe file with all resources embeded into the exe.
I'm building on windows. so that will explain some of the os.system calls i'm using.
First I tried converting all my images into bitmats and then all my data files into text strings.
but this caused the final exe to be very very large.
After googleing for a week i figured out how to alter py2exe script to meet my needs.
here is the patch link on sourceforge i submitted, please post comments so we can get it included in
the next distribution.
http://sourceforge.net/tracker/index.php?func=detail&aid=3334760&group_id=15583&atid=315583
this explanes all the changes made, i've simply added a new option to the setup line.
here is my setup.py.
i'll try to comment it as best I can.
Please know that my setup.py is complex do to the fact that i'm access the images by filename.
so I must store a list to keep track of them.
this is from a want-to-b screen saver I was trying to make.
I use exec to generate my setup at run time, its easyer to cut and paste like that.
exec "setup(console=[{'script': 'launcher.py', 'icon_resources': [(0, 'ICON.ico')],\
'file_resources': [%s], 'other_resources': [(u'INDEX', 1, resource_string[:-1])]}],\
options={'py2exe': py2exe_options},\
zipfile = None )" % (bitmap_string[:-1])
breakdown
script = py script i want to turn to an exe
icon_resources = the icon for the exe
file_resources = files I want to embed into the exe
other_resources = a string to embed into the exe, in this case a file list.
options = py2exe options for creating everything into one exe file
bitmap_strings = a list of files to include
Please note that file_resources is not a valid option untill you edit your py2exe.py file as described in the link above.
first time i've tried to post code on this site, if I get it wrong don't flame me.
from distutils.core import setup
import py2exe ##UnusedImport
import os
#delete the old build drive
os.system("rmdir /s /q dist")
#setup my option for single file output
py2exe_options = dict( ascii=True, # Exclude encodings
excludes=['_ssl', # Exclude _ssl
'pyreadline', 'difflib', 'doctest', 'locale',
'optparse', 'pickle', 'calendar', 'pbd', 'unittest', 'inspect'], # Exclude standard library
dll_excludes=['msvcr71.dll', 'w9xpopen.exe',
'API-MS-Win-Core-LocalRegistry-L1-1-0.dll',
'API-MS-Win-Core-ProcessThreads-L1-1-0.dll',
'API-MS-Win-Security-Base-L1-1-0.dll',
'KERNELBASE.dll',
'POWRPROF.dll',
],
#compressed=None, # Compress library.zip
bundle_files = 1,
optimize = 2
)
#storage for the images
bitmap_string = ''
resource_string = ''
index = 0
print "compile image list"
for image_name in os.listdir('images/'):
if image_name.endswith('.jpg'):
bitmap_string += "( " + str(index+1) + "," + "'" + 'images/' + image_name + "'),"
resource_string += image_name + " "
index += 1
print "Starting build\n"
exec "setup(console=[{'script': 'launcher.py', 'icon_resources': [(0, 'ICON.ico')],\
'file_resources': [%s], 'other_resources': [(u'INDEX', 1, resource_string[:-1])]}],\
options={'py2exe': py2exe_options},\
zipfile = None )" % (bitmap_string[:-1])
print "Removing Trash"
os.system("rmdir /s /q build")
os.system("del /q *.pyc")
print "Build Complete"
ok, thats it for the setup.py
now the magic needed access the images.
I developed this app without py2exe in mind then added it later.
so you'll see access for both situations. if the image folder can't be found
it tries to pull the images from the exe resources. the code will explain it.
this is part of my sprite class and it uses a directx. but you can use any api you want or just access the raw data.
doesn't matter.
def init(self):
frame = self.env.frame
use_resource_builtin = True
if os.path.isdir(SPRITES_FOLDER):
use_resource_builtin = False
else:
image_list = LoadResource(0, u'INDEX', 1).split(' ')
for (model, file) in SPRITES.items():
texture = POINTER(IDirect3DTexture9)()
if use_resource_builtin:
data = LoadResource(0, win32con.RT_RCDATA, image_list.index(file)+1) #windll.kernel32.FindResourceW(hmod,typersc,idrsc)
d3dxdll.D3DXCreateTextureFromFileInMemory(frame.device, #Pointer to an IDirect3DDevice9 interface
data, #Pointer to the file in memory
len(data), #Size of the file in memory
byref(texture)) #ppTexture
else:
d3dxdll.D3DXCreateTextureFromFileA(frame.device, ##UndefinedVariable
SPRITES_FOLDER + file,
byref(texture))
self.model_sprites[model] = texture
#else:
# raise Exception("'sprites' folder is not present!")
Any questions fell free to ask.

You should create an installer, as mentioned before. Even though it is also possible to let py2exe bundle everything into a single executable, by setting bundle_files option to 1 and the zipfile keyword argument to None, I don't recommend this for PyGTK applications.
That's because of GTK+ tries to load its data files (locals, themes, etc.) from the directory it was loaded from. So you have to make sure that the directory of your executable contains also the libraries used by GTK+ and the directories lib, share and etc from your installation of GTK+. Otherwise you will get problems running your application on a machine where GTK+ is not installed system-wide.
For more details read my guide to py2exe for PyGTK applications. It also explains how to bundle everything, but GTK+.

I'm told bbfreeze will create a single file .EXE, and is newer than py2exe.

I recently used py2exe to create an executable for post-review for sending reviews to ReviewBoard.
This was the setup.py I used
from distutils.core import setup
import py2exe
setup(console=['post-review'])
It created a directory containing the exe file and the libraries needed. I don't think it is possible to use py2exe to get just a single .exe file. If you need that you will need to first use py2exe and then use some form of installer to make the final executable.
One thing to take care of is that any egg files you use in your application need to be unzipped, otherwise py2exe can't include them. This is covered in the py2exe docs.

try
c_x freeze
it can create a good standalone