We Keep Coding

How to bundle system sonames into a cx_Freeze application?

I am building an application which would make use of htmltidy, a library that cleans up bad HTML. There is a python binding which makes use of the soname libtidy.so. However, I intend to distribute the application using cx_Freeze. How should I do this? Also, I would like this to be a multiplatform solution, rather than just for Linux.
I tried to simply copy the soname to the created /lib folder, but that does not seem to work, most likely because of the soname search paths.
I also tried to use the bin_includes option in cx_Freeze setup.py, but that does not work either.
Script: testtidy.py
from tidylib import tidy_document
document, errors = tidy_document('''<p>fõo <img src="bar.jpg">''',
options={'numeric-entities':1})
print(document)
print(errors)
setup.py
import sys
from cx_Freeze import setup, Executable
# Dependencies are automatically detected, but it might need fine tuning.
# "packages": ["os"] is used as example only
build_exe_options = {"packages": ["os"],
"excludes": ["tkinter"],
"bin_includes": ["libtidy.so"]
}
base = None
setup(
name="testtidy",
version="0.1",
description="My application",
options={"build_exe": build_exe_options},
executables=[Executable("testtidy.py", base=base)],
)

The bin_includes option presumably does not work because the given library name is not an existing path.
In your setup.py, try to build an existing path to the tidy library:
import ctypes
from distutils import sysconfig
import os
lib_dir = sysconfig.get_config_var('LIBDIR')
libtidy_name = ctypes.util.find_library('tidy')
libtidy_path = os.path.join(lib_dir, libtidy_name)
libtidy_path should now contain the (symlink) path /usr/lib64/libtidy.so.58, use it in the bin_includes option:
build_exe_options = {...
"bin_includes": [libtidy_path]
}
After freezing, you should now see the symlink libtidy.so.58 and the the library itself libtidy.so.5.8.0 in the lib subdirectory of the build directory.
This still might not be enough to solve your problem. Looking at the source code of the python binding tidylib, starting from line 86, you will notice that it does not only rely on the soname libtidy.so, it rather relies on ctypes.util.find_library('tidy') and a pre-configured LIB_NAMES list as fallback. I'm not sure whether the modification proposed above will suffice to make ctypes.util.find_library('tidy') work in the frozen executable, if not we will still need to teach tidylib to find the library by tweaking the LIB_NAMES in the frozen executable.
EDIT:
Regarding building a multiplatform solution, see the cx_Freeze documentation:
cx_Freeze works on Windows, Mac and Linux, but on each platform it only makes an executable that runs on that platform. So if you want to freeze your programs for Windows, freeze it on Windows; if you want to run it on Macs, freeze it on a Mac.
You will thus need to make your source code run on the target platform, including htmltidy and its python binding pytidylib, and build the running code into an executable using cx_Freeze on that platform. Tweaks will probably be necessary there as well to let cx_Freeze include all dependencies of tidylib correctly, and these tweaks may look different for each platform.

SwiftUI - packing with Python and a module library

My goal: Import via PythonKit Python 3.9.1 in my Swift Project, where I can use it to package it with the needed module "openpyxl" (for filling an excel sheet with values).
Here are the imports at the beginning of the specific python file:
# some imports
import sys
from openpyxl import load_workbook # Library for editing xlsx-files
import datetime # for current date-informations
import csv # for reading csv
As you read I need Python 3 and the most Macs haven't installed it. So I want to include it in my App. I am new to Swift and have no clue how to achieve this.
What I have done so far:
Added the Swift Package Depency for "PythonKit" (used the Github Link and chose "master" branch)
Browsing Google for how to include a specific Python version in swift projects (and use them to execute Python files)
Asked Google how to add a library like "openpyxl" to PythonKit/Swift
Can you take me further? I am grateful for any tips how to get these python thing work in my Swift App. Would be wonderful to integrate the python library and make it work on every machine.

Found a solution:
Create a Python .plugin bundle with py2app, load it into Xcode and work with the path of the python binary.
This is the setup.py you could use:
"""
This is a setup.py script to create a py2app plugin bundle
Usage:
python setup.py py2app
"""
from setuptools import setup
APP = ['RandomFile.py']
OPTIONS = {
# Any local packages to include in the bundle should go here.
# See the py2app documentation for more.
"includes": [],
}
setup(
plugin=APP,
options={'py2app': OPTIONS},
setup_requires=['py2app'],
install_requires=['openpyxl'],
)
The documentation for building with py2app you can find here: https://py2app.readthedocs.io/en/latest/index.html
After building the plugin, we can drag it into our Xcode Project and get the path of it in Swift like this:
let pluginPath = Bundle.main.path(forResource: "Bridge", ofType: "plugin")

What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?

For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.

You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.

Python - Trouble in building executable

I'm a python programmer and I'm trying to build an executable binary to distribute my software to my clients, even if it's not fully executable I want to be able to distribute my software in a way so that it is convenient for the end user.
I have already tried PyInstaller as well as Py2Exe and I'm facing the same problem with a particular software.
I used the splinter module for my program (which of course is a new high level framework to interact with other frameworks like Selenium) and every time I try to compile it there seems to be a file called "webdriver.xpi" that is always left out from the final package and therefore when the program attempts to execute the web-driver it fails with an IO Error saying that the file "webdriver.xpi" was not found....but other than that the GUI and everything works perfectly fine.
So is there a way to include it even manually? I tried including it manually by browsing to the specific folder # library.zip file but it didn't work.
I'm not really expert in this matter and I rely on GUI2Exe for building everything...and I would really appreciate some advice if possible on how to fix this.
Thanks.

I was at this all day and found a workaround, it's sneaky but it works. In the error message I was receiving I noticed that there was a space between in library .zip. I could not trace it down in the source code for py2exe or selenium. I too had tried putting the xpi file in the library zip and it did not work. The workaround is:
In your setup file use these options:
setup(
console=['yourFile.py'],
options={
"py2exe":{
"skip_archive": True,
"unbuffered": True,
"optimize": 2
}
}
)
Run the py2exe install
Copy the xpi file into the dist directory
That should do it.

You need an instruction in your setup.py to include any resource files in your distribution. There is a couple of ways of doing this (see distutils, setuptools, distribute - depending on what you are using to build your distribution), but the py2exe wiki has an example.
You may need to use this py2exe tip to find your resources if you're installing them into the same directory as your exe.
See this answer for some additional info on including resource files in your distribution.

Here is a solution of your question:
I have modify a code little and it should be work since I had a same issue and I solved it:
from distutils.core import setup
import py2exe
wd_base = 'C:\\Python27\\Lib\site-packages\\selenium-2.44.0-py2.7.egg \\selenium\\webdriver'
RequiredDataFailes = [
('selenium/webdriver/firefox', ['%s\\firefox\\webdriver.xpi'%(wd_base), '%s\\firefox\\webdriver_prefs.json'%(wd_base)])
]
setup(
windows=[{"script":"gui_final.py"}],options={"py2exe":{"skip_archive": True,"includes":["sip"]}},
data_files=RequiredDataFailes,
)

I know this is old, but I wanted to give an updated answer to avoid suggesting that programmers do something manually.
There is a py2exe option to specify a list of data files as tuples. (pathtocopyto, [list of files and where to get them])
Example:
from disutils.core import setup
import py2exe
wd_base = 'C:\\Python27\\Lib\\site-packages\\selenium\\webdriver'
RequiredDataFailes = [
('selenium/webdriver/firefox', ['%s\\firefox\\webdriver.xpi'%(wd_base), '%s\\firefox\\webdriver_prefs.json'%(wd_base)])
]
setup(
console=['MyScript.py'],
data_files=RequiredDataFiles,
options={
**mypy2exeopts
}
)
The only downside I am aware of currently is that you still need skip_archive = True. There are workarounds to get the data files in the library.zip, but I haven't had much luck with the webdriver's info.

Compiling an IronPython WPF project to exe

What is the best way to pack up an IronPython application for deployment? After scouring the web the best thing I've come up with (and what I'm currently doing) is using clr.CompileModules() to glue together my entire project's .py files into one .dll, and then having a single run.py do this to run the dll:
import clr
clr.AddReference('compiledapp.dll')
import app
This is still suboptimal, though, because it means I have to
distribute 3 files (the .dll, the .xaml, and the run.py launcher)
install IronPython on the host machine
Plus, this feels so... hacky, after the wonderful integration IronPython already has with Visual Studio 2010. I'm completely mystified as to why there is no integrated build system for IPy apps, seeing as it all boils down to IL anyway.
Ideally, I want to be able to have a single .exe with the .xaml merged inside somehow (I read that C# apps compile XAML to BAML and merge them in the executable), and without requiring IronPython to be installed to run. Is this at least halfway possible? (I suppose it's ok if the exe needs some extra .DLLs with it or something. The important part is that it's in .exe form.)
Some edits to clarify: I have tried pyc.py, but it seems to not acknowledge the fact that my project is not just app.py. The size of the exe it produces suggests that it is just 'compiling' app.py without including any of the other files into the exe. So, how do I tell it to compile every file in my project?
To help visualize this, here is a screenshot of my project's solution explorer window.
Edit II: It seems that unfortunately the only way is to use pyc.py and pass every single file to it as a parameter. There are two questions I have for this approach:
How do I possibly process a command line that big? There's a maximum of 256 characters in a command.
How does pyc.py know to preserve the package/folder structure? As shown in my project screenshot above, how will my compiled program know to access modules that are in subfolders, such as accessing DT\Device? Is the hierarchy somehow 'preserved' in the dll?
Edit III: Since passing 70 filenames to pyc.py through the command line will be unwieldy, and in the interest of solving the problem of building IPy projects more elegantly, I've decided to augment pyc.py.
I've added code that reads in a .pyproj file through the /pyproj: parameter, parses the XML, and grabs the list of py files used in the project from there. This has been working pretty well; however, the executable produced seems to be unable to access the python subpackages (subfolders) that are part of my project. My version of pyc.py with my .pyproj reading support patch can be found here: http://pastebin.com/FgXbZY29
When this new pyc.py is run on my project, this is the output:
c:\Projects\GenScheme\GenScheme>"c:\Program Files (x86)\IronPython 2.7\ipy.exe"
pyc.py /pyproj:GenScheme.pyproj /out:App /main:app.py /target:exe
Input Files:
c:\Projects\GenScheme\GenScheme\__init__.py
c:\Projects\GenScheme\GenScheme\Agent.py
c:\Projects\GenScheme\GenScheme\AIDisplay.py
c:\Projects\GenScheme\GenScheme\app.py
c:\Projects\GenScheme\GenScheme\BaseDevice.py
c:\Projects\GenScheme\GenScheme\BaseManager.py
c:\Projects\GenScheme\GenScheme\BaseSubSystem.py
c:\Projects\GenScheme\GenScheme\ControlSchemes.py
c:\Projects\GenScheme\GenScheme\Cu64\__init__.py
c:\Projects\GenScheme\GenScheme\Cu64\agent.py
c:\Projects\GenScheme\GenScheme\Cu64\aidisplays.py
c:\Projects\GenScheme\GenScheme\Cu64\devmapper.py
c:\Projects\GenScheme\GenScheme\Cu64\timedprocess.py
c:\Projects\GenScheme\GenScheme\Cu64\ui.py
c:\Projects\GenScheme\GenScheme\decorators.py
c:\Projects\GenScheme\GenScheme\DeviceMapper.py
c:\Projects\GenScheme\GenScheme\DT\__init__.py
c:\Projects\GenScheme\GenScheme\DT\Device.py
c:\Projects\GenScheme\GenScheme\DT\Manager.py
c:\Projects\GenScheme\GenScheme\DT\SubSystem.py
c:\Projects\GenScheme\GenScheme\excepts.py
c:\Projects\GenScheme\GenScheme\FindName.py
c:\Projects\GenScheme\GenScheme\GenScheme.py
c:\Projects\GenScheme\GenScheme\PMX\__init__.py
c:\Projects\GenScheme\GenScheme\PMX\Device.py
c:\Projects\GenScheme\GenScheme\PMX\Manager.py
c:\Projects\GenScheme\GenScheme\PMX\SubSystem.py
c:\Projects\GenScheme\GenScheme\pyevent.py
c:\Projects\GenScheme\GenScheme\Scheme.py
c:\Projects\GenScheme\GenScheme\Simulated\__init__.py
c:\Projects\GenScheme\GenScheme\Simulated\Device.py
c:\Projects\GenScheme\GenScheme\Simulated\SubSystem.py
c:\Projects\GenScheme\GenScheme\speech.py
c:\Projects\GenScheme\GenScheme\stdoutWriter.py
c:\Projects\GenScheme\GenScheme\Step.py
c:\Projects\GenScheme\GenScheme\TimedProcess.py
c:\Projects\GenScheme\GenScheme\UI.py
c:\Projects\GenScheme\GenScheme\VirtualSubSystem.py
c:\Projects\GenScheme\GenScheme\Waddle.py
Output:
App
Target:
ConsoleApplication
Platform:
ILOnly
Machine:
I386
Compiling...
Saved to App
So it correctly read in the list of files in the .pyproj... Great! But running the exe gives me this:
Unhandled Exception: IronPython.Runtime.Exceptions.ImportException:
No module named Cu64.ui
So even though Cu64\ui.py is obviously included in compilation, the exe, when run, can't find it. This is what I was afraid of in point #2 in the previous edit. How do I preserve the package hierarchy of my project? Perhaps compiling each package seperately may be needed?
I'll extend the bounty for this question. Ultimately my hope is that we can get a working pyc.py that reads in pyproj files and produces working exes in one step. Then maybe it could even be submitted to IronPython's codeplex to be included in the next release... ;]

Use pyc.py to produce app.exe and don't forget to include app.dll and IronPython libraries.
As for XAML - I've created project just for .xaml files that I compile in VS and then use them from IronPython. For example:
<ResourceDictionary.MergedDictionaries>
<ResourceDictionary Source="/CompiledStyle;component/Style.xaml" />
</ResourceDictionary.MergedDictionaries>

It "boils down to IL", but it isn't compatible with the IL that C# code produces, so it can't be directly compiled to a standalone .exe file.
You'll need to use pyc.py to compile your code to a stub EXE with the DLL that CompileModules creates.
Then distribute those files with IronPython.dll, IronPython.Modules.dll, Microsoft.Dynamic.dll, Microsoft.Scripting.Debugging.dll, Microsoft.Scripting.dll, and of course the XAML file.
To compile other files, just add them as arguments:
ipy.exe pyc.py /main:app.py /target:winexe another.py another2.py additional.py

I posted a Python script which can take an IronPython file, figure out its dependencies and compile the lot into a standalone binary at Ironpython 2.6 .py -> .exe. Hope you find it useful. It ought to work for WPF too as it bundles WPF support.

To create a set of assemblies for your IronPython application so that you can distribute it you can either use pyc.py or SharpDevelop.
To compile using pyc.py:
ipy.exe pyc.py /main:Program.py Form.py File1.py File2.py ... /target:winexe
Given the amount of files in your project you could try using SharpDevelop instead of maintaining a long command line for pyc.py. You will need to create a new IronPython project in SharpDevelop and import your files into the project. You will probably need to import the files one at a time since SharpDevelop lacks a way to import multiple files unless they are in a subfolder.
You can then use SharpDevelop to compile your application into an executable and a dll. All the other required files, such as IronPython.dll, Microsoft.Scripting.dll, will be in the bin/debug or bin/release folder. SharpDevelop uses clr.CompileModules and a custom MSBuild task behind the scenes to generate the binaries.
Any IronPython packages defined in your project should be usable from your application after compilation.
Packaging up the XAML can be done by embedding the xaml as a resource. Then using code similar to the following:
import clr
clr.AddReference('PresentationFramework')
clr.AddReference('System')
from System.IO import FileMode, FileStream, Path
from System.Reflection import Assembly
from System.Windows import Application
from System.Windows.Markup import XamlReader
executingAssemblyFileName = Assembly.GetEntryAssembly().Location
directory = Path.GetDirectoryName(executingAssemblyFileName)
xamlFileName = Path.Combine(directory, "Window1.xaml")
stream = FileStream(xamlFileName, FileMode.Open)
window = XamlReader.Load(stream)
app = Application()
app.Run(window)
SharpDevelop 3.2 does not embed resource files correctly so you will need to use SharpDevelop 4.
If you are using IronPython 2.7 you can use the new clr.LoadComponent method that takes an object and either a XAML filename or stream and wires up that object to the XAML.
Whilst the C# compiler can compile your XAML into a BAML resource doing the same with IronPython has a few problems. If you do not link the XAML to a class via the x:Class attribute then it is possible to compile the XAML into a BAML resource and have that embedded into your assembly. However you will not get any autogenerated code so you will need to create that code yourself. Another problem is that this will not work out of the box with SharpDevelop. You will need to edit the SharpDevelop.Build.Python.targets file and change the from Python to C#. Trying to use the x:Class attribute will not work since the BAML reader cannot access any associated IronPython class. This is because the generated IL in the compiled IronPython application is very different to that in a C# or VB.NET assembly.

I installed Visual Studio 2015 with PTVS (ironpython 2.7). I created a very simple WPF project and wasn't able to compile an exe. I always got the exception "ImportError: No module named wpf".
import clr
clr.AddReferenceToFileAndPath("c:\\path\\to\\IronPython.Wpf.dll")
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationCore.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationFramework.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\WindowsBase.dll')
from System.Windows import Application, Window
import wpf
class MyWindow(Window):
def __init__(self):
wpf.LoadComponent(self, 'RegExTester.xaml')
def OnSearch(self, sender, e):
self.tbOut.Text = "hello world"
if __name__ == '__main__':
Application().Run(MyWindow())
The fault I got was because the clr clause must be before the import wpf. Steps to compile it:
install pip for CPython 2.7 (not ironpython!)
install ipy2asm
python -m pip install ironpycompiler
compile the application like
ipy2asm compile -t winexe -e -s program.py