We Keep Coding

python modules in gcloud deployment manager template

Is it possible to use modules installed via python pip in gcloud deployment manager templates (python templates, not jinja)?
I have only being able to find reference of how to import .py files through a deployment manager schema file. e.g.
app.py.schema
info:
title: app
author: me
description: this is a description
imports:
- path: helper.py
i.e. i can only import a single .py at a time, so not useful for importing pip modules.
this link explains that to use libraries that is not explicitly supported we need to import the full library source. Although it does not mention if this full library source can actually be a pip module, or is it only referring to single .py files.
The module i'm trying to use inside my python templates is netaddr for manipulating ip address and subnets.
Any help is appreciated.

what you are looking for it not possible, you cannot install module using pip with interacting the the API, unless if you want to import the whole netaddr module as source code in your *.yaml config file (by adding the path for all the files related to the module) then importing which function your *.py file as Google mention in the documentation some library are supported, even with that some sys and network call will be rejected, you may think about using template_module
Original Answer:
Yes, you can check the link Here for importing multiple python files and using multiple templates.

Compiling an IronPython WPF project to exe

What is the best way to pack up an IronPython application for deployment? After scouring the web the best thing I've come up with (and what I'm currently doing) is using clr.CompileModules() to glue together my entire project's .py files into one .dll, and then having a single run.py do this to run the dll:
import clr
clr.AddReference('compiledapp.dll')
import app
This is still suboptimal, though, because it means I have to
distribute 3 files (the .dll, the .xaml, and the run.py launcher)
install IronPython on the host machine
Plus, this feels so... hacky, after the wonderful integration IronPython already has with Visual Studio 2010. I'm completely mystified as to why there is no integrated build system for IPy apps, seeing as it all boils down to IL anyway.
Ideally, I want to be able to have a single .exe with the .xaml merged inside somehow (I read that C# apps compile XAML to BAML and merge them in the executable), and without requiring IronPython to be installed to run. Is this at least halfway possible? (I suppose it's ok if the exe needs some extra .DLLs with it or something. The important part is that it's in .exe form.)
Some edits to clarify: I have tried pyc.py, but it seems to not acknowledge the fact that my project is not just app.py. The size of the exe it produces suggests that it is just 'compiling' app.py without including any of the other files into the exe. So, how do I tell it to compile every file in my project?
To help visualize this, here is a screenshot of my project's solution explorer window.
Edit II: It seems that unfortunately the only way is to use pyc.py and pass every single file to it as a parameter. There are two questions I have for this approach:
How do I possibly process a command line that big? There's a maximum of 256 characters in a command.
How does pyc.py know to preserve the package/folder structure? As shown in my project screenshot above, how will my compiled program know to access modules that are in subfolders, such as accessing DT\Device? Is the hierarchy somehow 'preserved' in the dll?
Edit III: Since passing 70 filenames to pyc.py through the command line will be unwieldy, and in the interest of solving the problem of building IPy projects more elegantly, I've decided to augment pyc.py.
I've added code that reads in a .pyproj file through the /pyproj: parameter, parses the XML, and grabs the list of py files used in the project from there. This has been working pretty well; however, the executable produced seems to be unable to access the python subpackages (subfolders) that are part of my project. My version of pyc.py with my .pyproj reading support patch can be found here: http://pastebin.com/FgXbZY29
When this new pyc.py is run on my project, this is the output:
c:\Projects\GenScheme\GenScheme>"c:\Program Files (x86)\IronPython 2.7\ipy.exe"
pyc.py /pyproj:GenScheme.pyproj /out:App /main:app.py /target:exe
Input Files:
c:\Projects\GenScheme\GenScheme\__init__.py
c:\Projects\GenScheme\GenScheme\Agent.py
c:\Projects\GenScheme\GenScheme\AIDisplay.py
c:\Projects\GenScheme\GenScheme\app.py
c:\Projects\GenScheme\GenScheme\BaseDevice.py
c:\Projects\GenScheme\GenScheme\BaseManager.py
c:\Projects\GenScheme\GenScheme\BaseSubSystem.py
c:\Projects\GenScheme\GenScheme\ControlSchemes.py
c:\Projects\GenScheme\GenScheme\Cu64\__init__.py
c:\Projects\GenScheme\GenScheme\Cu64\agent.py
c:\Projects\GenScheme\GenScheme\Cu64\aidisplays.py
c:\Projects\GenScheme\GenScheme\Cu64\devmapper.py
c:\Projects\GenScheme\GenScheme\Cu64\timedprocess.py
c:\Projects\GenScheme\GenScheme\Cu64\ui.py
c:\Projects\GenScheme\GenScheme\decorators.py
c:\Projects\GenScheme\GenScheme\DeviceMapper.py
c:\Projects\GenScheme\GenScheme\DT\__init__.py
c:\Projects\GenScheme\GenScheme\DT\Device.py
c:\Projects\GenScheme\GenScheme\DT\Manager.py
c:\Projects\GenScheme\GenScheme\DT\SubSystem.py
c:\Projects\GenScheme\GenScheme\excepts.py
c:\Projects\GenScheme\GenScheme\FindName.py
c:\Projects\GenScheme\GenScheme\GenScheme.py
c:\Projects\GenScheme\GenScheme\PMX\__init__.py
c:\Projects\GenScheme\GenScheme\PMX\Device.py
c:\Projects\GenScheme\GenScheme\PMX\Manager.py
c:\Projects\GenScheme\GenScheme\PMX\SubSystem.py
c:\Projects\GenScheme\GenScheme\pyevent.py
c:\Projects\GenScheme\GenScheme\Scheme.py
c:\Projects\GenScheme\GenScheme\Simulated\__init__.py
c:\Projects\GenScheme\GenScheme\Simulated\Device.py
c:\Projects\GenScheme\GenScheme\Simulated\SubSystem.py
c:\Projects\GenScheme\GenScheme\speech.py
c:\Projects\GenScheme\GenScheme\stdoutWriter.py
c:\Projects\GenScheme\GenScheme\Step.py
c:\Projects\GenScheme\GenScheme\TimedProcess.py
c:\Projects\GenScheme\GenScheme\UI.py
c:\Projects\GenScheme\GenScheme\VirtualSubSystem.py
c:\Projects\GenScheme\GenScheme\Waddle.py
Output:
App
Target:
ConsoleApplication
Platform:
ILOnly
Machine:
I386
Compiling...
Saved to App
So it correctly read in the list of files in the .pyproj... Great! But running the exe gives me this:
Unhandled Exception: IronPython.Runtime.Exceptions.ImportException:
No module named Cu64.ui
So even though Cu64\ui.py is obviously included in compilation, the exe, when run, can't find it. This is what I was afraid of in point #2 in the previous edit. How do I preserve the package hierarchy of my project? Perhaps compiling each package seperately may be needed?
I'll extend the bounty for this question. Ultimately my hope is that we can get a working pyc.py that reads in pyproj files and produces working exes in one step. Then maybe it could even be submitted to IronPython's codeplex to be included in the next release... ;]

Use pyc.py to produce app.exe and don't forget to include app.dll and IronPython libraries.
As for XAML - I've created project just for .xaml files that I compile in VS and then use them from IronPython. For example:
<ResourceDictionary.MergedDictionaries>
<ResourceDictionary Source="/CompiledStyle;component/Style.xaml" />
</ResourceDictionary.MergedDictionaries>

It "boils down to IL", but it isn't compatible with the IL that C# code produces, so it can't be directly compiled to a standalone .exe file.
You'll need to use pyc.py to compile your code to a stub EXE with the DLL that CompileModules creates.
Then distribute those files with IronPython.dll, IronPython.Modules.dll, Microsoft.Dynamic.dll, Microsoft.Scripting.Debugging.dll, Microsoft.Scripting.dll, and of course the XAML file.
To compile other files, just add them as arguments:
ipy.exe pyc.py /main:app.py /target:winexe another.py another2.py additional.py

I posted a Python script which can take an IronPython file, figure out its dependencies and compile the lot into a standalone binary at Ironpython 2.6 .py -> .exe. Hope you find it useful. It ought to work for WPF too as it bundles WPF support.

To create a set of assemblies for your IronPython application so that you can distribute it you can either use pyc.py or SharpDevelop.
To compile using pyc.py:
ipy.exe pyc.py /main:Program.py Form.py File1.py File2.py ... /target:winexe
Given the amount of files in your project you could try using SharpDevelop instead of maintaining a long command line for pyc.py. You will need to create a new IronPython project in SharpDevelop and import your files into the project. You will probably need to import the files one at a time since SharpDevelop lacks a way to import multiple files unless they are in a subfolder.
You can then use SharpDevelop to compile your application into an executable and a dll. All the other required files, such as IronPython.dll, Microsoft.Scripting.dll, will be in the bin/debug or bin/release folder. SharpDevelop uses clr.CompileModules and a custom MSBuild task behind the scenes to generate the binaries.
Any IronPython packages defined in your project should be usable from your application after compilation.
Packaging up the XAML can be done by embedding the xaml as a resource. Then using code similar to the following:
import clr
clr.AddReference('PresentationFramework')
clr.AddReference('System')
from System.IO import FileMode, FileStream, Path
from System.Reflection import Assembly
from System.Windows import Application
from System.Windows.Markup import XamlReader
executingAssemblyFileName = Assembly.GetEntryAssembly().Location
directory = Path.GetDirectoryName(executingAssemblyFileName)
xamlFileName = Path.Combine(directory, "Window1.xaml")
stream = FileStream(xamlFileName, FileMode.Open)
window = XamlReader.Load(stream)
app = Application()
app.Run(window)
SharpDevelop 3.2 does not embed resource files correctly so you will need to use SharpDevelop 4.
If you are using IronPython 2.7 you can use the new clr.LoadComponent method that takes an object and either a XAML filename or stream and wires up that object to the XAML.
Whilst the C# compiler can compile your XAML into a BAML resource doing the same with IronPython has a few problems. If you do not link the XAML to a class via the x:Class attribute then it is possible to compile the XAML into a BAML resource and have that embedded into your assembly. However you will not get any autogenerated code so you will need to create that code yourself. Another problem is that this will not work out of the box with SharpDevelop. You will need to edit the SharpDevelop.Build.Python.targets file and change the from Python to C#. Trying to use the x:Class attribute will not work since the BAML reader cannot access any associated IronPython class. This is because the generated IL in the compiled IronPython application is very different to that in a C# or VB.NET assembly.

I installed Visual Studio 2015 with PTVS (ironpython 2.7). I created a very simple WPF project and wasn't able to compile an exe. I always got the exception "ImportError: No module named wpf".
import clr
clr.AddReferenceToFileAndPath("c:\\path\\to\\IronPython.Wpf.dll")
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationCore.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationFramework.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\WindowsBase.dll')
from System.Windows import Application, Window
import wpf
class MyWindow(Window):
def __init__(self):
wpf.LoadComponent(self, 'RegExTester.xaml')
def OnSearch(self, sender, e):
self.tbOut.Text = "hello world"
if __name__ == '__main__':
Application().Run(MyWindow())
The fault I got was because the clr clause must be before the import wpf. Steps to compile it:
install pip for CPython 2.7 (not ironpython!)
install ipy2asm
python -m pip install ironpycompiler
compile the application like
ipy2asm compile -t winexe -e -s program.py

How to use nose with IronPython?

I installed nose using the 'setup.py install' on the command line , I am able to run 'nosetests' and any python file matching testMatch regular expression is picked up and tests are automated in the %python home%\Scripts directory. Now I want nose to work with my iron Python files , how do I install nose on the %Iron Python home% directory ? i noticed my Iron Python Home directory does not even have a Scripts folder.
If i try running 'nosetests' with iron python code , it throws all sorts of exception
for eg. no module named clr.
Is anybody using nose with iron python ? if yes , please guide me. I have been struggling with this since an entire day,
currently my only workaround has been adding the following in my IronPython code:
import nose
nose.main(argv=['<arguments>'])
is this is the only way to go about using nose in iron python files ?
if there is no other way , then I wanted to know how to use the several plugins that nose has ? especially the coverage plugin ? i installed it for python2.6 , but how to make it work for ironpython ?
The reason I am asking is because with python , it gets easy to use the plugins just by calling the command line , but with IronPython I don't know how to make it work.

Your solution is actually all nosetests does:
#!/usr/bin/env python
from nose import main
if __name__ == '__main__':
main()
You'll want to make sure you add your system's Python lib to the path for it to find the nose extensions:
>>>import sys
>>>sys.path.append(r'C:\Python26\lib')
And you'll need to make sure you're executing your script with ipy.exe and not your system's Python executable.

I've been trying to run the sqlalchemy test suite, which uses nose and a plugin. So, this may be useful if anyone is trying to run nose on ironpython with plugins.
this tends not to work transparently on ipy, because setuptools doesn't quite work on ironpython.
after a bit of diggin, i found the nose init.py instructions for registering a plugin manually - essentially, import the plugin class (which subclasses nose.plugins.Plugin), and add it to the call to main().
here's what my script ended up looking like:
import sys, os
#import ironclad #not needed. i think.
sys.path.append(r'C:\Python26\lib')
#now load Jeff Hardys sqlite dll which is in sqlite folder (sqlite not supported on ipy)
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)),'sqlite'))
import clr
clr.AddReference('IronPython.SQLite')
#load plugin
from sqlalchemy.test.noseplugin import NoseSQLAlchemy
from nose import main
if __name__ == '__main__':
main(addplugins=[NoseSQLAlchemy()])
Hope this helps someone!

Adding Version Control / Numbering (?) to Python Project

With my Java projects at present, I have full version control by declaring it as a Maven project. However I now have a Python project that I'm about to tag 0.2.0 which has no version control. Therefore should I come accross this code at a later date, I won't no what version it is.
How do I add version control to a Python project, in the same way Maven does it for Java?

First, maven is a build tool and has nothing to do with version control. You don't need a build tool with Python -- there's nothing to "build".
Some folks like to create .egg files for distribution. It's as close to a "build" as you get with Python. This is a simple setup.py file.
You can use SVN keyword replacement in your source like this. Remember to enable keyword replacement for the modules that will have this.
__version__ = "$Revision$"
That will assure that the version or revision strings are forced into your source by SVN.
You should also include version keywords in your setup.py file.

Create a distutils setup.py file. This is the Python equivalent to maven pom.xml, it looks something like this:
from distutils.core import setup
setup(name='foo',
version='1.0',
py_modules=['foo'],
)
If you want dependency management like maven, take a look at setuptools.

Ants's answer is correct, but I would like to add that your modules can define a __version__ variable, according to PEP 8, which can be populated manually or via Subversion or CVS, e.g. if you have a module thingy, with a file thingy/__init__.py:
___version___ = '0.2.0'
You can then import this version in setup.py:
from distutils.core import setup
import thingy
setup(name='thingy',
version=thingy.__version__,
py_modules=['thingy'],
)

Create plugins for python standalone executables

how to create a good plugin engine for standalone executables created with pyInstaller, py2exe or similar tools?
I do not have experience with py2exe, but pyInstaller uses an import hook to import packages from it's compressed repository. Of course I am able to import dynamically another compressed repository created with pyInstaller and execute the code - this may be a simple plugin engine.
Problems appears when the plugin (this what is imported dynamically) uses a library that is not present in original repository (never imported). This is because import hook is for the original application and searches for packages in original repository - not the one imported later (plugin package repository).
Is there an easy way to solve this problem? Maybe there exist such engine?

When compiling to exe, your going to have this issue.
The only option I can think of to allow users access with thier plugins to use any python library is to include all libraries in the exe package.
It's probably a good idea to limit supported libraries to a subset, and list it in your documentation. Up to you.
I've only used py2exe.
In py2exe you can specify libraries that were not found in the search in the setup.py file.
Here's a sample:
from distutils.core import setup
import py2exe
setup (name = "script2compile",
console=['script2compile.pyw'],
version = "1.4",
author = "me",
author_email="somemail#me.com",
url="myurl.com",
windows = [{
"script":"script2compile.pyw",
"icon_resources":[(1,"./ICONS/app.ico")] # Icon file to use for display
}],
# put packages/libraries to include in the "packages" list
options = {"py2exe":{"packages": [ "pickle",
"csv",
"Tkconstants",
"Tkinter",
"tkFileDialog",
"pyexpat",
"xml.dom.minidom",
"win32pdh",
"win32pdhutil",
"win32api",
"win32con",
"subprocess",
]}}
)
import win32pdh
import win32pdhutil
import win32api

PyInstaller does have a plugin system for handling hidden imports, and ships with several of those already in. See the webpage (http://www.pyinstaller.org) which says:
The main goal of PyInstaller is to be compatible with 3rd-party packages out-of-the-box. This means that, with PyInstaller, all the required tricks to make external packages work are already integrated within PyInstaller itself so that there is no user intervention required. You'll never be required to look for tricks in wikis and apply custom modification to your files or your setup scripts. Check our compatibility list of SupportedPackages.