We Keep Coding

Using embedded python, loading a *.pyd outside of DLLs directory fails

I have a C++ application (X-Plane) for which there is a plugin which permits the use of python scripts (XPPython3 plugin). The plugin is written in C, using python CAPI, and works great, allowing me to write python scripts which get executed within the C++ application.
On Windows 10, I want to extend my python features by importing imgui. I have a python cython-built pyd file (_imgui.cp39-win_amd64.pyd).
If I place the pyd file in C\Program Files\Python39\DLLs, it works as expected: C++ application calls CAPI to python, which loads script which imports and executes imgui code.
If I place the pyd file anywhere else, embedded python either reports "module not found" -- if the pyd isn't on sys.path(), or if it is on sys.path():
ImportError: DLL load failed while importing _imgui: The parameter is incorrect.'
Changes using: os.add_dll_directory(r'D:\somewhere\else')
Does not effect whether the module is found or not, nor does it change the 'parameter incorrect' error. (see https://bugs.python.org/issue36085 for details on this change. -- my guess is add_dll_directory changes lookup for DLLs, but not for pyd?) sys.path appears to be used for locating pyd.
Yes, the pyd is compiled with python3.9: I've compiled it both with mingw and with visual studio toolchains, in case that might be a difference.
For fun, I moved python-standard _zoneinfo.pyd from Python39\DLLs and it fails in the same way in embedded python: "The parameter is incorrect". So, that would appear to rule out my specific pyd file.
The key question is/are:
Other than placing a pyd file under PythonXX\DLLs, is there a way to load a PYD in an embedded python implementation? (I want to avoid having to tell users to move my pyd file into the Python39\DLLs directory... because they'll forget.)
Note that using IDLE or python.exe, I can load pyds without error -- anywhere on sys.path -- so they don't have to be under Python39\DLLs. It's only when trying to load from embedded python that the "Parameter is incorrect" appears. And of course, this works flawlessly on Mac.
(Bonus question: what parameter? It appears to be python passing through a windows error.)

There seems to be a simple answer, though I suspect it's better characterized as a python bug.
There is nothing magical about Python39\DLLs directory.
The problem is using absolute vs relative paths in sys.path.
Python can find modules using absolute or relative paths. So if zippy.py is in folder foobar,
sys.path.append('foobar')
import zippy
# Success
Python and find, BUT NOT LOAD pyd files using relative paths. For example, move _zoneinfo.pyd from PythonXX\LDDs to foobar
sys.path.append('foobar')
import _zoneinfo
# ImportError: DLL load failed while importing _zoneinfo: The parameter is incorrect.'
Instead, use absolute path, and it will find and load PYD:
sys.path.append(r'c:\MyTest\foobar')
import _zoneinfo
# Success

So, there is actually a way to do this—that is, ship your application with the desired libraries. The solution is to use an embedded distribution and ship this with your application. You can find the correct distribution on the official Python download page corresponding to your desired version (here's the link to the lastest 3.9 release which seems to be what you're using: https://www.python.org/downloads/release/python-392/). Look for the Windows Embeddable Package.
You can then simply drop in your .pyd file alongside the standard library files (note that if your third-party library is dependent on any other libraries, you will have to include them, as well). Shipping your application with an embeddable distribution should not only solve your current issue, but will also mean that your application will work regardless of which version of Python a user has installed (or without having Python installed at all).

Trying to import cjson in a Maya 2018 Python environment

I'd like to use cjson in Maya 2018 but I'm running into some trouble. The original tool seems to be 32-bit compiled (https://pypi.org/project/python-cjson/) so I looked around and found a version that was compiled in 64-bit from https://www.lfd.uci.edu/~gohlke/pythonlibs/.
Using "C:\Program Files\Autodesk\Maya2018\Python\Scripts\pip.exe" in admin mode I managed to pip install the 64-bit wheel file and that's given me cjson.pyd in my Maya 2018 site-packages directory.
However, when I try to import this I get the error:
// Error: root : [('<maya console>', 2, '<module>', None)] //
// Error: root : DLL load failed: The specified module could not be found. //
// Error: //
Has anyone had any experience importing this module into Maya before?
I have a feeling this could be to do DLL dependencies but I'm not sure where to begin solving it.

So, traditionally you'd check your dll for dependencies using something like Dependency Walker. .pyd files are just renamed .dll files, so should work fine with this. However, Dependency Walker is mostly full of noise - unless you spot something really obvious (and you can ignore all the API-MS-WIN*, EXT-MS-* dlls straight away) then you might find it easier to enable Loader Snaps (see this old but still relevant MSDN article) which should hopefully be able to tell you which loadlibrary call is failing. Alternatively, you can use Process Monitor (procmon) form Sysinternals to see what files Maya attempts to load when you import the module in to a script. I can't remember if Maya spawns a separate process to run it's python scripts in, or if they're executed in the context of the main maya process. This is important as you'll want to filter events in procmon to just maya / the python executable as otherwise you'll be swamped by them.
Edit: Having looked through the source code there doesn't appear to be much in the way of dependencies beyond the python headers themselves and some standard library includes. I suspect it's more likely that this is compiled with a different version of the toolchain than Maya's Python. In the Python interpreter, you should see a line like this when you start it:
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:25:58) [MSC v.1500 64 bit (AMD64)] on win32
It would be good to see what toolchain was used to compile it (the square brackets bit), and then build the extension from source again with that toolchain. Alternatively, you can try running the setup.py for this module from the project's GitHub page, which if your machine is set up to be able to compile python extensions correctly will build it from source. More details about building c extensions can be found on the Python docs page.

Python for .NET "unable to find assembly" error

I'm using CPython and I have a C# dll. I'm trying to use Python for .NET to make them talk. I can't use IronPython because I need to integrate this into an existing CPython system.
I'm completely new to Python for .NET, and I actually have very little experience with Python and no experience with C#. So please forgive me if my question seems very basic.
I'm using Python 2.7.3, and I downloaded
pythonnet-2.0-alpha2-clr2.0_131_py27_UCS2 and unzipped it into a folder named pyfornet_test, which also contains the dll I'm trying to use (called DotNet4Class.dll)
Then I run this:
import sys
import os
import clr
sys.path.append(r"C:\pyfornet_test")
clr.AddReference("DotNet4Class.dll")
Which gives me this error:
System.IO.FileNotFoundException: Unable to find assembly 'DotNet4Class.dll'.
at Python.Runtime.CLRModule.AddReference(String name) in C:\Users\Barton\Documents\Visual Studio 2008\Projects\PyShar
p\trunk\pythonnet\src\runtime\moduleobject.cs:line 375
Any advice would be much appreciated. Thank you!

One reason can be Windows was not enabling it to load from "external sources". To fix this:
Right-click on the .dll
"Properties"
Under "General", click "Unblock"

Try this (without extension .dll):
clr.AddReference(r"C:\pyfornet_test\DotNet4Class")

Is DotNet4Class.dll built against .NET 4? I assume so based on the naming of the dll.
Note the issue here: http://sourceforge.net/tracker/?func=detail&aid=3293169&group_id=162464&atid=823891
clr.AddReference fails when assembly is built with .NET 4.0 - ID: 3293169
I'd read the solution, but essentially, you need to rebuild and recompile the python for .NET project under .NET 4.
I'll also mention that projects like this, that aren't actively developed and used by lots of people, generally have subtle idiosyncrasies that make knowledge of the platform essential to work around problems such as this. It sounds like you're trying to hack this solution in without understanding much about python or .NET which is always going to be fraught with problems.

Did you try clr.FindAssembly?
import clr
import sys
assemblydir = r"C:\pyfornet_test"
assemblypath = r"C:\pyfornet_test\DotNet4Class.dll"
sys.path.append(assemblydir)
clr.FindAssembly(assemblypath)
I don't know why it works, but this code works on my computer (Python 2.7, .NET4)

I have code like this (I copied MyRightClickMenuService.dll to the same directory as my script.py). It is built against .Net 4.0.
# script.py
import clr
import os
import sys
sys.path.append(os.path.dirname(__file__))
clr.AddReference('MyRightClickMenuService')
clr.AddReference('System')
clr.AddReference('System.Security')
from MyRightClickMenuService import (
AclSecuredNamedPipeBinding,
MyMenuItem,
MyContextMenuService,
etc
)

Checklist
The folder(s) containing the DLL(s) is/are added to sys.path before loading. You may append, or sys.path.insert(0, dll_folder) to put it first on the list.
You call clr.AddReference('my_dll') without the dll extension (for my_dll.dll), after adding the folder to sys.path
The DLL Target Architecture is the same as the CPython version bitness. That is, if Architecture is x64, use 64-bit python, and if Architecture is x86, use 32-bit python. (instructions for this below)
How to check target Architecture for DLL?
I Used ILSpy (free and open source) -> Open DLL -> Check the output. Below example output.

What worked for me was to Unblock the dll file.
if u download the dll file or took it from different computer it might be blocked. So unblocked solved the issue for me.
To unblock right click on the properties if the dll file and check the Unblock box at the bottom

Importing a dll in python on Ubuntu

I am using python 2.6.5 on an Ubuntu intalled server.
I need to integrate an API for our applicaion, in that case, i needed to use a DLL given to me by the API provider. Their example of code about api integration is written in Visual Basic... I made a search on google and found some examples of using ctypes , and i try using cdll and pydll, which caused the following error...
OSError: /home//some.dll: invalid ELF header
One possibility is using IronPython, but i do not have much information about ironpython so i am not sure if it will handle my needs completely..
Is there any available module that let me use that dll on python (or aynthing that i am missing from the exixting ones). It is hard to upgrade my python version?

DLLs may be windows creatures, but if a DLL is 'pure .NET' and doesn't utilize executables specific to windows etc., then it can work often in Linux, through Mono. (mono ipy.exe).
Ironpython's System and similiar windows modules are customized to be os agnostic (to a untested degree).
I have successfully run NHibernate, FluentNHibernate, log4net, and a few other commonly used DLLS in Ubuntu.
import clr
import sys
sys.path.append(os.path.abspath('./DLL')) #where your dlls are
clr.AddReference('System')
clr.AddReference('FluentNHibernate')
from FluentNHibernate.Cfg.Db import PostgreSQLConfiguration
The key seems to be to import DLLs in this fashion. If a dll imports another (fluentnhibernate imports nhibernate), you don't need to import Nhibernate for example.

DLLs are Windows creatures. The only way you'll be able to use a DLL is by using a Windows build of Python. You'll be able to run Windows Python on Ubuntu by having Windows installed inside a virtual machine. You also might be able to run it using Wine.
An alternative, of course, is to ask your API provider if they have a Linux version of the API.

First, check if your DLL is a .NET Assembly file. An "Assembly DLL file" has nothing to do with the assembler. It's simply a way the .NET framework stores its bytecode inside a DLL file!
Do file library.dll in Linux. If it says something like this:
PE32 executable (DLL) (console) Intel 80386 Mono/.Net assembly, for MS Windows
then you're lucky: it's an assembly file. You can run it on Linux.
Install Mono. Install Python.NET. Forget IronPython: it's dead.
Now, in Python.NET, you can do this:
import clr
clr.AddReference('./library.dll')
# the library has just registered a namespace we can use
from LibraryName import *
but how do you know what to import?
Auto-complete.
Or use monop tool to inspect the DLL like this:
$ monop -r library.dll
Assembly Information:
LibraryName
Version=9.9.3.0
Culture=neutral
PublicKeyToken=null
LibraryName.ClassName
...
$ monop -r library.dll LibraryName.ClassName
public class ClassName {
public ClassName (string inputString);
...
}
and it will tell you everything about that library

Compiling an IronPython WPF project to exe

What is the best way to pack up an IronPython application for deployment? After scouring the web the best thing I've come up with (and what I'm currently doing) is using clr.CompileModules() to glue together my entire project's .py files into one .dll, and then having a single run.py do this to run the dll:
import clr
clr.AddReference('compiledapp.dll')
import app
This is still suboptimal, though, because it means I have to
distribute 3 files (the .dll, the .xaml, and the run.py launcher)
install IronPython on the host machine
Plus, this feels so... hacky, after the wonderful integration IronPython already has with Visual Studio 2010. I'm completely mystified as to why there is no integrated build system for IPy apps, seeing as it all boils down to IL anyway.
Ideally, I want to be able to have a single .exe with the .xaml merged inside somehow (I read that C# apps compile XAML to BAML and merge them in the executable), and without requiring IronPython to be installed to run. Is this at least halfway possible? (I suppose it's ok if the exe needs some extra .DLLs with it or something. The important part is that it's in .exe form.)
Some edits to clarify: I have tried pyc.py, but it seems to not acknowledge the fact that my project is not just app.py. The size of the exe it produces suggests that it is just 'compiling' app.py without including any of the other files into the exe. So, how do I tell it to compile every file in my project?
To help visualize this, here is a screenshot of my project's solution explorer window.
Edit II: It seems that unfortunately the only way is to use pyc.py and pass every single file to it as a parameter. There are two questions I have for this approach:
How do I possibly process a command line that big? There's a maximum of 256 characters in a command.
How does pyc.py know to preserve the package/folder structure? As shown in my project screenshot above, how will my compiled program know to access modules that are in subfolders, such as accessing DT\Device? Is the hierarchy somehow 'preserved' in the dll?
Edit III: Since passing 70 filenames to pyc.py through the command line will be unwieldy, and in the interest of solving the problem of building IPy projects more elegantly, I've decided to augment pyc.py.
I've added code that reads in a .pyproj file through the /pyproj: parameter, parses the XML, and grabs the list of py files used in the project from there. This has been working pretty well; however, the executable produced seems to be unable to access the python subpackages (subfolders) that are part of my project. My version of pyc.py with my .pyproj reading support patch can be found here: http://pastebin.com/FgXbZY29
When this new pyc.py is run on my project, this is the output:
c:\Projects\GenScheme\GenScheme>"c:\Program Files (x86)\IronPython 2.7\ipy.exe"
pyc.py /pyproj:GenScheme.pyproj /out:App /main:app.py /target:exe
Input Files:
c:\Projects\GenScheme\GenScheme\__init__.py
c:\Projects\GenScheme\GenScheme\Agent.py
c:\Projects\GenScheme\GenScheme\AIDisplay.py
c:\Projects\GenScheme\GenScheme\app.py
c:\Projects\GenScheme\GenScheme\BaseDevice.py
c:\Projects\GenScheme\GenScheme\BaseManager.py
c:\Projects\GenScheme\GenScheme\BaseSubSystem.py
c:\Projects\GenScheme\GenScheme\ControlSchemes.py
c:\Projects\GenScheme\GenScheme\Cu64\__init__.py
c:\Projects\GenScheme\GenScheme\Cu64\agent.py
c:\Projects\GenScheme\GenScheme\Cu64\aidisplays.py
c:\Projects\GenScheme\GenScheme\Cu64\devmapper.py
c:\Projects\GenScheme\GenScheme\Cu64\timedprocess.py
c:\Projects\GenScheme\GenScheme\Cu64\ui.py
c:\Projects\GenScheme\GenScheme\decorators.py
c:\Projects\GenScheme\GenScheme\DeviceMapper.py
c:\Projects\GenScheme\GenScheme\DT\__init__.py
c:\Projects\GenScheme\GenScheme\DT\Device.py
c:\Projects\GenScheme\GenScheme\DT\Manager.py
c:\Projects\GenScheme\GenScheme\DT\SubSystem.py
c:\Projects\GenScheme\GenScheme\excepts.py
c:\Projects\GenScheme\GenScheme\FindName.py
c:\Projects\GenScheme\GenScheme\GenScheme.py
c:\Projects\GenScheme\GenScheme\PMX\__init__.py
c:\Projects\GenScheme\GenScheme\PMX\Device.py
c:\Projects\GenScheme\GenScheme\PMX\Manager.py
c:\Projects\GenScheme\GenScheme\PMX\SubSystem.py
c:\Projects\GenScheme\GenScheme\pyevent.py
c:\Projects\GenScheme\GenScheme\Scheme.py
c:\Projects\GenScheme\GenScheme\Simulated\__init__.py
c:\Projects\GenScheme\GenScheme\Simulated\Device.py
c:\Projects\GenScheme\GenScheme\Simulated\SubSystem.py
c:\Projects\GenScheme\GenScheme\speech.py
c:\Projects\GenScheme\GenScheme\stdoutWriter.py
c:\Projects\GenScheme\GenScheme\Step.py
c:\Projects\GenScheme\GenScheme\TimedProcess.py
c:\Projects\GenScheme\GenScheme\UI.py
c:\Projects\GenScheme\GenScheme\VirtualSubSystem.py
c:\Projects\GenScheme\GenScheme\Waddle.py
Output:
App
Target:
ConsoleApplication
Platform:
ILOnly
Machine:
I386
Compiling...
Saved to App
So it correctly read in the list of files in the .pyproj... Great! But running the exe gives me this:
Unhandled Exception: IronPython.Runtime.Exceptions.ImportException:
No module named Cu64.ui
So even though Cu64\ui.py is obviously included in compilation, the exe, when run, can't find it. This is what I was afraid of in point #2 in the previous edit. How do I preserve the package hierarchy of my project? Perhaps compiling each package seperately may be needed?
I'll extend the bounty for this question. Ultimately my hope is that we can get a working pyc.py that reads in pyproj files and produces working exes in one step. Then maybe it could even be submitted to IronPython's codeplex to be included in the next release... ;]

Use pyc.py to produce app.exe and don't forget to include app.dll and IronPython libraries.
As for XAML - I've created project just for .xaml files that I compile in VS and then use them from IronPython. For example:
<ResourceDictionary.MergedDictionaries>
<ResourceDictionary Source="/CompiledStyle;component/Style.xaml" />
</ResourceDictionary.MergedDictionaries>

It "boils down to IL", but it isn't compatible with the IL that C# code produces, so it can't be directly compiled to a standalone .exe file.
You'll need to use pyc.py to compile your code to a stub EXE with the DLL that CompileModules creates.
Then distribute those files with IronPython.dll, IronPython.Modules.dll, Microsoft.Dynamic.dll, Microsoft.Scripting.Debugging.dll, Microsoft.Scripting.dll, and of course the XAML file.
To compile other files, just add them as arguments:
ipy.exe pyc.py /main:app.py /target:winexe another.py another2.py additional.py

I posted a Python script which can take an IronPython file, figure out its dependencies and compile the lot into a standalone binary at Ironpython 2.6 .py -> .exe. Hope you find it useful. It ought to work for WPF too as it bundles WPF support.

To create a set of assemblies for your IronPython application so that you can distribute it you can either use pyc.py or SharpDevelop.
To compile using pyc.py:
ipy.exe pyc.py /main:Program.py Form.py File1.py File2.py ... /target:winexe
Given the amount of files in your project you could try using SharpDevelop instead of maintaining a long command line for pyc.py. You will need to create a new IronPython project in SharpDevelop and import your files into the project. You will probably need to import the files one at a time since SharpDevelop lacks a way to import multiple files unless they are in a subfolder.
You can then use SharpDevelop to compile your application into an executable and a dll. All the other required files, such as IronPython.dll, Microsoft.Scripting.dll, will be in the bin/debug or bin/release folder. SharpDevelop uses clr.CompileModules and a custom MSBuild task behind the scenes to generate the binaries.
Any IronPython packages defined in your project should be usable from your application after compilation.
Packaging up the XAML can be done by embedding the xaml as a resource. Then using code similar to the following:
import clr
clr.AddReference('PresentationFramework')
clr.AddReference('System')
from System.IO import FileMode, FileStream, Path
from System.Reflection import Assembly
from System.Windows import Application
from System.Windows.Markup import XamlReader
executingAssemblyFileName = Assembly.GetEntryAssembly().Location
directory = Path.GetDirectoryName(executingAssemblyFileName)
xamlFileName = Path.Combine(directory, "Window1.xaml")
stream = FileStream(xamlFileName, FileMode.Open)
window = XamlReader.Load(stream)
app = Application()
app.Run(window)
SharpDevelop 3.2 does not embed resource files correctly so you will need to use SharpDevelop 4.
If you are using IronPython 2.7 you can use the new clr.LoadComponent method that takes an object and either a XAML filename or stream and wires up that object to the XAML.
Whilst the C# compiler can compile your XAML into a BAML resource doing the same with IronPython has a few problems. If you do not link the XAML to a class via the x:Class attribute then it is possible to compile the XAML into a BAML resource and have that embedded into your assembly. However you will not get any autogenerated code so you will need to create that code yourself. Another problem is that this will not work out of the box with SharpDevelop. You will need to edit the SharpDevelop.Build.Python.targets file and change the from Python to C#. Trying to use the x:Class attribute will not work since the BAML reader cannot access any associated IronPython class. This is because the generated IL in the compiled IronPython application is very different to that in a C# or VB.NET assembly.

I installed Visual Studio 2015 with PTVS (ironpython 2.7). I created a very simple WPF project and wasn't able to compile an exe. I always got the exception "ImportError: No module named wpf".
import clr
clr.AddReferenceToFileAndPath("c:\\path\\to\\IronPython.Wpf.dll")
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationCore.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationFramework.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\WindowsBase.dll')
from System.Windows import Application, Window
import wpf
class MyWindow(Window):
def __init__(self):
wpf.LoadComponent(self, 'RegExTester.xaml')
def OnSearch(self, sender, e):
self.tbOut.Text = "hello world"
if __name__ == '__main__':
Application().Run(MyWindow())
The fault I got was because the clr clause must be before the import wpf. Steps to compile it:
install pip for CPython 2.7 (not ironpython!)
install ipy2asm
python -m pip install ironpycompiler
compile the application like
ipy2asm compile -t winexe -e -s program.py