I am currently working on a project to embed a Python interpreter in my website--I would like users to be able to execute their own Python code on the server-side in a secure Python sandbox. I have been using PyPy's sandboxing features and have set my virtual tmp/ directory (which corresponds to the real directory that the sandbox can read from) to my server-side files. So ideally, users will be able to run their own code on the server and use whatever libraries I make available to them (with no ability to write to any of these files/make system calls/do anything in general that would mess up my files).
The issue is that within my server-side code I use standard Python libraries as well as others I have installed separately (e.g. "import datetime", "import pandas", etc.). While the sandbox can read my server-side code it ends up failing whenever I try to import additional libraries within this code. I have thought of setting my virtual tmp/ directory to include my server-side files + the standard Python libraries + separately installed libraries such as PANDAS. However, when I tried setting it to just the Python libraries it ended up not working, not to mention that this solution seems pretty inelegant.
Any thoughts on how to go about fixing this? I appreciate whatever advice anyone may have. Thanks!
EDIT: I think part of the answer lies in the following portion of pypy_interact.py (the full source code can be found here: https://github.com/ojii/sandlib/blob/master/sandlib/pypy_interact.py):
def build_virtual_root(self):
# build a virtual file system:
# * can access its own executable
# * can access the pure Python libraries
# * can access the temporary usession directory as /tmp
exclude = ['.pyc', '.pyo']
if self.tmpdir is None:
tmpdirnode = Dir({})
else:
tmpdirnode = RealDir(self.tmpdir, exclude=exclude)
libroot = str(LIB_ROOT)
return Dir({
'bin': Dir({
'pypy-c': RealFile(self.executable),
'lib-python': RealDir(os.path.join(libroot, 'lib-python'),
exclude=exclude),
'lib_pypy': RealDir(os.path.join(libroot, 'lib_pypy'),
exclude=exclude),
}),
'tmp': tmpdirnode,
})
I am relatively new to Python and can't seem to figure out how to use this method properly. Any suggestions would be greatly appreciated.
Related
I am new with programming, so it is maybe harder for me to understand but I have the following issue:
I have a script that imports "eurostag.dll", which according to its manual, should work until Python 3.6. (but the manual is not updated, so it may work also with later updates, I assume).\ The issue is that I have Python 3.8. and when running the script I receive the following message:
"Failed to load EUROSTAG library (Could not find module 'D:\Eurostag\eustag_esg.dll' (or one of its dependencies). Try using the full path with constructor syntax.)"
I have tried to move the .dll library where the script is, but nothing changed. I tried also changing the directory with os.chdir, but the same message appears (with the same 'D:\Eurostag\eustag_esg.dll', so the directory was not changed.
Does anybody know if there is any workaround for this?
Thank you!
I have a C++ application (X-Plane) for which there is a plugin which permits the use of python scripts (XPPython3 plugin). The plugin is written in C, using python CAPI, and works great, allowing me to write python scripts which get executed within the C++ application.
On Windows 10, I want to extend my python features by importing imgui. I have a python cython-built pyd file (_imgui.cp39-win_amd64.pyd).
If I place the pyd file in C\Program Files\Python39\DLLs, it works as expected: C++ application calls CAPI to python, which loads script which imports and executes imgui code.
If I place the pyd file anywhere else, embedded python either reports "module not found" -- if the pyd isn't on sys.path(), or if it is on sys.path():
ImportError: DLL load failed while importing _imgui: The parameter is incorrect.'
Changes using: os.add_dll_directory(r'D:\somewhere\else')
Does not effect whether the module is found or not, nor does it change the 'parameter incorrect' error. (see https://bugs.python.org/issue36085 for details on this change. -- my guess is add_dll_directory changes lookup for DLLs, but not for pyd?) sys.path appears to be used for locating pyd.
Yes, the pyd is compiled with python3.9: I've compiled it both with mingw and with visual studio toolchains, in case that might be a difference.
For fun, I moved python-standard _zoneinfo.pyd from Python39\DLLs and it fails in the same way in embedded python: "The parameter is incorrect". So, that would appear to rule out my specific pyd file.
The key question is/are:
Other than placing a pyd file under PythonXX\DLLs, is there a way to load a PYD in an embedded python implementation? (I want to avoid having to tell users to move my pyd file into the Python39\DLLs directory... because they'll forget.)
Note that using IDLE or python.exe, I can load pyds without error -- anywhere on sys.path -- so they don't have to be under Python39\DLLs. It's only when trying to load from embedded python that the "Parameter is incorrect" appears. And of course, this works flawlessly on Mac.
(Bonus question: what parameter? It appears to be python passing through a windows error.)
There seems to be a simple answer, though I suspect it's better characterized as a python bug.
There is nothing magical about Python39\DLLs directory.
The problem is using absolute vs relative paths in sys.path.
Python can find modules using absolute or relative paths. So if zippy.py is in folder foobar,
sys.path.append('foobar')
import zippy
# Success
Python and find, BUT NOT LOAD pyd files using relative paths. For example, move _zoneinfo.pyd from PythonXX\LDDs to foobar
sys.path.append('foobar')
import _zoneinfo
# ImportError: DLL load failed while importing _zoneinfo: The parameter is incorrect.'
Instead, use absolute path, and it will find and load PYD:
sys.path.append(r'c:\MyTest\foobar')
import _zoneinfo
# Success
So, there is actually a way to do this—that is, ship your application with the desired libraries. The solution is to use an embedded distribution and ship this with your application. You can find the correct distribution on the official Python download page corresponding to your desired version (here's the link to the lastest 3.9 release which seems to be what you're using: https://www.python.org/downloads/release/python-392/). Look for the Windows Embeddable Package.
You can then simply drop in your .pyd file alongside the standard library files (note that if your third-party library is dependent on any other libraries, you will have to include them, as well). Shipping your application with an embeddable distribution should not only solve your current issue, but will also mean that your application will work regardless of which version of Python a user has installed (or without having Python installed at all).
I'm trying to use python in a new OS X application for plugin scripting. I'm looking to offload some of the program logic to python for easy, on-the-fly modification. I had started by looking at the Big Nerd Ranch tutorial. That seemed to work, but it suggested an older method of linking Python. It appears that since Xcode 5, we are supposed to install python using this Mac Developer Library Tech Note. This process, however, seems to create a linked python instance, not an embedded one. I've tried following the guidelines in the answer to this question but it seems to break down.
So my question is this: what are the current best practices for building python for use as a plugin runtime in an Objective C Mac OS X app? How does one go about making sure that it is bundled with the application, and that it is possible to install any additional libraries that one might want before the final Objective C app is built?
I've come up with a method that seems to work fairly well.
First, download source of the python version you want to use from the official website. Extract the archive somewhere. I'm using Python 3.4.2. Adjust the commands on your system for the specific version you're using.
Create a build directory that you will use for this development python version. The entire directory should have no spaces in it to make sure that bash interprets the she-bang (#!) lines correctly. I used /Users/myaccount/development/python/devbuild/python3.4.2.
Go into the extracted Python directory and run the following commands:
./configure --prefix="/Users/myaccount/development/python/devbuild/python3.4.2"
make
make install
This will install python in that development build directory. Setup the Python path to use the correct directory:
export PYTHONPATH="/Users/myaccount/development/python/devbuild/python3.4.2/lib/python3.4/site-packages/"
Go into the python bin directory (/Users/myaccount/development/python/devbuild/python3.4.2/bin) and use the pip3 there to install any modules that you need. The $PYTHONPATH setting will ensure that the modules get installed into the correct site-packages directory.
Find a handy home for the PyObjC repository and clone it there. Then checkout the latest version tag and install it, making sure that your $PYTHONPATH is still correct:
hg clone https://bitbucket.org/ronaldoussoren/pyobjc
cd pyobjc
hg tags
hg checkout [the id of the latest version from the tag list]
/Users/myaccount/development/python/devbuild/python3.4.2/bin/python3 ./install.py
Whenever you need to update the python modules, just make sure to use the correct python bin and $PYTHONPATH.
Now to add python to an Xcode project.
Drag the /Users/myaccount/development/python/devbuild/python3.4.2 directory into the Xcode project, setting it to not copy items as necessary, and to create a folder reference.
Add /Users/myaccount/development/python/devbuild/python3.4.2/include/python3.4m to the Header Search Paths setting in the Xcode project's Build Settings. Not sure if there's a way to do this as a generalized step to just search the folder referenced directory we had just added.
Drag the `/Users/myaccount/development/python/devbuild/python3.4.2/lib/libpython3.4m.a library into the Xcode project, setting it to be added as a reference without copying as well.
The code from the Big Nerd Ranch scripting tutorial repository can now be used with a few modifications.
The Plugin Manager code will need an NSString extension to work with the wchar_t strings that the Python API seems to like so much:
#interface NSString (WcharEncodedString)
- (wchar_t*) getWideString;
#end
#implementation NSString (WcharEncodedString)
- (wchar_t*) getWideString {
const char* tmp = [self cStringUsingEncoding:NSUTF8StringEncoding];
unsigned long buflen = strlen(tmp) + 1;
wchar_t* buffer = malloc(buflen * sizeof(wchar_t));
mbstowcs(buffer, tmp, buflen);
return buffer;
}
#end
The Python header should be included as follows:
#include "Python.h"
The following code needs to be run before Py_Initialize() is called in order to set up the correct python executable, PYTHONPATH, and PYTHONHOME as suggested by Zorg on that other question.
NSString* executablePath = [[NSBundle mainBundle] pathForResource:#"python3.4" ofType:nil inDirectory:#"python3.4.2/bin"];
Py_SetProgramName([executablePath getWideString]);
NSString* pythonDirectory = [[NSBundle mainBundle] pathForResource:#"python3.4" ofType:nil inDirectory:#"python3.4.2/lib"];
Py_SetPath([pythonDirectory getWideString]);
Py_SetPythonHome([pythonDirectory getWideString]);
Finally, the python path needs to be expanded in the PluginExecutor.py file to include the various subdirectories of the high level lib path. Add the following code to the top of the Plugin Executor file:
import sys
from os import walk
path = sys.path.copy()
for p in path:
for root,dirs,files in walk(p):
if p is not root:
sys.path.append(root)
I'll post updates if things start to break down, but this seems a working solution for now.
I am having this problem that we compiled our python project into an exe with py2exe and the resulting exe does not work unless it is executed as an administrator.
I wanted to ask that is this supposed to happen ?? As in there are so many applications that we can run as not an administrator so is there any way I can convert my python code to such an application ...
Thanks a lot..
It sounds like your application is attempting to write to a directory for which the basic user doesn't have access; most likely the "Program Files" directory. I believe in Vista/Win7 this is not allowed, and the standard convention is to write to the user's appdata folder, for any user data you might wish to store.
You can reliably obtain the location of this directory using the ctypes module, here's an example:
import ctypes
from ctypes import wintypes
def get_appdata_directory():
CSIDL_APPDATA = 0x001a
dll = ctypes.windll.shell32
app_data_directory = ctypes.create_unicode_buffer(wintypes.MAX_PATH)
found = dll.SHGetFolderPathW(0, CSIDL_APPDATA, 0, 0, app_data_directory)
# FYI: if `found` is False, then it failed to locate the appdata directory
# and app_data_directory.value is empty. So you might want to add some
# code here to verify that a valid path is going to be returned.
# This would probably only happen on older versions of windows,
# but, this is just a guess as I don't have any older OSs available
# for testing. (see my note below)
return app_data_directory.value
appdata = get_appdata_directory()
print(appdata)
# outputs something such as: 'C:\Users\bob\AppData'
Note: I believe the appdata folder was introduced with WinXP/Win2k. Not sure about WinME and prior, however, I don't believe you have to worry about the administrator restrictions on these earlier OSs. If you really care to support them, you could use python's builtin platform module along with some conditionals, then simply write user data to the "Program Files" directory for the archaic versions of Windows.
I've had serious problems with getting py2exe to work. Luckily I found the excellent PyInstaller which not only works, but also creates smaller executables. I haven't ran into the problem you mention with PyInstaller so I recommend trying it.
http://www.pyinstaller.org/
What is the best way to pack up an IronPython application for deployment? After scouring the web the best thing I've come up with (and what I'm currently doing) is using clr.CompileModules() to glue together my entire project's .py files into one .dll, and then having a single run.py do this to run the dll:
import clr
clr.AddReference('compiledapp.dll')
import app
This is still suboptimal, though, because it means I have to
distribute 3 files (the .dll, the .xaml, and the run.py launcher)
install IronPython on the host machine
Plus, this feels so... hacky, after the wonderful integration IronPython already has with Visual Studio 2010. I'm completely mystified as to why there is no integrated build system for IPy apps, seeing as it all boils down to IL anyway.
Ideally, I want to be able to have a single .exe with the .xaml merged inside somehow (I read that C# apps compile XAML to BAML and merge them in the executable), and without requiring IronPython to be installed to run. Is this at least halfway possible? (I suppose it's ok if the exe needs some extra .DLLs with it or something. The important part is that it's in .exe form.)
Some edits to clarify: I have tried pyc.py, but it seems to not acknowledge the fact that my project is not just app.py. The size of the exe it produces suggests that it is just 'compiling' app.py without including any of the other files into the exe. So, how do I tell it to compile every file in my project?
To help visualize this, here is a screenshot of my project's solution explorer window.
Edit II: It seems that unfortunately the only way is to use pyc.py and pass every single file to it as a parameter. There are two questions I have for this approach:
How do I possibly process a command line that big? There's a maximum of 256 characters in a command.
How does pyc.py know to preserve the package/folder structure? As shown in my project screenshot above, how will my compiled program know to access modules that are in subfolders, such as accessing DT\Device? Is the hierarchy somehow 'preserved' in the dll?
Edit III: Since passing 70 filenames to pyc.py through the command line will be unwieldy, and in the interest of solving the problem of building IPy projects more elegantly, I've decided to augment pyc.py.
I've added code that reads in a .pyproj file through the /pyproj: parameter, parses the XML, and grabs the list of py files used in the project from there. This has been working pretty well; however, the executable produced seems to be unable to access the python subpackages (subfolders) that are part of my project. My version of pyc.py with my .pyproj reading support patch can be found here: http://pastebin.com/FgXbZY29
When this new pyc.py is run on my project, this is the output:
c:\Projects\GenScheme\GenScheme>"c:\Program Files (x86)\IronPython 2.7\ipy.exe"
pyc.py /pyproj:GenScheme.pyproj /out:App /main:app.py /target:exe
Input Files:
c:\Projects\GenScheme\GenScheme\__init__.py
c:\Projects\GenScheme\GenScheme\Agent.py
c:\Projects\GenScheme\GenScheme\AIDisplay.py
c:\Projects\GenScheme\GenScheme\app.py
c:\Projects\GenScheme\GenScheme\BaseDevice.py
c:\Projects\GenScheme\GenScheme\BaseManager.py
c:\Projects\GenScheme\GenScheme\BaseSubSystem.py
c:\Projects\GenScheme\GenScheme\ControlSchemes.py
c:\Projects\GenScheme\GenScheme\Cu64\__init__.py
c:\Projects\GenScheme\GenScheme\Cu64\agent.py
c:\Projects\GenScheme\GenScheme\Cu64\aidisplays.py
c:\Projects\GenScheme\GenScheme\Cu64\devmapper.py
c:\Projects\GenScheme\GenScheme\Cu64\timedprocess.py
c:\Projects\GenScheme\GenScheme\Cu64\ui.py
c:\Projects\GenScheme\GenScheme\decorators.py
c:\Projects\GenScheme\GenScheme\DeviceMapper.py
c:\Projects\GenScheme\GenScheme\DT\__init__.py
c:\Projects\GenScheme\GenScheme\DT\Device.py
c:\Projects\GenScheme\GenScheme\DT\Manager.py
c:\Projects\GenScheme\GenScheme\DT\SubSystem.py
c:\Projects\GenScheme\GenScheme\excepts.py
c:\Projects\GenScheme\GenScheme\FindName.py
c:\Projects\GenScheme\GenScheme\GenScheme.py
c:\Projects\GenScheme\GenScheme\PMX\__init__.py
c:\Projects\GenScheme\GenScheme\PMX\Device.py
c:\Projects\GenScheme\GenScheme\PMX\Manager.py
c:\Projects\GenScheme\GenScheme\PMX\SubSystem.py
c:\Projects\GenScheme\GenScheme\pyevent.py
c:\Projects\GenScheme\GenScheme\Scheme.py
c:\Projects\GenScheme\GenScheme\Simulated\__init__.py
c:\Projects\GenScheme\GenScheme\Simulated\Device.py
c:\Projects\GenScheme\GenScheme\Simulated\SubSystem.py
c:\Projects\GenScheme\GenScheme\speech.py
c:\Projects\GenScheme\GenScheme\stdoutWriter.py
c:\Projects\GenScheme\GenScheme\Step.py
c:\Projects\GenScheme\GenScheme\TimedProcess.py
c:\Projects\GenScheme\GenScheme\UI.py
c:\Projects\GenScheme\GenScheme\VirtualSubSystem.py
c:\Projects\GenScheme\GenScheme\Waddle.py
Output:
App
Target:
ConsoleApplication
Platform:
ILOnly
Machine:
I386
Compiling...
Saved to App
So it correctly read in the list of files in the .pyproj... Great! But running the exe gives me this:
Unhandled Exception: IronPython.Runtime.Exceptions.ImportException:
No module named Cu64.ui
So even though Cu64\ui.py is obviously included in compilation, the exe, when run, can't find it. This is what I was afraid of in point #2 in the previous edit. How do I preserve the package hierarchy of my project? Perhaps compiling each package seperately may be needed?
I'll extend the bounty for this question. Ultimately my hope is that we can get a working pyc.py that reads in pyproj files and produces working exes in one step. Then maybe it could even be submitted to IronPython's codeplex to be included in the next release... ;]
Use pyc.py to produce app.exe and don't forget to include app.dll and IronPython libraries.
As for XAML - I've created project just for .xaml files that I compile in VS and then use them from IronPython. For example:
<ResourceDictionary.MergedDictionaries>
<ResourceDictionary Source="/CompiledStyle;component/Style.xaml" />
</ResourceDictionary.MergedDictionaries>
It "boils down to IL", but it isn't compatible with the IL that C# code produces, so it can't be directly compiled to a standalone .exe file.
You'll need to use pyc.py to compile your code to a stub EXE with the DLL that CompileModules creates.
Then distribute those files with IronPython.dll, IronPython.Modules.dll, Microsoft.Dynamic.dll, Microsoft.Scripting.Debugging.dll, Microsoft.Scripting.dll, and of course the XAML file.
To compile other files, just add them as arguments:
ipy.exe pyc.py /main:app.py /target:winexe another.py another2.py additional.py
I posted a Python script which can take an IronPython file, figure out its dependencies and compile the lot into a standalone binary at Ironpython 2.6 .py -> .exe. Hope you find it useful. It ought to work for WPF too as it bundles WPF support.
To create a set of assemblies for your IronPython application so that you can distribute it you can either use pyc.py or SharpDevelop.
To compile using pyc.py:
ipy.exe pyc.py /main:Program.py Form.py File1.py File2.py ... /target:winexe
Given the amount of files in your project you could try using SharpDevelop instead of maintaining a long command line for pyc.py. You will need to create a new IronPython project in SharpDevelop and import your files into the project. You will probably need to import the files one at a time since SharpDevelop lacks a way to import multiple files unless they are in a subfolder.
You can then use SharpDevelop to compile your application into an executable and a dll. All the other required files, such as IronPython.dll, Microsoft.Scripting.dll, will be in the bin/debug or bin/release folder. SharpDevelop uses clr.CompileModules and a custom MSBuild task behind the scenes to generate the binaries.
Any IronPython packages defined in your project should be usable from your application after compilation.
Packaging up the XAML can be done by embedding the xaml as a resource. Then using code similar to the following:
import clr
clr.AddReference('PresentationFramework')
clr.AddReference('System')
from System.IO import FileMode, FileStream, Path
from System.Reflection import Assembly
from System.Windows import Application
from System.Windows.Markup import XamlReader
executingAssemblyFileName = Assembly.GetEntryAssembly().Location
directory = Path.GetDirectoryName(executingAssemblyFileName)
xamlFileName = Path.Combine(directory, "Window1.xaml")
stream = FileStream(xamlFileName, FileMode.Open)
window = XamlReader.Load(stream)
app = Application()
app.Run(window)
SharpDevelop 3.2 does not embed resource files correctly so you will need to use SharpDevelop 4.
If you are using IronPython 2.7 you can use the new clr.LoadComponent method that takes an object and either a XAML filename or stream and wires up that object to the XAML.
Whilst the C# compiler can compile your XAML into a BAML resource doing the same with IronPython has a few problems. If you do not link the XAML to a class via the x:Class attribute then it is possible to compile the XAML into a BAML resource and have that embedded into your assembly. However you will not get any autogenerated code so you will need to create that code yourself. Another problem is that this will not work out of the box with SharpDevelop. You will need to edit the SharpDevelop.Build.Python.targets file and change the from Python to C#. Trying to use the x:Class attribute will not work since the BAML reader cannot access any associated IronPython class. This is because the generated IL in the compiled IronPython application is very different to that in a C# or VB.NET assembly.
I installed Visual Studio 2015 with PTVS (ironpython 2.7). I created a very simple WPF project and wasn't able to compile an exe. I always got the exception "ImportError: No module named wpf".
import clr
clr.AddReferenceToFileAndPath("c:\\path\\to\\IronPython.Wpf.dll")
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationCore.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\PresentationFramework.dll')
clr.AddReferenceToFileAndPath('c:\\path\\to\\WindowsBase.dll')
from System.Windows import Application, Window
import wpf
class MyWindow(Window):
def __init__(self):
wpf.LoadComponent(self, 'RegExTester.xaml')
def OnSearch(self, sender, e):
self.tbOut.Text = "hello world"
if __name__ == '__main__':
Application().Run(MyWindow())
The fault I got was because the clr clause must be before the import wpf. Steps to compile it:
install pip for CPython 2.7 (not ironpython!)
install ipy2asm
python -m pip install ironpycompiler
compile the application like
ipy2asm compile -t winexe -e -s program.py