Related
Short version:
Is it possible to create a standalone program that can be distributed to computers that don't have python installed, which writes a python script during runtime and executes it during or shortly after?
Long version:
A project I've been wanting to do for a while is to create a visual programming interface, that lets people use Machine Learning without needing to know python/keras/numpy syntax. Programs like lobe or rapidminer already do this, but they are all bound to their own interfaces and servers. What i would like to do is create a program that:
anyone can use without needing python/anaconda installed
allow the user to create visual scripts like in scratch or google-blocs
generates python code behind the scenes containing keras or tf code
is able to execute the script
is able to show the code to the user for:
educational purposes
allowing the user to use it as a base for a more complex program
Since the generation part would just need to create a text file almost any interface and language like python/java using Qt or C# / javascript using Unity, would qualify. I think that should definitely be doable, probably just very long, but that is not my largest concern.
The problem:
I have tried to search everywhere on the internet about things like standalone python programs which led me to for instance py2exe. Those kind of python-to-exe's work great, but they all require the script to be compiled on a machine containing a python environment. In my case i want to be able to generate a python script on the user's PC, and run it directly after from there.
The alternative
If this isn't possible i might just create the whole AI part myself, not using python nor libraries like keras or tensorflow, but in a unity game for instance. The downside to this would be that it exists already(like rapidminer), it would be less optimized/versatile/customizable and doesn't show what the "real" machine learning script would look.
If there are any other alternatives i would be happy to hear them
PS.
I have mostly Python & Keras, moderate Java and little Unity3D or web JS experience
I'm using Windows with anaconda
Distributing to linux/mac would be nice, but not required
maybe you should check the Orange Data Mining software, it's written in python and it has the same purpose as your project (https://orange.biolab.si/). On the other hand, some time ago I tried to compile my app that contains machine learning libraries but to date, I have not been able to achieve it. Instead, what I use is the WINPYTHON project, this is a portable Anaconda software that allows you to run projects on any PC without the need to install anything (https://winpython.github.io/).
Yes, it is possible!
I had the same requirement, so I wrote my visual programming language and IDE...
...that could generate an almost python-like script, and which is compiled natively inside the app, without the use of any external compilers or libraries.
My target architecture was mobile devices, but it also works on browsers via the unity plugin.
...and yes, that's correct, it runs natively on your phone or tablet in a simulated sandbox, with its own built-in IDE.
It's written in C#, and implemented in Unity3d
You can check it out at https://aiBoard.blog
..and see the videos at https://youtu.be/DIDgu9jrdLc
I want to add scripting capabilities to my application and I just came across python4delphi http://code.google.com/p/python4delphi/, which seems to be stable.
At first sight it looks very easy to use on the developers side, but I couldn't find a way to debug a script. Would be great if I could embed the IDLE IDE in my application, I just have no idea how to do it.
Is there a easy way to add debugging capabilities with python4delphi? Should I use other script engines like pascalScript (seems to have been discontinued)?
Edit: After having a look at PySctipt and pdb I realized that what I am looking is more like a visual editor with embedded debugging and the capability to read python4delphi custom modules (the ones where I export my delphi objects. Being a python noob I have no idea how p4d does this). I am probably asking too much, but would be great to find an implementation of that and not having to code it all from scratch.
Python is really a great language.
But if you need to embed script, with debug, in a Delphi application you should considere Pascal scripting, which is more available in Delphi.
I use the excellent PascalScipt, in the exemples you will see editor, debugger and a lot of other functions.
Is it possible to deploy python applications such that you don't release the source code and you don't have to be sure the customer has python installed?
I'm thinking maybe there is some installation process that can run a python app from just the .pyc files and a shared library containing the interpreter or something like that?
Basically I'm keen to get the development benefits of a language like Python - high productivity etc. but can't quite see how you could deploy it professionally to a customer where you don't know how there machine is set up and you definitely can't deliver the source.
How do professional software houses developing in python do it (or maybe the answer is that they don't) ?
You protect your source code legally, not technologically. Distributing py files really isn't a big deal. The only technological solution here is not to ship your program (which is really becoming more popular these days, as software is provided over the internet rather than fully installed locally more often.)
If you don't want the user to have to have Python installed but want to run Python programs, you'll have to bundle Python. Your resistance to doing so seems quite odd to me. Java programs have to either bundle or anticipate the JVM's presence. C programs have to either bundle or anticipate libc's presence (usually the latter), etc. There's nothing hacky about using what you need.
Professional Python desktop software bundles Python, either through something like py2exe/cx_Freeze/some in-house thing that does the same thing or through embedding Python (in which case Python comes along as a library rather than an executable). The former approach is usually a lot more powerful and robust.
Yes, it is possible to make installation packages. Look for py2exe, cx_freeze and others.
No, it is not possible to keep the source code completely safe. There are always ways to decompile.
Original source code can trivially be obtained from .pyc files if someone wants to do it. Code obfuscation would make it more difficult to do something with the code.
I am surprised no one mentioned this before now, but Cython seems like a viable solution to this problem. It will take your Python code and transpile it into CPython compatible C code. You also get a small speed boost (~25% last I checked) since it will be compiled to native machine code instead of just Python byte code. You still need to be sure the user has Python installed (either by making it a pre-requisite pushed off onto the user to deal with, or bundling it as part of the installer process). Also, you do need to have at least one small part of your application in pure Python: the hook into the main function.
So you would need something basic like this:
import cython_compiled_module
if __name__ == '__main__':
cython_compiled_module.main()
But this effectively leaks no implementation details. I think using Cython should meet the criteria in the question, but it also introduces the added complexity of compiling in C, which loses some of Python's easy cross-platform nature. Whether that is worth it or not is up to you.
As others stated, even the resulting compiled C code could be decompiled with a little effort, but it is likely much more close to the type of obfuscation you were initially hoping for.
Well, it depends what you want to do. If by "not releasing the source code" you mean "the customer should not be able to access the source code in any way", well, you're fighting a losing battle. Even programs written in C can be reverse engineered, after all. If you're afraid someone will steal from you, make them sign a contract and sue them if there's trouble.
But if you mean "the customer should not care about python files, and not be able to casually access them", you can use a solution like cx_Freeze to turn your Python application into an executable.
Build a web application in python. Then the world can use it via a browser with zero install.
I want to create a GUI application which should work on Windows and Mac. For this I've chosen Python.
The problem is on Mac OS X.
There are 2 tools to generate an ".app" for Mac: py2app and pyinstaller.
py2app is pretty good, but it adds the source code in the package. I
don't want to share the code with the final users.
Pyinstaller generates UNIX executable, so how to run it on Mac? I
created a bundles with this executable, but the resulted ".app" is
not working.
The questions are:
How to configure py2app to include the source code in the
executable, so the final users will not have access to my program?
How to convert UNIX executable to Mac ".app" ?
Is there a way to compile Python code with GCC ?
In Windows it's easy, I created an "exe" file from Python code and
it works. Is it possible to create a single file "app" for Mac ?
P.S. I use two computers (Windows and for Mac), Python 2.7, wxPython, py2exe, py2app and pyinstaller.
Also, I have checked out these sites:
http://svn.pythonmac.org/py2app/py2app/trunk/doc/index.html
http://www.pyinstaller.org/export/develop/project/doc/Manual.html?format=raw
http://www.pyinstaller.org/wiki/Features/MacOsCompatibility
http://www.stackoverflow.com/questions/2933/an-executable-python-app
How to configure py2app to include the source code in the executable,
so the final users will not have access to my program?
Unless you very seriously hack the python interpreter (and include the mangled version) there is no really good way to hide the source from a moderately skilled and determined user. I strongly believe this is true on Windows also. Basically, whether you include true source or bytecode, a pretty clean version of the source can be recovered. More importantly, in my opinion, unless you include the actual source code (as opposed to bytecode, you will introduce a possible dependency on the interpreter version).
How to convert UNIX executable to Mac ".app" ?
What do you mean by a UNIX executable? A Darwin (OS X) binary [which isn't actually UNIX]? That can be done using the kinds of tools you already mentioned, but it must be done carefully to avoid library dependencies.
If all you want it a simple wrapper to put a command-line binary into a window, it's pretty easy to accomplish and the free XCode suite has several examples that would serve (depending on what output
you wan to deliver, if any).
Is there a way to compile Python code with GCC ?
GCC does not compile Python. It's a different language (although there tools in the gcc family rthat support multiple language front-ends, but not Python). There are tools that attempt to translate Python into C, and then you can compile that into a true binary, but this only works for programs that avoid certain types of construct, and the process (and restrictions) need to apply your libraries as well.
One project to allow this is Cython. It works well for some types
of code, mostly numerical code, but it is not trivial to install and
exploit, very especially if you want to produce something that runs on multiple
different computers.
In Windows it's easy, I created an "exe" file from Python code and it
works. Is it possible to create a single file "app" for Mac ?
I would have to say I am skeptical -- very skeptical -- about this. Just like the OS X case, the exe almost certainly has the source code trivially accessible within it.
One fairly easy trick is to encrypt the source code and then decrypt it on the fly, but this
seems to me like more trouble than it's worth.
PyInstaller will automatically create bundles under Mac OSX for windowed executables. When running ypinstaller.py, make sure to pass the option "--windowed".
This feature is documented in the website of pyinstaller
If you're not completely committed to wxPython (and for anyone else looking for a cross platform Python GUI framework), I recommend you check out Kivy. It's cross platform, GPU accelerated, and it will do the app packaging for you. It's easy to jump into, has a well thought-out architecture, and gives you an incredible amount of flexibility in terms of the interface. It's the best way I've found to make a cross platform Python GUI app.
cxFreeze was the choice.
I use it pack my python program to a Mac OS X app. Which works like a charm.
Automator was already mentioned as a quick and simple solution for Pythons scripts that are contained in a single file, but since the Automator UI has so many options, and it is not obvious how to actually do it, I'll provide step-by-step instructions (verified to work on Yosemite):
In Automator select File > New and pick Application as document type.
Next, make sure Actions tab is selected on the left, and then in the search box type run. Among other options you'll see Run Shell Script — doubleclick it, and an editor window will appear in the right panel.
From the Shell dropdown menu select /usr/bin/python.
Paste your Python code into the edit window and then pick File > Save.
By default, the app will be saved under $HOME/Applications and will appear in Spotlight.
If you want to be able to set your own icon and have some fancy features, like task bar icons with a menu, log windows etc, then have a look at Platypus — an open-source app for creating MacOS native bundles.
2: You can't "convert" it, but you can move the executable to App.app/Contents/MacOS/something in a .app file, with CFBundleExecutable set to "something". This would not generally be recommended.
A motivated person could probably reconstruct usable source code from the Python bytecode in your app, so you might reconsider your opposition to py2app. If you don't trust your final users, why are you doing business with them?
Having used py2exe for windows users so they wouldn't have to deal with library versions, I've torn apart the compiled programs, they include the python bytecode files. While you can make it a violation of the license to look inside those, the fact is that if a computer can execute them, I can read them. It is possible to compile python programs with gcc, via a C preprocessor (try looking for 2c.py on google), I don't know if any of them support GCC. Again, you don't gain any security through using them, but you can get a significant speed improvement.
I haven't tried it with big Python projects, but for my own scripts, the easiest way I found was to use Automator
You can interactively create an app project with Run Shell Script action, then paste in your script in its editor, select your shell program (/usr/bin/python), finally save the project. And you have yourself a Mac native app.
Automator can also be driven by AppleScript. So you can pipeline this py-2-app conversion process to your build scripts.
I've never tested a GUI program with it so I don't know if you'll be happy with it. But I'd give it a try since you may wonder how well all the cited 3rd-party python modules/applications are maintained, and how long they are gonna last. Coming bundled with OS X, Automator will likely stay, unless Apple got REALLY tired of it.
cxFreeze is best solution available, first create your program or application using python and than make setup file for your application, and than build the app using build command python setup.py build, according to your requirement you need to make some changes.
The only way is py2app. You have no other way. Sorry.
The research you did seems very solid and you did not miss anything.
I've got some experience with Bash, which I don't mind, but now that I'm doing a lot of Windows development I'm needing to do basic stuff/write basic scripts using
the Windows command-line language. For some reason said language really irritates me, so I was considering learning Python and using that instead.
Is Python suitable for such things? Moving files around, creating scripts to do things like unzipping a backup and restoring a SQL database, etc.
Python is well suited for these tasks, and I would guess much easier to develop in and debug than Windows batch files.
The question is, I think, how easy and painless it is to ensure that all the computers that you have to run these scripts on, have Python installed.
Summary
Windows: no need to think, use Python.
Unix: quick or run-it-once scripts are for Bash, serious and/or long life time scripts are for Python.
The big talk
In a Windows environment, Python is definitely the best choice since cmd is crappy and PowerShell has not really settled yet. What's more Python can run on several platform so it's a better investment. Finally, Python has a huge set of library so you will almost never hit the "god-I-can't-do-that" wall. This is not true for cmd and PowerShell.
In a Linux environment, this is a bit different. A lot of one liners are shorter, faster, more efficient and often more readable in pure Bash. But if you know your quick and dirty script is going to stay around for a while or will need to be improved, go for Python since it's far easier to maintain and extend and you will be able to do most of the task you can do with GNU tools with the standard library. And if you can't, you can still call the command-line from a Python script.
And of course you can call Python from the shell using -c option:
python -c "for line in open('/etc/fstab') : print line"
Some more literature about Python used for system administration tasks:
The IBM lab point of view.
A nice example to compare bash and python to script report.
The basics.
The must-have book.
Sure, python is a pretty good choice for those tasks (I'm sure many will recommend PowerShell instead).
Here is a fine introduction from that point of view:
http://www.redhatmagazine.com/2008/02/07/python-for-bash-scripters-a-well-kept-secret/
EDIT: About gnud's concern: http://www.portablepython.com/
Are you aware of PowerShell?
Anything is a good replacement for the Batch file system in windows. Perl, Python, Powershell are all good choices.
#BKB definitely has a valid concern. Here's a couple links you'll want to check if you run into any issues that can't be solved with the standard library:
Pywin32 is a package for working with low-level win32 APIs (advanced file system modifications, COM interfaces, etc.)
Tim Golden's Python page: he maintains a WMI wrapper package that builds off of Pywin32, but be sure to also check out his "Win32 How Do I" page for details on how to accomplish typical Windows tasks in Python.
Python is certainly well suited to that. If you're going down that road, you might also want to investigate SCons which is a build system itself built with Python. The cool thing is the build scripts are actually full-blown Python scripts themselves, so you can do anything in the build script that you could otherwise do in Python. It makes make look pretty anemic in comparison.
Upon rereading your question, I should note that SCons is more suited to building software projects than to writing system maintenance scripts. But I wouldn't hesitate to recommend Python to you in any case.
As a follow up, after some experimentation the thing I've found Python most useful for is any situation involving text manipulation (yourStringHere.replace(), regexes for more complex stuff) or testing some basic concept really quickly, which it is excellent for.
For stuff like SQL DB restore scripts I find I still usually just resort to batch files, as it's usually either something short enough that it actually takes more Python code to make the appropriate system calls or I can reuse snippets of code from other people reducing the writing time to just enough to tweak existing code to fit my needs.
As an addendum I would highly recommend IPython as a great interactive shell complete with tab completion and easy docstring access.
I've done a decent amount of scripting in both Linux/Unix and Windows environments, in Python, Perl, batch files, Bash, etc. My advice is that if it's possible, install Cygwin and use Bash (it sounds from your description like installing a scripting language or env isn't a problem?). You'll be more comfortable with that since the transition is minimal.
If that's not an option, then here's my take. Batch files are very kludgy and limited, but make a lot of sense for simple tasks like 'copy some files' or 'restart this service'. Python will be cleaner, easier to maintain, and much more powerful. However, the downside is that either you end up calling external applications from Python with subprocess, popen or similar. Otherwise, you end up writing a bunch more code to do things that are comparatively simple in batch files, like copying a folder full of files. A lot of this depends on what your scripts are doing. Text/string processing is going to be much cleaner in Python, for example.
Lastly, it's probably not an attractive alternative, but you might also consider VBScript as an alternative. I don't enjoy working with it as a language personally, but if portability is any kind of concern then it wins out by virtue of being available out of the box in any copy of Windows. Because of this I've found myself writing scripts that were unwieldy as batch files in VBScript instead, since I can't usually depend on Python or Perl or Bash being available on Windows.
Python, along with Pywin32, would be fine for Windows automation. However, VBScript or JScript used with the Windows Scripting Host works just as well, and requires nothing additional to install.
I've been using a lot of Windows Script Files lately. More powerful than batch scripts, and since it uses Windows scripting, there's nothing to install.
As much as I love python, I don't think it a good choice to replace basic windows batch scripts.
I can't see see someone having to import modules like sys, os or getopt to do basic things you can do with shell like call a program, check environment variable or an argument.
Also, in my experience, goto is much easier to understand to most sysadmins than a function call.