Short version:
Is it possible to create a standalone program that can be distributed to computers that don't have python installed, which writes a python script during runtime and executes it during or shortly after?
Long version:
A project I've been wanting to do for a while is to create a visual programming interface, that lets people use Machine Learning without needing to know python/keras/numpy syntax. Programs like lobe or rapidminer already do this, but they are all bound to their own interfaces and servers. What i would like to do is create a program that:
anyone can use without needing python/anaconda installed
allow the user to create visual scripts like in scratch or google-blocs
generates python code behind the scenes containing keras or tf code
is able to execute the script
is able to show the code to the user for:
educational purposes
allowing the user to use it as a base for a more complex program
Since the generation part would just need to create a text file almost any interface and language like python/java using Qt or C# / javascript using Unity, would qualify. I think that should definitely be doable, probably just very long, but that is not my largest concern.
The problem:
I have tried to search everywhere on the internet about things like standalone python programs which led me to for instance py2exe. Those kind of python-to-exe's work great, but they all require the script to be compiled on a machine containing a python environment. In my case i want to be able to generate a python script on the user's PC, and run it directly after from there.
The alternative
If this isn't possible i might just create the whole AI part myself, not using python nor libraries like keras or tensorflow, but in a unity game for instance. The downside to this would be that it exists already(like rapidminer), it would be less optimized/versatile/customizable and doesn't show what the "real" machine learning script would look.
If there are any other alternatives i would be happy to hear them
PS.
I have mostly Python & Keras, moderate Java and little Unity3D or web JS experience
I'm using Windows with anaconda
Distributing to linux/mac would be nice, but not required
maybe you should check the Orange Data Mining software, it's written in python and it has the same purpose as your project (https://orange.biolab.si/). On the other hand, some time ago I tried to compile my app that contains machine learning libraries but to date, I have not been able to achieve it. Instead, what I use is the WINPYTHON project, this is a portable Anaconda software that allows you to run projects on any PC without the need to install anything (https://winpython.github.io/).
Yes, it is possible!
I had the same requirement, so I wrote my visual programming language and IDE...
...that could generate an almost python-like script, and which is compiled natively inside the app, without the use of any external compilers or libraries.
My target architecture was mobile devices, but it also works on browsers via the unity plugin.
...and yes, that's correct, it runs natively on your phone or tablet in a simulated sandbox, with its own built-in IDE.
It's written in C#, and implemented in Unity3d
You can check it out at https://aiBoard.blog
..and see the videos at https://youtu.be/DIDgu9jrdLc
I am wondering how to intelligently manage the building and installation for some of our 3rd party C++ dependencies on Linux(Ubuntu). The way I currently have it set up is a git-lfs with all the necessary compressed 3rd party sources. I then use a shell script I wrote to install all the necessary system dependencies and then unzip and build the desired library. This shell script also takes care setting up all the paths so that our source code can easily link to the 3rd party libraries.
Example commands for our script are ./install opencv or ./install everything
However, over the months the script has gotten quite large and breaks sometimes when certain libraries are already installed or other minor issues. Thus I would like to replace it with something a bit more intelligent and useful. I have currently been looking into writing some kind of python script, but just changing the language from shell to python is not that big of an advantage. So I am looking if there are any specific python libraries that can help me with managing these libraries.
I have looked into things like chef and other automated builds stuff, but that is overkill for the small project I am working on.
I was wondering what other people used for this 3rd party management stuff, as sadly C++ does not have anything like pip.
I use jhbuild for this kind of thing (if I understand what you are doing correctly). It came out of the GNOME project (they use it to build the whole desktop from source), but it's easy to customize for any set of projects. The jhbuild packaged in recent Ubuntus works fine.
You write a little XML to describe each project: where to download the sources, what patches to apply, what configure flags to use, what projects it depends on, and so on; then when you enter jhbuild build mything it works out what to build and in what order and gets on with it. It's reasonably smart about changes, so if you edit a source file in one of the projects that makes up your stack, it'll only rebuild the affected parts.
For example, I have this for fftw3, the excellent fast Fourier transform library:
<autotools id="fftw3"
autogen-sh="configure"
autogenargs="--disable-static --enable-shared --disable-threads"
>
<branch
repo="fftw"
module="fftw-3.3.4.tar.gz"
/>
<dependencies>
<dep package="libiconv"/>
</dependencies>
</autotools>
With probably obvious meanings. That's building from a release tarball, but it can build from git as well. It's happy with cmake projects. jhbuild is written in Python, so it's simple to customize. Thanks to GNOME, many common libraries are included.
I actually use it to build Windows binaries. You can set it up to build everything with a cross-compiler, then put it inside Docker. It makes it a one-liner for anyone to be able to build a large and complex application on (almost) any platform. You can use it to do automated nightly builds as well, of course.
There are probably better things around, but it has worked well for me.
I'm sure this is answered somewhere else, but I can't for the life of me find it!
I have some python files and I want to automate the linting, testing and running of them. This is going to be part of a CI dependency chain, and I know how I would do this in e.g. Jenkins.
However the problem is that I would like developers to be able to run the above change locally before they push changes to the remote repository. I have come from a C++ team where we used team and I'm used to running something like:
scons test
That would run the linter, the tests, and maybe then output some files.
How can I replicate this functionality on a Python code base? I realise I could do this with a bash script or something, but it seems that there should be a standard build tool to do this. Thanks!
I have quite a bit experience in Python, but I haven't yet used any Build tools as such.
My technology :
Python
Django
Python devs talks about Continues Integration which needs the following :
VCS [ I use GIT ]
Build Tool [ I knowing nothing about ]
CI System like Jenkins [ I am learning ]
So, Why use a Build Tool what does it provides ?
Is it same as packaging tools ?
I heard of maven and ant but never heard any using them in the python world, why is that ?
What will you recommend for first timer with Build tools ?
How does that helps in deployment ?
The wiki for Continuous Integration pretty much explains it.
Build tools relate to what is necessary to build your entire project. If your project is 100% pure python then you probably don't have any build tools as its all interpreted scripts. You would keep committing to the project, and it would keep building (if building is neccessary), and testing.
Compiled languages (or if your python project needs to be packaged somehow to be tested) would require build tools to produce a program that can be run and be tested by the CI system.
If, for instance, your project were C++, you might be using make as your build tool. When you commit new code, make has to be run to build the project and be tested.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
Currently I'm using a Windows batch file to build my software. It does things like running MSBuild, copying files, creating a ZIP file, running some tests, including the subversion revision number, etc.
But the problem is, batch files are evil. So I would like to change to something better. I was planning to recreate my build script in Python. Is that a smart choice? What about all those build systems, like Ant, SCons, Maven, Rake, etc. Would using any of those be a better choice?
Note: I'm not planning to replace my Visual Studio solution/project files. I only want to script everything else that's required to create a release of my software.
Edit: I have good reasons to move away from batch, that's not what my question is about. I would like to know (for example) what SCons gives me, over a normal Python script.
For a tool that is scripted with Python, I happen to think Paver is a more easily-managed and more flexible build automator than SCons. Unlike SCons, Paver is designed for the plethora of not-compiling-programs tasks that go along with managing and distributing a software project.
Batch files aren't evil - they've actually come quite a long way from the brain-dead days of command.com. The command language can be pretty expressive nowadays, it just requires a bit of effort on your part to learn it.
Unless there's an actual problem with your build script that you can't fix (and, if that's the case, that's the question you should be asking rather than some wishy-washy "What's the best replacement?" :-), my approach would be to stick with what you've got.
A vague feeling of evilness would not be reason for me to waste effort 'fixing' something that isn't broken. And it would be wasted effort unless there's a clear advantage to changing ("less evil" is not something I'd consider a clear advantage).
As you're mentioning Python and SCons, I'd say go for SCons. It is Python after all. And yes, any of the above would be a better choice than hand-rolled build scripts.
I've seen python scripts used for building releases elsewhere so it can't be bad. Actually, I've personally used perl scripts to automate release building. I guess any scripting language could easily automate that procedure. If it's gonna be easy to do (and probably better than batch scripts), why not try it?
I would suggest using NAnt for your build script instead of python.
My reasons for this are:
It has the tasks defined already, all you need to do is write the XML and point it to the right places. If you are working with people who do not know python, XML may be a little less scary than learning a new language.
NAnt is designed to work in the windows .Net environment, so it can already do MSBuild and NUnit tasks.
If you are already writing in C#, if you need to extend NAnt to do new tasks you are not adding another language to the mix of your project.
You can hook into Cruise Control .Net (for continuous builds). Which I think is the main reason why you would use NAnt.
Why should you use python? If your build script isn't broke don't fix it. If your having issues updating it to deal with new aditions to the project then you may want to look at rewriting it. I wouldn't use Python though tools like NANT or MSBuild do the job. I don't see the point in using a general purpis programming language to do something that tools have already been written to do unless you have a lot of obscure requirements existing tools can't deal with. Second what happens if you get hit by a bus or win the lotto? If you are determined to script everything I'd use powershell or some other Microsoft specific technology since your already wedded to Microsoft. If you leave will there be enough Python programmers to maintain the build scripts?
I would strongly suggest to take a look at waf. It's kind of what you want: "a Python-based framework for configuring, compiling and installing applications"
Personally I would use scripting as a last resort given that
With a bit of work you can get MSBuild to do all those things for you by extending it with additional components
There are third party equivalents to MSBuild like NANT that can do the same thing
There are entire tools like FinalBuilder that also do the same thing, and are easier to configure and extend
However, if I had to go the scripting route I would use Powershell for a couple of reasons:
Complete access to file system
You can easily access .NET objects
You can easily access COM objects
You can create custom makefiles for Microsoft nmake tool which you already have installed. Using a tool like that (SCons, Maven, etc. fall into the same category) gives you much more than regular scripts.
The main benefit is that dependencies between files are tracked and also the timestamps of changes. For example, you can make your .zip file depend on some other files, so .zip only gets repacked if some of those files have changed in the meantime. Just like with source code and its compiled form.
Python is very portable. SCons is field tested and reliable. Given what you know (from what you explained), why even ask the question?
If your maintaining something, its not just about getting it to build, its also about explaining to the user why it can NOT build, which saves you a ton of very frustrating questions while helping users to help themselves.
I can not think of a modern, production operating system that lacks Python, unless you get into the embedded / research arena.
So, I'm answering to say, you answered your own question :)
It depends on what technology your software uses. If you're building C++ programs, I'd probably say go for scons without question (unless you have weird requirements scons can't meet). On the other hand, consider the instructions for building C#: CSharpBuilder.
I would like to know (for example) what SCons gives me, over a normal Python script.
Think of scons as being more of a library than a program. It provides you with code that will prevent a lot of tedium that you will have to deal with without it. In my opinion, vanilla Python isn't the best option for any kind of shell scripting stuff (not that it can't do it).
But the problem is, batch files are evil.
Lastly, batch files are evil if they're used for a project they're not suited to handle. For the one or two file project, batch files do just fine.
It does things like running MSBuild, copying files, creating a ZIP file, running some tests, including the subversion revision number, etc.
MSBuild and PowerShell can easily do all of this with reasonably clean succinct code. You're then sticking to purely M$ products which managers tend to like. Otherwise I would suggest you could look into Rake if not only for its large community. It has a nice syntax and iron ruby support (irake).
To be honest all but the last task you have mentioned are easily done in MSBuild alone. I would suggest learning the tools you have before going elsewhere.
Check out http://msbuildtasks.tigris.org/ for some good add ons to MSBuild