Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
Currently I'm using a Windows batch file to build my software. It does things like running MSBuild, copying files, creating a ZIP file, running some tests, including the subversion revision number, etc.
But the problem is, batch files are evil. So I would like to change to something better. I was planning to recreate my build script in Python. Is that a smart choice? What about all those build systems, like Ant, SCons, Maven, Rake, etc. Would using any of those be a better choice?
Note: I'm not planning to replace my Visual Studio solution/project files. I only want to script everything else that's required to create a release of my software.
Edit: I have good reasons to move away from batch, that's not what my question is about. I would like to know (for example) what SCons gives me, over a normal Python script.
For a tool that is scripted with Python, I happen to think Paver is a more easily-managed and more flexible build automator than SCons. Unlike SCons, Paver is designed for the plethora of not-compiling-programs tasks that go along with managing and distributing a software project.
Batch files aren't evil - they've actually come quite a long way from the brain-dead days of command.com. The command language can be pretty expressive nowadays, it just requires a bit of effort on your part to learn it.
Unless there's an actual problem with your build script that you can't fix (and, if that's the case, that's the question you should be asking rather than some wishy-washy "What's the best replacement?" :-), my approach would be to stick with what you've got.
A vague feeling of evilness would not be reason for me to waste effort 'fixing' something that isn't broken. And it would be wasted effort unless there's a clear advantage to changing ("less evil" is not something I'd consider a clear advantage).
As you're mentioning Python and SCons, I'd say go for SCons. It is Python after all. And yes, any of the above would be a better choice than hand-rolled build scripts.
I've seen python scripts used for building releases elsewhere so it can't be bad. Actually, I've personally used perl scripts to automate release building. I guess any scripting language could easily automate that procedure. If it's gonna be easy to do (and probably better than batch scripts), why not try it?
I would suggest using NAnt for your build script instead of python.
My reasons for this are:
It has the tasks defined already, all you need to do is write the XML and point it to the right places. If you are working with people who do not know python, XML may be a little less scary than learning a new language.
NAnt is designed to work in the windows .Net environment, so it can already do MSBuild and NUnit tasks.
If you are already writing in C#, if you need to extend NAnt to do new tasks you are not adding another language to the mix of your project.
You can hook into Cruise Control .Net (for continuous builds). Which I think is the main reason why you would use NAnt.
Why should you use python? If your build script isn't broke don't fix it. If your having issues updating it to deal with new aditions to the project then you may want to look at rewriting it. I wouldn't use Python though tools like NANT or MSBuild do the job. I don't see the point in using a general purpis programming language to do something that tools have already been written to do unless you have a lot of obscure requirements existing tools can't deal with. Second what happens if you get hit by a bus or win the lotto? If you are determined to script everything I'd use powershell or some other Microsoft specific technology since your already wedded to Microsoft. If you leave will there be enough Python programmers to maintain the build scripts?
I would strongly suggest to take a look at waf. It's kind of what you want: "a Python-based framework for configuring, compiling and installing applications"
Personally I would use scripting as a last resort given that
With a bit of work you can get MSBuild to do all those things for you by extending it with additional components
There are third party equivalents to MSBuild like NANT that can do the same thing
There are entire tools like FinalBuilder that also do the same thing, and are easier to configure and extend
However, if I had to go the scripting route I would use Powershell for a couple of reasons:
Complete access to file system
You can easily access .NET objects
You can easily access COM objects
You can create custom makefiles for Microsoft nmake tool which you already have installed. Using a tool like that (SCons, Maven, etc. fall into the same category) gives you much more than regular scripts.
The main benefit is that dependencies between files are tracked and also the timestamps of changes. For example, you can make your .zip file depend on some other files, so .zip only gets repacked if some of those files have changed in the meantime. Just like with source code and its compiled form.
Python is very portable. SCons is field tested and reliable. Given what you know (from what you explained), why even ask the question?
If your maintaining something, its not just about getting it to build, its also about explaining to the user why it can NOT build, which saves you a ton of very frustrating questions while helping users to help themselves.
I can not think of a modern, production operating system that lacks Python, unless you get into the embedded / research arena.
So, I'm answering to say, you answered your own question :)
It depends on what technology your software uses. If you're building C++ programs, I'd probably say go for scons without question (unless you have weird requirements scons can't meet). On the other hand, consider the instructions for building C#: CSharpBuilder.
I would like to know (for example) what SCons gives me, over a normal Python script.
Think of scons as being more of a library than a program. It provides you with code that will prevent a lot of tedium that you will have to deal with without it. In my opinion, vanilla Python isn't the best option for any kind of shell scripting stuff (not that it can't do it).
But the problem is, batch files are evil.
Lastly, batch files are evil if they're used for a project they're not suited to handle. For the one or two file project, batch files do just fine.
It does things like running MSBuild, copying files, creating a ZIP file, running some tests, including the subversion revision number, etc.
MSBuild and PowerShell can easily do all of this with reasonably clean succinct code. You're then sticking to purely M$ products which managers tend to like. Otherwise I would suggest you could look into Rake if not only for its large community. It has a nice syntax and iron ruby support (irake).
To be honest all but the last task you have mentioned are easily done in MSBuild alone. I would suggest learning the tools you have before going elsewhere.
Check out http://msbuildtasks.tigris.org/ for some good add ons to MSBuild
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I am planning on creating a software that will be used to analyse biological data, some of us biologists would not have much programming skills yet we do alot of repetitive tasks and are required to write more scripts or programs that we always use using.
i have decided to narrow it down to only one very important task, its a searching script, i.e to query a database of genes-(data), there is already a program that does that called ncbi-blast, usually one needs to be familiar with command line and also requires ubuntu or windows, but its a hassle if you have to analyse the huge quantities of results generated.
so i have want to package some of my python scripts into a software so that others can find it convenient to analyse their work
Open vs. Closed Source (Licensing)
i intend to make some of the functionalities freely available to the user in the software, although at a later stage, i intend to incorpaorate pipelines that would require an affordable license
testing
so with this one function, it makes testing alot simpler-i dont know yet, and i would not mind starting with linux-ubuntu and windows as the platforms.
I appreciate your advice on choosing just one language, and i will go with python, how ever i would like tyhe software to atleast support other free programs created in other languages like R, as in plugins that a client installs when they need them.
I hope this shades more light to my already complex situation
Thank you
Sounds like you're at the beginning of an adventure and some new learning curves. I will applaud your willingness to create. Here are some things to consider as you get started on your journey.
fyi: "software" is typically a single word, no space.
Based on the context of "a soft ware" it seems like you mean "application".
I mention "software" and "application" not to give you a hard time about it, but because I think 1) it will help you in future searches when you look for ways to get something done (e.g. you'll get more useful hits on "python application installer" than "python soft ware installer", and 2) you can more clearly explain to people what you are attempting.
So. Your application(s) will need a user interface for your collection of scripts. Do you want a graphical user interface (GUI)? Or a command line interface (CLI)?
GUI: more complex than CLI:
For a GUI it will be more complex to get something that runs on both Windows & Linux.
So far as I know there isn't a trivial way to create a single GUI that runs on both Windows & Linux without adding another tool. Maybe python has easy GUI user-interface stuff in it and you could use that.
For perl I will point you here: "Perl GUI programming on Windows" Perl GUI programming on Windows
This lists some Perl approaches to consider, you will have to research them and see which ones are also able to run on Linux. Be careful about including additional libraries you'll need to package (or document how to install) with you application.
CLI: more simple than a GUI:
Since they are just scripts today, I would suggest starting with a CLI which probably means cleaning up your argument handling ( #ARGV in Perl, I can't comment on Python).
Read through your code and find literals that you change when you run a script for something new, those things will become your arguments. And if it starts to seem like you are creating too many arguments maybe you want to look into a configuration file (properties file of some kind).
It will be easier for you to get started with a CLI and you can always come back and add a GUI option in a future release. I find that designing a well thought out CLI makes it easier to focus on what is important for a GUI so it should help your eventual GUI be even better.
Packaging
Packaging your application is going to be a challenge.
Do your scripts need any libraries that aren't part of default with python & perl installs? If so you need to work out a way to supply those (e.g. include them with your distribution or include documentation that your users can follow to download and install the libs).
I can't comment on pyinstaller.
For Perl I will point you to this question on distributing a Perl Application : Distributing a Perl Application
User Skill
How much skill do you expect your users to have?
Will they be ok with installing a python and perl interpreters if necessary? Not every user has that technical skill.
Do you want to make sure your scripts verify the minimum versions of python & perl they need?
Documentation
See User Skill, above. You will need to make documentation available.
This will be driven by who is going to use your scripts. What is your target user like? Can you write a "5 line" summary that would make me want to download them and try them? That will be a helpful exercise to help you focus on how you want to present the scripts.
Open vs. Closed Source (Licensing)
This also ties into licensing. Are you aiming for a commercial product? Giving the world another gift of open source? Those are both fine things to do, but you'll want to choose a suitable license for your application. You also have some work to do if you want to avoid distributing your script's source code (I'm not suggesting you pursue one approach or the other, just that you make the decision before you put it on the internet at large).
Testing
Testing your application is going to be a challenge.
If you don't have automated tests for your scripts you will go crazy trying to verify it runs everywhere you want it to run.
Do you have unit tests & test suites for your scripts so you can verify they run correctly on different versions of Windows and various Linux distributions?
If your scripts are pretty simple maybe it doesn't matter if they're running on 32 bit vs. 64 bit operating systems.
Maybe it doesn't matter if they run on Windows 10, Windows 8.x, Windows 7.x, maybe various versions of Windows Server (2012, 2008, ...).
Maybe it doesn't matter if they run on RedHat, Suse, Ubuntu, Mint, ...
You probably want some sanity checks to verify that your install program worked correctly and the environment is suitable.
Without knowing what your scripts do it is kind of hard to say how much testing they benefit from.
Free Advice: choose just one scripting language and run with that
I will end with some completely free advice (worth what you're paying for it :-) )
Think really hard about just choosing one scripting language and writing everything in that.
You are going to have a LOT of additional complexity from supporting two scripting systems, if I were doing something like you describe I would Seriously Consider consolidating into a single language and just to Python or just do Perl. The time and energy it takes you to rewrite your Python scripts as Perl (or the other way around) may very well be easier compared to learning about creating installation guides and packaging your application using two different languages.
During developing one of my applications, I've come to a point where I'd like to give the users a more powerful filter. Therefore, I'd like to provide a simple scripting interface to the users. The scripting language would be Python.
For obvious reasons, I'd like to tighten the scope of the language to match my particular purposes (I don't want the users to touch the server's HDD files etc.). I also don't want to write a Python interpreter myself (which would be reinventing the wheel and the "new" wheel would be rectangular in the end). However, I haven't found any suitable library or module for this purpose.
Groovy's approach with its Compilation Customizers and Compiler Configuration would be exactly what I want, does something similar exist for Python?
What you're looking for is called a "sandbox" or "restricted execution." This wiki page discusses some of the details.
In a nutshell, there have been several efforts by Python geeks and gurus to build a sandbox on top of Python but they all failed eventually.
The main reason is that Python offers so many paths to do something that the sandbox would either have to forbid common use cases (rendering a lot of the library and Python code useless) or it would have to have holes in the sandbox which would make the concept useless.
So while it looks like a good and simple idea, so far, there is no solution. AFAIK, there are no hooks in Python to tweak the byte code compiler to achieve something like Groovy Sandbox.
Related:
How can I sandbox Python in pure Python?
Is there a "safe" subset of Python for use as an embedded scripting language?
Is it possible to deploy python applications such that you don't release the source code and you don't have to be sure the customer has python installed?
I'm thinking maybe there is some installation process that can run a python app from just the .pyc files and a shared library containing the interpreter or something like that?
Basically I'm keen to get the development benefits of a language like Python - high productivity etc. but can't quite see how you could deploy it professionally to a customer where you don't know how there machine is set up and you definitely can't deliver the source.
How do professional software houses developing in python do it (or maybe the answer is that they don't) ?
You protect your source code legally, not technologically. Distributing py files really isn't a big deal. The only technological solution here is not to ship your program (which is really becoming more popular these days, as software is provided over the internet rather than fully installed locally more often.)
If you don't want the user to have to have Python installed but want to run Python programs, you'll have to bundle Python. Your resistance to doing so seems quite odd to me. Java programs have to either bundle or anticipate the JVM's presence. C programs have to either bundle or anticipate libc's presence (usually the latter), etc. There's nothing hacky about using what you need.
Professional Python desktop software bundles Python, either through something like py2exe/cx_Freeze/some in-house thing that does the same thing or through embedding Python (in which case Python comes along as a library rather than an executable). The former approach is usually a lot more powerful and robust.
Yes, it is possible to make installation packages. Look for py2exe, cx_freeze and others.
No, it is not possible to keep the source code completely safe. There are always ways to decompile.
Original source code can trivially be obtained from .pyc files if someone wants to do it. Code obfuscation would make it more difficult to do something with the code.
I am surprised no one mentioned this before now, but Cython seems like a viable solution to this problem. It will take your Python code and transpile it into CPython compatible C code. You also get a small speed boost (~25% last I checked) since it will be compiled to native machine code instead of just Python byte code. You still need to be sure the user has Python installed (either by making it a pre-requisite pushed off onto the user to deal with, or bundling it as part of the installer process). Also, you do need to have at least one small part of your application in pure Python: the hook into the main function.
So you would need something basic like this:
import cython_compiled_module
if __name__ == '__main__':
cython_compiled_module.main()
But this effectively leaks no implementation details. I think using Cython should meet the criteria in the question, but it also introduces the added complexity of compiling in C, which loses some of Python's easy cross-platform nature. Whether that is worth it or not is up to you.
As others stated, even the resulting compiled C code could be decompiled with a little effort, but it is likely much more close to the type of obfuscation you were initially hoping for.
Well, it depends what you want to do. If by "not releasing the source code" you mean "the customer should not be able to access the source code in any way", well, you're fighting a losing battle. Even programs written in C can be reverse engineered, after all. If you're afraid someone will steal from you, make them sign a contract and sue them if there's trouble.
But if you mean "the customer should not care about python files, and not be able to casually access them", you can use a solution like cx_Freeze to turn your Python application into an executable.
Build a web application in python. Then the world can use it via a browser with zero install.
I've got some experience with Bash, which I don't mind, but now that I'm doing a lot of Windows development I'm needing to do basic stuff/write basic scripts using
the Windows command-line language. For some reason said language really irritates me, so I was considering learning Python and using that instead.
Is Python suitable for such things? Moving files around, creating scripts to do things like unzipping a backup and restoring a SQL database, etc.
Python is well suited for these tasks, and I would guess much easier to develop in and debug than Windows batch files.
The question is, I think, how easy and painless it is to ensure that all the computers that you have to run these scripts on, have Python installed.
Summary
Windows: no need to think, use Python.
Unix: quick or run-it-once scripts are for Bash, serious and/or long life time scripts are for Python.
The big talk
In a Windows environment, Python is definitely the best choice since cmd is crappy and PowerShell has not really settled yet. What's more Python can run on several platform so it's a better investment. Finally, Python has a huge set of library so you will almost never hit the "god-I-can't-do-that" wall. This is not true for cmd and PowerShell.
In a Linux environment, this is a bit different. A lot of one liners are shorter, faster, more efficient and often more readable in pure Bash. But if you know your quick and dirty script is going to stay around for a while or will need to be improved, go for Python since it's far easier to maintain and extend and you will be able to do most of the task you can do with GNU tools with the standard library. And if you can't, you can still call the command-line from a Python script.
And of course you can call Python from the shell using -c option:
python -c "for line in open('/etc/fstab') : print line"
Some more literature about Python used for system administration tasks:
The IBM lab point of view.
A nice example to compare bash and python to script report.
The basics.
The must-have book.
Sure, python is a pretty good choice for those tasks (I'm sure many will recommend PowerShell instead).
Here is a fine introduction from that point of view:
http://www.redhatmagazine.com/2008/02/07/python-for-bash-scripters-a-well-kept-secret/
EDIT: About gnud's concern: http://www.portablepython.com/
Are you aware of PowerShell?
Anything is a good replacement for the Batch file system in windows. Perl, Python, Powershell are all good choices.
#BKB definitely has a valid concern. Here's a couple links you'll want to check if you run into any issues that can't be solved with the standard library:
Pywin32 is a package for working with low-level win32 APIs (advanced file system modifications, COM interfaces, etc.)
Tim Golden's Python page: he maintains a WMI wrapper package that builds off of Pywin32, but be sure to also check out his "Win32 How Do I" page for details on how to accomplish typical Windows tasks in Python.
Python is certainly well suited to that. If you're going down that road, you might also want to investigate SCons which is a build system itself built with Python. The cool thing is the build scripts are actually full-blown Python scripts themselves, so you can do anything in the build script that you could otherwise do in Python. It makes make look pretty anemic in comparison.
Upon rereading your question, I should note that SCons is more suited to building software projects than to writing system maintenance scripts. But I wouldn't hesitate to recommend Python to you in any case.
As a follow up, after some experimentation the thing I've found Python most useful for is any situation involving text manipulation (yourStringHere.replace(), regexes for more complex stuff) or testing some basic concept really quickly, which it is excellent for.
For stuff like SQL DB restore scripts I find I still usually just resort to batch files, as it's usually either something short enough that it actually takes more Python code to make the appropriate system calls or I can reuse snippets of code from other people reducing the writing time to just enough to tweak existing code to fit my needs.
As an addendum I would highly recommend IPython as a great interactive shell complete with tab completion and easy docstring access.
I've done a decent amount of scripting in both Linux/Unix and Windows environments, in Python, Perl, batch files, Bash, etc. My advice is that if it's possible, install Cygwin and use Bash (it sounds from your description like installing a scripting language or env isn't a problem?). You'll be more comfortable with that since the transition is minimal.
If that's not an option, then here's my take. Batch files are very kludgy and limited, but make a lot of sense for simple tasks like 'copy some files' or 'restart this service'. Python will be cleaner, easier to maintain, and much more powerful. However, the downside is that either you end up calling external applications from Python with subprocess, popen or similar. Otherwise, you end up writing a bunch more code to do things that are comparatively simple in batch files, like copying a folder full of files. A lot of this depends on what your scripts are doing. Text/string processing is going to be much cleaner in Python, for example.
Lastly, it's probably not an attractive alternative, but you might also consider VBScript as an alternative. I don't enjoy working with it as a language personally, but if portability is any kind of concern then it wins out by virtue of being available out of the box in any copy of Windows. Because of this I've found myself writing scripts that were unwieldy as batch files in VBScript instead, since I can't usually depend on Python or Perl or Bash being available on Windows.
Python, along with Pywin32, would be fine for Windows automation. However, VBScript or JScript used with the Windows Scripting Host works just as well, and requires nothing additional to install.
I've been using a lot of Windows Script Files lately. More powerful than batch scripts, and since it uses Windows scripting, there's nothing to install.
As much as I love python, I don't think it a good choice to replace basic windows batch scripts.
I can't see see someone having to import modules like sys, os or getopt to do basic things you can do with shell like call a program, check environment variable or an argument.
Also, in my experience, goto is much easier to understand to most sysadmins than a function call.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I write tons of python scripts, and I find myself reusing lots code that I've written for other projects. My solution has been to make sure the code is separated into logical modules/packages (this one's a given). I then make them setuptools-aware and publish them on PyPI. This allows my other scripts to always have the most up-to-date code, I get a warm fuzzy feeling because I'm not repeating myself, and my development, in general, is made less complicated. I also feel good that there MAY be someone out there that finds my code handy for something they're working on, but it's mainly for selfish reasons :)
To all the pythonistas, how do you handle this? Do you use PyPI or setuptools (easy_install)? or something else?
I have been doing the same thing. Extract common functionality, pretty the code up with extra documentation and unit tests/ doctests, create an easy_install setup.py, and then release on PyPi. Recently, I created a single Google Code site where I manage the source and keep the wiki up to date.
What kind of modules are we talking about here? If you're planning on distributing your projects to other python developers, setuptools is great. But it's usually not a very good way to distribute apps to end users. Your best bet in the latter case is to tailor your packaging to the platforms you're distributing it for. Sure, it's a pain, but it makes life for end users far easier.
For example, in my Debian system, I usually don't use easy_install because it is a little bit more difficult to get eggs to work well with the package manager. In OS X and windows, you'd probably want to package everything up using py2app and py2exe respectively. This makes life for the end user better. After all, they shouldn't know or care what language your scripts are written in. They just need them to install.
I store it all offline in a logical directory structure, with commonly used modules grouped as utilities. This means it's easier to control which versions I publish, and manage. I also automate the build process to interpret the logical directory structure.