at my work i write numerous small python scripts for DB management, most scripts use one or two common libraries which i sometimes update,
before distribution i freeze the scripts with cxfreeze and copy over some resources and upload to a server.
i would like to set up some up some system which would allow me to automatically rebuild/freeze all of the scripts
, copy over some files, archive and upload to server.
I'm not sure where even to look for such a system, because most of what if found is complicated server based systems like AHP for compiled languages, while i need something small for a single computer
obviously i can write something fast and dirty in python but it seems illogical that there isn't something ready made for these simple requirements.
please forgive my ignorance, I'm still learning.
Have you considered using make? If you're familiar with it from another language, it might be the easiest. There's also a list on the python wiki here.
Have a look at buildout. From their site:
Buildout is a Python-based build system for creating, assembling and deploying applications from multiple parts, some of which may be non-Python-based. It lets you create a buildout configuration and reproduce the same software later.
We are using it to do exactly what you describe. It is flexible and easy to extend through recipes.
Related
Background:
I am still new to Python so apologies if this is a naive question. I was originally looking to develop a tool for Windows with a GUI, however, reading around I get the impression that it may be simpler to have my compiled python code somehow 'wrapped' in simpleHTTPServer so users can interact via localhost (much like Jupyter notebook/labs).
Question:
Is there a standard/recommended way to have a compiled python program interface with the user via a localhost port in their web browser? I assume this is a fairly common approach and that a package may already even exist but I've not been able to find anything that gives guidance on how to tackle it.
Alternatively, is is feasible to somehow compile a Django development project to run on localhost?
Ideally, I want an executable file that users can run and their browser opens up at the localhost port required.
If you wish to quickly make a GUI based Python tool, you should consider using Django, a very straightforward framework for WebApps. It is very quick to learn and implement, and you shall very easily be able to integrate highly advanced functionalities too into your tool with ease.
When it comes to distribution of code, you can convert the entire Django App into a windows exe file for ease of distribution and use. There are many python packages that allow you to port python to windows executable files. I would recommend using PyInstaller. You may find this link useful: Recipe Executable From Django.
Alternatively, as someone suggested, you may use Docker, but that would be heavier in terms of space utilized, and will require the user to install the same.
Hope this helps!
I am looking for a flexible solution for uploading generic builds to an artifact repository (in my case it would be Artifactory but I would not mind if it would also support others, like Nexus)
Because I am not building java code adding maven to the process would only add some unneeded complexity to the game.
Still, the entire infrastructure already supports bash and python everywhere (including Windows) which makes me interested on finding something that involves those two.
I do know that I could code it myself, but now I am looking for a way to make it as easy and flexible as possible.
Gathering the metadata seems simple, only publishing it in the format required by the artefact repository seems to be the issue.
After discovering that the two existing Python packages related to Artifactory are kinda useless as both not being actively maintained, one being only usable as a query interface and the other two having serious bugs that prevent it use, I discovered something than seems to close that what I was looking: http://teamfruit.github.io/defend_against_fruit/
Still, it seems that was designed to deal only with python packages, not with generic builds.
Some points to consider:
Tools like Maven and Gradle are capable of building more than Java projects. Artifactory already integrates with them and this includes gathering the metadata and publishing it together with the build artifacts.
The Artifactory Jenkins plugin supports generic (freestyle) builds. You can use this integration to deploy whatever type of files you like.
You can create your own integration based on the Artifactory's open integration layer for CI build servers - build-info. This is an open source project and all the implementations are also open sourced.The relevant Artifactory REST APIs are documented here.
Disclaimer: I'm affiliated with Artifactory
I'm looking for a tool to keep track of "what's running where". We have a bunch of servers, and on each of those a bunch of projects. These projects may be running on a specific version (hg tag/commit nr) and have their requirements at specific versions as well.
Fabric looks like a great start to do the actual deployments by automating the ssh part. However, once a deployment is done there is no overview of what was done.
Before reinventing the wheel I'd like to check here on SO as well (I did my best w/ Google but could be looking for the wrong keywords). Is there any such tool already?
(In practice I'm deploying Django projects, but I'm not sure that's relevant for the question; anything that keeps track of pip/virtualenv installs or server state in general should be fine)
many thanks,
Klaas
==========
EDIT FOR TEMP. SOLUTION
==========
For now, we've chosen to simply store this information in a simple key-value store (in our case: the filesystem) that we take great care to back up (in our case: using a DCVS). We keep track of this store with the same deployment tool that we use to do the actual deploys (in our case: fabric)
Passwords are stored inside a TrueCrypt volume that's stored inside our key-value store.
==========
I will still gladly accept any answer when some kind of Open Source solution to this problem pops up somewhere. I might share (part of) our solution somewhere myself in the near future.
pip freeze gives you a listing of all installed packages. Bonus: if you redirect the output to a file, you can use it as part of your deployment process to install all those packages (pip can programmatically install all packages from the file).
I see you're already using virtualenv. Good. You can run pip freeze -E myvirtualenv > myproject.reqs to generate a dependency file that doubles as a status report of the Python environment.
Perhaps you want something like Opscode Chef.
In their own words:
Chef works by allowing you to write
recipes that describe how you want a
part of your server (such as Apache,
MySQL, or Hadoop) to be configured.
These recipes describe a series of
resources that should be in a
particular state - for example,
packages that should be installed,
services that should be running, or
files that should be written. We then
make sure that each resource is
properly configured, only taking
corrective action when it's
neccessary. The result is a safe,
flexible mechanism for making sure
your servers are always running
exactly how you want them to be.
EDIT: Note Chef is not a Python tool, it is a general purpose tool, written in Ruby (it seems). But it is capable of supporting various "cookbooks", including one for installing/maintaining Python apps.
Which framework has the most mature, flexible, intergrated, centralized and easy-to-use plugins/extension system.
My main requirements are:
a centralized system/repository where i could find a extension i need
no need to make changes in the source code, the plugin should be easily enabled and disabled
large plugin/extension database
something like http://wordpress.org/extend/plugins/
http://www.symfony-project.org/plugins/
I can't speak for Django, but I can tell you about Rails' open source community. GitHub is the central location for all Rails open source code.
Most ruby libraries/plugins these days are packaged as "gems", which are easy to install, update, and remove. RubyGems is the place to go for these pre-packaged gems, when you care less about the code and more about dropping the functionality into your application.
There is now a new tool called RVM that keeps the gems (and even rails version) isolated from one application to the next, on your system. That way if one app uses version 1.0 of a gem, and another uses version 2.0, they don't conflict with each other.
All in all, a pretty sweet setup.
There are lots of reusable django apps around. You can find many on the CheeseShop, but even more on GitHub and BitBucket.
There is also django-packages, which is a bit like the CheeseShop, but just for django packages.
VirtualEnv is like RVM (or rather, RVM is like VirtualEnv), which is a great way to isolate your python packages (I even use it in production). It has been around for ages, and works well with pip (the best python package installer).
Both of them are mature frameworks. I don't use ruby so I don't know about the rails plugin land. Given how popular it is (and my information from my lurking time on local Ruby lists), it's pretty good.
With Django, you have (like Matthew mentioned) django-packages and a few other places. I've been working on a largish Django project and it's pretty easy to just search for something like "django facebook" on google and get what you need. The Pinax project is an integrated collection of Django apps that lets you have most things out of the box. That's another thing you might want to consider. The packaging of the plugins are using the standard Python distutils libraries so installation is a single command (or if you're using pip/virtualenv, directly off the net).
VirtualEnv and related tools are not really Django specific. They're good practice if you're doing any python development though.
You should take a step back and evaluate both languages as well in my opinion. Python and Ruby are quite different in their approach to good code and it's likely that one will fit your brain better than the other.
I'm wondering if there's such a thing as Django-like ease of web app development combined with good deployment, debugging and other tools?
Django is a very productive framework for building content-heavy sites; the best I've tried and a breath of fresh air compared to some of the Java monstrosities out there. However it's written in Python which means there's little real support in the way of deployment/packaging, debugging, profilers and other tools that make building and maintaining applications much easier.
Ruby has similar issues and although I do like Ruby much better than I like Python, I get the impression that Rails is roughly in the same boat at Django when it comes to managing/supporting the app.
Has anyone here tried both Django and Grails (or other web frameworks) for non-trivial projects? How did they compare?
You asked for someone who used both Grails and Django. I've done work on both for big projects. Here's my Thoughts:
IDE's:
Django works really well in Eclipse, Grails works really well in IntelliJ Idea.
Debugging:
Practically the same (assuming you use IntelliJ for Grails, and Eclipse for Python). Step debugging, inspecting variables, etc... never need a print statement for either. Sometimes django error messages can be useless but Grails error messages are usually pretty lengthy and hard to parse through.
Time to run a unit test:
django: 2 seconds.
Grails: 20 seconds (the tests themselves both run in a fraction of a second, it's the part about loading the framework to run them that takes the rest... as you can see, Grails is frustratingly slow to load).
Deployment:
Django: copy & paste one file into an apache config, and to redeploy, just change the code and reload apache.
Grails: create a .war file, deploy it on tomcat, rinse and repeat to redeploy.
Programming languages:
Groovy is TOTALLY awesome. I love it, more so than Python. But I certainly have no complaints.
Plugins:
Grails: lots of broken plugins (and can use every java lib ever).
Django: a few stable plugins, but enough to do most of what you need.
Database:
Django: schema migrations using South, and generally intuitive relations.
Grails: no schema migrations, and by default it deletes the database on startup... WTF
Usage:
Django: startups (especially in the Gov 2.0 space), independent web dev shops.
Grails: enterprise
Hope that helps!
However it's written in Python which
means there's little real support in
the way of deployment/packaging,
debugging, profilers and other tools
that make building and maintaining
applications much easier.
Python has:
a great interactive debugger, which makes very good use of Python REPL.
easy_install anv virtualenv for dependency management, packaging and deployment.
profiling features comparable to other languages
So IMHO you shouldn't worry about this things, use Python and Django and live happily :-)
Lucky for you, newest version of Django runs on Jython, so you don't need to leave your whole Java ecosystem behind.
Speaking of frameworks, I evaluated this year:
Pylons (Python)
webpy (Python)
Symfony (PHP)
CakePHP (PHP)
None of this frameworks comes close to the power of Django or Ruby on Rails. Based on my collegue opinion I could recommend you kohana framework. The downside is, it's written in PHP and, as far as I know, PHP doesn't have superb tools for debugging, profiling and packaging of apps.
Edit: Here is a very good article about packaging and deployment of Python apps (specifically Django apps). It's a hot topic in Django community now.
The statement that grails deletes the database on start-up is completely wrong. It's behavior on start-up is completely configurable and easy to configure. I generally use create-drop when running an app in dev mode. I use update when I run in test and production.
I also love the bootstrap processing that lets me pre-configure test users, data, etc by environment in Grails.
I'd love to see someone who has really built and deployed some commercial projects comment on the pros / cons. Be a really interesting read.
Grails.
Grails just looks like Rails (Ruby),but it uses groovy which is simpler than java. It uses java technology and you can use any java lib without any trouble.
I also choose Grails over simplicity and there are lots of java lib (such as jasper report, jawr etc) and I am glad that now they join with SpringSource which makes their base solid.
I have two friends who originally started writing an application using Ruby on Rails, but ran into a number of issues and limitations. After about 8 weeks of working on it, they decided to investigate other alternatives.
They settled on the Catalyst Framework, and Perl. That was about 4 months ago now, and they've repeatedly talked about how much better the application is going, and how much more flexibility they have.
With Perl, you have all of CPAN available to you, along with the large quantity of tools included. I'd suggest taking a look at it, at least.
The "good deployment" issue -- for Python -- doesn't have the Deep Significance that it has for Java.
Python deployment for Django is basically "move the files". You can run straight out of the subversion trunk directory if you want to.
You can, without breaking much of a sweat, using the Python distutils and build yourself a distribution kit that puts your Django apps into Python's site-packages. I'm not a big fan of it, but it's really easy to do.
Since my stuff runs in Linux, I have simple "install.py" scripts that move stuff out of the Subversion directories into /opt/this and /opt/that directories. I use an explicit path settings in my Apache configuration to name those directories where the applications live.
Patching can be done by editing the files in place. (A bad policy.) I prefer to edit in the SVN location and rerun my little install to be sure I actually have all the files under control.
cakephp.org
Cakephp is really good, really close to ruby on rails (1.2). It is in php, works very well on shared hosts and is easy to implement.
The only downside is that the documentation is somewhat lacking, but you quickly get it and quickly start doing cool stuff.
I totally recommend cakephp.
Personally I made some rather big projects with Django, but I can compare only with said "montrosities" (Spring, EJB) and really low-level stuff like Twisted.
Web frameworks using interpreted languages are mostly in its infancy and all of them (actively maintained, that is) are getting better with every day.
By "good deployment" are you comparing it with Java's EAR files, which allow you to deploy web applications by uploading a single file to a J2EE server? (And, to a lesser extent, WAR files; EAR files can have WAR files for dependent projects)
I don't think Django or Rails have gotten quite to that point yet, but I could be wrong... zuber pointed out an article with more details on the Python side.
Capistrano may help out on the Ruby side.
Unfortunately, I haven't really worked with either Python or Ruby that much, so I can't help out on profilers or debuggers.