Ideally I’d like to build a package to deploy to Debian. Ideally the installation process would check the system has the required dependencies installed, as well as configure Cronjobs, set up users etc.
I’ve tried googling around and I understand a .deb is the format I can distribute in - but that is as far as I got since I’m getting confused now with the tooling I need to get up to speed with. The other option is to just git clone on the server and configure the environment manually… but that’s not preferable for obvious reasons.
How can I get started with building a Debian package and is that the right direction for deploying web applications? If anyone could point me in the right direction tools-wise and perhaps a tutorial that would be massively appreciated :) also if you advise to just take the simple route with git, happy to take that advice as well if you explain why. if it makes any difference I’m deploying one nodejs and one python web application
You can for sure package everything as a Linux application; for example using pyinstaller for your python webapp.
Besides that, it depends on your use case.
I will focus on the second part of your question,
How can I get started with building a Debian package and is that the right direction for deploying web applications?
as that seems to be what you are after when considering other alternatives to .dev already in your question.
I want to deploy 1-2 websites on my linux server
In this case, I'd say manually git clone and configure everything. Its totally fine when you know that there won't be much more running on the server and is pretty hassle free.
Why spend time packaging when noone will need the package ever again after you just installed it on your server?
I want to distribute my webapps to others on Debian
Here a .deb would make total sense. For example Plex media server and other applications are shipped like this.
If the official Debian wiki is too abstract, there are also other more hands on guides to get you started quickly. You could also get other .deb Packages and extract them to see what they are made up from. You mentioned one of your websites is using python, so I just suspect it might be flask or Django. If it's Django, there is an example repository you might want to check out.
I want to run a lot of stuff on my server / distribute to other devs and platforms / or scale soon
In this case I would make the webapps into docker containers. They are easy to build, share, and deploy. On top you can easily bundle all dependencies and scripts to make sure everything is setup right. Also they are easy to run and stop. So you have a simple "on/off" switch if your server is running low on resources while you want to run something else. I highly favour this solution, as it also allows you to easily control what is running on what ip when you deploy more and more applications to your server. But, as you pointed out, it runs with a bit of overhead and is not the best solution on weak hardware.
Also, if you know for sure what will be running on the server long term and don't need the flexibility I would probably skip Docker as well.
Easy deployment of a Flask API, how do we do that? What is the best way?
I would like to deploy my Flask API on a single server, in the beginning. I just got started with a new project and I don't want to spend too much time on Docker and scalability. I am even a bit scared to use Docker in production at the beginning anyway.
With PHP there are a ton of options, I just saw they even have "deployer" now, which makes things even easier.
What I am looking for:
with one command, deploying my project to the server (using git). But depending on "deploy dev" or "deploy prod" command, the server needs to know from which branch to pull. So I do need to merge branches before deploying.
create a new "release" folder on the server and symlink the www folder to the new release.
keep at least 5 release folders, remove the 5th on every deploy.
make it possible to rollback, so change symlink to a previous release folder.
I saw I can use Fabric, but it seems kinda complicated and perhaps overkill (like capistrano). I searched quite a lot on the web, but couldn't find a very clear answer/solution. Or a solution which most people agree on.
Any thoughts or people who would like share their experience?
I will post an answer, cause I see I've been given an answer 9 months ago already, without actually answering the thread.
Like Sayse already told: plenty of ways but GIT and CI are both good ways to implement Continuous Deployment on a VPS.
I've been trying CI, with much success!
I am studying TDD and developing an API in Django Rest Framework, and I had a need that I researched and could not find some tools to solve my problem, I am trying to find out how much my tests cover my application in %.
For know the number of possibilities and possible suggestions of what is missing cover, I found the coverage lib, but it generates a report with lots of data, which are not very useful for my case, I just want to know the coverage of my tests that I created. Does anyone know of any tool or plugin for pycharm that does this coverage of the tests?
I know that in visual studio there is Ncrunch that does this, but I do not know if there is something similar in pycharm.
I was struggling with the same question.
Especially I wanted to visualize the execution path of each test and run only affected tests.
I created a tool that sits in the background and runs only impacted tests:
(You will need PyCharm plugin and pycrunch-engine from pip)
https://pycrunch.com
https://github.com/gleb-sevruk/pycrunch-engine
This is how it looks like:
It is currently in beta, and may not support all usage scenarios, but I use it every day for development, without major issues.
I found a tool in the professional pycharm of which does what I need, is the functionality of running the tests with coverage, there is an option that runs the tests again to check if everything is ok:
And in this tool there is also another feature that shows the coverage of your tests against the existing code:
I hope I can help someone who has the same doubt! Thanks!
I have a django site that needs to be rebuilt every night. I would like to check out the code from the Git repo and then begin doing the stuff like setting up the virtual environment, downloading the packages, etc. This would have no manual intervention as this would be run from cron
I'm really confused as to what to use for this. Should I write a Python script or a Shell script? Are there any tools that assist in this?
Thanks.
So what I'm looking for is CI and from what I've seen I'll probably end up using Jenkins or Buildbot for it. I've found the docs to be rather cryptic for someone who's never attempted anything like this before.
Do all CI like Buildbot/Jenkins simply run tests and more test and send you reports or do they actually set up a working Django environment that you can access through your browser?
You'll need to create some sort of build script that does everything but the GIT checkout. I've never used any Python build tools, but perhaps something like: http://www.scons.org/.
Once you've created a script you can use Jenkins to schedule a nightly build and report success/failure: http://jenkins-ci.org/. Jenkins will know how to checkout your code and then you can have it run your script.
There are litterally 100's of different tools to do this. You can write python scripts to be run from cron, you can write shell scripts, you can use one of the 100's of different build tools.
Most python/django shops would likely recommend Fabric. This really is a matter of you running through and making sure you understand everything that needs to be done and how to script it. Do you need to run a test suite before you deploy to ensure it doesn't really break everything? Do you need to run South database migrations? You really need to think about what needs to be done and then you just write a fabric script to do those things.
None of this even touches the fact that overall what you're asking for is continuous integration which itself has a whole slew of tools to help manage that.
What you are asking for is Continuous Integration.
There are many CI tools out there, but in the end it boils down to your personal preferences (like always, hopefully) and which one just works for you.
The Django project itself uses buildbot.
If you would ask me, then I would recommend you continuous.io, which works ouf the box with Django applications.
You can manually set how many times you would like to build your Django project, which is great.
You can, of course, write a shell script which rebuilds your Django project via cron, but you should deserve better than that.
I'm wondering if there's such a thing as Django-like ease of web app development combined with good deployment, debugging and other tools?
Django is a very productive framework for building content-heavy sites; the best I've tried and a breath of fresh air compared to some of the Java monstrosities out there. However it's written in Python which means there's little real support in the way of deployment/packaging, debugging, profilers and other tools that make building and maintaining applications much easier.
Ruby has similar issues and although I do like Ruby much better than I like Python, I get the impression that Rails is roughly in the same boat at Django when it comes to managing/supporting the app.
Has anyone here tried both Django and Grails (or other web frameworks) for non-trivial projects? How did they compare?
You asked for someone who used both Grails and Django. I've done work on both for big projects. Here's my Thoughts:
IDE's:
Django works really well in Eclipse, Grails works really well in IntelliJ Idea.
Debugging:
Practically the same (assuming you use IntelliJ for Grails, and Eclipse for Python). Step debugging, inspecting variables, etc... never need a print statement for either. Sometimes django error messages can be useless but Grails error messages are usually pretty lengthy and hard to parse through.
Time to run a unit test:
django: 2 seconds.
Grails: 20 seconds (the tests themselves both run in a fraction of a second, it's the part about loading the framework to run them that takes the rest... as you can see, Grails is frustratingly slow to load).
Deployment:
Django: copy & paste one file into an apache config, and to redeploy, just change the code and reload apache.
Grails: create a .war file, deploy it on tomcat, rinse and repeat to redeploy.
Programming languages:
Groovy is TOTALLY awesome. I love it, more so than Python. But I certainly have no complaints.
Plugins:
Grails: lots of broken plugins (and can use every java lib ever).
Django: a few stable plugins, but enough to do most of what you need.
Database:
Django: schema migrations using South, and generally intuitive relations.
Grails: no schema migrations, and by default it deletes the database on startup... WTF
Usage:
Django: startups (especially in the Gov 2.0 space), independent web dev shops.
Grails: enterprise
Hope that helps!
However it's written in Python which
means there's little real support in
the way of deployment/packaging,
debugging, profilers and other tools
that make building and maintaining
applications much easier.
Python has:
a great interactive debugger, which makes very good use of Python REPL.
easy_install anv virtualenv for dependency management, packaging and deployment.
profiling features comparable to other languages
So IMHO you shouldn't worry about this things, use Python and Django and live happily :-)
Lucky for you, newest version of Django runs on Jython, so you don't need to leave your whole Java ecosystem behind.
Speaking of frameworks, I evaluated this year:
Pylons (Python)
webpy (Python)
Symfony (PHP)
CakePHP (PHP)
None of this frameworks comes close to the power of Django or Ruby on Rails. Based on my collegue opinion I could recommend you kohana framework. The downside is, it's written in PHP and, as far as I know, PHP doesn't have superb tools for debugging, profiling and packaging of apps.
Edit: Here is a very good article about packaging and deployment of Python apps (specifically Django apps). It's a hot topic in Django community now.
The statement that grails deletes the database on start-up is completely wrong. It's behavior on start-up is completely configurable and easy to configure. I generally use create-drop when running an app in dev mode. I use update when I run in test and production.
I also love the bootstrap processing that lets me pre-configure test users, data, etc by environment in Grails.
I'd love to see someone who has really built and deployed some commercial projects comment on the pros / cons. Be a really interesting read.
Grails.
Grails just looks like Rails (Ruby),but it uses groovy which is simpler than java. It uses java technology and you can use any java lib without any trouble.
I also choose Grails over simplicity and there are lots of java lib (such as jasper report, jawr etc) and I am glad that now they join with SpringSource which makes their base solid.
I have two friends who originally started writing an application using Ruby on Rails, but ran into a number of issues and limitations. After about 8 weeks of working on it, they decided to investigate other alternatives.
They settled on the Catalyst Framework, and Perl. That was about 4 months ago now, and they've repeatedly talked about how much better the application is going, and how much more flexibility they have.
With Perl, you have all of CPAN available to you, along with the large quantity of tools included. I'd suggest taking a look at it, at least.
The "good deployment" issue -- for Python -- doesn't have the Deep Significance that it has for Java.
Python deployment for Django is basically "move the files". You can run straight out of the subversion trunk directory if you want to.
You can, without breaking much of a sweat, using the Python distutils and build yourself a distribution kit that puts your Django apps into Python's site-packages. I'm not a big fan of it, but it's really easy to do.
Since my stuff runs in Linux, I have simple "install.py" scripts that move stuff out of the Subversion directories into /opt/this and /opt/that directories. I use an explicit path settings in my Apache configuration to name those directories where the applications live.
Patching can be done by editing the files in place. (A bad policy.) I prefer to edit in the SVN location and rerun my little install to be sure I actually have all the files under control.
cakephp.org
Cakephp is really good, really close to ruby on rails (1.2). It is in php, works very well on shared hosts and is easy to implement.
The only downside is that the documentation is somewhat lacking, but you quickly get it and quickly start doing cool stuff.
I totally recommend cakephp.
Personally I made some rather big projects with Django, but I can compare only with said "montrosities" (Spring, EJB) and really low-level stuff like Twisted.
Web frameworks using interpreted languages are mostly in its infancy and all of them (actively maintained, that is) are getting better with every day.
By "good deployment" are you comparing it with Java's EAR files, which allow you to deploy web applications by uploading a single file to a J2EE server? (And, to a lesser extent, WAR files; EAR files can have WAR files for dependent projects)
I don't think Django or Rails have gotten quite to that point yet, but I could be wrong... zuber pointed out an article with more details on the Python side.
Capistrano may help out on the Ruby side.
Unfortunately, I haven't really worked with either Python or Ruby that much, so I can't help out on profilers or debuggers.