Deployment is a big part of the web application development process that is frequently overlooked, or taken for granted — but when it’s not handled properly, it can cause very expensive problems.
When you’re a single developer creating a small application, it’s not unusual to simply edit your source files, and upload them directly to the server. That’s it. No complex deployment necessary.
However, when you’re working on a website with millions of monthly visitors and hundreds of dollars per minute at stake, that strategy is out of the question.
In order to avoid a lot of lost revenue, a more robust strategy is called for.
I once worked for a company that hosted hundreds of websites on a proprietary content management system. When I got there, their idea of a deployment strategy was to keep daily backups of the whole system — operating system, database, and all. If anything went wrong, they would restore content from the previous nightly.
The problem came one day when something did go wrong. I didn’t want to just restore from backup. What would happen to all the changes users made that day? They would vanish into thin air. Hours of work, wasted. How would you feel if that happened to your data?
I wanted to fix the bugs without resorting to the backups, but my opinion didn’t matter — that’s the way they had always done things. I wasn’t about to stand in the way of a well established system. They restored from backup, I was flooded with user complaints, and the blame for the system failure was on my head. I gave my notice the next day. I had a feeling that a company with such disregard for customer happiness wasn’t going very far.
I just did a search to see if they’re still around. You can’t make this stuff up: Their website is still active, but currently down. Their traffic rankings are in the toilet. Looks like I made the right call.
Don’t get me wrong — I’m all for data redundancy and nightly backups. However, relying on them to fix your blunders is nothing short of corporate suicide.
Version control, AKA revision control, revision management, or version management, is the process of storing a history of every change to the source. You edit your source code like you normally would, but when you’re done with a new feature or bug fix, you commit the changes to your version control system.
Now you have an easy way to fix something if your changes introduce bugs. Just rewind to the previous version and try again. Nothing else gets impacted, and you don’t have to worry about screwing up the operating system, server configuration, or user data changes that occurred in the mean time.
This is all well and good — but what if you have a complex code base with a lot of contributors? Sorting out exactly when and how things got broken is not always as simple as rolling back to the previous version. There is a chance that two different people editing two completely different (but interdependent) files made changes that conflict with each other over the course of several commits.
That could take some time to track down, and if you’re editing against the production code, that could mean some very costly downtime.
The next step is to deploy a staging environment. Staging is a mirror of the production environment that you can use to do test deployments. Now you develop your new features against the staging environment, and keep a stable build in production.
The problem with this strategy comes when you need to make hotfixes on the live production code. You should make those bug fixes in a copy of the live environment so that you can test your fixes before you commit them back. If you only have one other environment, where do you make your hotfixes?
This is where branching comes in. Create a master branch, which is always a stable mirror of the production build. When you need to make a hotfix on a bug in production, you check out the master branch, make your fix, commit it, and push it into production. If the bug fix is screwed up, rewind the change and try again.
New features are added to a development branch. Development is frequently an unstable version of your application, which might break and get fixed several times in a busy development day. While developers are strongly encouraged to test their code before committing to development, sometimes severely broken code gets checked in accidentally. After all – developers are often under pressure and time constraints.
For this reason, the development branch should never get pushed directly into production without first being passed through QA. Whether you’re a one-man developer team, or a large organization, you should not forget to test.
When the next version is feature complete — meaning that all of the features described in the specification document (you have one of those, right?) are implemented, the code is under feature freeze, and all further changes are made for the purpose of fixing bugs and polishing the branch for production release.
Meanwhile, new development can still take place concurrently in a future version branch.
Serious bugs that get discovered and fixed in the course of new development can be selectively back-ported to the development and production branches. Likewise, bugs that get fixed in the master branch that also affect the development branches can also get merged into the other branches. Distributed version control systems like git make cherry picking changes a fairly straightforward process, once you wrap your head around it.
Just don’t forget to test!
Most projects should have a thorough test suite that encompasses both automated unit tests, and high-level tests that get passed via visual inspection.
Unit tests are sanity checks that make assertions like,
myMethodReturnValue == 1. If it’s truthy, the assertion passes and you get a green light. If it isn’t, it fails, and you know something’s broken in the code.
High level tests might make assertions like “Cart item subtotals line up with item descriptions on the printed receipt.” Essentially, they’re a checklist of tests that can’t be automated that the QA team needs to run through to make sure that the release is ready to move into production.
Many teams will have a dedicated testing environment for this purpose, but don’t let that stop you if you’re a one-man show. Being thorough about testing can save even a lone coder a lot of lost revenues if they don’t have to wait for users to find (and maybe report the bugs. In the testing environment, all the debug switches will be turned on, and the application will do lots of logging. The unit test suite will be used to monitor committed changes, and frequently there will be a dashboard full of green lights (or red lights, if something goes awry). Use your checklist and an automated testing suite to guide you through the application wizard-style for the final “all systems go for launch!”
I didn’t invent any of this stuff. The concept of release and hotfix branching has been around for years, and I have used them on a number of projects. However, there are some new tools, and a formalized branching strategy standard developed specifically for the distributed version control system, Git.
This post owes thanks to Vlad and Douglas at Zumba for introducing me to Git Flow. You can read Vlad’s thoughts on CakePHP at Nuts and Bolts of CakePHP.