4 reasons why agile teams should stage deployments each iteration

There are a few shops like Etsy that use continuous deployment/delivery to have code go straight into production, but otherwise the rest of us have some sort of environment between the developer’s machine and the live production environment.  Some call this Staging or QA, or there may actually be MANY of these environments that a build needs to be promoted through before it is pushed to production.  We’ve all seen this, we’ve all had to go through “deployment day”, and we’ve all hated it.

Whether you run agile/lean or waterfall, this idea of acceptance environments is the same.  However, for some reason, these environments only seem to get updated at “critical milestones” on projects.  In a waterfall pattern, this makes sense.  You finish your work, you pass it to the next environment and let it continue its path.  However, why do so many agile teams not push through these environments every iteration? Why are we pushing to acceptance environments so late in our projects?

Perhaps it is because we have embedded QA on our teams during the iteration and believe that everything is ready for production by the end of the iteration so we don’t need to worry about testing in all the various environments. We’ve already done our testing, so why do it again? Doesn’t that seem more like a waterfall testing approach? Absolutely not, and here’s why:

Reason #1: Fear

No matter the courage of your deployment team, in some organizations there will always be fear of the deployment to production. What if it goes horribly wrong? What if there are still bugs? To quote Frank Herbert:

Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear.

For the Fear reason alone, we need to have an environment where we can practice pushing from one environment to another.  No amount of assurance from the QA team that all is good in the code will stop this organizational fear if you haven’t proven to them that you’ve done it before.

It’s not just about organizational fear either.  The team also needs to be confident in the deployment process as well!

Reason #2: Practice makes perfect

How do we face this fear? With practice. If you’ve been using automated deployments to deploy your code to a centralized QA environment where your testers have been validating stories during an iteration, you’ve already done hundreds of deployments. At this point, the team will have practiced taking the source code and pushing it to that centralized location.  However, this process builds in a false sense of security.  Production is not the same as an internal test environment.  It has a different IP, different database servers, different file paths.  How can we be sure that the we don’t have some sort of dependency built in for our test environment?  We need to practice pushing from one environment to another, not the same environment all the time, in order to make sure that our code works in multiple environments.  By doing this every iteration, by the time we get to launch date, we can accurately say “we’ve done this before, and we know what to watch for”.

I also recommend creating a new branch at this point as well.  If you’ve been developing in a single branch for an entire iteration there are certain paths that are sometimes depended upon in the source control.  The simple matter of moving the code to a new release branch for going to the next environment finds these problems.  Doing this every iteration gets the team used to branching often, and making their code work in any branch.

Reason #3: Scaling

In larger organizations, the team working on a given iteration may only be a single piece of a much larger overall delivery.  They may be a component team or a feature team, and their work needs to be deployed along with many other features or components being developed by other teams.  In this case, the internal QA process is not good enough and we need to be able to practice integrating one team’s work with the work packages from other teams.  Doing this every iteration ensures we get feedback on the integrated components and features earlier, and on our deployment process for integrating with the other features.

Reason #4: Smaller Batch Sizes

Lean principles have taught us that the smaller the batch size of our work package, the faster it flows through our system.  This applies to our software development, but also our delivery.   To quote Damon Edwards from his blog post Small Batches Improve Flow:

Break down large releases into small units of deployment that employ standardized packaging and configuration management mechanisms. These units of deployment should be aligned towards the things that are changed (i.e. application services) rather than large project releases that change many things. In addition to reducing deployment and configuration woes, this also has the effect of standardizing batch sizing across lifecycle by determining the appropriate unit of change for your infrastructure.

If the team waits to do their deployments to other environments after several iterations have gone by, or at the end of the project, the batch size at that point in time greatly increases the complexity of the deployment.  The more often we deploy, the less risk we have in our deployment.

This is also why we should be updating production more often, but that’s a whole other story 😉

Where do I start?

The obvious first place to start is by making sure you have at least one environment that is not production that you can deploy to regularly if for no other reason than to practice deploying an iteration release to an environment.  Here are some other tips:

1. Create a release branch.  As mentioned before, branching your code when you are about to ‘close off’ the iteration and deploy to another environment allows you to also identify dependencies in your source control.
2. Build into the iteration. Make part of your iteration tasks the deployment.  For example, you likely are doing a demo at the end of the iteration to interested stakeholders and possibly even some follow-up client acceptance testing or usability testing.  Plan for this to happen in an isolated environment and it will force you to deploy every iteration and make it part of the iteration Definition of Done.
3. Change it up. Do something in the deployment target environment that makes it different from the test environment that is used during the iteration.  Change the operating system, firewall settings, IIS version, .NET version, or even just install your application in a different folder on the drive.  This lets you find hidden dependencies that might not otherwise be apparent.
4. Plan it out.  If you have multiple teams deploying to the same environment, make sure you all have your deployment plans in place and identify the dependencies between the teams.  Everybody has to get their pieces done and pushed, and this requires a good plan and some solid communication.
5. Automation.  Most teams these days use some form of automation to pull together the build and deploy it into the test environment for use during the iteration.  Leveraging this knowledge towards environment deployments allows the team to have even easier deployments and eventually automated production deployments!