Iteration vs Planning
It's all about tradeoffs
What if I told you that the Waterfall methodology wasn’t that bad? Before I lose all the Agile Scrum Masters, hear me out. This isn’t really about comparing Agile vs Waterfall, but rather about the overarching structures. They are surprisingly similar, but one of the biggest differences between the two methodologies is that one is optimized for iteration and unknowns, while the other for having all the information up front to be able to build everything at once. If you don’t know which is which, Agile is more focused on iterative principles and Waterfall is all about the big system. Both work for different reasons, and even though I admittedly haven’t practiced Waterfall in my day-to-day work, I really just want to talk about the tradeoffs of iteration vs planning.
Like I said, I haven’t practiced the Waterfall methodology formally in my work (though I learned all about it in engineering school), but the structures and decisions still present themselves when building systems. The thought process around iterating towards the end goal or putting together a plan and executing on that plan without deviation is present every day as an engineer. Each has its tradeoffs. Also, to be painstakingly clear, this is not a definitive comparison of Agile and Waterfall. I am just using them as reference points to explain my overall point. Let’s chat through the two approaches over an 8 week timeline:
The Iterative Approach
You plan to deliver something every week. Your first week delivery is just a small feature. The design is subpar, and you are still unsure if the architecture and data model are going to work. It probably has some bugs (maybe even a lot). It’s clunky at best, customers are frustrated because it is affecting their normal workflow, and some even churn. By the second and third weeks, it is at least usable since the QA team has caught some of the worst bugs. You start to get some quality customer feedback during the fourth week while you are connecting this new feature with some of the other features in the product. In the fifth and sixth weeks, you fix a few of the weird things the customers have been reporting. The architecture is mostly solidified and the design is starting to fit in with the rest of the product. By the seventh and eighth weeks, the product is in a really good state, your marketing team has finished their materials, and they are telling the world about it. But that doesn’t change anything for you, onto the next week and adding more to the feature set.
The Planning Approach
You start planning out the full eight weeks. You get all of the architecture and design fleshed out in weeks one and two. You and the team put your heads down and build for the next four weeks. QA catches some bugs in week 6 that you are able to fix pretty quickly. Nothing earth shattering. Week 8 means the marketing team gets to announce it to the world and you get customer feedback. They hate it. The design doesn’t quite fit with the rest of the product like you thought, and they are asking for a “small” tweak to the feature that would completely break the data model. Back to the drawing board for eight weeks of planning and building just to fix that one small thing (even though you’ll fix it in two weeks, it still took longer and some customers churn while waiting for the update).
Okay, so I didn’t exactly paint the rosiest picture for the planning approach, but I wanted to illustrate a point and then tear it down a bit. This is the story you often hear for why the iterative approach provided by Agile is better than the planning approach touted by Waterfall. However, there is a lot missing from these two situations. First, the planning ahead of time lead to only four weeks of building, instead of eight while iterating. Sure, you got customer feedback sooner, but the product was “broken” for longer while you continued to build on it. Ultimately, the product got done in sooner while iterating, but took the full time because you had to pivot in the middle of building.
As an engineer, it becomes a balancing act. You want to incorporate customer feedback as soon as possible, but you don’t want to have to constantly context switch. The focus time that you get from planning means you don’t waste time context switching. When you are doing one week sprints, it is really easy to just say “yeah we can throw it on the board for next week”, but you don’t always feel like you get to focus and bring one feature to complete fruition. It gets there slowly, but you learn along the way. Most engineers have looked at code they wrote six months ago and said “wow, that’s a pile of hot garbage”. That’s mostly because you learned more by building that feature and others along the way. This means that iteration can really help you learn along the way while still being flexible enough to rewrite things now that you know more. Sometimes the new knowledge is customer feedback about how that feature should be an autocomplete with multi-select options rather than an open-ended text input. Being able to maintain that context and rewrite it is huge for performance (see previous post about context switching).
This may mostly just be a problem at early stage startups where everyone wears lots of hats and kind of works on everything, but at a certain point, the iteration causes too much context switching and a lack of focus. One week, you are working on a feature involving some complicated CSS magic, and the next week it’s a data migration for a completely different part of the product. Lack of focus anyone? Sometimes, it keeps work fresh and new so you don’t get bored. Other times, it’s like having the attention span of a—ooo shiny thing! The chance to slow down and plan out how a feature is going to work, even just a little bit, will pay dividends. Sure, you want to be able to learn and pivot along the way if necessary, but being able to focus and deliver the feature before moving onto the next thing will mean a more complete feature gets delivered. It goes right back to balance and tradeoffs.
ShapeUp to the Rescue
I think that the ShapeUp process that the Basecamp team has come up with is a bit of the best of both worlds. It is iterative but requires enough planning to make each iteration fully-baked. I chose eight weeks for the example on purpose, as the cycle process they outline is eight weeks long. Six weeks of feature work with two weeks of “cooldown” where the team gets to fix bugs, clean up tech debt, plan for the next feature, and work on other improvements that they think will help going forward. The six weeks is entirely focused on feature work that is scoped to what can get done in the allotted six week timeframe, and the scope doesn’t change (with very few exceptions). The important part happens before that six week building time. Specifically, getting early customer feedback on designs and ensuring that we will have the correct architecture in place before building. If a feature doesn’t go as well as expected, the worst case scenario is that you lose six weeks of development (and some planning technically, but you were going to do that regardless). In the grand scheme of product development and business, that isn’t a lot.
Again, I’m not saying that the ShapeUp process is the end-all-be-all solution, but I have really enjoyed it (having done it at my previous job and now introducing it to the team at my current one). The team was expressing a lack of focus and direction, and it was feeling a bit like floundering some days. Our team is doing a modified version of it where we cut the timeframes in half, since we want to iterate faster (being an early stage startup). We also are admittedly not following every principle quite yet, as we are just now entering the cooldown week of the first cycle. However, just from that first three weeks, the team has already expressed a much better feeling of focus and direction (we were doing Agile 1 week sprints before and were kind of all over the place). We’ll see how it continues to develop and evolve as time goes on, but I have hopes that it will get us to where we want to go faster and more sustainably.