Technical Debt

Technical Debt  is one of my favorite work-related phrases because it succinctly and metaphorically sums up the obligation we incur when we cut corners during software development.  The phrase was coined by Ward Cunningham, the developer of the first wiki and, incidentally, a major contributor to Extreme Programming Agile methodology.

The concept of technical debt can be thought of as the accumulation of product risk introduced by any number of poor development practices.  Examples include poorly designed code, loose development practices, and, in Scrum terms, the delivery of “undone work”.   The payment terms on this debt can be onerous.  Usually the cost is time or productivity, sometimes both.  Dealing with this debt during the product release can increase the work required before the product can ship, often delaying the release.  Or, if you decide to postpone payment until after the release, the cost may be customer-discovered defects which can cause unexpected maintenance escalations and a higher ongoing product maintenance burden, ultimately reducing productivity and velocity for future development.

Regardless, like credit card interest or the National Debt, at some point in the future the bill comes due - someone is going to have to pay off the debt.

For new teams adopting Scrum, Technical Debt can occur without the team being fully aware that it is happening.    For example, teams with a loose definition of Done can introduce technical debt.  Stories may be accepted by the Product owner without having code reviews, minimal automated testing, or no code coverage or memory leak verification.  Any of which could leave unexposed defects in a seemingly “done” feature.  Another possible cause of debt can be caused by user stories that not properly decomposed such that Sprint backlogs do not yield a potentially shippable product increment at the end of a Sprint.  In this case, more work, either known or unknown, is created for “later” in the release.

As a Product Owner, you should keep track of your accumulating debt.  If the debt is significant, I recommend putting it in your Product Backlog, to be prioritized appropriately.  One way that I have found to identify and track any technical debt your team is accruing is to plan periodic “releases” to customers.  Product early releases, in the form of early access programs or Technical Previews, can not only provide valuable customer input that can feed into your backlog, but it also keep the Scrum team in the practice of delivering potentially shippable code.  When engineers realize their code is going to be used by actual customers, often “forgotten” undone work can suddenly appear.

Note that not all debt is “bad” - there are often valid reasons for accumulating technical debt.  Perhaps the underlying architecture has some scalability constraints, but getting the product to market, in other words, harvesting “business value” ($, customers), is more important to the business.  The key is for the Product Owner and Team to explicitly know that this debt was incurred, and accept that you may have to pay in the future be willing to accept that obligation.

The Product Owner should be aware at all times of work that is needed to release the software, including the technical debt that is accruing.  With a complete and publicly visible Product Backlog, there should be no surprises. Keep an eye on your debt - make sure you know how much debt interest is accumulating...

Cycling and Scrum

Every year I join a group of friends and ride a 100K charity bike ride.  This ride takes about 4 hours and has turned into a fun yearly tradition. This year, as I was battling my way up the hills of Chilmark (the ride covers most of Martha’s Vineyard), it occurred to me that there were a lot of similarities between the cycle ride I was a part of, and a Scrum team.

Some of the similarities I came up with were as follows:

• Once you know the basics, you can ride a bike a long distance, or participate on a Scrum team.  However, having solid training for each makes doing them much easier and more efficient.

• Developing a regular cadence is valued in both Scrum and cycling.  Going to hard and fast early on can quickly burn you out, making it hard to finish the job.

• Buying a Scrum tool doesn’t make you a better Agile organization, just like buying a new carbon-fiber bike doesn’t make you a better, faster rider.

• I’m a long distance runner, and not a great cycler.  However, riding with my friends, many of whom are serious cyclers, makes me a better rider.  As the ride progressed, I rode smoother, tighter and faster.  Working on a Scrum team has a similar effect for team members.  Knowledge readily spreads to the teammates through frequent communication via daily stand-ups and backlog grooming activities, making them stronger, better engineers.

• Due to the fact that the ride occurs mid-Spring, each member of the team arrives with a different level of fitness, depending on how much training we put in on indoor trainers during the long New England winter.   As such those of us less capable rely on help from the stronger riders. With cycling, teamwork is core to the effort.  Riders work together, take turns leading the group, breaking the wind for the rest of the team,  pulling the group, allowing others (slower, tired riders!) to draft behind them.    Similarly, Scrum is focused on teamwork and less about solo achievements.  Sprints (and thus product releases) can only efficiently succeed if the team works together and arrives at the end together.

• There was a strong team commitment to finish the full 100K. During the ride there were several opportunities during the ride to cut it short - to quit early.  Additionally, the event offers 2 shorter rides, of 10 and 30 miles. But we, as a team, committed to completing the 100K ride regardless of the weather or any other adversity that came our way.    Scrum teams develop a similar strong focus and commitment to delivering potentially shippable product increments and overcoming impediments.

My friends and I had great weather and a fun ride.  We reached the 100K mark, the “finish line” together with a team velocity of 17 [mph] story points.  I’m looking forward to next year’s ride.

Transparency: Keep it visible!

“Openness is required for Scrum to work.”

-Ken Schwaber, Agile Software Development with Scrum

Openness, transparency, visibility - no matter what synonym you use, it is critical to Scrum.  My favorite term is transparency: Scrum’s openness makes everything that is going on within the project transparent to the world.   With Scrum there are no hidden priorities, no hidden work, everyone should (and generally does) know what is currently being worked on and what will be delivered by the team.

On the surface, it may seem that transparency is more important for management and the business owners rather than the team.  Management primarily cares about when we can actually ship and start selling a product (and of course, that the product does what it needs to do - that it meets market expectations).  Numerous business-related activities such as marketing, support, training, etc,  are coordinated around a product release.  

But transparency is also extremely important for the team and individuals.  Nowhere is this more apparent than in a Waterfall development environment. There have been several instances in my career where a peer has asked me for workplace advice.  The conversation zeros in on a boss who is giving them grief about their productivity.  After listening to them vent, a common thread usually appears:  these individuals often are doing a lot of work, usually juggling many different activities, but their manager is only tangentially aware of the scope of responsibilities this person is shouldering.  The manager is not aware of their employee’s “backlog”.    

My advice in these situations is simple:  make a list of work you are doing, order it according to priority, and discuss the list with your manager.  In particular, I suggest that they work with their manager to help set the right priority of the various work items.  This conversation often opens the eyes of both the manager and the employee, quickly addressing the tension and frustration experienced by both parties.

With Scrum, this communication activity is commonplace.  In fact it is core to the Scrum framework:  the prioritized product backlog and the Sprint backlog are visible to everyone. As a result, the work being done by the team is obvious and very transparent.  Further, project progress is demonstrated to the business owners regularly during the Sprint Review, via a (potentially shippable!) working code demonstration.

The Inevitable New Unplanned “Top Priority” Work

In a Waterfall environment, the impact of new, mid-project “high priority” work is often not immediately felt.  Because the product release can be many months away, incoming hot issues can be dealt with in a timely manner and everyone feels good, at least in the short-term. However, this new work causes the team to accrue a form of “schedule debt”.  The painful balance comes due at some point in the future, usually in the form of a schedule slip.  And often, sadly, that schedule slip is a surprise to management.

In a Scrum environment, transparency is the proverbial Big Stick when unplanned “critical” work is added to the project.  When management (let’s face it, it usually is management) adds new new work to the team’s plate, the impact to the team is immediately obvious:  1) the new work will either displace existing Sprint priorities causing the Sprint to be abnormally terminated, something everyone wants to avoid, or 2) the work will be added to the Product Backlog and prioritized according to business value.  There’s no magic, no silver bullet, and everyone Scrum-enabled, will should understand that!

Surprise!

In software development, we don’t like surprises - we like predicable.  Usually a surprise means new unplanned work for the team or a schedule slip.  Someone is is going to be unhappy!  

Scrum’s transparency helps organizations eliminate surprises or at a minimum deal with them effectively.   Surprises in the form of schedule slips are eliminated because what the team is working on is very visible and demonstrated at the end of every Sprint.  Work, delivered via Sprints, is time-boxed and ends up being very predictable.  Surprises in the form of new, unplanned work is dealt with effectively by adding it to the prioritized Product Backlog.  With a prioritizing Backlog, the team is always working on top business value items.  

Transparency, via a published, readily viewable backlogs allows everyone to understand what is currently being worked on as well as the impact of new unplanned work.    
 

“Scrum keeps everything about a project visible to everyone.”

-Ken Schwaber, Agile Software Development with Scrum

Hyper-Communication: Scrum’s Secret Sauce

Com com communicate

Communicate communicate

Communicate communicate

Via satellite and solid state

Never never hesitate

Communicate communicate

Communicate communicate

Never never hesitate

-Pete Townshend, Communication, from the album All The Best Cowboys Have Chinese Eyes

Effective and continuous communication is the cornerstone of any high-performance team. It may be obvious, but we need good communication to do our job well.  But good communication doesn’t come for free: you have to continuously work at it.   Further, when developing software, your choice of development process can hamper or promote high levels of communication.

During our Waterfall to Agile Pilot project effort, I’ve observed that Scrum has “forced” us to dramatically communicate more, and at more more regular intervals, than we have done in the past.  Scrum forced our teams to develop a communication habit - one where where I would estimate that we communicate twice as much, if not more, than we did with our current Waterfall process.  

Why is this so?  It’s because Scrum prescribes a set of frequently repeating ceremonies (meetings) that require effective team-wide communication.  Teams that adhere to Scrum develop a work cadence, a habit of working, that fosters regular, focused communication on the work at hand.  Scrum promotes this team habit through a the following ceremonies:

• Release Planning Meeting - held at the start of a project.
• Sprint Planning Meeting - held prior to the start of a Sprint.
• Daily Scrum (stand-up) - held every day of the Sprint.
• Sprint Review - held at the end of every Sprint to demonstrate working software to the stakeholders.
• Sprint Retrospective - held after every Sprint to inspect and adapt.

Several additional conversations generally occur regularly as part of developing software guided by Scrum.  One, the subject of an earlier blog, surrounds the team agreement on what “Done” means.  Another equally important conversation occurs with the Product Owner about the Product Backlog. Called Backlog Grooming, the team works closely with the Product Owner to focus on, understand, refine, and size top priority user stories over the course of every Sprint, a “just in time” process that helps feed the next Sprint Planning Meeting.  The Scrum framework doesn’t require these conversations, but my observation is that they are regular ceremonies for most Scrum teams, and they were with our Scrum Pilot teams.

Contrasted Communication with Waterfall

Compared to Scrum, Waterfall-based software development seems lazy with regards to communication.  A majority of team communication happen up front, when the least is known about the project.  Up-front requirements, analysis, and design discussions help formulate a project plan and project schedule, then the team goes off and develops according to that plan.  Regular project status meetings, usually weekly, are held to track the project and resolve dependencies. Sometimes, in fact sometimes quite frequently in my experience, status meetings are cancelled because everyone is “heads down” working, an ironic often-used phrase - If someone is head’s down, an image comes to mind of a someone cloistered, cranking out code, only coming up for air, for a bathroom break.

During past Waterfall development efforts, I have observed situations where an engineer, working “heads down” delivering according to his own schedule, is suddenly informed that a feature he previously completed needs to be re-implemented in a substantially different way, thereby putting him unexpectedly behind schedule.   As a Project Leader I recall feeling surprised that this happened, after so much time being spent up-front identifying dependencies and coordinating task schedules.  Something slipped through the cracks, causing the delivered functionality to based on stale, incorrect information.  We simply didn’t find out that we went down the wrong path until much later in the release, a point in the project where it was more costly to fix the issue. These were situations that could have been easily avoided with just a little bit more communication with the appropriate parties, a situation much less common with Scrum.

Continuous Communication with Scrum

During two week Sprints, the Scrum Team is meeting and communicating at least once daily, with a 10% of the Sprint spent on planning, demonstrating product, and adapting and improving their process.  Further, there are daily Scrums, discussions with subject-matter expert, and design discussions and activities between team members.  In other words, plenty of opportunities where team members are  in essence, forced to communicate.

One benefit I observed during our Pilot Scrum Team effort was the impact that the Daily Scrum had on team communication.  The daily Scrum meeting frequently spawned ad hoc discussions immediately following the 15 minute stand-up.  Team members responding to the daily Scrum “3 questions” fostered additional, unplanned, clarifying discussions, just by team members actively listening to teammates.  Project questions and issues were often resolved immediately, or “just in time” rather than laying dormant until some point in the future when they would surprisingly be discovered.

To those used to Waterfall, the communication required by Scrum may seem like hyper-communication, perhaps overkill.  But ultimately it works well.  Teams quickly develop a  communication rhythm, involving Stakeholders (via Sprint Reviews), Product Owners (via the Product Backlog Grooming) and fellow team members (via the Daily Scrum).  With the Team focused on the same goal, delivering on Sprint commitments, continuous communication is the secret sauce that allows the Team to succeed.  It’s the catalyst to turning a team into a high performance team.  

Pete Townshend said it best:

“Communicate communicate, Never never hesitate”

What does “Done” mean?

It starts with a team conversation...

One of my favorite team conversations that generated by the Scrum framework is one that occurs at the beginning of a Scrum project.  It’s the team discussion and subsequent agreement of what the word “Done” means.

A Scrum Team is supposed to produce potentially shippable code at the end of every Sprint.  This means that any stories (requirements) that the team commits to must be full implemented, documented, tested, etc within the time-boxed Sprint.  The stories must be DONE. Done can and often is defined differently by each team.  Minimally, Done means that the story committed to is coded and fully tested.  A story really can’t be potentially shippable if it isn’t coded and tested.  

Done is demonstrated during the Sprint Review, where working code is demonstrated the business owners and everyone else in attendance.   A story isn’t complete unless it satisfies the definition of “done”.  In fact, Scrum will not credit the team with story points unless a story is, in fact, “done” (and accepted by the Product Owner).  

Each Pilot team that I observed went through similar discussions about “done”.  Because Scrum was new to the teams, the initial definition of “Done” was a bit loose:  there was general agreement that a story was “Done” if it was coded and tested.    Over the course of the first Sprint Planning and the start of the Sprint 1, it often became apparent how loose that definition of Done actually was.  

The Product Owner and Team conversation for story acceptance criteria was one of the main catalysts for Done refinement.   As each member of the cross-functional team discussed the stories they were committing to, it was quickly realized that the initial definition of done left many questions unanswered.  Product areas such as documentation (what kind? what level?), training (again, what kind?), automated tests (how much should be automated? what platforms should be tested? what web browser? what about performance testing?) and bugs (should all be fixed?  just blockers? more than just blockers?) all had questions that needed answers.  

Over the period of a month or so, the teams each refined the meaning of Done.  The process of discussing, debating, and ultimately answering these questions helped the group grow into a high performance team.  This was very important to the success of the effort, as with Scrum the commitment to delivering a story is a team commitment, rather than an individual one.

A Great Habit

I think most would agree that it’s great to get in the habit of fully completing work, versus completing 80% of the work and leaving the remaining 20% for some time “later”, during the release end-game, or perhaps never (thereby wasting effort).  With Scrum, producing done work is essential to the process. In the pilots, I’ve noticed a tendency for teams to commit to too many story points during a Sprint, ultimately leaving one or two stories partially finished, though the team put in often heroic effort to meet the original commitment.  The unfinished stories were fortunately completed in the next sprint, but the fact that they were not complete often defeated the “potential” from the potentially shippable product.   One way to deal with this is for the team to sequentially complete stories within the sprint, never starting a new story until the current story is Done.  This isn’t always practical, or efficient, but the team should be aware of the story bleed and velocity drag that partially implemented stories create.  Alternately, it is perhaps better to commit to fewer story points per sprint and fill in any capacity with product maintenance or backlog grooming or architectural spikes.    These are team decisions that can be considered during Sprint Planning or during the Sprint Retrospective.

Agility

For me, the concept of Done is the key foundational principal that  makes Scrum agile.  At the end of a Sprint all stories are fully completed, thus potentially shippable.  This fact, coupled with a short development iteration, offers the business frequent predictable time windows in which to re-prioritize work or introducing completely new requirements.  Scrum allows an organization frequent opportunities to, in contemporary terms,  pivot.  Easily and readily shifting priorities with minimal impact to productivity is one of the primary mechanisms through which an engineering organization gains agility.