Back on the Front Line of Agility...

Sorry that it has been a while since I posted here - my professional life has changed considerably since I last wrote!

Last Fall I join an exciting startup, VoltDB, leaving behind a "Waterfall to Agile" transformation environment, diving head first into the front lines of a pure Agile start-up environment.

It's been a pretty exciting 6 months working with a great team of engineers.  We're delivering a high velocity, scale-out NewSQL relational database. This product fits at the fire hose intake edge of Big Data and is capable of ingesting hundreds of thousands of transactions per second on commodity hardware.   Low latency, high velocity.  Lots of challenges, both technical and process-wise.

Here's an overview of the VoltDB Development environment:

• Open Source Database.  Because it is open source, anyone can see the our code (see our Git repository at  https://github.com/organizations/VoltDB).
• It's a database.  It has to work, can't corrupt or lose data.  Quality is paramount.  
• We have Continuous Integration system running a large suite of automated tests.
• VoltDB is in production for a significant number of customers.  There's no dedicated Support team; the Engineering Team provides 24x7 support.
• We have transparent development: our Product Backlog and Sprint Backlog is hosted (publicly) in JIRA.  We use Greenhopper to plan and track iterations.
• 3 week iteration.  The product is released almost every iteration!  The Team truly produces a potentially shippable product increment most iterations.

Of course, like every Agile team, we have our share of challenges.  Some of the challenges we experience include:

• Dealing with unplanned customer support issues as well as pre- and post-sales assistance.
• Dealing with technical debt.
• Story sizing: specifically defining small enough slices of potentially shippable functionality.
• Scaling the team: we're growing fast!

We've adopting Scrum to help us meet these challenges. I'm a firm believer in Agile software development, specifically an Inspect and Adapt framework like Scrum.  Because we're a start-up with a product smack in the middle of a rapidly growing, and evolving, Big Data market, product requirements can change often, sometimes daily. Scrum provides us the framework to adapt to changing business conditions while still regularly producing business value.

It's great being back on the front line of Agility. I hope to find time to blog about these challenges as well as other topics in the coming months, time permitting.

Scrum Cadence

It’s unfortunate that the the Scrum product iteration increment is called a Sprint, implying that each increment is delivered by the team in a mad rush, full throttle.  The image I have in my head is a 100 yard dash, everyone running as fast as they can to reach the finish line.  Developing software is more like running a marathon than it is a back-to-back series of 100 yard dashes.  In order to successfully complete a marathon you have to find a one mile pace you are comfortable running at, and keep running at that speed for 26 miles.  Sprinting hard for the first several miles often means you are walking (or even crawling) the last mile, if you even make it that far.

Me, still running at mile 26, in the 1998 Boston Marathon

One of the challenges many new Scrum teams face is developing a sustainable pace, sometimes called cadence, over multiple sprints over the course of a project.    

New to Scrum, teams often struggle to find a sustainable cadence for delivering potentially shippable product increments.  What I’ve observed happens is this:  Early in a new Scrum project the teams is (I hope) excited to adopt a new way of working.  They understand the Scrum principles and the time-boxed Sprint iteration, and more importantly, they feel a strong commitment to succeed on their newly committed Sprint deliverables within the first few Sprints. After all, at the end of the Sprint they will be proudly demonstrating their work to their stakeholders, which usually includes there boss and department head.  However, since everyone is new to this different way of working, it is inevitable that certain work, usually that requiring specialized skills such as QA and documentation writing skills, ends up being delivered late in the game.  Undeterred, the team works extra hours, heroically delivering the Sprint goals.  This extra effort delivers a much needed early success, builds organization confidence, and can kick-start viral adoption.  However, this type heroic effort doesn’t scale - after 2 or 3 Sprints the team starts feeling burned out and may fail to deliver Sprint-committed work.

In an ideal situation, the Team inspects and adapts, and, over the course of several sprints, build a sustainable cadence.  However, many teams will continue to struggle, and can fall back to the “old way of doing things”.

While every team, project and environment is different, here are some points to consider to help get to a sustainable cadence:

• Work, as a full team, on only one story at a time.  Make sure that story is DONE before starting the next story.  In other words, implement stories sequentially.
• Avoid picking stories targeted for one individual due to their specialized skills.  Make sure the whole team is involved in delivering each story.
• Develop an “in it together” feeling among the team.  If you aren’t feeling this, your team isn’t operating as a true TEAM.  
• Commit to less stories to make sure your Sprint backlog really is DONE at the end of a Sprint.
• Become more of a generalist. Scrum certainly favors “jack of all trades” type of developers.  Coming from a Waterfall environment, where there are separate groups for QA and documentation, work can fall to those with that domain specialization.  As a team member, don’t look at work as “that person’s”, view it as a team deliverable. Learn a new skill to help a team member (and thus the Team) out, be it coding, QA or documentation.
• Be honest and innovative during your Sprint Retrospectives.  It’s your chance to change how you work together to deliver software.
• Get additional training or coaching.  Read Scrum blogs and ask questions on Agile forums.  Likely you are not the first team to run into the issues you are experiencing.
• If you are manager reading this, realize that team cadence, a predictable velocity, is key to delivering on corporate objectives.  Work hard to remove team obstacles and allow them to focus and succeed.  This is the true value of a manager in an Agile world.

Developing a sustainable cadence is one of the keys to the success of the Scrum Team and project effort.  Remember, running a marathon full throttle is unsustainable, unless, of course, you are an ultra-elite runner.  I know I’m not one.

Inspect and Adapt

“SCRUM assumes that the systems development process is an unpredictable, complicated process that can only be roughly described as an overall progression.”

-Ken Schwaber, Scrum Development Process, 1995

One of the things that I like about Scrum is that its creators defined it knowing that no development process is perfect.  The Scrum Guide does not tell us how to build software, rather, it openly acknowledges that building software is complicated and unpredictable.  This may seem odd, with Scrum being thought of as a development process (it’s not).  In a recent post to the Yahoo! Scrum Development group, Ken Schwaber stated that “we purposefully constructed Scrum with a lot of holes in which a person or team or organization has to continually figure out the best thing to do.”

So how does a Scrum team “continually figure out” how to do the best thing?   To facilitate iterative and ongoing improvement, Scrum specifies that each Scrum Team hold a Sprint Retrospectives. Further, these retrospectives must happen frequently, occurring immediately following each Sprint Review.  

It is often overlooked when starting out with Scrum, but I believe the Sprint Retrospective is the most important meetings a Scrum team participates in.  While the Sprint Review is a valuable “public” meeting where working product is demonstrated to stakeholders and customers, the Sprint Retrospective is a “private” team meeting, allowing the team to safely discuss what’s working and want could be improved with regards to how they are delivering their work.  In the Sprint Retrospective, anything that relates to how the team operates can be discussed and possibly tuned or improved.  Up for debate could be how the team communicates, technology issues, done-ness criteria, or even choice of tools, as was recently discussed for one of our new Scrum teams in my organization (They reverted back to whiteboard task charts and spreadsheet burndown charts!).

What this ultimately means is that each Scrum team is evaluating, refining, and improving how they work together and build software every 2 to 4 weeks.  Almost continuously improving throughout the life of the project!  Over a 12 month project there can be upwards of 24 opportunities to “do things better” and improve how the software is built and delivered.   Compare that to our our Waterfall-like development model where we generally conduct formal retrospectives twice, once after Beta and once after we release.   In our current model, we, at best, have one predefined opportunity to improve our practices during the release, at Beta when there is usually no more than 3-4 months of (QA and bug fixing time) time remaining before the software is released.  Process improvement is clearly not highly integrated into our Waterfall process.

With frequent inspection and adaptation we can become more efficient at building and delivering high quality software that meets or even exceeds customer expectation.  So my advice is: take full advantage of your Sprint Retrospectives.   As a Scrum team member, it’s your (frequent!) opportunity to make a difference...

kaizen  (kaɪˈzɛn) - n  a philosophy of continuous improvement 

of working practices that underlies total quality management 

and just-in-time business techniques [Dictionary.com]

Velocity and Release Planning

“An iterative, story-driven process makes it easy to fix a date but difficult to fix what will be included by a given date.”

-Mike Cohn, User Stories Applied

Predicting a release date is often a big challenge in software development, particularly Waterfall-type development projects.  Actually, making the prediction is the easy part - predicting accurately is the challenge.  

Assuming a team is in place, determining when a software project will be released is a function of time, features, and quality.  Further, by fixing any one of those variables, you can can expect to, perhaps greatly,  impact the other two.   

As my organization transitions from Waterfall to Agile, we’re challenged with how to accurately perform Agile product release planning and reflect that back to the wider (non-development) organization.   With the “old way of doing things” there is a general comfort with holding a written plan, a project schedule, even if everyone knows that plan has a high probability of being wrong.

Waterfall Release Planning

In today’s hyper-competitive business world, a product release date is often defined by company management.  This is true in both Agile and Waterfall development environments.  In my experience, it’s been true for most, if not all, of the large waterfall development project that I have been a part of the past 25 years.

Once a release date has been set and communicated to the field, the development team heroically tries to meet that date.  If you are a lucky engineer, your management team gives you some time to flesh out requirements, create a architecture and design documents, define tasks and a schedule, and ultimately allow your data to influence the release schedule.  Even with that luxury, challenges abound to meet this “defined up front” schedule: as you implement more, you learn more, the work remaining becomes more defined, ultimately impacting the project release date - often it is later than originally predicted, rarely is it earlier.   

When faced with more work remaining than time allowed, a hard delivery date often forces you to cut features to meet the schedule.  The problem is that features may be partially implemented (waste) and often most of the remaining work is quality assurance testing.  In those situations, the choice becomes tougher: slip the schedule or reduce testing (and thus quality).  It turns into a Kobayashi Maru (“no win”, for those who aren’t Star Trek fans) situation.  We’ve all been there, and we all bear the battle scars.

Release planning is different with Scrum

With a Scrum Team producing potentially shippable code with every Sprint iteration, we’ve essentially fixed one of the release planning variables: quality.  In order for a product to be potentially shippable, it must have a level of customer-expected quality.  This leaves the organization two remaining adjustable dials: time and features.  


The amount of completed work (user stories “done”) a Scrum team can deliver in a given iteration is identified as the team’s Velocity.  Velocity is a key metric for release planning - it quantifies the capacity of the team for a Sprint.  Knowing a team’s velocity, coupled with a sized Product Backlog, the Product Owner can get an approximate idea of what features will be delivered by a specific time.   [You can reference numerous books and blogs for further details about story sizing and velocity, with Mike Cohn’s User Stories Applied, and  Agile Estimating and Planning being two of the more highly regarded references.]

Time-based Release Planning

If you’re organization is going to mandate a release date, you must readily accept that you won’t initially know the full feature set that will be delivered on that date.  But, by using the Scrum Team’s velocity, you can get an approximate idea of the feature set.  

Knowing a team’s velocity, how much work a team can get “done” in a Sprint, will help predict how much of product backlog can be implemented by the time the mandated product release date arrives.   By using the team’s velocity, the number of sprints that can be completed before the release date can be computed (remember, Sprints are of fixed time).  Applying that to the backlog can give you a very good idea of what features will make the release and what are on the “bubble”.  Usually this is better than good enough.   Features that are on the delivery bubble, those that may or may not make the release, are much lower in priority than those that will be fully “done” prior to the release date.  These backlog items shouldn’t be make or break features, and can be saved for a subsequent release.

When the surrounding corporate environment is not Agile and you need to publish a future product feature list to a wider audience, the majority of the the features on the Backlog that comfortably fit into the velocity capacity can be communicated outward, setting expectations and enabling the coordination of public release-type activities such as PR, marketing, etc.

Feature-based Release Planning

If you product must complete a specific set of features before you can release, then using team Velocity can help you predict your release date.  By using the team’s velocity, and mapping it to the product backlog, you can predict many sprints it will take to complete the required backlog items.   Release burndown charts are great at visualizing how much work is remaining in a release and identifying how many sprints are likely required before you can ship the product.

Because the team velocity can vary from sprint to sprint a firmed up average, team velocity usually isn’t firmed up until the team achieves a regular Sprint cadence - usually 3 or so sprints.  To determine a release date early on, you will need to consider velocity variability.  Mapping out a release date window can be computed by using the predicted best velocity and predicted worse velocity, which will give you a release window as your target release date.  As more sprints are delivered, this release window can be refined.  

Product Backlog Considerations

The Product Backlog is a living entity.  As you learn more about your customer needs, and hopefully get customer feedback on early sprints, you likely will add and remove items as well as re-prioritize items, all of which can effect your release date.  Additionally, the team velocity likely will fluctuate - but with short (2-4 week) sprints, you will have your finger on the pulse, having up-to-date velocity measures as well as average and historical values.  Using this data, you can frequently map out how many sprints will be required to deliver true product value to the market.


Continuous Planning

Release planning within Scrum doesn’t happen just once, at the beginning of the release, as it does with waterfall development.  The release plan is evaluated and possibly adjusted at the end of every iteration, as more is known and team velocity is refined.  Realize that it may not be the project release date that is adjusted, rather it could be the work that will be delivered.  As more potentially shippable product increments are delivered by the team, the organization has  the opportunity to gather customer feedback, and use that feedback to influence (to add or remove items from) the product backlog.  Doing so may change your release date, or the features ultimately delivered.  Further, performing this activity regularly, doing frequent customer validation and backlog/release grooming, gives the organization the opportunity to deliver a product that better matches market needs.

For my organization, Scrum has given us a higher level of confidence with our release planning.  This year we’re part of a new corporate initiative requires that requires product by the end of 2011.  After the recent success of our Scrum pilots, we have a new found confidence in our planning such that we believe we’ll be able to deliver working shippable product by that deadline by using Scrum.  We can’t currently say what the complete functionality set will be with absolute certainty, however, but that’s okay.  We know we’ll deliver shippable product by the end of the year.

Ship it! - Scrum’s “Potentially Shippable” Product Increment

One of the problems many teams struggle with when beginning to adopt Scrum is delivering potentially shippable increments of product at the end of every Sprint.  Delivering a potentially shippable increment of code is one of the fundamental concepts of Scrum - a requirement that is specifically called out in the Scrum Guide:

“Scrum requires Teams to build an increment of product functionality every Sprint. This increment must be potentially shippable...the increment must be a complete slice of the product. It must be “done.” Each increment should be additive to all prior increments and thoroughly tested, ensuring that all increments work together.”

Scrum Guide: http://www.scrum.org/scrumguideenglish/

Producing a potentially shippable product in 2 or 4 weeks involves a semi-radical mindset change to those used to building software in a traditional Waterfall-like manner.  Making this mindset shift enables an organization to experience numerous benefits and efficiencies, from the ability to deliver software to customers at more frequent intervals, to being able to strategically pivot product direction while minimizing waste.

At first, most teams new to Scrum struggle with delivering software in this manner - our Agile Pilot teams were no different.  The first challenge was integrating cross-functional team members, development, doc, QA, into a new team.  Agreeing on the definition of “Done” helped with this.  Another hurdle involved Product Backlog user stories - often user stories were too large, or difficult to break down into small single feature product slices.  Backlog grooming quickly rose in importance for most teams.  Team member specialization also presented challenges. Tensions around creating a shippable product often developes when team members such as QA and documentation have to complete there work the last day or two during the Sprint.  At present this is our biggest challenge - the teams are still learning and adapting, as is the organization as a whole.

Over the course of a handful of sprints the Pilot teams put in a valiant effort, overcoming many of these challenges.  Teams adjusted the way they worked, via Sprint Retrospectives, to address some of these concerns, and ended up producing high quality software and documentation.  One of the Pilot teams even released the results of one of their early Sprints to customers via a Technical Preview (TP) pre-release program.

Last week I was fortunate enough to attend an executive briefing scheduled with one of our customers who has been participating in this Technical Preview program.  As part of this TP, we supplied a documented script that the customers should follow; this script defined the usage pattern we used as a final test before releasing the product to this program.  The customer, being a self-proclaimed power user, told us that he followed the script, but really wanted to see what else the product offered.  He strayed off script and began investigating how the tool’s functionality could help his applications.  I think a few jaws dropped in the meeting, as some were expecting the worse - straying off the prescribed path was an “all bets are off” scenario for pre-released software in our Waterfall development environment.  Surprisingly to most, the unexpected happened:  The customer told us that the software had all the functionality he expected and had very few bugs.  Further, he stated that he was able to use the product to map how he would make use of it in his enterprise. He did mentioned, in a proud tone, that he found 2 or 3 bugs that he wanted to demonstrate to us. Sure enough, he brought up the software and proceeded to demonstrate how he used the software, along the way, attempting to show two issues he found.  His subsequent focus on the issues, one minor (a browser-related issue), the other an issue he was unable to reproduce (but did note there was an easy work-around), didn’t minimized his earlier claim, now stuck in my mind:  “the product had all the functionality [he] needed”!

It’s always great to see a Customer demo your software to you. Our Product Owner had an opportunity to gather real-world product feedback from a power-user customer, one who personally demonstrated how he would use our pre-pre-Alpha, but potentially shippable, version of our product.   This customer’s feedback subsequently influenced our product backlog for an upcoming sprint - agility in action!

During this whole “customer demonstrating our product to us” segment of the meeting, I was smiling proudly: this was software that we, in comparison to a Waterfall-type release, had essentially “tossed out there”: no code freezes, no bug triage, no end-game QA bug fix cycle.  It was product that was the Sprint 4 output from our new Pilot Scrum Team, software that the Team had deemed “potentially shippable”.  And it was just that: potentially shippable.