Management’s Role in Scrum

In one of my earlier Blog postings, I responded to a comment and stated that I believed that “Scrum aims to make managing software development easy”.  I believe this is true because the core of Scrum is the self-empowered, self-organizing Team.  In its simplest form, a Scrum Team is fed a prioritized list of user stories and the Team commits to delivering some set of top stories in a fixed amount of time.  Because the team is empowered, it is up to the team itself to figure out to divide the work between themselves and deliver “done” stories by way of a potentially shippable product at the end of the sprint.   

As an Engineering Manager, I can sit back and watch an established Scrum team crank through their work.  I can readily track the Team’s work via burn down charts and the Scrum task board.  In fact, I was able to do just that during our recent Pilot with the Tool team.

A fully-functioning Scrum team will ask for help from the appropriate people, usually Subject Matter Experts (SME) and Product Owners (PO), and will inspect and adapt to make their work more efficient.  When any blocking issues occur, often called impediments, the team will raise these issue to their ScrumMaster.  Nowhere in the Agile Manifesto or Scrum Guide is a role for Engineering Manager defined.

Because there is no need for day to day management and oversight of a Scrum team, the traditional software engineering manager may feel marginalized, perhaps assuming that their primary responsibility is writing annual performance reviews.  Ugh, what a horrible role that would be.  Thankfully this couldn’t be further from the truth, particularly during the process of transforming from a Waterfall process to an Agile one.

A Manager’s Top Priority

A manager’s top priority is to put their employees in a position such that they have the best opportunity to succeed.  In a Waterfall environment, this usually means breaking down and assigning the work, working to identify and resolve cross-team dependencies, managing the team’s schedule, and communicating release status to upper management.  The common thread for these activities is that they facilitate the Waterfall development process.

In an Agile environment, a manager’s goal is the same, to create the best environment possible for that allows their employees to succeed.  But facilitating the development process in Scrum is very different.  Initially with Scrum, a manager has to be a champion for the process and help employees unlearn roles and responsibilities they’ve grown accustom to in Waterfall.  With Scrum, first and foremost, it is a Team effort versus an individual one.  The Team makes a commitment to work together and do all the work necessary to deliver “done” user-stories.  This can’t be understated. It may be hard for a manager to defer to the team and let the team self-organize, but once a manager “let’s go of the reins” and lets the team “figure it out”, their real work begins.

An Agile Manager’s Real Work

The first job of a manager is to assemble the right people for each Scrum team.  The skills and personalities needed for a waterfall-oriented team are often drastically different than needed for an Agile team.  Because Scrum teams deliver vertical increments of functionality teams members may need to be more cross-functional than siloed.   For example, rather than a team of User Interface engineers, a more appropriate team would include UI, middleware and database engineers, covering features from the front-end all the way to back-end.   This may require a more matrix-oriented organizational reporting structure or changes to the existing organization, something that may ruffle territorial feathers.  But such changes may ultimately what is best for the organization, and a manager should accept and adapt.

Managers have a broader understanding of the business, and can also act as subject matter experts (SME), providing coaching to the team as requested, and additionally participating in the Sprint Reviews where feedback on delivered functionality can be voiced.   

In addition, managers should also consider and account for external influences that may impact the team.  Because the team has committed to a delivering features in a time-boxed iteration, managers should make it one of their top goals to keep the team from being interrupted.  Team priorities are managed via the Product Backlog and short sprints give Product Owners frequent opportunity to re-prioritize work.  Managers should strive to block interrupts during a Sprint, by either pushing back on the intrusion or pre-identify organizational capacity for predicted interrupts such as software maintenance.  I could go on and on about this as I feel strongly about this point.  The Team makes a strong commitment to deliver on the stories.  Managers should respect that and equally make a strong commitment to protect them from interrupts and priority-shifts during the Sprint.

Perhaps one of the most important roles a manager has in Scrum is to address team impediments. A manager should regularly connect with the ScrumMaster to understand and work to eliminate impediments identified by the team.  Impediments may be easy to remove, such as finding budget money for new hardware (perhaps not so easy!), or may sound impossible, as in fixing the timezone issues between the U.S. and India.  Regardless, addressing impediments of all shapes is critical to the health and productivity of the team and it is ultimately managements responsibility to address these issues that can’t be addressed by the team.  Impediments offer an opportunity to think out of the box and to operationally improve organizational efficiency.   Here, out of the box thinking and sometimes creative organizational changes can make a huge impact to the team.  It’s a perfect opportunity for the manager to challenge “the way things have always been done.”   Removing impediments could almost be a full-time job, particularly for an organization transforming from Waterfall to Agile.

Some final thoughts on a manager’s role in Scrum:  A manager should never be the ScrumMaster for teams containing direct reports, as this, in practice, breaks team self-organization. Some companies have, however, had success with managers acting as co-Product Owner.  Engineering Managers partner with Product Managers, with the Engineering Managers focusing inward on the team, and the Product Managers facing outward to the customers.  This role may work where the manager has a strong technical background and the team is without an Architect or strong full-time Product Manager.

Agents of Transformation

Over the past year we’ve asked each Agile pilot team to drastically change the way they work, by holding daily stand-ups, tracking and publishing Sprint burndown and regularly delivering potentially shippable software every 2 weeks.  But as managers we’ve essentially kept our day-to-day work environment unchanged.

Converting from Waterfall to Scrum forces engineering managers to change their day-to-day activities within the organization.  Rather than “owning” the project schedule, their role shifts to enabling and coaching self-organizing teams as well as removing impediments.   But I’d offer up that the role goes a step further: as Managers, the organizational transformation starts with us,  the business owners.   

Agile development challenges us to see beyond the “this is the way we’ve always done it” mantra. Just because it “is”, that doesn’t mean it “has to be”.  And there lies the opportunity for transformation.  As the management leaders of this company, it is our responsibility to not only facilitate this transformation, but to also partake in it ourselves.

The Burn Down Chart: A Picture is Worth a Thousand Words

Back in the halcyon days of Digital Equipment Corporation (DEC) my team was “volunteered” to  participate in a pilot process improvement effort called Software Metrics In Action (SMIA). As project leader of 10 engineers working on an 18 month project, we were a perfect candidate for this effort of inspecting and adapting and ultimately improving our software development methodology via Software Metrics.  My participation in this process gave me valuable experience with process improvement.  What I didn’t realize at the time was that I was being exposed to many of the key elements of the yet-to-be named Agile software development philosophy.

As I look back to that period of time, I recall measuring every piece of the software development process we could identify. Our team motto was “Data tells the truth about perception.” Seriously!  We measured everything.  We compared task effort ‘actuals’ with ‘predicted’.  We tracked our time for development, test, vacation, dentist and doctor visits, even bathroom breaks, and we had graphs for every data series.  But in the midst of this data collection and evaluation there was one metric that stood out. It was the one ultimate metric that the business owners, our management, was most focused on: the “work remaining”, specifically: when can we ship?   

To track the answer to this question we kept a Microsoft Project schedule that contained only the remaining work.   Each day we adjusted the start date of the project each day to the current day and recomputed the remaining schedule.  As more project work got completed, our knowledge grew and so did our confidence in our task estimates and our release date.   By continually refining our estimates based on new knowledge, we were able to track, in “real time”, the effective project completion date on a daily basis.

My teams effort in our1990’s pilot organically stumbled upon one of the key mechanisms Scrum uses today to track project status. Scrum monitors “work remaining” by using the concept of Burn Down, and in particular, Burn Down Charts.  

Burn Down Charts

There are two types of burn downs that are generally tracked by a Sprint team, the Sprint Burn Down and the Release Burn Down.  They both operate similarly, but have a different scope.  The Release Burn Down most closely matched my DEC pilot schedule release date monitoring, as it tracks work remaining for the entire project.  The Sprint Burn Down tracks work remaining within a single Sprint, usually 2-4 weeks long.  Because the Progress Agile pilots were of such short duration, each Progress pilot focused solely on Sprint Burn Down Charts.

Understanding how to interpret these charts is important. As a business owner they provide a valuable and near-instant view into the health (or status), of the current effort, be it a Sprint or Release.  

Reading the Charts

A burn down chart is laid out simply: The x-axis denotes number of days in the iteration (or project), the y-axis, for a Sprint Burn, is amount of available time in hours (for a release burn down, it is story points - topic for another blog).  The numbers on this axis are computed by multiplying 8 (hours/day) times the number of team members, assuming each team member is full-time on the effort.
Reading a burn down chart is fairly simple once you understand what to look for.   Burn down charts show a the rate of progress of work being completed compared to the ideal rate of completion of that work.  All burn down charts have one thing in common: the ideal burn down line with the ideal slope.  Note the pink line in the Sprint burn down chart taken from one of our Agile pilot teams.  This pink line shows the ideal rate of progress, specifically that one day of work (8 hours times the number of team members) is completed each day.  There are 5 days of work, the Sprint iteration is 5 days long, and at the end of the sprint, 5 days of team work should be completed.

The pink line is boring. It’s boring because it is our control value, it’s slope is always the same.  In this chart, the interesting line is the blue line.  The blue line denotes the predicted remaining work for the iteration, in essence the current remaining workload of the Sprint.  If everything works out as we hope, at the end of the sprint, day 5 in this example, the blue line will converge exactly with the pink line, at zero work remaining (since there is zero time remaining!).    If we examine this chart on day 2 and day 3, we can see that the team is ahead of schedule, because it is below our control line.  Possibly the team under-committed to work, but regardless, at day 2 it looks like they are on track to meet their commitment to complete everything they said they would.   By day 4 the team is exactly on track.  But something happens by day 5 because they are half-day behind.  Perhaps a team member became sick or perhaps the team discovered some major unknown work at the last day.  Either way, this is likely a bad example I have chosen to discuss because the team missed their commitment and everyone only found out about it the last day.  The retrospective for this Sprint must surely have been interesting!  This chart introduces more questions than it answers, but it encapsulates a interesting story.  It turns out that this sprint was a bug fix sprint, in essence a “release sprint”, designed to produce a customer-shippable technical preview kit.  Bug fixing uncovered further bugs and the team was unable to fix everything they found in time for the release, the end of the sprint.  

Yes, that burn down chart tells quite a story, a thousand words, as they say, and if you are the business owner, it’s your quickest and best view into each project team.

More Examples
Let’s look at a couple more burn down charts.  In this chart the team is tracking close to the ideal progress line, the black line in this chart format.

The remaining work, Task To Do blue bars, are flirting above and below the ideal.  With 2 days remaining, it is likely this team complete all of their committed work.  In fact, because the remaining work amount hasn’t changed the last few days, it suggests that an inquiry to the ScrumMaster might be needed to see if people simply haven’t been updating work remaining in their effort to complete the Sprint.

In this final chart below, it looks like the team has over-committed, almost from the start, and true to form, they were unable to complete a bunch of work, just over two days of work.
As you become more comfortable with read burning down charts, you will be able to make project status assessments in a matter of seconds.  Tools such as Rally or VersionOne automatically generate these charts based on user data, or, as with one of our Agile pilots, a standard spreadsheet can be created to plot the team data.

Scrum Metrics

Metrics capture what happened in the past.  With historical metrics, we hope that we can understand the past and adapt more efficiently to similar conditions in the future.  While Scrum values inspecting and adapting, Scrum looks forward for iteration and release status using burn down charts.  

Incidentally, my Digital Equipment Corp Software Metrics in Action pilot was deemed a success: our 18 months development project was released 1 day ahead of our original schedule.  You can’t change the past, but sometimes you can predict the future...

What does “Done” mean?

It starts with a team conversation...

One of my favorite team conversations that generated by the Scrum framework is one that occurs at the beginning of a Scrum project.  It’s the team discussion and subsequent agreement of what the word “Done” means.

A Scrum Team is supposed to produce potentially shippable code at the end of every Sprint.  This means that any stories (requirements) that the team commits to must be full implemented, documented, tested, etc within the time-boxed Sprint.  The stories must be DONE. Done can and often is defined differently by each team.  Minimally, Done means that the story committed to is coded and fully tested.  A story really can’t be potentially shippable if it isn’t coded and tested.  

Done is demonstrated during the Sprint Review, where working code is demonstrated the business owners and everyone else in attendance.   A story isn’t complete unless it satisfies the definition of “done”.  In fact, Scrum will not credit the team with story points unless a story is, in fact, “done” (and accepted by the Product Owner).  

Each Pilot team that I observed went through similar discussions about “done”.  Because Scrum was new to the teams, the initial definition of “Done” was a bit loose:  there was general agreement that a story was “Done” if it was coded and tested.    Over the course of the first Sprint Planning and the start of the Sprint 1, it often became apparent how loose that definition of Done actually was.  

The Product Owner and Team conversation for story acceptance criteria was one of the main catalysts for Done refinement.   As each member of the cross-functional team discussed the stories they were committing to, it was quickly realized that the initial definition of done left many questions unanswered.  Product areas such as documentation (what kind? what level?), training (again, what kind?), automated tests (how much should be automated? what platforms should be tested? what web browser? what about performance testing?) and bugs (should all be fixed?  just blockers? more than just blockers?) all had questions that needed answers.  

Over the period of a month or so, the teams each refined the meaning of Done.  The process of discussing, debating, and ultimately answering these questions helped the group grow into a high performance team.  This was very important to the success of the effort, as with Scrum the commitment to delivering a story is a team commitment, rather than an individual one.

A Great Habit

I think most would agree that it’s great to get in the habit of fully completing work, versus completing 80% of the work and leaving the remaining 20% for some time “later”, during the release end-game, or perhaps never (thereby wasting effort).  With Scrum, producing done work is essential to the process. In the pilots, I’ve noticed a tendency for teams to commit to too many story points during a Sprint, ultimately leaving one or two stories partially finished, though the team put in often heroic effort to meet the original commitment.  The unfinished stories were fortunately completed in the next sprint, but the fact that they were not complete often defeated the “potential” from the potentially shippable product.   One way to deal with this is for the team to sequentially complete stories within the sprint, never starting a new story until the current story is Done.  This isn’t always practical, or efficient, but the team should be aware of the story bleed and velocity drag that partially implemented stories create.  Alternately, it is perhaps better to commit to fewer story points per sprint and fill in any capacity with product maintenance or backlog grooming or architectural spikes.    These are team decisions that can be considered during Sprint Planning or during the Sprint Retrospective.

Agility

For me, the concept of Done is the key foundational principal that  makes Scrum agile.  At the end of a Sprint all stories are fully completed, thus potentially shippable.  This fact, coupled with a short development iteration, offers the business frequent predictable time windows in which to re-prioritize work or introducing completely new requirements.  Scrum allows an organization frequent opportunities to, in contemporary terms,  pivot.  Easily and readily shifting priorities with minimal impact to productivity is one of the primary mechanisms through which an engineering organization gains agility.

Scrum @ Progress

Earlier in 2010, my company began the process of defining a Unified Development Process for the 500+ engineers in the Product Group.  We formed a committee containing key representatives from across the product group. After extensive investigation and deliberation, the committee chose to focus on an Agile process, and in particular, Scrum.

What is Scrum and how does it fit within our Company?   

Scrum is a software development framework, and not a prescribed process. It’s a framework that focuses on cross-functional, self-organizing teams producing potentially shippable business value in increments called Sprints.   A quick Google search will uncover reams of literature on what Scrum is, with the Scrum Alliance has a nice short overview here: http://www.scrumalliance.org/learn_about_scrum.  

What’s important about this process effort is how such an Agile framework fits within our existing 25+ year old corporate environment.  On the surface it sounds simple:  educate the engineering teams, flip a switch or shout”on your mark, get set, go!” and voila!, we’re now Agile!  In reality, evolving from a Waterfall software development lifecycle to Agile is a monumental undertaking, but one that is, in my opinion and experience, worth taking.  

As part of the our effort, we’ve began running a number of Scrum Pilot Projects, projects in each division who have “volunteered” to build software using the Scrum framework.  I’ve had the good fortune to “own” one of the pilots and to also observe others.  Most of the benefits and challenges that I have observed during this process matched my own experience as well as those I have heard about from friends or read about from others in the software industry.  And some of what I observed surprised me.  I hope to capture these observations as the subject this blog, aptly titled “Agile making Progress”.

Scrum Roles

There are three roles that the Scrum framework identifies:  ScrumMaster, Product Owner, Team Member.  How the roles do or could fit into my company's Product Group is discussed below, ihowever it should be noted that every group, every product, has potentially different needs.

The ScrumMaster.  The ScrumMaster is the person who ensures that the team is functional and productive.  The ScrumMaster role can be viewed as the facilitator or ringmaster.  At my company the role of a GEM (General Engineering Manager, a project coordinator) may align well with ScrumMaster.  For the Pilot Tooling team, the pilot project that I own, we’ve got a great ScrumMaster who is in fact a GEM.  However, this shouldn’t be a hard and fast role mapping.  Other teams have ScrumMasters who are engineers, or even engineering managers (just not the manager of anyone on the team).  The ScrumMaster cannot hold a position of reporting authority over any of the team members.  On any team, it is very likely that the “right” person for the role will rise to the occasion - someone who is a facilitator or coordinator of the group.  With the pilots, sometimes the choice was obvious, as it was with the Tooling team, sometimes after a sprint or two, the team realizes that someone else on the team might be better suited as the ScrumMaster and the team adjusts, as happened with another Pilot team.

The Product Owner.  The Product Owner is responsible for the business value of the project, and represents the “voice of the customer.”  The PO is also identified as the “single wring-able neck”, having ultimate responsibility for the project.  On of the most important tasks for the Product Owner is for identifying and prioritizing the “stories” (release requirements) that the team will be working on. Here in in my company, one initial thought expressed was that a Product Manager should be a part of every team in the role of Product Owner.  In theory this makes a lot of sense.  In practice, however, it quickly becomes problematic.  We’ve seen in the Pilots that the user stories and product backlog require near-constant refinement (also known as “grooming”).  Additionally, some user stories are very technical in nature, due to the type of software we are building.  A high degree of low-level technical understanding has often pulled in a Software Architect to help with backlog grooming and prioritization.   The Tooling team found that the backlog grooming and story refinement regularly required deeper technical expertise. Scrum favors empowered and self-organizing teams, and a this team quickly readjusted to solve this need by making the Product Owner a joint role, shared between a Product Manager (outward view) and a Software Architect (inward, technical view).  To date that partnership has worked well over 4 sprints.  It will be interesting to see how the Product Owner role will scale within the larger organization where there could be a dozen or so Scrum teams working on the same product line.  Perhaps we’ll see some form of Product Council or co-Product Owner or a hybrid approach.  I look forward to seeing how each team self-organizes!

The Team.  The team consists of the people doing the work. As a general rule, the team should be cross-functional, because at the end of each Sprint a potentially shippable piece of software is produced.  For software to be shippable it must be written, tested, documented, packaged, and have training materials created.  As you can imagine, this generally means that you need Developers, QA, Doc and Readiness on the team.  The Scrum team is given the authority and empowerment to self-organize and do what they need to do to get the job done.  In return for this self-governing power, the business owner asks for a team commitment to deliver some level of potentially shippable software in a predetermined amount of time.  In other words, the team commits to implementing a some number of stories (requirements prioritized by the Product Owner) within some predetermined amount of time (generally 2-4 weeks - for the pilots it was 2 weeks).  Because I had a high level of confidence in Scrum to begin with, I wasn’t personally surprised by the (tremendous) output of the pilot teams.  I was surprised, however, by a higher than expected commitment by each Team to make this effort work.  This was Scrum magic, in my eyes.

Organizational Transformation

Due to corporate evolution, the environment I work in makes it hard for many us to focus on one project.   Because we’ve all got a number (often different) “top priority” work items, we must often time slice our attention across these efforts to achieve some level of success.  This strategy is at odds with Scrum, as Scrum requires a Team to commit to deliver an unchanging list of functionality, over a fixed period of time.  In other words, Scrum teams need unwavering focus on commitments they have made.  For a Scrum team to succeed, Business Owners (management) need to leave the team alone during that time-boxed sprint so that the team can succeed. Here, to me, lies the biggest organizational challenge.

The pilot teams demonstrated to me that even in the face of major impediments and overriding “external” priorities that they were able to “figure it out” on their own and make it work.  This is a good thing and gives me hope for the success of our ongoing organizational transformation.