Scrum: It’s a full-time job.

“An adequate ScrumMaster can handle two or three teams at a time...A great ScrumMaster can handle one team at a time.”

-Michael James

One of the lessons we learned early in our Scrum roll-out was that the Product Owner and Scrum Master roles required much more involvement than originally thought.  Coming from a Waterfall development environment, where releases were long in duration and milestones spread out, people in the organization were used to having key roles on multiple projects, context switching as needed.   Juggling responsibilities in a Waterfall environment seemed to work well, or at least acceptable.  Our Product Managers, for example, could own several product lines, including associated customer meetings and field interactions, and easily satisfy occasional project touch-points such as requirements definition (at the beginning of the release) and Beta and FCS duties (at the end of the release).  These project touch-points required their full-time involvement periodically, a few times a year, making it relatively easy to context-switch periodically and satisfy the needs of each effort.



With Scrum, however, things turned out a bit different.  Though we “took the training” and “read the literature” which repeatedly said the roles of Scrum Master and Product Owner were full time jobs, we weren’t true believers.  Our “muscle memory” way of working had everyone juggling multiple roles and responsibilities, it was the way we were used to working. During the course of our Agile Pilot projects, however, we got convinced pretty quickly that Scrum roles could not be part-time.

Scrum is more intensive than waterfall development.   

With frequent “potentially shippable product increments” created every 2-4 weeks,  there’s essentially a full product release at least once a month if not twice.   During our Agile pilots, our Product Managers, filling the role of Product Owners, soon realized that creating and maintaining a Product Backlog required a significant amount of time.  Every few weeks a new set of fully-defined “product requirements” need to be ready for the team for the next Sprint. Product Owners found it challenging to satisfying their Scrum PO role and concurrently complete other project work.  Our Scrum Masters, too, found themselves booked full-time, facilitating team interactions, producing Sprint Burn Down charts, working impediments, and delivering frequent product releases at regular Sprint Review.  Team members, often with other-project responsibilities, found themselves spending more time delivering on Sprint Goals, often at the expense of their other responsibilities.  

As we scale Scrum within the organization, we’ve (obviously!) needed to identify new Scrum Masters and Product Owners.  We’ve held fast to the one project per Scrum Master recommendation. Identifying new dedicated Scrum Masters has been fairly easy - our Pilot teams was that the efforts produced several Scrum Master candidates, a nice side-benefit!   Finding full-time Product Owners for each team was more challenging.  Because we’re applying Scrum to large multi-team projects, our mapping of Product Manager positions to Product Owners simply didn’t scale.    Our solution was to look to our lead engineers. Most of our new teams have technical Product Owners, usually Software Architects, and they are dedicated full-time to the team.  Product Managers are involved at either a co-Product Owner (in an outward facing role) or at a higher level in the project effort, via a Product Council-type role that involves them as stakeholders.  

I expect our organization to fine tune (via inspect and adapt) these non-Scrum roles as we mature in our Agility.  But regardless of the organizational “noise” around the Scrum Teams, I believe that participation at the Scrum Team-level will continue to require full-time effort from all members in order to be successful.

Why Agile?

We’re at what I perceive to be a major “transition point” in our Agile transformation.  Most of our Agile pilot teams and projects have successfully completed and we are rolling out internal Scrum training throughout the Product Group.  We’re on the cusp of taking a big step forward with Scrum adoption within our 500 person organization.  It’s at transition points like this when I find myself reevaluating our direction: “Why are we doing this? Are we ready for this next big step?”   Our existing processes do, in fact, lead to shipping product and generating revenue, so: “Why Agile?”

Our original goal, defined we began this transformation, still rings true:

The goal is not to release things faster for the sake of it. It is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes.   It is about creating more business opportunities for ourselves rather than being cased into a model that requires our resource commitment for a time window that is longer than what the market we’re competing in requires.

Specifically, the software development group’s reason for moving from Waterfall to Agile is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes.  We want the ability to efficiently deliver strategic products to market in a timely manner.

Today we have challenges delivering on this goal.  As a mature software organizations our development practices have been hardened over many years of delivering software to tens of thousands of customers.   Our processes have been tempered with the successes (and often, the failures) of the past.    Because of this, it’s difficult to change the current way of doing things.  This is an understandable, and completely human, reaction - but it can dangerous over the long-term.  Fortifying the walls, via policies and procedures as well as organizational divisions, has the result of business being less reactive to change, less agile, which ultimately impacts corporate revenue.   New software releases often take longer to get out the door and may not fully meet customer needs.  Entering new markets in a timely manner becomes difficult, allowing younger more nimble companies a market advantage.   It is for these reasons that we’ve embarked on our development process transformation.

As part of our migration from Waterfall to Agile software development, we need to build a new way of working, including a new corporate culture: one that is nimble and responsive to ever changing business conditions. A culture that values responding to change over following a plan.  One that regularly inspects and adapts, and becomes better over time.  Scrum, with its “inspect and adapt” framework,  is one of the catalysts that can greatly assist us in this journey.

Agile Pilot Teams - A Retrospective

Early in our journey from Waterfall development to Agile, we identify 4 pilot projects, each from different product lines, to test the Agile waters via “Pilot Projects”.   Even though there were several Agile champions (myself included) in the organization, we decided to take a “let’s try it before we buy it” approach to Agile.  Each of the pilot teams consisted of cross-functional team members. Our strategy was to formally train members of the 4 pilot teams as Certified ScrumMasters, Certified Product Owners or Certified Scrum Developers.  

As the overseeing Unified Development Process (UDP) Committee, we identified our in-house Agile champions as coaches for each team, but ultimately each team had to self-organize, inspect and adapt, and deliver on their commitments following the Scrum framework.  

Once trained, we sent the teams off to build software for a minimum of three two-week long sprints.

When it was all over...
When all Pilot teams completed their the minimum 3 Sprints we conducted a multi-team retrospective.  Those of us that were coaching the pilot teams had already observed many of the positives and negatives identified during this retrospective, some of which are recounted in this blog.  However, it was very heartening to hear similar sentiments expressed nearly unanimously by all Pilot teams.

What Worked Well?
The teams enthusiastically offered up a long list of “what worked”, but the first two items mentioned are worth noting here.  Improved communication was unanimously agreed upon by all teams as the top benefit they experienced during the pilot.  [Ironically, this conversation occurred mere hours after I had written my “Hyper-Communication: Scrum’s Secret Sauce” blog!  I was mentally smiling as they spoke.]  The teams stated that the Scrum framework fostered communication, mentioning that the cross-functional teams, specifically the inclusion of QA engineers, was extremely beneficial.  Also singled out was the Product Backlog Grooming - it was a key team communication activity valued by all.

It’s interesting to note that the Waterfall process used in our Product Group does not prohibited nor dissuade people from communicating with others when developing software.  I believe it is simply that often there is no immediacy to engage in these types of discussions in a Waterfall environment.  The Waterfall process does not foster this type of regular extended team communication. For example, there is often a buffer, perhaps a project leader or supervisor between Product Management and the developers writing the software.  Further, with Waterfall, the “are we writing the correct software” check-point is often distant, in terms of time, from the requirements definition, occurring at Beta perhaps, where as Scrum’s are much more frequent, occurring at the end of every Sprint.

The second notable benefit brought up was the feeling of empowerment experienced by the participants in the pilot.   This one is interesting and caught me a bit by surprise as it is hard to visually identify.  The teams actually appreciated the complete ownership, not only for the delivery of working product, but for user story refinement, quality, documentation, etc. Success (or failure) was almost completely within their, the Team’s, control, and they found this to be extremely valued.

In Waterfall software development, most teams are used to be being led.  Often a project leader or project manager is identified as the chief point of contact.  That person manages the schedule, works dependencies, reports status for the team, and often runs interference, allowing the team members work on their individual tasks with minimal interruption.  Teams are typically not cross-functional and often the onus of communication is often on the individuals themselves.

Scrum’s emphasis on “The Team” was appreciated by all of our pilot teams.  The team members valued the empowerment and responsibility - essentially the  confidence the organization put in them.  As management, of which I am a part, it seems we really did succeed in stepping back and letting the Pilot teams operate without traditional management meddling.

What Didn’t Work Well?
The Pilot teams also brought up a healthy list of things that didn’t work.  Being a new process that most had just learned, coupled with introducing Agile in a Waterfall environment, a long list of issues was not unexpected.

The number one issue the teams noted was that most Product Owners did not spend enough time with the team.   For our Pilots, we identified Product Managers as Product Owners for each team.  Because our Product Managers were few in numbers and had many other responsibilities most ended up being only part-time members of the teams.  The teams felt this strongly that the Product Owners needed to participate more as a full-time team member, attending the daily Scrum as well as spending more time creating and refining the Product Backlog and working with the Team on backlog grooming.  

This issue was a direct result of mapping a Waterfall role into a Scrum role.  In Waterfall, a Product Manager can manage many projects, defining the requirements up front, then moving on to the next project’s requirements.  Not so with Scrum, where the Product Owner is a full-time job, a fact that became obvious very quickly for most Pilot teams.  It’s clear going forward that our organization needs to re-examine our staffing and role needs, as well as acknowledge that some existing roles won’t readily map into new roles within Agile.

One other notable “didn’t work well” issue was the handling of customer escalations.  With a large customer base, an escalation can come in and derail the team’s Sprint commitment.  This situation can and does occur during our Waterfall-run projects as well.  It’s an issue that likely will need an organizational solution, as it introduces tension between specialized engineers developing new software in time-boxed sprints and the need to support a significant and demanding and customer base.

We thought it was all over, but it wasn’t...
Much to my delight, all four of our Pilot Teams decided on their own to continue with Scrum beyond their initial 3-sprint Pilot.  One Team is now on Sprint 7, and has successfully released the output of an earlier Sprint to customers in a Technology Preview program.

Beyond the team enthusiasm and product successes, the biggest outcome of our Pilot effort was that we created new Agile champions.  Each Pilot Team has several members who have been positively promoting their Pilot Team experience.  As we roll out Agile to more teams, we’ll leveraging these new Agile advocates as mentors and champions to assist with our organization Agile Transformation.  To me, that was the biggest “what worked well?” of all.

Hyper-Communication: Scrum’s Secret Sauce

Com com communicate

Communicate communicate

Communicate communicate

Via satellite and solid state

Never never hesitate

Communicate communicate

Communicate communicate

Never never hesitate

-Pete Townshend, Communication, from the album All The Best Cowboys Have Chinese Eyes

Effective and continuous communication is the cornerstone of any high-performance team. It may be obvious, but we need good communication to do our job well.  But good communication doesn’t come for free: you have to continuously work at it.   Further, when developing software, your choice of development process can hamper or promote high levels of communication.

During our Waterfall to Agile Pilot project effort, I’ve observed that Scrum has “forced” us to dramatically communicate more, and at more more regular intervals, than we have done in the past.  Scrum forced our teams to develop a communication habit - one where where I would estimate that we communicate twice as much, if not more, than we did with our current Waterfall process.  

Why is this so?  It’s because Scrum prescribes a set of frequently repeating ceremonies (meetings) that require effective team-wide communication.  Teams that adhere to Scrum develop a work cadence, a habit of working, that fosters regular, focused communication on the work at hand.  Scrum promotes this team habit through a the following ceremonies:

• Release Planning Meeting - held at the start of a project.
• Sprint Planning Meeting - held prior to the start of a Sprint.
• Daily Scrum (stand-up) - held every day of the Sprint.
• Sprint Review - held at the end of every Sprint to demonstrate working software to the stakeholders.
• Sprint Retrospective - held after every Sprint to inspect and adapt.

Several additional conversations generally occur regularly as part of developing software guided by Scrum.  One, the subject of an earlier blog, surrounds the team agreement on what “Done” means.  Another equally important conversation occurs with the Product Owner about the Product Backlog. Called Backlog Grooming, the team works closely with the Product Owner to focus on, understand, refine, and size top priority user stories over the course of every Sprint, a “just in time” process that helps feed the next Sprint Planning Meeting.  The Scrum framework doesn’t require these conversations, but my observation is that they are regular ceremonies for most Scrum teams, and they were with our Scrum Pilot teams.

Contrasted Communication with Waterfall

Compared to Scrum, Waterfall-based software development seems lazy with regards to communication.  A majority of team communication happen up front, when the least is known about the project.  Up-front requirements, analysis, and design discussions help formulate a project plan and project schedule, then the team goes off and develops according to that plan.  Regular project status meetings, usually weekly, are held to track the project and resolve dependencies. Sometimes, in fact sometimes quite frequently in my experience, status meetings are cancelled because everyone is “heads down” working, an ironic often-used phrase - If someone is head’s down, an image comes to mind of a someone cloistered, cranking out code, only coming up for air, for a bathroom break.

During past Waterfall development efforts, I have observed situations where an engineer, working “heads down” delivering according to his own schedule, is suddenly informed that a feature he previously completed needs to be re-implemented in a substantially different way, thereby putting him unexpectedly behind schedule.   As a Project Leader I recall feeling surprised that this happened, after so much time being spent up-front identifying dependencies and coordinating task schedules.  Something slipped through the cracks, causing the delivered functionality to based on stale, incorrect information.  We simply didn’t find out that we went down the wrong path until much later in the release, a point in the project where it was more costly to fix the issue. These were situations that could have been easily avoided with just a little bit more communication with the appropriate parties, a situation much less common with Scrum.

Continuous Communication with Scrum

During two week Sprints, the Scrum Team is meeting and communicating at least once daily, with a 10% of the Sprint spent on planning, demonstrating product, and adapting and improving their process.  Further, there are daily Scrums, discussions with subject-matter expert, and design discussions and activities between team members.  In other words, plenty of opportunities where team members are  in essence, forced to communicate.

One benefit I observed during our Pilot Scrum Team effort was the impact that the Daily Scrum had on team communication.  The daily Scrum meeting frequently spawned ad hoc discussions immediately following the 15 minute stand-up.  Team members responding to the daily Scrum “3 questions” fostered additional, unplanned, clarifying discussions, just by team members actively listening to teammates.  Project questions and issues were often resolved immediately, or “just in time” rather than laying dormant until some point in the future when they would surprisingly be discovered.

To those used to Waterfall, the communication required by Scrum may seem like hyper-communication, perhaps overkill.  But ultimately it works well.  Teams quickly develop a  communication rhythm, involving Stakeholders (via Sprint Reviews), Product Owners (via the Product Backlog Grooming) and fellow team members (via the Daily Scrum).  With the Team focused on the same goal, delivering on Sprint commitments, continuous communication is the secret sauce that allows the Team to succeed.  It’s the catalyst to turning a team into a high performance team.  

Pete Townshend said it best:

“Communicate communicate, Never never hesitate”

Agile Transformation and Me

I've worked for big companies like Digital Equipment Corporation, medium, like Sybase and Progress (my current job) all the way down to small start-ups, like EasyAsk and Novera Software.  Each job has offered me different opportunity to grow, from create value from nothing at start-ups, to managing product lines generating hundreds of millions of dollars.  Through it all, one thing has remained constant regardless as to the size of the company or my role in it:  continuously improve and become more efficient.  The only difference I've experienced is that with smaller companies it is generally easy, or quicker, to change what (and how) you are doing.  The larger the company, the more difficult it is to change.  Pure size, inertia and existing customers and revenue provide significant reasons for doing things the way they have always been done.  

It is in this type of "bigger company" environment that the greatest challenge and perhaps the greatest rewards lie.  Improvements in the efficiency of the software development process can reap significant dividends.  Over the past decade+, a major evolution in software development practice, in the name of Agile Software Development, has been taking hold in the industry.  I've been fortunate to have experience with delivering product using Agile frameworks and methodologies, both as an individual contributor, project leader, up to VP of Engineering.  In my present role of Director of Engineering for 3 product lines generating over $300M in revenue, I am one of the leaders working to convert my the engineering organization of over 500 software engineers from a Waterfall-centric development process to Agile (Scrum) process.  As this transformation got underway and I began coaching our pilot teams, I found there were many seemingly random thoughts and ideas bouncing around my head that I started writing them down, and ultimately started blogging within my organization.   I've decided to post these blogs publicly, charting my organization's journey, our Progress, from Waterfall to Agile.

Thanks for reading...

John Piekos