-Ken Schwaber, Agile Software Development with Scrum
Openness, transparency, visibility - no matter what synonym you use, it is critical to Scrum. My favorite term is transparency: Scrum’s openness makes everything that is going on within the project transparent to the world. With Scrum there are no hidden priorities, no hidden work, everyone should (and generally does) know what is currently being worked on and what will be delivered by the team.
On the surface, it may seem that transparency is more important for management and the business owners rather than the team. Management primarily cares about when we can actually ship and start selling a product (and of course, that the product does what it needs to do - that it meets market expectations). Numerous business-related activities such as marketing, support, training, etc, are coordinated around a product release.
But transparency is also extremely important for the team and individuals. Nowhere is this more apparent than in a Waterfall development environment. There have been several instances in my career where a peer has asked me for workplace advice. The conversation zeros in on a boss who is giving them grief about their productivity. After listening to them vent, a common thread usually appears: these individuals often are doing a lot of work, usually juggling many different activities, but their manager is only tangentially aware of the scope of responsibilities this person is shouldering. The manager is not aware of their employee’s “backlog”.
My advice in these situations is simple: make a list of work you are doing, order it according to priority, and discuss the list with your manager. In particular, I suggest that they work with their manager to help set the right priority of the various work items. This conversation often opens the eyes of both the manager and the employee, quickly addressing the tension and frustration experienced by both parties.
With Scrum, this communication activity is commonplace. In fact it is core to the Scrum framework: the prioritized product backlog and the Sprint backlog are visible to everyone. As a result, the work being done by the team is obvious and very transparent. Further, project progress is demonstrated to the business owners regularly during the Sprint Review, via a (potentially shippable!) working code demonstration.
The Inevitable New Unplanned “Top Priority” Work
In a Waterfall environment, the impact of new, mid-project “high priority” work is often not immediately felt. Because the product release can be many months away, incoming hot issues can be dealt with in a timely manner and everyone feels good, at least in the short-term. However, this new work causes the team to accrue a form of “schedule debt”. The painful balance comes due at some point in the future, usually in the form of a schedule slip. And often, sadly, that schedule slip is a surprise to management.
In a Scrum environment, transparency is the proverbial Big Stick when unplanned “critical” work is added to the project. When management (let’s face it, it usually is management) adds new new work to the team’s plate, the impact to the team is immediately obvious: 1) the new work will either displace existing Sprint priorities causing the Sprint to be abnormally terminated, something everyone wants to avoid, or 2) the work will be added to the Product Backlog and prioritized according to business value. There’s no magic, no silver bullet, and everyone Scrum-enabled, will should understand that!
In software development, we don’t like surprises - we like predicable. Usually a surprise means new unplanned work for the team or a schedule slip. Someone is is going to be unhappy!
Scrum’s transparency helps organizations eliminate surprises or at a minimum deal with them effectively. Surprises in the form of schedule slips are eliminated because what the team is working on is very visible and demonstrated at the end of every Sprint. Work, delivered via Sprints, is time-boxed and ends up being very predictable. Surprises in the form of new, unplanned work is dealt with effectively by adding it to the prioritized Product Backlog. With a prioritizing Backlog, the team is always working on top business value items.
Transparency, via a published, readily viewable backlogs allows everyone to understand what is currently being worked on as well as the impact of new unplanned work.
“Scrum keeps everything about a project visible to everyone.”
-Ken Schwaber, Agile Software Development with Scrum
One of the problems many teams struggle with when beginning to adopt Scrum is delivering potentially shippable increments of product at the end of every Sprint. Delivering a potentially shippable increment of code is one of the fundamental concepts of Scrum - a requirement that is specifically called out in the Scrum Guide:
“Scrum requires Teams to build an increment of product functionality every Sprint. This increment must be potentially shippable...the increment must be a complete slice of the product. It must be “done.” Each increment should be additive to all prior increments and thoroughly tested, ensuring that all increments work together.”
Producing a potentially shippable product in 2 or 4 weeks involves a semi-radical mindset change to those used to building software in a traditional Waterfall-like manner. Making this mindset shift enables an organization to experience numerous benefits and efficiencies, from the ability to deliver software to customers at more frequent intervals, to being able to strategically pivot product direction while minimizing waste.
At first, most teams new to Scrum struggle with delivering software in this manner - our Agile Pilot teams were no different. The first challenge was integrating cross-functional team members, development, doc, QA, into a new team. Agreeing on the definition of “Done” helped with this. Another hurdle involved Product Backlog user stories - often user stories were too large, or difficult to break down into small single feature product slices. Backlog grooming quickly rose in importance for most teams. Team member specialization also presented challenges. Tensions around creating a shippable product often developes when team members such as QA and documentation have to complete there work the last day or two during the Sprint. At present this is our biggest challenge - the teams are still learning and adapting, as is the organization as a whole.
Over the course of a handful of sprints the Pilot teams put in a valiant effort, overcoming many of these challenges. Teams adjusted the way they worked, via Sprint Retrospectives, to address some of these concerns, and ended up producing high quality software and documentation. One of the Pilot teams even released the results of one of their early Sprints to customers via a Technical Preview (TP) pre-release program.
Last week I was fortunate enough to attend an executive briefing scheduled with one of our customers who has been participating in this Technical Preview program. As part of this TP, we supplied a documented script that the customers should follow; this script defined the usage pattern we used as a final test before releasing the product to this program. The customer, being a self-proclaimed power user, told us that he followed the script, but really wanted to see what else the product offered. He strayed off script and began investigating how the tool’s functionality could help his applications. I think a few jaws dropped in the meeting, as some were expecting the worse - straying off the prescribed path was an “all bets are off” scenario for pre-released software in our Waterfall development environment. Surprisingly to most, the unexpected happened: The customer told us that the software had all the functionality he expected and had very few bugs. Further, he stated that he was able to use the product to map how he would make use of it in his enterprise. He did mentioned, in a proud tone, that he found 2 or 3 bugs that he wanted to demonstrate to us. Sure enough, he brought up the software and proceeded to demonstrate how he used the software, along the way, attempting to show two issues he found. His subsequent focus on the issues, one minor (a browser-related issue), the other an issue he was unable to reproduce (but did note there was an easy work-around), didn’t minimized his earlier claim, now stuck in my mind: “the product had all the functionality [he] needed”!
It’s always great to see a Customer demo your software to you. Our Product Owner had an opportunity to gather real-world product feedback from a power-user customer, one who personally demonstrated how he would use our pre-pre-Alpha, but potentially shippable, version of our product. This customer’s feedback subsequently influenced our product backlog for an upcoming sprint - agility in action!
During this whole “customer demonstrating our product to us” segment of the meeting, I was smiling proudly: this was software that we, in comparison to a Waterfall-type release, had essentially “tossed out there”: no code freezes, no bug triage, no end-game QA bug fix cycle. It was product that was the Sprint 4 output from our new Pilot Scrum Team, software that the Team had deemed “potentially shippable”. And it was just that: potentially shippable.
Early in our journey from Waterfall development to Agile, we identify 4 pilot projects, each from different product lines, to test the Agile waters via “Pilot Projects”. Even though there were several Agile champions (myself included) in the organization, we decided to take a “let’s try it before we buy it” approach to Agile. Each of the pilot teams consisted of cross-functional team members. Our strategy was to formally train members of the 4 pilot teams as Certified ScrumMasters, Certified Product Owners or Certified Scrum Developers. As the overseeing Unified Development Process (UDP) Committee, we identified our in-house Agile champions as coaches for each team, but ultimately each team had to self-organize, inspect and adapt, and deliver on their commitments following the Scrum framework. Once trained, we sent the teams off to build software for a minimum of three two-week long sprints. When it was all over... When all Pilot teams completed their the minimum 3 Sprints we conducted a multi-team retrospective. Those of us that were coaching the pilot teams had already observed many of the positives and negatives identified during this retrospective, some of which are recounted in this blog. However, it was very heartening to hear similar sentiments expressed nearly unanimously by all Pilot teams. What Worked Well? The teams enthusiastically offered up a long list of “what worked”, but the first two items mentioned are worth noting here. Improved communication was unanimously agreed upon by all teams as the top benefit they experienced during the pilot. [Ironically, this conversation occurred mere hours after I had written my “Hyper-Communication: Scrum’s Secret Sauce” blog! I was mentally smiling as they spoke.] The teams stated that the Scrum framework fostered communication, mentioning that the cross-functional teams, specifically the inclusion of QA engineers, was extremely beneficial. Also singled out was the Product Backlog Grooming - it was a key team communication activity valued by all. It’s interesting to note that the Waterfall process used in our Product Group does not prohibited nor dissuade people from communicating with others when developing software. I believe it is simply that often there is no immediacy to engage in these types of discussions in a Waterfall environment. The Waterfall process does not foster this type of regular extended team communication. For example, there is often a buffer, perhaps a project leader or supervisor between Product Management and the developers writing the software. Further, with Waterfall, the “are we writing the correct software” check-point is often distant, in terms of time, from the requirements definition, occurring at Beta perhaps, where as Scrum’s are much more frequent, occurring at the end of every Sprint. The second notable benefit brought up was the feeling of empowerment experienced by the participants in the pilot. This one is interesting and caught me a bit by surprise as it is hard to visually identify. The teams actually appreciated the complete ownership, not only for the delivery of working product, but for user story refinement, quality, documentation, etc. Success (or failure) was almost completely within their, the Team’s, control, and they found this to be extremely valued. In Waterfall software development, most teams are used to be being led. Often a project leader or project manager is identified as the chief point of contact. That person manages the schedule, works dependencies, reports status for the team, and often runs interference, allowing the team members work on their individual tasks with minimal interruption. Teams are typically not cross-functional and often the onus of communication is often on the individuals themselves. Scrum’s emphasis on “The Team” was appreciated by all of our pilot teams. The team members valued the empowerment and responsibility - essentially the confidence the organization put in them. As management, of which I am a part, it seems we really did succeed in stepping back and letting the Pilot teams operate without traditional management meddling. What Didn’t Work Well? The Pilot teams also brought up a healthy list of things that didn’t work. Being a new process that most had just learned, coupled with introducing Agile in a Waterfall environment, a long list of issues was not unexpected. The number one issue the teams noted was that most Product Owners did not spend enough time with the team. For our Pilots, we identified Product Managers as Product Owners for each team. Because our Product Managers were few in numbers and had many other responsibilities most ended up being only part-time members of the teams. The teams felt this strongly that the Product Owners needed to participate more as a full-time team member, attending the daily Scrum as well as spending more time creating and refining the Product Backlog and working with the Team on backlog grooming. This issue was a direct result of mapping a Waterfall role into a Scrum role. In Waterfall, a Product Manager can manage many projects, defining the requirements up front, then moving on to the next project’s requirements. Not so with Scrum, where the Product Owner is a full-time job, a fact that became obvious very quickly for most Pilot teams. It’s clear going forward that our organization needs to re-examine our staffing and role needs, as well as acknowledge that some existing roles won’t readily map into new roles within Agile. One other notable “didn’t work well” issue was the handling of customer escalations. With a large customer base, an escalation can come in and derail the team’s Sprint commitment. This situation can and does occur during our Waterfall-run projects as well. It’s an issue that likely will need an organizational solution, as it introduces tension between specialized engineers developing new software in time-boxed sprints and the need to support a significant and demanding and customer base. We thought it was all over, but it wasn’t... Much to my delight, all four of our Pilot Teams decided on their own to continue with Scrum beyond their initial 3-sprint Pilot. One Team is now on Sprint 7, and has successfully released the output of an earlier Sprint to customers in a Technology Preview program. Beyond the team enthusiasm and product successes, the biggest outcome of our Pilot effort was that we created new Agile champions. Each Pilot Team has several members who have been positively promoting their Pilot Team experience. As we roll out Agile to more teams, we’ll leveraging these new Agile advocates as mentors and champions to assist with our organization Agile Transformation. To me, that was the biggest “what worked well?” of all.