Software Engineering with Microsoft Visual Studio Team System (Microsoft .NET Development Series) [1 ed.] 9780321278722, 0321278720

Citation preview

Software Engineering with Microsoft Visual Studio Team System By Sam Guckenheimer, Juan J. Perez ............................................... Publisher : Addison W e sle y Pr ofe ssion a l Pub Dat e: M a y 0 9 , 2 0 0 6 Print I SBN- 10: 0 - 3 2 1 - 2 7 8 7 2 - 0 Print I SBN- 13: 9 7 8 - 0 - 3 2 1 - 2 7 8 7 2 - 2 Pages: 3 0 4

Table of Cont ent s | I ndex

Soft w a r e En gin e e r in g w it h M icr osoft Visu a l St u dio Te a m Syst e m is writ t en for any soft ware t eam t hat is considering running a soft ware proj ect using Visual St udio Team Syst em ( VSTS) , or evaluat ing m odern soft ware developm ent pract ices for it s use.

I t is about t he value- up paradigm of soft ware developm ent , which form s t he basis of VSTS: it s guiding ideas, why t hey are present ed in cert ain ways, and how t hey fit int o t he process of m anaging t he soft ware lifecycle. This book is t he next best t hing t o having an onsit e coach who can lead t he t eam t hrough a consist ent set of processes.

Sam Guckenheim er has been t he chief cust om er advocat e for VSTS, responsible for it s end- t o- end ext ernal design. He has writ t en t his book as a fram ework for t hinking about soft ware proj ect s in a way t hat can be direct ly t ooled by VSTS. I t present s essent ial t heory and pract ical exam ples t o describe a realist ic process for I T proj ect s.

Readers will learn what t hey need t o know t o get st art ed wit h VSTS, including The role of t he value- up paradigm ( versus work- down) in t he soft ware developm ent lifecycle, and t he m eanings and im port ance of " flow" The use of MSF for Agile Soft ware Developm ent and MSF for CMMI Process I m provem ent Work it em s for planning and m anaging backlog in VSTS Mult idim ensional, daily m et rics t o m aint ain proj ect flow and enable est im at ion Creat ing requirem ent s using personas and scenarios Proj ect m anagem ent wit h it erat ions, t rust wort hy t ransparency, and frict ion- free m et rics Archit ect ural design using a value- up view, service- orient ed archit ect ure, const raint s, and qualit ies of service Developm ent wit h unit t est s, code coverage, profiling, and build aut om at ion

Test ing for cust om er value wit h scenarios, qualit ies of service, configurat ions, dat a, explorat ion, and m et rics Effect ive bug report ing and bug assessm ent Troubleshoot ing a proj ect : recognizing and correct ing com m on pit falls and ant ipat t erns This is a book t hat any t eam using or considering VSTS should read.

" This is first and forem ost a book about soft ware engineering. I n discussing flash point s such as planning, docum ent at ion, governance, audit abilit y, and organizat ion, Sam present s t he case for bot h agile and m ore form al pract ices, as well as describing t he opt im al condit ions for each. Even t hough t he m at erial is present ed in t he cont ext of VSTS, t he guidance is universal." Dr. Bill Curt is chief process officer, Borland Soft ware Corporat ion

" Sam Guckenheim er ushers in t he era of t rust wort hy t ransparency t hat will revolut ionize t he way we m anage soft ware developm ent proj ect s." David J. Anderson aut hor of Agile Managem ent for Soft ware Engineering

" This book is an eye opener: a door t o a new era of soft ware engineering." Francis T. Delgado senior program m anager, Avanade

Software Engineering with Microsoft Visual Studio Team System By Sam Guckenheimer, Juan J. Perez ............................................... Publisher : Addison W e sle y Pr ofe ssion a l Pub Dat e: M a y 0 9 , 2 0 0 6 Print I SBN- 10: 0 - 3 2 1 - 2 7 8 7 2 - 0 Print I SBN- 13: 9 7 8 - 0 - 3 2 1 - 2 7 8 7 2 - 2 Pages: 3 0 4

Table of Cont ent s | I ndex

Copyright Praise for Software Engineering with Microsoft Visual Studio Team System Microsoft .NET Development Series About the Author Foreword Preface Acknowledgments Chapter 1. A Value-Up Paradigm A Paradigm Shift Contrasting Paradigms Attention to Flow One Work Item Database Fit the Process to the Project Summary Endnotes Chapter 2. Value-Up Processes Microsoft Solutions Framework Iteration Risk Management Fit the Process to the Project Summary Endnotes Chapter 3. Requirements What's Your Vision? When to Detail Requirements Personas and Scenarios Personas and Scenarios and Their Alternatives Exciters, Satisfiers, and Dissatisfiers Qualities of Service Kano Analysis Summary Endnotes Chapter 4. Project Management Understanding Variation Using Descriptive Rather Than Prescriptive Metrics Many Dimensions of Project Health

Answering Everyday Questions Estimating an Iteration Triage Satisfying the Auditor Summary Endnotes Chapter 5. Architectural Design A Value-Up View of Architecture Service-Oriented Architecture Constraints with Degrees of Freedom VSTS and Service-Oriented Architecture Quality of Service Mindset Citizenship Mindset Design for Operations Summary Endnotes Chapter 6. Development A Value-Up View of Development Quality from a Developer's Point of View Using Test-Driven Development to Ensure Requirements Clarity Addressing Programming Errors with Code Reviews, Automated and Manual Providing Immediate Feedback with Unit Tests and Code Coverage Making Unit Tests Better Preventing Version Skew Making Work Transparent Summary Endnotes Chapter 7. Testing A Value-Up View of Testing Basic Questions Are We Delivering the Customer Value? Are the Qualities of Service Fit for Use? Have We Tested the Changes? What Haven't We Tested? Does It Work in Production as Well as in the Lab? Are We Testing Enough? When Should We Test? Which Tests Should Be Automated? How Efficient Is Our Team, or Our Outsourced Team? Summary Endnotes Chapter 8. Reporting Bugs A Cautionary Tale A (Software) Bug's Life Bug Reporting Is Like Journalism Subjective Data Objective Data Assessment Data Plan Summary

Endnotes Chapter 9. Troubleshooting the Project Underestimating Development Practices Too Loose Tests Passing; Solution Doesn't Work Solution Stuck in Testing Summary Endnotes Chapter 10. Conclusion Expected Criticisms Value-Up, Again Endnotes Index

Copyright Many of t he designat ions used by m anufact urers and sellers t o dist inguish t heir product s are claim ed as t radem arks. Where t hose designat ions appear in t his book, and t he publisher was aware of a t radem ark claim , t he designat ions have been print ed wit h init ial capit al let t ers or in all capit als. The aut hor and publisher have t aken care in t he preparat ion of t his book, but m ake no expressed or im plied warrant y of any kind and assum e no responsibilit y for errors or om issions. No liabilit y is assum ed for incident al or consequent ial dam ages in connect ion wit h or arising out of t he use of t he inform at ion or program s cont ained herein. The publisher offers excellent discount s on t his book when ordered in quant it y for bulk purchases or special sales, which m ay include elect ronic versions and/ or cust om covers and cont ent part icular t o your business, t raining goals, m arket ing focus, and branding int erest s. For m ore inform at ion, please cont act :

U. S. Corporat e and Governm ent Sales ( 800) 382- 3419 corpsales@pearsont echgroup.com For sales out side t he U. S., please cont act :

I nt ernat ional Sales int ernat [email protected] Visit us on t he Web: www.awprofessional.com Library of Congress Cat alog Num ber:

Guckenheim er, Sam , 1956Soft ware engineering wit h Visual st udio t eam syst em / Sam Guckenheim er, Juan J. Perez. p. cm . I SBN 0- 321- 27872- 0 ( pbk. : alk. paper) 1. Microsoft Visual st udio. 2. Soft ware engineering. 3. Microsoft .NET Fram ework. I . Perez, Juan J. I I . Tit le. QA76.758.G82 2006 005.1- - dc22 2006004369 Copyright © 2006 Sam Guckenheim er All right s reserved. Print ed in t he Unit ed St at es of Am erica. This publicat ion is prot ect ed by copyright , and perm ission m ust be obt ained from t he publisher prior t o any prohibit ed reproduct ion, st orage in a ret rieval syst em , or t ransm ission in any form or by any m eans, elect ronic, m echanical, phot ocopying, recording, or likewise. For inform at ion regarding perm issions, writ e t o:

Pearson Educat ion, I nc. Right s and Cont ract s Depart m ent 75 Arlingt on St reet , Suit e 300 Bost on, MA 02116 Fax: ( 617) 848- 7047 Text print ed in t he Unit ed St at es on recycled paper at R. R. Donnelley in Crawfordsville, I ndiana. First print ing, May 2006

Dedication To m y wife, Monica, whose support m ade t his book possible.

Praise for Software Engineering with Microsoft Visual Studio Team System " Fascinat ing! This book is packed wit h det ails about VSTS' capabilit ies, and t he reason behind why t hese capabilit ies were included in t he VSTS product inform at ion t hat only an int ernal t eam m em ber could provide. Perhaps m ore im port ant ly, each t echnical capabilit y or how- t o inst ruct ion is encased in t he explanat ion of why t he funct ionalit y is crit ical t o you. The book discards t he pit falls wit hin processes of t he past while am plifying t he sweet spot s wit hin t hose sam e processes. I n so doing, it defines t he m et hodology direct ion of t he fut ure and ident ifies t he m et rics for refining and cust om izing t hat m et hodology wit hin your own proj ect s." Mark Michaelis, aut hor of Essent ial C# 2.0 " This book is a m ust read for anyone hoping t o em brace Visual St udio Team Syst em and Microsoft Solut ions Fram ework 4.0 as int ended by t heir creat ors. One of it s key t hem es is 'agilit y wit h account abilit y.' I t explains t he paradigm shift t o a value- up proj ect approach, and describes how Team Syst em enables t his shift . The m any exam ples of how t his approach was applied t o t he developm ent of VSTS bring t he m essage t o life on a m eaningful scale." Aaron Kowall, EDS Applicat ions Port folio Developm ent , I nnovat ion Engineering " Sam Guckenheim er ushers in t he era of t rust wort hy t ransparency t hat will revolut ionize t he way we m anage soft ware developm ent proj ect s. Don't j ust buy Visual St udio Team Syst em ; learn how t o use it t o drive change and reap t he rewards. Sam shows you how." David J. Anderson, aut hor of Agile Managem ent for Soft ware Engineering " I n 250 pages, Sam has capt ured t he essence of Visual St udio Team Syst em . I f you are involved in t he process of m aking soft ware or m anaging soft ware proj ect sas a developer, t est er, proj ect m anager, archit ect , or CI Oyou'll want a copy for everyone on your t eam . The book bot h m akes m odern soft ware engineering pract ices approachable, and does so wit h clear exam ples of how t o im plem ent t hem wit h Team Syst em t ools. Unlike previous books on soft ware m et hodology, t his one does not shy away from put t ing t he principles int o pract ice. Whet her you already have VSTS, are considering it , or j ust want t o im prove your soft ware product ivit y and business alignm ent , you'll find t his full of insight . The book is enj oyable, approachable, and easy t o read in a weekend." Rick LaPlant e, general m anager, Visual St udio Team Syst em , Microsoft " Sam Guckenheim er has been one of t he int ellect ual powerhouses and m ent ors for t he soft ware t est ing com m unit y for years. I t is a pleasure t o see a book from him at last , especially one t hat illust rat es his vision as well as t his one does." Cem Kaner, J.D., Ph.D., professor of soft ware engineering, Florida I nst it ut e of Technology; lead aut hor of Lessons Learned in Soft ware Test ing and Test ing Com put er Soft ware " I n Soft ware Engineering wit h Microsoft Visual St udio Team Syst em , Sam Guckenheim er capt ures t he gest alt of Team Syst em and t he em erging soft ware process paradigm of value- up. Measuring t he value delivered, inst ead of t he long- held paradigm of m easuring work accom plished, is core t o Team Syst em 's design and im plem ent at ion. As a result , you will find

t hat t he unprecedent ed proj ect t ransparency Team Syst em provides im proves t eam int eract ion and proj ect predict abilit y. Moreover, it does so wit hout burdening t eam m em bers wit h t im erobbing overhead. You m ust read t his book t o appreciat e fully t he vision behind Team Syst em and t he virt uous cycle of value- up soft ware developm ent it m akes possible." Rob Caron, cont ent archit ect , Microsoft ; aut hor of Team Syst em Nexus " Sam Guckenheim er is a t echnical diplom at . I n a world where t he guerilla forces of agile m et hods are aligned against t he arm ored legions of CMMI , Sam provides a pat h for coexist ence. This is first and forem ost a book about soft ware engineering. I n discussing flash point s such as planning, docum ent at ion, governance, audit abilit y, and organizat ion, Sam present s t he case for bot h agile and m ore form al pract ices, as well as describing t he opt im al condit ions for each. Even t hough t he m at erial is present ed in t he cont ext of VSTS, t he guidance is universal. Sam writ es t o each of t he roles on a proj ect , providing t hem wit h sound advice regardless of t he 'weight ' t hey have chosen for t heir pract ices. The m at erial is current and t im ely, wit h discussions of service orient ed archit ect ures, Test - Driven Developm ent , and design t echniques developed in t he user int erface com m unit y. Sam 's book is a Very Superior Text on Soft ware." Dr. Bill Curt is, chief process officer, Borland Soft ware Corporat ion; lead aut hor of People Capabilit y Mat urit y Model " Sam Guckenheim er is a t rue advocat e for t he user. Buoyed by Team Syst em , a plat form t hat provides process in a way t hat is aut om at ed by t ools, m anaged by m et rics, and nearly t ransparent t o t he user, he present s an approach t o soft ware engineering t hat is pract ical and achievable wit hout ignoring t he fact t hat we have hard problem s t o solve." Jam es Behling, Accent ure Delivery Met hods lead archit ect , Accent ure " Sam Guckenheim er and I have always walked a com m on road t o im proving support bet ween developm ent and operat ions t eam s. Sam 's book delivers an easy t o underst and, processcent ered approach t o t he best pract ices of soft ware developm ent em bodied in MSF and delivered t hrough Visual St udio Team Syst em . The 'wat erfall' is a failure, but Sam 's book can guide you t hrough t he use of Visual St udio Team Syst em t o rapid developm ent wit h j ust enough process t o get t he j ob done." Brian Whit e, senior direct or of product m anagem ent , iConclude, I nc., aut hor of Soft ware Configurat ion Managem ent St rat egies and Rat ional ClearCase: A Pract ical I nt roduct ion " Transparency is a crit ical elem ent in t oday's agile environm ent . Sam was and st ill is inst rum ent al in creat ing t he overall archit ect ure t hat provides t he level of int egrat ion and t ransparency in Team Syst em necessary t o scale agile proj ect s t o larger t eam s. This t ransparency, if used in an environm ent fost ering t rust and personal safet y, can creat e m ore product ive developm ent t eam s while propagat ing t he discipline of agile m et hods. Report ing inform at ion such as velocit y becom es effort less. Now t he ent ire soft ware developm ent t eam , including business analyst s, archit ect s, and t est ers, can j oin in t he agile process." Granville " Randy" Miller, co- aut hor of A Pract ical Guide t o eXt rem e Program m ing and Advanced Use Case Modeling " Can you im agine having a Business Process Re- engineering ( BPR) t ool for soft ware engineering ( SE) ? A t ool t hat could act ually help t he I T indust ry get leaner? This is what t his book is all about ! I t 's an eye opener: a door t o a new era of SE. The quest ion at st ake in t his book is sim ple: Could MSFT VSTS em power our I T indust ry t o becom e m ore of a science and less of an art t hat it has been up t o now? Sam Guckenheim er explains not only why t his could be t he case, but also gives m any t ips on how an ent ire SE t eam could evolve t o be m ore product ive and efficient , wit hout m anual overhead."

Francis T. Delgado, senior program m anager, Avanade, I nc.

Microsoft .NET Development Series John Mont gom ery, Series Advisor Don Box, Series Advisor Mart in Heller, Series Edit or The M icr osoft .N ET D e ve lopm e n t Se r ie s is support ed and developed by t he leaders and expert s of Microsoft developm ent t echnologies including Microsoft archit ect s and DevelopMent or inst ruct ors. The books in t his series provide a core resource of inform at ion and underst anding every developer needs in order t o writ e effect ive applicat ions and m anaged code. Learn from t he leaders how t o m axim ize your use of t he .NET Fram ework and it s program m ing languages.

Titles in the Series Brad Abram s, .NET Fram ework St andard Library Annot at ed Reference Volum e 1: Base Class Library and Ext ended Num erics Library, 0- 321- 15489- 4 Brad Abram s and Tam ara Abram s, .NET Fram ework St andard Library Annot at ed Reference, Volum e 2: Net working Library, Reflect ion Library, and XML Library, 0- 321- 19445- 4 Keit h Ballinger, .NET Web Services: Archit ect ure and I m plem ent at ion, 0- 321- 11359- 4 Bob Beauchem in, Niels Berglund, Dan Sullivan, A First Look at SQL Server 2005 for Developers, 0321- 18059- 3 Don Box wit h Chris Sells, Essent ial .NET, Volum e 1: The Com m on Language Runt im e, 0- 201- 73411- 7 Keit h Brown, The .NET Developer's Guide t o Windows Securit y , 0- 321- 22835- 9 Eric Cart er and Eric Lippert , Visual St udio Tools for Office: Using C# wit h Excel, Word, Out look, and I nfoPat h, 0- 321- 33488- 4 Eric Cart er and Eric Lippert , Visual St udio Tools for Office: Using Visual Basic 2005 wit h Excel, Word, Out look, and I nfoPat h, 0- 321- 41175- 7 Mahesh Chand, Graphics Program m ing wit h GDI + , 0- 321- 16077- 0 Krzyszt of Cwalina and Brad Abram s, Fram ework Design Guidelines: Convent ions, I diom s, and Pat t erns for Reusable .NET Libraries, 0- 321- 24675- 6 Anders Hej lsberg, Scot t Wilt am ut h, Pet er Golde, The C# Program m ing Language, 0- 321- 15491- 6 Alex Hom er, Dave Sussm an, Mark Fussell, ADO.NET and Syst em .Xm l v. 2.0The Bet a Version, 0- 32124712- 4 Alex Hom er, Dave Sussm an, Rob Howard, ASP.NET v. 2.0The Bet a Version, 0- 321- 25727- 8 Jam es S. Miller and Susann Ragsdale, The Com m on Language I nfrast ruct ure Annot at ed St andard, 0-

321- 15493- 2 Christ ian Nagel, Ent erprise Services wit h t he .NET Fram ework: Developing Dist ribut ed Business Solut ions wit h .NET Ent erprise Services, 0- 321- 24673- X Brian Noyes, Dat a Binding wit h Windows Form s 2.0: Program m ing Sm art Client Dat a Applicat ions wit h .NET, 0- 321- 26892- X Frit z Onion, Essent ial ASP.NET wit h Exam ples in C# , 0- 201- 76040- 1 Frit z Onion, Essent ial ASP.NET wit h Exam ples in Visual Basic .NET, 0- 201- 76039- 8 Ted Pat t ison and Dr. Joe Hum m el, Building Applicat ions and Com ponent s wit h Visual Basic .NET, 0201- 73495- 8 Dr. Neil Roodyn, eXt rem e .NET: I nt roducing eXt rem e Program m ing Techniques t o .NET Developers, 0321- 30363- 6 Chris Sells, Windows Form s Program m ing in C# , 0- 321- 11620- 8 Chris Sells and Just in Geht land, Windows Form s Program m ing in Visual Basic .NET, 0- 321- 12519- 3 Paul Vick, The Visual Basic .NET Program m ing Language, 0- 321- 16951- 4 Dam ien Wat kins, Mark Ham m ond, Brad Abram s, Program m ing in t he .NET Environm ent , 0- 20177018- 0 Shawn Wilderm ut h, Pragm at ic ADO.NET: Dat a Access for t he I nt ernet World, 0- 201- 74568- 2 Paul Yao and David Durant , .NET Com pact Fram ework Program m ing wit h C# , 0- 321- 17403- 8 Paul Yao and David Durant , .NET Com pact Fram ework Program m ing wit h Visual Basic .NET, 0- 32117404- 6 For m or e in for m a t ion go t o w w w .a w pr ofe ssion a l.com / m sdot n e t se r ie s/

About the Author Sa m Gu ck e n h e im e r has 25 years of experience as archit ect , developer, t est er, product m anager, proj ect m anager, and general m anager in t he soft ware indust ry in t he U.S. and Europe. Current ly, Sam is t he group product planner for Microsoft Visual St udio Team Syst em . I n t his capacit y, he act s as chief cust om er advocat e, responsible for t he end- t o- end ext ernal design of t he next releases of t hese product s. Prior t o j oining Microsoft in 2003, Sam was direct or of Product Line St rat egy at Rat ional Soft ware Corporat ion, now t he Rat ional Division of I BM. He holds five pat ent s on soft ware lifecycle t ools. A frequent speaker at indust ry conferences, Sam is a Phi Bet a Kappa graduat e of Harvard Universit y. Sam lives in t he Puget Sound area wit h his wife and t hree of his four children.

Foreword For alm ost t en years I 've been encouraging Sam Guckenheim er t o writ e a book about soft ware engineering. " Oh no, I 'm not ready," was t he invariable reply. Wit h t he release of Visual St udio Team Syst em , Sam no longer had an excuse: he really had t o explain his ideas about soft ware engineering t o help people m ake sense of t he product t hat em bodies t hem . I t 's great t o see t hat t urn int o a book t hat put s equal weight on pract icum and t heory, rat her t han a book- lengt h product advert isem ent or a vague discussion of t he philosophy of soft ware engineering. I like t he concret e exam ples here: t hey m ake t he concept s com e alive. One key concept in t his book is t hat of value- up processes. Sam believes t hat we are facing a huge paradigm shift in t he way we approach soft ware, which rings t rue. The work- down paradigm has led t o a num ber of problem s wit h t he soft ware developm ent process and ult im at ely t o a high rat e of failed proj ect s. Whet her t he value- up paradigm will solve t he problem s wit hout creat ing new ones, of course, rem ains t o be seen. I n t he past , t he pract ice of soft ware m et rics has not kept up wit h it s pot ent ial, largely because of t he high cost of collect ing dat a. As Sam explains in t his book, inst rum ent ing daily act ivit ies t o allow painless dat a collect ion opens up a new set of opport unit ies for m eaningful m et rics. Sam hasn't st opped t here; he has applied som e of t he m ore int erest ing t echniques from lean proj ect m anagem ent t o dem onst rat e how t o t roubleshoot soft ware proj ect s on a daily basis. That also enables t he reliable applicat ion of value- up processes. For alm ost a decade, a num ber of ideas have percolat ed in t he various areas of soft ware engineering: in program m ing, user experience, t est ing, and archit ect ure. Sam has pulled t he best of t hese t oget her t o apply across t he ent ire soft ware lifecycle. I t rust t hat you'll enj oy t hem as m uch as I have.

I var Jacobson, Ph.D. I var Jacobson Consult ing LLC

Preface Why I Wrote This Book I j oined Microsoft in 2003 t o work on Visual St udio Team Syst em ( VSTS) , t he new product line t hat was j ust released at t he end of 2005. As t he group product planner, I have played chief cust om er advocat e, a role t hat I have loved. I have been in t he I T indust ry for t went y- som e years, spending m ost of m y career as a t est er, proj ect m anager, analyst , and developer. As a t est er, I 've always underst ood t he t heoret ical value of advanced developer pract ices, such as unit t est ing, code coverage, st at ic analysis, and m em ory and perform ance profiling. At t he sam e t im e, I never underst ood how anyone had t he pat ience t o learn t he obscure t ools t hat you needed t o follow t he right pract ices. As a proj ect m anager, I was always t roubled t hat t he only decent dat a we could get was about bugs. Driving a proj ect from bug dat a alone is like driving a car wit h your eyes closed and only t urning t he wheel when you hit som et hing. You really want t o see t he right indicat ors t hat you are on course, not j ust feel t he bum ps when you st ray off it . Here t oo, I always underst ood t he value of m et rics, such as code coverage and proj ect velocit y, but I never underst ood how anyone could realist ically collect all t hat st uff. As an analyst , I fell in love wit h m odeling. I t hink visually, and I found graphical m odels com pelling ways t o docum ent and com m unicat e. But t he m odels always got out of dat e as soon as it cam e t im e t o im plem ent anyt hing. And t he m odels j ust didn't handle t he key concerns of developers, t est ers, and operat ions. And in all t hese cases, I was frust rat ed by how hard it was t o connect t he dot s for t he whole t eam . I loved t he idea in Scrum ( one of t he agile processes) of a " single product backlog" one place where you could see all t he workbut t he t ools people could act ually use would fragm ent t he work every which way. What do t hese requirem ent s have t o do wit h t hose t asks, and t he m odel elem ent s here, and t he t est s over t here? And where's t he source code in t hat m ix? From a hist orical perspect ive, I t hink I T t urned t he corner when it st opped t rying t o aut om at e m anual processes and inst ead asked t he quest ion, " Wit h aut om at ion, how can we reengineer our core business processes?" That 's when I T st art ed t o deliver real business value. They say t he cobbler's children go shoeless. That 's t rue for I T, t oo. While we've been busy aut om at ing ot her business processes, we've largely neglect ed our own. Virt ually all t ools t arget ed for I T professionals and t eam s seem t o st ill be aut om at ing t he old m anual processes. Those processes required high overhead before aut om at ion, and wit h aut om at ion, t hey st ill have high overhead. How m any t im es have you gone t o a one- hour proj ect m eet ing where t he first ninet y m inut es were an argum ent about whose num bers were right ? Now, wit h Visual St udio Team Syst em , we are seriously asking, " Wit h aut om at ion, how can we reengineer our core I T processes? How can we rem ove t he overhead from following good process? How can we m ake all t hese different roles individually m ore product ive while int egrat ing t hem as a high- perform ance t eam ?"

Who Should Read This Book This book is writ t en for a soft ware t eam t hat is considering running a soft ware proj ect using VSTS. This book is about t he why, not t he how .[ 1 ] What are t he guiding ideas behind VSTS? Why are cert ain ideas present ed in cert ain ways? For exam ple, why are so m any t hings called work it em s? What does t he m et rics warehouse m easure? Why would you use t hose part icular report s? I t has been m y experience t im e and t im e again t hat knowledgeable, skillful, experienced people bring uneven st art ing assum pt ions t o soft ware proj ect s. What appear t o be self- evident t rut hs t o one person are folk m yt hs t o anot her, and one person's com m on wisdom is anot her's discovery. This issue is exacerbat ed by a nat ural em phasis on funct ional roles, which are oft en baked int o career ladders. I cert ainly believe t hat t here are expert developers, expert t est ers, expert archit ect s, expert business analyst s, and expert proj ect m anagers, but delivering cust om er value requires collaborat ion across all disciplines. At t em pt s t o opt im ize one special role in isolat ion from t he ot hers do not necessarily im prove t he delivery of result s as t he cust om er sees t hem . One way of solving t he discrepancies has been t o have an onsit e coach who can lead t he t eam t hrough a consist ent set of processes. Coaches are great , but not everyone has t he luxury of working wit h one. So, because I cannot ship you an on- dem and coach, I 've writ t en t his book. This is not a st ep- by- st ep t ut orial in t he sense of a user m anual t hat t ells you where t o click in what sequence. Plent y of good docum ent at ion on t hose t opics is available wit h VSTS, and I reference it where appropriat e. Rat her, t his book offers a fram ework for t hinking about soft ware proj ect s in a way t hat can be direct ly t ooled by VSTS. I ndeed, we built VSTS t o run soft ware proj ect s t his way. This book is also not a survey of soft ware engineering lit erat ure. Dozens, perhaps hundreds, of books have been writ t en about soft ware engineering in t he last fort y years. I do not recap t hem here, and I do not cover all t he m at erial t hat t hey do. I expect t he crit icism from m any expert s t hat som e of m y argum ent s go wit hout saying nowadays. Unfort unat ely, as Freud point ed out , what goes wit hout saying is oft en not said at all. As a result , differences in t eam m em bers' assum pt ions are not exposed unt il t he wrong argum ent happens. So if you want t o fault m e for st at ing t oo m any obvious t hings, I 'm guilt y as charged. I present enough Team Syst em t heory and pract ice exam ples t o describe a realist ic process for m ost m ainst ream I T proj ect s and t eam s. I t m ay not be form al enough for avionics soft ware t hat requires FAA approval; it m ay not be loose enough for a t hree- person t eam co- locat ed in a garage.

How to Read This Book VSTS includes process guidance called Microsoft Solut ions Fram ework ( MSF) , which includes t he cent ral concept of a t eam m odel based on a t eam of peers. The t eam m odel allows for different scales of specializat ion. MSF defines seven const it uencies, or point s of view, t hat m ust be represent ed on a successful proj ect , and it includes recom m endat ions for scaling up and down. I call out t hese point s of view t hroughout t he book wit h icons t hat look like t his:

Figu r e p.1 .

M icr osoft Solu t ion s Fr a m e w or k in t r odu ce s a m ode l of a t e a m of pe e r s, a n ch or e d in se ve n vie w poin t s t h a t n e e d t o be r e pr e se n t e d for a su cce ssfu l pr oj e ct . M SF for CM M I Pr oce ss

I m pr ove m e n t spe cia lize s t h e se ve n in t o e igh t e e n , w h e r e a s M SF for Agile Soft w a r e D e ve lopm e n t r e a lize s t h e se ve n w it h six r ole s, clu st e r in g pr odu ct m a n a ge m e n t a n d u se r e x pe r ie n ce , a n d offe r s gu ide lin e s t o r e du ce t h e list t o t h r e e .

This book is writ t en for t he t eam as a whole. I t present s inform at ion in a st yle t hat will help all t eam m em bers get a sense of each ot her's viewpoint . However, role- specific sect ions are called out so t hat you can focus on or skim over port ions as needed for your specific roles. I 've t ried t o keep t he t opics at a level t hat is engaging t o all t eam m em bers and not arcane for any. ( For som e, t his choice m ay reinforce t he crit icism of sim plicit y.) I n t his age of specializat ion, I t hink it is im port ant t o have at least t his level of cont ract wit h and expect at ions of your colleagues in ot her specialt ies. I f you're in a hurry, you can use t he const it uency icons as a guide t o t he role- relat ed t opics t hat int erest you m ost .

Pointers to Documentation As I said, t his is not a how- t o book. Where det ails of VSTS or it s docum ent at ion are appropriat e, you will see a point er t o it , like t his exam ple:

Microsoft Developer Network Subscription (MSDN) Every VSTS includes a subscript ion t o Microsoft Developer Net work. St art at ht t p: / / m sdn.m icrosoft .com / t eam syst em and follow t he links for Reference ® Product Docum ent at ion. There m ay be a num ber of t erm s for which you want t o check t he usage. To look t hem up, please consult t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Visual St udio Team Syst em Glossary

I m ade t his choice because I assum e t hat m ost of t he t im e t hat you are reading t his book, you are not sit t ing in front of a com put er, and occasionally you will want t o go back and t ry som et hing hands- on. When you're j ust reading, you can skip t hese point ers.

Other People's Ideas My goal in t his book is t o int roduce t he ideas behind VSTS, not t o claim t he ideas as original. VSTS was designed from t he ground up t o enable different processes t o be applied t o fit different organizat ions and proj ect s. VSTS, and correspondingly t his book, m ake liberal use of good pract ices t hat have been developed by t he soft ware com m unit y. Where possible, I 've t ried t o capt ure relevant sources in endnot es. I f you're not int erest ed in t he references, you don't need t o read t he endnot es.

Enough About VSTS to Get You Started Reviewers of early draft s of t his book com plained t hat I did not explain clearly enough what is in VSTS, so I 'm put t ing a product int roduct ion in t he following sidebar, st raight from a Microsoft Web page. There are current ly four VSTS client product s, and t here m ay be m ore in t he fut ure, but I do not dist inguish t hem because t he value is in t he Team Suit e. Of course, Microsoft let s you buy t he funct ionalit y à la cart e, but I 'm going t o keep t hings sim ple. So when I writ e about " VSTS" or " Team Syst em ," assum e t hat I am writ ing about t he Team Suit e. Part of VSTS is Microsoft Solut ions Fram ework ( MSF) , shown as t he " Process Guidance" box in Figure P.2. MSF com es in t wo form s out of t he box and can be cust om ized int o count less variat ions. The t wo st andard ones are • MSF for Agile Soft ware Developm ent • MSF for CMMI Process I m provem ent I 'll describe t hese in a lit t le m ore dept h lat er, but basically, if your organizat ion is new t o soft ware process, st art wit h MSF for Agile Soft ware Developm ent . I f you need m ore form alit y due t o geographic dist ribut ion, process im provem ent program s, com pliance audit s, or t he desire for CMMI appraisal, t hen you should consider MSF for CMMI Process I m provem ent .

Figu r e P.2 .

Visu a l St u dio 2 0 0 5 Te a m Syst e m is com pr ise d of fou r clie n t pr odu ct s a n d on e se r ve r pr odu ct .

Unless it 's necessary t o dist inguish t hese variant s, I will I st ick t o concept s t hat are com m on t o bot h.

Disclaimer

Finally, I need t o be clear t hat t he views in t his book are m y own and not necessarily t hose of Microsoft Corporat ion. Alt hough I am a Microsoft em ployee, I am writ ing on m y own behalf, and not for t he com pany. Don't blam e Microsoft for t he opinions I express and t he errors I m ake here ( unless you want t o t ell t hem t hey m ade a bad hire) , but blam e m e inst ead. You can flam e m e on m y blog direct ly at ht t p: / / blogs.m ircosoft .com / sam / .

Endnotes 1 . For a how- t o book, see Will St ot t and Jam es Newkirk, Visual St udio Team Syst em Bet t er Soft ware Developm ent for Agile Team s ( Bost on, MA: Addison- Wesley, 2006) .

Acknowledgments There are several people whose prodding dist inct ly helped m e writ e t his book. Let m e st art by acknowledging m y edit or, Karen Get t m an, who was willing t o consider a first - t im e aut hor wit h a vision and proposal. I var Jacobson and Cem Kaner have separat ely been im port ant m ent ors, who have encouraged m e t o writ e for m any years. Next is Rick LaPlant e, who is st ill m y boss at m y day j ob. Rick t ook a bet on m e when he hired m e as group product planner for Visual St udio Team Syst em , and he has been a com plet ely support ive m anager. Beyond Rick are a couple hundred colleagues who have m ade VSTS t he product t hat it is. Every cont act wit h t hem has been and cont inues t o be an int ellect ual charge. As you will see, I rely very heavily on t he work of Granville ( " Randy" ) Miller and David J. Anderson, who creat ed MSF for Agile Soft ware Developm ent and MSF for CMMI Process I m provem ent . We had endless debat es and discoveries in creat ing t he inst ances of MSF v4, and t hat learning has shaped what you read here in a m aj or way. Juan J. Perez, m y coaut hor, and Kim Tapia St . Am ant of Personify Design m ade t he rich exam ples and illust rat ions here possible. Working wit h t hem has been a great pleasure. Finally, I am indebt ed t o a large num ber of reviewers, including Jeff Beehler, Jam es Behling, Charlie Bess, Rossen Blagoev, Rob Caron, Wendy Chun, Kevin P. Davis, Crist of Falk, Linda Fernandez, Ken Garove, Bill Gibson, Mart in Heller, Bij an Javidi, Yulin Jin, Cem Kaner, Chris Kinsm an, Aaron Kowall, Clem ent ino Mendonca, Thom as Murphy, Gary Pollice, Tom as Rest repo, Johanna Rot hm an, Joel Sem eniuk, Will St ot t , Dan Sullivan, David Trowbridge, Mike Turner, Kum ar Vadapart y, and Pet er William s. Kim Boedigheim er, Ben Lawson, and Michael Thurst on of Addison- Wesley were ext rem ely helpful in t he endgam e. Wit hout t he advice and suggest ions of all t hese reviewers, t his book would have been a sm all fract ion of what it becam e.

Sam Guckenheim er Redm ond, WA January 2006

Chapter 1. A Value-Up Paradigm " A t heory should be as sim ple as possible, but no sim pler." Albert Einst ein

Figu r e 1 .1 .

Ein st e in 's Th e or y of Spe cia l Re la t ivit y w a s t h e foca l poin t of a pa r a digm sh ift in ou r u n de r st a n din g of ph ysics. I t ca ppe d for t y ye a r s of de ba t e on t h e m ost ve x in g t e ch n ica l ch a lle n ge s of h is da yh ow t o syn ch r on ize clock s a n d h ow t o a ccu r a t e ly dr a w m a ps ove r lon g dist a n ce s.

[View full size image]

A Paradigm Shift Paradigm shift s com e in fit s and st art s, as old t heories can no longer explain t he world as observed. [ 1 ] A post er child for t he scient ific paradigm shift is Albert Einst ein's Theory of Special Relat ivit y, published in 1905. Einst ein's work reduced Newt onian m echanics t o a special case, set t led fort y years of debat e on t he nat ure of t im e and synchronicit y, and set t he agenda for m uch of science, t echnology, and world affairs of t he t went iet h cent ury. According t o a post hum ous legend m any of us learned in school, Einst ein was a solit ary t heoret ician whose day j ob reviewing pat ent applicat ions was a m ere dist ract ion from his passionat e pursuit of physics. Yet t his popular view of Einst ein is m isguided. I n fact , t he m aj orit y of pat ent applicat ions t hat Einst ein reviewed concerned t he very physics problem t hat fascinat ed him how t o synchronize t im e over dist ance for m ult iple pract ical purposes, such as creat ing railroad schedules, m arit im e chart s, and accurat e t errit orial m aps in an age of colonial expansion. I ndeed, t he synchronizat ion of t im e was a great t echnological problem of t he age, for which special relat ivit y becam e a m at hem at ical solut ion, capping decades of debat e. Einst ein was not t he only person t o solve t he m at hem at ical problem in 1905t he far m ore prom inent Henri Poincaré produced an alt ernat ive t hat has long since been forgot t en. [ 2 ] Why is Einst ein's solut ion t he one t aught in every physics class t oday? Poincaré's calculat ions relied on t he " et her," a supposed m edium of space t hat had pervaded ninet eent h- cent ury physics. Einst ein's Special Relat ivit y, on t he ot her hand, used m uch sim pler calculat ions t hat required no et her. This was t he first not able exam ple of t he principle at t ribut ed t o Einst ein, also post hum ously, t hat " a t heory should be as sim ple as possible, but no sim pler."

Three Forces to Reconcile A shift sim ilar t o t he cont rast ing views of physics 100 years ago has been occurring t oday in soft ware developm ent . On a weekend in 2001, sevent een soft ware lum inaries convened t o discuss " light weight m et hods." At t he end of t he weekend, t hey launched t he Agile Alliance, init ially charged around t he Agile Manifest o. [ 3 ] I nit ially, it was a rallying cry for t hose who saw cont em porary soft ware processes as sim ilar t o t he " et her" of ninet eent h- cent ury physicsan unnecessary com plexit y and an im pedim ent t o product ivit y. Five years lat er, " agilit y" is m ainst ream . Every indust ry analyst advocat es it , every business execut ive espouses it , and everyone t ries t o get m ore of it . At t he sam e t im e, t wo ext ernal econom ic fact ors cam e int o play. One is global com pet it ion . The convergence of econom ic liberalizat ion, increased com m unicat ions bandwidt h, and a highly skilled labor force in em erging m arket s m ade t he out sourcing of soft ware developm ent t o lower- wage count ries ( especially I ndia) profit able. [ 4 ] The I ndian consult ancies, in t urn, needed t o guarant ee t heir qualit y t o Am erican and European cust om ers. Many lat ched ont o Capabilit y Mat urit y Model I nt egrat ion ( CMMI ) from t he Soft ware Engineering I nst it ut e at Carnegie Mellon Universit y. [ 5 ] CMMI epit om ized t he heavyweight processes against which t he agilist s rebelled, and it was considered t oo expensive t o be pract ical out side of t he defense indust ry. The offshorers, wit h t heir cost advant age, did not m ind t he expense and could t urn t he credent ial of a CMMI appraisal int o a com pet it ive advant age. The second econom ic fact or is increased at t ent ion t o regulat ory com pliance aft er t he lax business pract ices of t he 1990s. I n t he Unit ed St at es, t he Sarbanes- Oxley Act of 2002 (SOX) epit om izes t his em phasis by holding business execut ives crim inally liable for financial m isrepresent at ions. This m eans t hat soft ware and syst em s t hat process financial inform at ion are subj ect t o a level of scrut iny and audit m uch great er t han previously known.

These forcesagilit y, out sourcing/ offshoring, and com pliancecannot be resolved wit hout a paradigm shift in t he way we approach t he soft ware lifecycle. The m odern econom ics require agilit y wit h account abilit y. Closing t he gap requires a new approach, bot h t o process it self and t o it s t ooling.

What Software Is Worth Building? To overcom e t he gap, you m ust recognize t hat soft ware engineering is not like ot her engineering. When you build a bridge, road, or house, for exam ple, you can safely st udy hundreds of very sim ilar exam ples. I ndeed, m ost of t he t im e, econom ics dict at e t hat you build t he current one alm ost exact ly like t he last t o t ake t he risk out of t he proj ect . Wit h soft ware, if som eone has built a syst em j ust like you need, or close t o what you need, t hen chances are you can license it com m ercially ( or even find it as freeware) . No sane business is going t o spend m oney on building soft ware t hat it can buy m ore econom ically. Wit h t housands of soft ware product s available for com m ercial license, it is alm ost always cheaper t o buy. Because t he decision t o build soft ware m ust be based on sound ret urn on invest m ent and risk analysis, t he soft ware proj ect s t hat get built will alm ost invariably be t hose t hat are not available com m ercially. This business cont ext has a profound effect on t he nat ure of soft ware proj ect s. I t m eans t hat soft ware proj ect s t hat are easy and low risk, because t hey've been done before, don't get funded. The only new soft ware developm ent proj ect s undert aken are t hose t hat haven't been done before or t hose whose predecessors are not publicly available. This business realit y, m ore t han any ot her fact or, is what m akes soft ware developm ent so hard and risky, which m akes at t ent ion t o process so im port ant . [ 6 ]

Contrasting Paradigms The inherent uncert aint y in soft ware proj ect s m akes it difficult t o est im at e t asks correct ly, which creat es a high variance in t he accuracy of t he est im at es. A com m on m isconcept ion is t hat t he variance is accept able because t he posit ive and negat ive variat ions average out . However, because soft ware proj ect s are long chains of dependent event s, t he variat ion it self accum ulat es in t he form of downst ream delays. [ 7 ] Unfort unat ely, m ost accept ed proj ect m anagem ent wisdom com es from t he world of roads and bridges. I n t hat world, design risks are low, design cost is sm all relat ive t o build cost , and t he opport unit y t o deliver increm ent al value is rare. ( You can't drive across a half- finished bridge! ) Wit h t his st yle of proj ect m anagem ent , you det erm ine an engineering design early, carefully decom pose t he design int o im plem ent at ion t asks, schedule and resource t he t asks according t o t heir dependencies and resource availabilit y, and m onit or t he proj ect by checking off t asks as com plet ed ( or t racking percent ages com plet ed) . For sim plicit y, I 'll call t his st yle of proj ect m anagem ent t he work- down approach because it is easily envisioned as burning down a list of t asks. The work- down approach succeeds for engineering proj ect s wit h low risk, low variance, and wellunderst ood design. Many I T proj ect s, for exam ple, are cust om izat ions of com m ercial- off- t he- shelf soft ware ( COTS) , such as ent erprise resource planning syst em s. Oft en, t he developm ent is a sm all part of t he proj ect relat ive t o t he business analysis, proj ect m anagem ent , and t est ing. Typically, t hese proj ect s have lower variabilit y t han new developm ent proj ect s, so t he wisdom of roads and bridges works bet t er for t hem t han for new developm ent . Since 1992, [ 8 ] t here has been a growing challenge t o t he work- down wisdom about soft ware process. No single t erm has capt ured t he em erging paradigm , but for sim plicit y, I 'll call t his t he value- up approach. And as happens wit h new paradigm s, t he value- up view has appeared in fit s and st art s ( see Figure 1.2 ) .

Figu r e 1 .2 .

Th e a t t it u din a l diffe r e n ce be t w e e n w or k - dow n a n d va lu e - u p is in t h e pr im a r y m e a su r e m e n t . W or k - dow n t r e a t s t h e pr oj e ct a s a fix e d st ock of t a sk s a t som e cost t h a t n e e d com ple t ion a n d m e a su r e s t h e e x pe n dit u r e a ga in st t h ose t a sk s. Va lu e - u p m e a su r e s va lu e de live r e d a t e a ch poin t in t im e a n d t r e a t s t h e in pu t s a s va r ia ble flow s r a t h e r t h a n a fix e d st ock .

An exam ple of t he value- up school is t he agile proj ect m anagem ent m anifest o Declarat ion of I nt erdependence. [ 9 ] I t st at es six principles t hat charact erize value- up: We increase ret urn on invest m ent by m aking cont inuous flow of value our focus. We deliver reliable result s by engaging cust om ers in frequent int eract ions and shared ownership. We expect uncert aint y and m anage for it t hrough it erat ions, ant icipat ion, and adapt at ion. We unleash creat ivit y and innovat ion by recognizing t hat individuals are t he ult im at e source of value, and creat ing an environm ent where t hey can m ake a difference. We boost perform ance t hrough group account abilit y for result s and shared responsibilit y for t eam effect iveness. We im prove effect iveness and reliabilit y t hrough sit uat ionally specific st rat egies, processes, and pract ices. Behind t hese principles is a significant ly different point of view about pract ices bet ween t he work- down and value- up m indset s. Table 1.1 below sum m arizes t he differences.

Ta ble 1 .1 . At t it u din a l D iffe r e n ce s Be t w e e n W or k - D ow n a n d Va lu e - Up Pa r a digm s Cor e Assu m pt ion W or k - D ow n At t it u de

Va lu e - Up At t it u de

Planning and change process

Planning and design are t he m ost im port ant act ivit ies t o get right . You need t o do t hese init ially, est ablish account abilit y t o plan, m onit or against t he plan, and carefully prevent change from creeping in.

Change happens; em brace it . Planning and design will cont inue t hrough t he proj ect . Therefore, you should invest in j ust enough planning and design t o underst and risk and t o m anage t he next sm all increm ent .

Prim ary m easurem ent

Task com plet ion. Because we know t he st eps t o achieve t he end goal, we can m easure every int erm ediat e deliverable and com put e earned value running as t he percent age of hours planned t o be spent by now versus t he hours planned t o be spent t o com plet ion.

Only deliverables t hat t he cust om er values ( working soft ware, com plet ed docum ent at ion, et c.) count . You need t o m easure t he flow of t he work st ream s by m anaging queues t hat deliver cust om er value and t reat all int erim m easures skept ically.

Cor e Assu m pt ion W or k - D ow n At t it u de

Va lu e - Up At t it u de

Definit ion of qualit y Conform ance t o specificat ion. That 's why you need t o get t he specs right at t he beginning.

Value t o t he cust om er. This percept ion can ( and probably will) change. The cust om er m ight not be able t o art iculat e how t o deliver t he value unt il working soft ware is init ially delivered. Therefore, keep opt ions open, opt im ize for cont inual delivery, and don't specify t oo m uch t oo soon.

Accept ance of variance

Tasks can be ident ified and est im at ed in a det erm inist ic way. You don't need t o pay at t ent ion t o variance.

Variance is part of all process flows, nat ural and m an- m ade. To achieve predict abilit y, you need t o underst and and reduce t he variance.

I nt erm ediat e work product s

Docum ent s, m odels, and ot her int erm ediat e art ifact s are necessary t o decom pose t he design and plan t asks, and t hey provide t he necessary way t o m easure int erm ediat e progress.

I nt erm ediat e docum ent at ion should m inim ize t he uncert aint y and variat ion in order t o im prove flow. Beyond t hat , t hey are unnecessary.

Troubleshoot ing approach

The const raint s of t im e, resource, funct ionalit y, and qualit y det erm ine what you can achieve. I f you adj ust one, you need t o adj ust t he ot hers. Cont rol change carefully t o m ake sure t hat t here are no unm anaged changes t o t he plan.

The const raint s m ay or m ay not be relat ed t o t im e, resource, funct ionalit y, or qualit y. I nst ead, ident ify t he prim ary bot t leneck in t he flow of value, work it unt il it is no longer t he prim ary one, and t hen at t ack t he next one. Keep reducing variance t o ensure sm oot her flow.

Approach t o t rust

People need t o be m onit ored and com pared t o st andards. Managem ent should use incent ives t o reward individuals for t heir perform ance relat ive t o t he plan.

Pride of workm anship and t eam work are m ore effect ive m ot ivat ors t han individual incent ives. Trust wort hy t ransparency, where all t eam m em bers can see t he overall t eam 's perform ance dat a, works bet t er t han m anagem ent direct ives.

Attention to Flow Cent ral t o t he value- up paradigm is an em phasis on flow . There are t wo discret e m eanings of flow, and bot h are significant in planning soft ware proj ect s. First , flow is t he hum an experience of perform ing expert ly, as Mihaly Csikszent m ihalyi explains in Flow: The Psychology of Opt im al Experience: We have seen how people describe t he com m on charact erist ics of opt im al experience: a sense t hat one's skills are adequat e t o cope wit h t he challenges at hand, in a goal- direct ed, rule- bound act ion syst em t hat provides clear clues as t o how well one is perform ing. Concent rat ion is so int ense t hat t here is no at t ent ion left over t o t hink about anyt hing irrelevant , or t o worry about problem s. Self- consciousness disappears, and t he sense of t im e becom es dist ort ed. An act ivit y t hat produces such experiences is so grat ifying t hat people are willing t o do it for it s own sake, wit h lit t le concern for what t hey will get out of it , even when it is difficult , or dangerous.[ 10] This m eaning of flow is cit ed heavily by advocat es of eXt rem e Program m ing ( XP) and ot her pract ices t hat focus on individual perform ance. The second m eaning of flow is t he flow of cust om er value as t he prim ary m easure of t he syst em of delivery. David J. Anderson sum m arizes t his view in Agile Managem ent for Soft ware Engineering: Flow m eans t hat t here is a st eady m ovem ent of value t hrough t he syst em . Client - valued funct ionalit y is m oving regularly t hrough t he st ages of t ransform at ionand t he st eady arrival of t hroughput wit h working code being delivered. [ 11] I n t his paradigm , you do not m easure planned t asks com plet ed as t he prim ary indicat or of progress; you count unit s of value delivered. Your rat es of progress in t hroughput of delivered value and st age of com plet ion at t he unit s of value are t he indicat ors t hat you use for planning and m easurem ent . Correspondingly, t he flow- of- value approach forces you t o underst and t he const raint s t hat rest rict t he flow. You t une t he end- t o- end flow by ident ifying t he m ost severe bot t leneck or inefficiency your process, fixing it , and t hen t ackling t he next m ost severe one. As Anderson explains: The developm ent m anager m ust ensure t he flow of value t hrough t he t ransform at ion processes in t he syst em . He is responsible for t he rat e of product ion out put from t he syst em and t he t im e it t akes t o process a single idea t hrough t he syst em . To underst and how t o im prove t he rat e of product ion and reduce t he lead t im e, t he developm ent m anager needs t o underst and how t he syst em works, be able t o ident ify t he const raint s, and m ake appropriat e decisions t o prot ect , exploit , subordinat e, and elevat e t he syst em processes.[ 12] A flow- based approach t o planning and proj ect m anagem ent requires keeping int erm ediat e work- inprocess t o a m inim um , as shown in Figure 1.3 . This m it igat es t he risk of lat e discovery of problem s and unexpect ed bubbles of required rework.

Figu r e 1 .3 .

M e a su r in g flow of sce n a r io com ple t ion on a da ily ba sis sh ow s t h e r h yt h m of pr ogr e ss a n d qu ick ly ide n t ifie s bot t le n e ck s t h a t ca n be a ddr e sse d a s t h e y a r ise .

[View full size image]

Figure 1.3 shows how t he cont inuous m easurem ent of flow can illum inat e bot t lenecks as t hey are form ing. Planned work for t he it erat ion is progressing well t hrough developm ent ( Act ive t urning t o Resolved) , but is increasingly get t ing st uck in t est ing ( Resolved t o Closed) . This accum ulat es as t he bulge of work- in- process in t he m iddle band. I f you t racked developm ent only ( t he reduct ion in Act ive work it em s) , you would expect com plet ion of t he work by t he expect ed end dat e; but because of t he bot t leneck, you can see t hat t he slope of t he Closed t riangle is not st eep enough t o finish t he work on t im e. This let s you drill int o t he bot t leneck and det erm ine whet her t he problem is inadequat e t est ing resources or poor qualit y of work from developm ent .

Contrast to Work-Down An icon of t he work- down paradigm is t he widely t aught " iron t riangle" view of proj ect m anagem ent . This is t he not ion t hat t here are only t hree variables t hat a proj ect m anager can work wit h: t im e, resources ( of which people are by far t he m ost im port ant ) , and funct ionalit y. I f you acknowledge qualit y as a fourt h dim ension ( which m ost people do now) , t hen you have a t et rahedron, as shown in Figure 1.4 .

Figu r e 1 .4 .

Th e " ir on t r ia n gle " ( or t e t r a h e dr on ) t r e a t s a pr oj e ct a s a fix e d st ock of w or k , in cla ssic

w or k - dow n t e r m s. To st r e t ch on e fa ce of t h e t e t r a h e dr on , you n e e d t o st r e t ch t h e ot h e r s.

I n Rapid Developm ent , St eve McConnell sum m arizes t he iron t riangle as follows: To keep t he t riangle balanced, you have t o balance schedule, cost , and product . I f you want t o load up t he product corner of t he t riangle, you also have t o load up cost or schedule or bot h. The sam e goes for t he ot her com binat ions. I f you want t o change one of t he corners of t he t riangle, you have t o change at least one of t he ot hers t o keep it in balance. [ 13] According t o t his view, a proj ect m anager has an init ial st ock of resources and t im e. Any change t o funct ionalit y or qualit y requires a corresponding increase in t im e or resources. You cannot st ret ch one face wit hout st ret ching t he ot hers because t hey are all connect ed. Alt hough widely pract iced, t his paradigm does not work well. Just as Newt onian physics is now known t o be a special case, t he iron t riangle is a special case t hat assum es t he process is flowing sm oot hly t o begin wit h. I n ot her words, it assum es t hat resource product ivit y is quit e uniform ly dist ribut ed, t hat t here is lit t le variance in t he effect iveness of t ask com plet ion, and t hat no spare capacit y exist s t hroughout t he syst em . These condit ions exist som et im es, not ably on low- risk proj ect s. Unfort unat ely, for t he t ypes of soft ware proj ect s usually undert aken, t hey are oft en unt rue. Many users of agile m et hods have dem onst rat ed experiences t hat pleasant ly cont radict t o t his viewpoint . For exam ple, in m any cases, if you im prove qualit ies of service, such as reliabilit y, you can short en t im e. Significant im provem ent s in flow are possible wit hin t he exist ing resources and t im e. [ 14]

Transparency I t 's no secret t hat m ost soft ware proj ect s are lat e, bot h in t he execut ion and in t he discovery t hat t hey are lat e. [ 15] This phenom enon has m any consequences, which are discussed in alm ost every chapt er of t his book. One of t he consequences is a vicious cycle of groupt hink and denial t hat underm ines effect ive flow. Lat e delivery leads t o request s for replanning, which lead t o pressure for ever m ore opt im ist ic est im at es, which lead t o m ore lat e delivery, and so on. And m ost part icipant s in t hese proj ect s plan opt im ist ically, replan, and replan furt her but wit h lit t le visibilit y int o t he effect s. Of course, t he all- t oo- frequent result is a deat h m arch. This is not because people can't plan or m anage t heir t im e. The problem is m ore com m only t he disparit y am ong priorit ies and expect at ions of different t eam m em bers. Most approaches t o soft ware engineering have lot s of places t o t rack t he workspreadsheet s, Microsoft Proj ect Plans, requirem ent s dat abases, bug dat abases, t est m anagem ent syst em s, t riage m eet ing not es, and so on. When t he inform at ion is scat t ered t his way, it is pret t y hard t o get a whole pict ure of t he proj ect you need t o look in t oo m any sources, and it 's hard t o balance all t he inform at ion int o one schedule. And when t here are so m any sources, t he inform at ion you find is oft en obsolet e when you find it .

Things don't need t o be t hat way. Som e com m unit y proj ect s post t heir developm ent schedules on t he Web, effect ively m aking individual cont ribut ors creat e expect at ions am ong t heir com m unit y peers about t heir t asks. Making all t he work in a proj ect visible can creat e a virt uous cycle. Of course, t his assum es t hat t he proj ect is st ruct ured it erat ively, t he scheduling and est im at ion are m ade at t he right granularit y, and t riage is effect ive at keeping t he work it em priorit ies in line wit h t he available resources in t he it erat ion. SCRUM, one of t he agile processes, cham pioned t he idea of a t ransparent ly visible product backlog, as shown in Figure 1.5 . Here's how t he founders of SCRUM, Ken Schwaber and Mike Beedle, define t he product backlog: Product Backlog is an evolving, priorit ized queue of business and t echnical funct ionalit y t hat needs t o be developed int o a syst em . The Product Backlog represent s everyt hing t hat anyone int erest ed in t he product or process has t hought is needed or would be a good idea in t he product . I t is a list of all feat ures, funct ions, t echnologies, enhancem ent s and bug fixes t hat const it ut e t he changes t hat will be m ade t o t he product for fut ure releases. Anyt hing t hat represent s work t o be done on t he product is included in Product Backlog. [ 16]

Figu r e 1 .5 .

Th e ce n t r a l gr a ph ic of t h e SCRUM m e t h odology is a gr e a t illu st r a t ion of flow in t h e m a n a ge m e n t se n se . N ot su r pr isin gly, SCRUM pion e e r e d t h e con ce pt of a sin gle pr odu ct ba ck log a s a m a n a ge m e n t t e ch n iqu e .

[View full size image]

This t ransparency is enorm ously effect ive for m ult iple reasons. I t creat es a " single set of books," or in ot her words, a unique, m aint ained source of inform at ion on t he work com plet ed and rem aining.

Com bined wit h flow m easurem ent , as shown in Figure 1.3 , it creat es t rust am ong t he t eam because everyone sees t he sam e dat a and plan. And finally, it creat es a virt uous cycle bet ween t eam responsibilit y and individual account abilit y. Aft er all, an individual is m ost likely t o com plet e a t ask when he or she knows exact ly who is expect ing it t o be done. [ 17]

One Work Item Database Visual St udio Team Syst em ( VSTS) t akes t he idea of a t ransparent product backlog even furt her ( see Figure 1.6 ) . Team Syst em uses a com m on product backlog t o t rack all planned, act ive, and com plet ed work for t he t eam and a hist ory of t he m aj orit y of act ions t aken and decisions m ade regarding t hat work. I t calls t hese unit s " work it em s" and let s t he user view and edit t hem in a dat abase view inside Visual St udio, in Microsoft Excel, and in Microsoft Proj ect , all t he while synchronizing t hem t o a com m on dat abase.

Figu r e 1 .6 .

VSTS e n a ct s a n d in st r u m e n t s t h e pr oce ss, t yin g sou r ce code , t e st in g, w or k it e m s, a n d m e t r ics t oge t h e r . W or k it e m s in clu de a ll t h e w or k t h a t n e e ds t o be t r a ck e d on a pr oj e ct , su ch a s sce n a r ios, qu a lit y of se r vice r e qu ir e m e n t s, de ve lopm e n t t a sk s, t e st t a sk s, bu gs, a n d r isk s. Th e se ca n be vie w e d a n d e dit e d in t h e Te a m Ex plor e r , Visu a l St u dio, M icr osoft Ex ce l, or M icr osoft Pr oj e ct .

One dat abase behind t he com m on, fam iliar t ools defragm ent s t he inform at ion. I nst ead of cut t ing and past ing am ong random ly dist ribut ed art ifact s, proj ect m anagers, business analyst s, developers, and t est ers all see t he sam e work, whet her planned in advance or scheduled on t he fly, and whet her from well- underst ood requirem ent s or discovered while fixing a bug ( see Figure 1.7 ) . And unlike separat e proj ect t racking t ools and t echniques, m uch of t he dat a collect ion in VSTS is aut om at ic.

Figu r e 1 .7 .

Th is is a n e x a m ple of t h e w or k it e m s a s t h e y a ppe a r e it h e r in t h e Te a m Ex plor e r of VSTS or in Visu a l St u dio. N ot e t h a t t a sk s, r e qu ir e m e n t s, a n d bu gs ca n a ll be vie w e d in on e pla ce .

[View full size image]

Because VSTS uses a com m on dat abase t o t rack work it em s, it exposes t hem not j ust in Team Explorer but also in Microsoft Excel ( see Figures 1.8 and 1.9) . The use of Excel and Proj ect is convenient but not necessary. All t he funct ionalit y is available t hrough t he Team Explorer, which is t he client for Team Foundat ion. I f you're using any Visual St udio Team Syst em client edit ion or Visual St udio Professional, t hen t he Team Explorer appears as a set of windows inside t he developm ent environm ent .

Figu r e 1 .8 .

W it h VSTS, t h e sa m e da t a ca n be vie w e d a n d e dit e d in M icr osoft Ex ce l. Th e w or k it e m s, r e ga r dle ss of t ype , a r e st or e d in t h e sa m e Te a m Fou n da t ion da t a ba se .

[View full size image]

Figu r e 1 .9 .

M icr osoft Pr oj e ct le t s you pla n a n d m a n a ge som e or a ll of t h e w or k it e m s w it h fu ll r ou n d t r ippin g t o t h e Te a m Fou n da t ion da t a ba se .

[View full size image]

Offline Editing, Management, and "What-Ifs" of Work Items For t hese t asks, use Excel or Proj ect . Your files are st ored locally on your client m achine while you work. The changes are writ t en back in Team Foundat ion when you next synchronize wit h t he dat abase, and any pot ent ial m erge conflict s are highlight ed at t hat t im e. On t he ot her hand, when you use Team Explorer, changes are saved t o t he dat abase during t he session.

The ext ensibilit y of Team Syst em m akes it possible for Microsoft part ners t o add funct ionalit y. For exam ple, Personify Design Team look [ 18] provides t eam m em bers a view of t heir Team Proj ect s on m ult iple Team Foundat ion Servers from wit hin Microsoft Office Out look. Team Foundat ion Server ext ensibilit y enables Team look t o t rack work it em s wit h full account abilit y in t he fam iliar com m unicat ions t ool, Out look ( see Figure 1.10) .

Figu r e 1 .1 0 .

W it h Te a m look fr om Pe r son ify D e sign , you ca n a lso u se Ou t look a s a clie n t for t h e Te a m Fou n da t ion se r ve r .

[View full size image]

Instrument Daily Activities

The t ransparent backlog relies on accurat e dat a t o be useful. Oft en, collect ing t he dat a becom es a m aj or act ivit y in it self t hat relies on willing com pliance of large num bers of part icipant s. This disciplined at t ent ion t o t he bookkeeping is rarely sust ained in pract ice, especially during periods of int ense act ivit y. The irony is t hat t he vast m aj orit y of t he dat a t hat a t eam needs is direct ly correlat ed t o ot her act ions t hat are already m anaged by soft ware. Developers check in code, builds parse t hat code, t est ers writ e and run t est s, and all t heir act ivit ies are t racked som ewherein Proj ect , Excel, t he bug dat abase, or t im esheet s. What if you could gat her all t hat dat a aut om at ically, correlat e it , and use it t o m easure t he process? Team Syst em t akes t hat approach. I t inst rum ent s t he daily act ivit ies of t he t eam m em bers t o collect process dat a wit h no overhead. For exam ple, every t im e a developer checks updat ed code int o version cont rol, work it em s are updat ed t o reflect t he t asks and scenarios updat ed by t his code. The relat ionships are capt ured in a " changeset ," and when t he next build runs, it ident ifies t he change set s included and updat es work it em s again wit h t he build num ber. When t est s execut e, t hey use t he sam e build num ber. Then t est result s, code changes, and work it em s are all correlat ed aut om at ically by Team Syst em ( see Figure 1.11) .

Figu r e 1 .1 1 .

Th e m e t r ics w a r e h ou se colle ct s t h e da t a fr om a ll t h e a ct ion s on t h e pr oj e ct t o pr ovide r e por t s t h a t cor r e la t e t h e diffe r e n t sou r ce s a n d dim e n sion s of da t a .

I n addit ion t o keeping t he backlog current and visible, t his aut om at ic dat a collect ion populat es a dat a

warehouse wit h m et rics t hat reveal t rends and com parisons of qualit y from m any dim ensions on a daily basis. Just like a dat a warehouse t hat provides business int elligence on funct ions such as a sales or product ion process, t his one provides int elligence on t he soft ware developm ent process.

Simple Observations Wit h t his dat a warehouse, basic quest ions becom e easy t o answer: I s t he proj ect com ing in on t im e, or how far off is it ? How m uch has t he plan changed? Who's over or under and needs t o have work rebalanced? What rat es should we use t o est im at e rem aining work? How effect ive are our t est s? Most proj ect m anagers would love t o answer t hese basic quest ions wit h hard dat a. When t he dat a collect ion is aut om at ed, t he answers becom e st raight forward.

Project "Smells" More significant ly, m ost proj ect m anagers would love t o find blind spot splaces where dat a indicat es a likely problem . I t is now com m on t o t alk about " sm ells" for suspicious areas of code. [ 19] Problem s for t he proj ect as a whole also appear oft en as hard- t o- pin- down sm ells, which are not well exposed by exist ing m et rics. I 'll cover sm ells in som e det ail in Chapt er 9, " Troubleshoot ing t he Proj ect ," but for now I 'll share a com m on exam ple. I m agine a graph t hat shows you t hese bug and t est pass rat es ( see Figure 1.12) .

Figu r e 1 .1 2 .

Th e X- a x is ide n t ifie s diffe r e n t com pon e n t s of you r pr oj e ct ; t h e ba r s sh ow you t h e t e st pa ss r a t e for e a ch com pon e n t , w h ile t h e poin t s a n d lin e sh ow t h e a ct ive bu g cou n t .

[View full size image]

Based on Figure 1.12, what would you conclude? Probably t hat t he I nst ore Pickup Kiosk code is in great shape, so you should look for problem s elsewhere. At t he sam e t im e, t here's a danger of relying on t oo few m et rics. Consider t he graph in Figure 1.13, which overlays code churn ( t he num ber of lines added, m odified, and delet ed) and code coverage from t est ing ( t he percent age of code lines or blocks exercised during t est ing) on t he sam e axes.

Figu r e 1 .1 3 .

Ove r la yin g code cove r a ge a n d code ch u r n for t h e com pon e n t s pr ovide s a ve r y diffe r e n t pe r spe ct ive on t h e da t a .

[View full size image]

Suddenly t he pict ure is reversed. There's really high code churn in I nst ore Pickup Kiosk, and t he code is not being covered by t he t est s t hat supposedly exercise t hat com ponent . This pict ure reveals t hat we m ay have st ale t est s t hat aren't exercising t he new funct ionalit y. Could t hat be why t hey're passing and not covering t he act ual code in t his com ponent ?

Multidimensional Metrics and Smells The abilit y t o see m ore dim ensions of t he proj ect dat a is a direct benefit of t he m et rics warehouse, which collect s and correlat es dat a from daily act ivit ies. I t provides a quant it at ive, visual t ool t o pursue t he sm ells. I n t his way, you can achieve t he visibilit y level needed for t he st rict est com pliance report ing while working in an agile m anner and having t he sam e visibilit y int o a rem ot e proj ect , even out sourced, t hat you would have in a local one.

Fit the Process to the Project I nst rum ent ing daily act ivit ies and aut om at ically collect ing dat a m ake it m uch easier t o follow a consist ent soft ware process. Team Syst em aut om at es t he process guidance and inst rum ent s t he process so t hat m ost of t he overhead associat ed wit h process and m ost of t he resist ance t o com pliance are elim inat ed. However, t his quickly exposes a valid concern, which is t hat no one process fit s all soft ware proj ect s, even wit hin one organizat ion. Regulat ory environm ent , business risk, business upside, t echnology risk, t eam skills, geographic dist ribut ion, and proj ect size all play a role in det erm ining t he right fit of a process t o a proj ect . Team Syst em t akes t he diversit y of process int o account , enabling t he proj ect t eam t o choose or adapt it s m et hodology for cont ext ual realit ies. When you st art a t eam proj ect in VSTS, you pick a process t em plat e, as shown in Figure 1.14. I n effect , you can choose and cust om ize t he process furt her for each proj ect , det erm ining not only guidance but also workflow, policies, t em plat es, report s, and perm issions.

Figu r e 1 .1 4 .

W h e n you st a r t a Te a m Pr oj e ct , you r fir st ch oice is w h ich " Pr oce ss Te m pla t e " t o a pply for t h e pr oj e ct . Th e Pr oce ss Te m pla t e de fin e s t h e pr oce ss gu ida n ce w e b sit e , t h e w or k it e m t ype s a n d t h e ir w or k flow , st a r t e r w or k it e m s, t h e se cu r it y gr ou ps a n d pe r m ission s, a n d r e por t s a n d t e m pla t e s for w or k pr odu ct s.

[View full size image]

Summary I n pract ice, m ost soft ware processes require m anual enact m ent , where collect ing dat a and t racking progress are expensive. Up front , such processes need lot s of docum ent at ion, t raining, and m anagem ent , and t hey have high operat ing and m aint enance cost s. Most significant ly, t he process art ifact s and effort do not cont ribut e in any direct way t o t he delivery of cust om er value. Proj ect m anagers can oft en spend 40 hours a week cut t ing and past ing t o report st at us. This const raint has left process as an exercise for m anagers, specialist Program Managem ent Offices, and skilled pract it ioners, who som et im es define m et rics and act ivit ies quit e divorced from t he int erest s of t he pract it ioners or t he t ools used t o im plem ent t hem . The dom inant paradigm in t his world has been t he work- down view, where soft ware engineering is a det erm inist ic exercise, sim ilar t o ot her engineering pursuit s. I n cont rast , t he business forces driving soft ware engineering t oday require a different paradigm . I n keeping wit h t he dict um " As sim ple as possible, but no sim pler," a t eam t oday needs t o em brace t he paradigm of cust om er value, change, variance, and sit uat ionally specific act ions as a part of everyday pract ice. This is equally t rue whet her proj ect s are in- house or out sourced and whet her t hey are local or geographically dist ribut ed. Managing such a process usually requires a value- up approach inst ead. Typically, t he value- up approach requires t ooling. Collect ing, m aint aining, and report ing t he dat a wit hout overhead is sim ply not pract ical ot herwise. I n sit uat ions where regulat ory com pliance and audit are required, t he t ooling is necessary t o provide t he change m anagem ent and audit t rails. Team Syst em is designed from t he ground up t o support t he value- up approach in an audit able way. The rest of t his book describes t he use of Team Syst em t o support t his paradigm .

Endnotes 1.

Thom as Kuhn, The St ruct ure of Scient ific Revolut ions ( Universit y of Chicago Press, 1962) .

2.

Pet er Galison, Einst ein's Clocks, Poincaré's Maps ( New York: Nort on, 2003) , 40.

3.

www.agilem anifest o.org

4.

See Thom as L. Friedm an, The World I s Flat : A Brief Hist ory of t he Twent y- First Cent ury ( New York: Farrar, St rauss & Giroux, 2005) for a discussion of t he enabling t rends.

5.

ht t p: / / www.sei.cm u.edu/ cm m i/

6.

There are ot her argum ent s as well, such as t he design com plexit y of soft ware relat ive t o m ost engineering pursuit s. See, for exam ple, Boris Beizer, " Soft ware I s Different ," in Soft ware Qualit y Professional I : 1 ( Am erican Societ y for Qualit y, Decem ber 1998) .

7.

The negat ive consequence of t he int erplay of variat ion and dependent event s is cent ral t o t he Theory of Const raint s. For exam ple, see Eliyahu M. Goldrat t , The Goal ( Nort h River Press, 1986) .

8.

The first m aj or work t o highlight what I call t he value- up approach is Gerald M. Weinberg, Qualit y Soft ware Managem ent , Volum e I : Syst em s Thinking ( New York: Dorset House, 1992) .

9.

The Agile Proj ect Manifest o is available at ht t p: / / www.pm doi.org/ . I t is anot her exam ple of t he value- up approach.

1 0 . Mihaly Csikszent m ihalyi, Flow: The Psychology of Opt im al Experience ( New York: HarperCollins, 1990) , 71.

1 1 . David J. Anderson, Agile Managem ent for Soft ware Engineering: Applying t he Theory of Const raint s for Business Result s ( Upper Saddle River, NJ: Prent ice Hall, 2004) , 77.

1 2 . I bid., 77. 1 3 . St eve McConnell, Rapid Developm ent ( Redm ond, WA: Microsoft Press, 1996) , 126. 1 4 . For a m ore det ailed discussion of t his subj ect , using t he nom enclat ure of t he Theory of Const raint s, see David J. Anderson and Dragos Dum it riu, " From Worst t o Best in 9 Mont hs: I m plem ent ing a Drum - Buffer- Rope Solut ion in Microsoft 's I T Depart m ent ," present ed at t he TOCI CO Conference, Novem ber 2005, available at ht t p: / / www.agilem anagem ent .net / Art icles/ Papers/ From _Worst _t o_Best _in_9_Mont hs_Final_1_2.pdf.

1 5 . The St andish Group ( www.st andishgroup.com ) publishes a biennial survey called " The Chaos Report ." According t o t he 2004 dat a, 71% of proj ect s were lat e, over budget , and/ or canceled.

1 6 . Ken Schwaber and Mike Beedle, Agile Soft ware Developm ent wit h Scrum ( Upper Saddle River, NJ: Prent ice Hall, 2001) , 323.

1 7 . Bellot t i, V., Dalal, B., Good, N., Bobrow, D. G., and Ducheneaut , N., " What a t o- do: st udies of t ask m anagem ent t owards t he design of a personal t ask list m anager." ACM Conference on Hum an Fact ors in Com put ing Syst em s ( CHI 2004) ; 2004 April 2429; Vienna; Aust ria. NY: ACM; 2004; 735742.

1 8 . ht t p: / / www.personifydesign.com / 1 9 . Originally used for code in Mart in Fowler, Refact oring: I m proving t he Design of Exist ing Code ( Reading, MA: Addison- Wesley, 1999) , 75.

Chapter 2. Value-Up Processes

Pr ogr a m M a n a ge r Pr oj e ct M a n a ge r " One m et hodology cannot possibly be t he " right " one, but ...t here is an appropriat e, different way of working for each proj ect and proj ect t eam ." [ 1 ] Alist air Cockburn

Figu r e 2 .1 .

Th e r h yt h m of a cr e w r ow in g in u n ison is a pe r fe ct e x a m ple of flow in bot h t h e h u m a n a n d m a n a ge m e n t se n se s. I n dividu a ls e x pe r ie n ce t h e e la t ion of pe r for m in g opt im a lly, a n d t h e coor din a t e d t e a m w or k e n a ble s t h e syst e m a s a w h ole ( h e r e , t h e boa t ) t o a ch ie ve it s opt im u m pe r for m a n ce .

I n t he last chapt er, I argued t he im port ance of t he value- up paradigm . I n t his chapt er, I cover t he next level of det ailt he charact erist ics of such processes, t he " sit uat ionally specific" cont ext s t o consider, and t he exam ples t hat you can apply.

Software Process Options in VSTS Two processes are shipped wit h VSTS, bot h of which are variant s of Microsoft Solut ions Fram ework ( MSF) : MSF for Agile Soft ware Developm ent MSF for CMMI Process I m provem ent You can download and preview t hese from ht t p: / / m sdn.m icrosoft .com / m sf/ .This web page also includes links t o t he m any processes t hat part ner com panies deliver for VSTS, including SCRUM, FDD, and EUP. You can also t ailor your own process t em plat es from t he MSF ones.

Microsoft Solutions Framework Dozens of docum ent ed soft ware processes exist . [ 2 ] Over t he last t hirt y years, m ost of t hese have com e from t he work- down paradigm , and t hey have spawned ext ensive am ount s of docum ent at ion t o cover all possible sit uat ions and cont ingencies. [ 3 ] Managers have want ed t o play it safe wit hout underst anding t he corresponding drain on product ivit y. Unfort unat ely, t he idea has backfired. When t eam s don't know what t hey can safely do wit hout , t hey include everyt hing in t heir planning and execut ion process. Barry Boehm and Richard Turner have described t he problem well: Bu ild You r M e t h od Up, D on 't Ta ilor I t D ow n Plan- driven m et hods have had a t radit ion of developing all- inclusive m et hods t hat can be t ailored down t o fit a part icular sit uat ion. Expert s can do t his, but nonexpert s t end t o play it safe and use t he whole t hing, oft en at considerable unnecessary expense. Agilist s offer a bet t er approach of st art ing wit h relat ively m inim al set s of pract ices and only adding ext ras where t hey can be clearly j ust ified by cost - benefit . [ 4 ] Sim ilarly, m ost processes and t ools have not allowed adequat ely for appropriat e diversit y am ong proj ect s and have forced a " one size fit s all" approach on t heir t eam s. I n cont rast , VSTS is a collaborat ion and developm ent environm ent t hat allows a process per proj ect . VSTS also assum es t hat a t eam will " st ret ch t he process t o fit " t hat is, t o t ake a sm all core of values and pract ices and t o add m ore as necessary. This approach has been m uch m ore successful, as t he previous quot e not es. I nside Team Syst em are t wo fully enact ed process inst ances, bot h based on a com m on core called Microsoft Solut ions Fram ework ( MSF) : M SF for Agile Soft w a r e D e ve lopm e n t . A light weight process t hat adheres t o t he principles of the Agile Alliance. [ 5 ] Choose t he MSF Agile process for proj ect s wit h short lifecycles and result sorient ed t eam s who can work wit hout lot s of int erm ediat e docum ent at ion. The MSF Agile process is a flexible guidance fram ework t hat helps creat e an adapt ive syst em for soft ware developm ent . I t ant icipat es t he need t o adapt t o change, em phasizes t he delivery of working soft ware, and prom ot es cust om er validat ion as key success m easures. M SF for CM M I Pr oce ss I m pr ove m e n t . A process designed t o facilit at e CMMI Level 3 as defined by t he Soft ware Engineering I nst it ut e. [ 6 ] This ext ends MSF Agile wit h m ore form al planning, m ore docum ent at ion and work product s, m ore sign- off gat es, and m ore t im e t racking. MSF for CMMI clearly m aps it s pract ices t o t he Pract ice Areas and Goals in order t o assist organizat ions t hat are using t he CMMI as t he basis for t heir process im provem ent or t hat are seeking CMMI appraisal. However, unlike previous at t em pt s at CMMI processes, MSF uses t he value- up paradigm t o enable a low- overhead, agile inst ant iat ion of t he CMMI fram ework. [ 7 ] Bot h inst ances of MSF are value- up processes. I n bot h cases, MSF harvest s proven pract ices from Microsoft , it s cust om ers, and indust ry expert ise. The prim ary differences bet ween t he t wo are t he level of approval form alit y, t he level of account ing for effort spent , and t he dept h of m et rics used. For exam ple, MSF for CMMI Process I m provem ent considers t he appraiser or audit or as an explicit role and provides act ivit ies and report s t hat t he audit or can use t o assess process com pliance. I n it s agile sibling, com pliance is not a considerat ion. I n bot h t he agile and CMMI inst ances, MSF's sm oot h int egrat ion in Team Syst em support s rapid it erat ive developm ent wit h cont inuous learning and refinem ent . The com m on work it em dat abase and m et rics warehouse answer quest ions on proj ect healt h in near real t im e, and t he coupling of process

guidance t o t ools m akes it possible t o see t he right guidance in t he cont ext of t he t ools and act on it as you need it .

Iteration MSF is an it erat ive and increm ent al process. For m ore t han t went y years, t he soft ware engineering com m unit y has underst ood t he need for it erat ive developm ent . This is t he generally defined as t he " soft ware developm ent t echnique in which requirem ent s definit ion, design, im plem ent at ion and t est ing occur in an overlapping, it erat ive ( rat her t han sequent ial) m anner, result ing in increm ent al com plet ion of t he overall soft ware product ." [ 8 ] I t erat ive developm ent arose as an ant idot e t o sequent ial " wat erfall" developm ent . Fred Brooks, whose Myt hical Man Mont h is st ill am ong t he world's m ost widely adm ired soft ware engineering books, sum s up t he wat erfall experience as follows: The basic fallacy of t he wat erfall m odel is t hat it assum es a proj ect goes t hrough t he process once, t hat t he archit ect ure is excellent and easy t o use, t he im plem ent at ion design is sound, and t he realizat ion is fixable as t est ing proceeds. Anot her way of saying it is t hat t he wat erfall m odel assum es t he m ist akes will all be in t he realizat ion, and t hus t hat t heir repair can be sm oot hly int erspersed wit h com ponent and syst em t est ing. [ 9 ]

Why Iterate? There are m any highly com pelling argum ent s for it erat ive developm ent :

1 . Risk m a n a ge m e n t . The desired result is unknowable in advance. To m anage risks, you m ust prove or disprove your requirem ent s and design assum pt ions by increm ent ally im plem ent ing t arget syst em elem ent s, st art ing wit h t he highest - risk elem ent s. 2 . Econ om ics. I n an uncert ain business clim at e, it is im port ant t o review priorit ies frequent ly and t reat invest m ent s as t hough t hey were financial opt ions. The m ore flexibilit y gained t hrough early payoff and frequent checkpoint s, t he m ore valuable t he opt ions becom e. [ 10] 3 . Focu s. People can only ret ain so m uch in t heir brains. By bat ching work int o sm all it erat ions, all t he t eam players focus m ore closely on t he work at handbusiness analyst s can do a bet t er j ob wit h requirem ent s, archit ect s wit h design, developers wit h code, and so on. 4 . M ot iva t ion . The m ost energizing phenom enon on a soft ware t eam is seeing early releases perform ( or be dem o'd) in act ion. No am ount of spec review can subst it ut e for t he value of working bit s. 5 . Con t r ol t h e or y. Sm all it erat ions enable you t o reduce t he m argin of error in your est im at es and provide fast feedback about t he accuracy of your proj ect plans. 6 . St a k e h olde r in volve m e n t . St akeholders ( cust om ers, users, m anagem ent ) see result s quickly and becom e m ore engaged in t he proj ect , offering m ore t im e, insight , and funding. 7 . Con t in u ou s le a r n in g. The ent ire t eam learns from each it erat ion, im proving t he accuracy, qualit y, and suit abilit y of t he finished product . A sim ple sum m ary of all t his wisdom is t hat " I ncrem ent alism is a good idea for all proj ect s . . . and a m ust when risks are high."[ 11]

Nonet heless, it erat ive developm ent st ruggles t o be adopt ed in m any I T organizat ions. I n pract ice, it erat ive developm ent requires t hat t he t eam and it s proj ect m anagers have a t ot al view of t he work t o be done and t he abilit y t o m onit or and priorit ize frequent ly at short it erat ion boundaries. This frequent updat ing requires a readily visible backlog, preferably wit h aut om at ed dat a collect ion, such as t he VSTS work it em dat abase provides. I n t he value- up view of it erat ive developm ent , t here are m any cycles in which act ivit ies overlap in parallel. The prim ary planning cycle is t he " it erat ion." An it erat ion is t he fixed num ber of weeks, som et im es called a " t im e box," used t o schedule a t ask set . Use it erat ions as t he int erval in which t o schedule int ended scenarios, m easure t he flow of value, assess t he act ual process, exam ine bot t lenecks, and m ake adj ust m ent s ( see Figure 2.2 ) . I n t he chapt ers on developm ent and t est ing, I 'll cover t he finer- grained cycles of t he check- in and daily build, which are oft en called " increm ent s" and which nat urally drive t he it erat ion. [ 12]

Figu r e 2 .2 .

Soft w a r e pr oj e ct s pr oce e d on m a n y in t e r lock in g cycle s, r a n gin g fr om t h e code - e dit - t e st de bu g- ch e ck in cycle , m e a su r e d in m in u t e s, t o t h e it e r a t ion of a fe w w e e k s, t o a pr ogr a m t h a t m igh t spa n ye a r s. I f t h e cycle s con n e ct , t h e pr oce ss a s a w h ole ca n be u n de r st ood.

Length

I n pract ice, t he lengt h of it erat ions varies from proj ect t o proj ect , usually t wo t o six weeks. Again, t he it erat ion det erm ines t he size of t he bat ch t hat you use for m easuring a chunk of deliverable cust om er value, such as scenarios and qualit y of service ( QoS) requirem ent s. You need t o keep t he bat ch size as sm all as possible while m eet ing t his obj ect ive, as David Anderson explains in Agile Managem ent for Soft ware Engineering: Sm all bat ch sizes are essent ial for flow! Sm all bat ch sizes are also essent ial for qualit y. I n soft ware developm ent hum an nat ure creat es a t endency for engineers t o be less exact ing and pay less at t ent ion t o det ail when considering larger bat ches. For exam ple, when code reviews are perform ed on sm all bat ches, t he code review t akes only a short t im e t o prepare and a short t im e t o conduct . Because t here is only a sm all am ount of code t o review, t he reviewers t end t o pay it great er at t ent ion and spot a great er num ber of errors t han t hey would on a larger bat ch of code. I t is t herefore bet t er t o spend 4 hours per week doing code reviews t han 2 days at t he end of each m ont h or, even worse, one week at t he end of each quart er. A large bat ch m akes code reviewing a chore. Sm all bat ches keep it light . The result is higher qualit y. Higher qualit y leads direct ly t o m ore product ion. . . . Reducing bat ch sizes im proves ROI ! [ 13]

Different Horizons, Different Granularity One of t he benefit s of using it erat ions is t hat it provides clear levels of granularit y in planning. You can plan t asks t o one- day det ail for t he current it erat ion while m aint aining a rough candidat e list of scenarios and QoS requirem ent s for fut ure it erat ions. I n ot her words, you only t ry t o go int o det ail for t he current it erat ion. ( By t he way, t he person who should plan and est im at e t he det ailed t asks is t he one who is going t o do t hem .) Not only does t his keep t he t eam focused on a m anageable set of work it em s, but it also gives t he m axim um benefit from t he experience of act ual com plet ion of delivered value when planning t he next set . Apply com m on sense t o t he dept h of planning and priorit izat ion. Obviously, you want t o m it igat e m aj or risks early, so you will priorit ize archit ect ural decisions int o early it erat ions, even if t hey are incom plet e from t he st andpoint of visible cust om er value.

Prioritization There are t wo com plem ent ary ways t o plan work: by priorit y and by st ack rank. VSTS support s bot h. Ranking by priorit y involves rat ing t he candidat e scenarios and QoS int o broad bucket s, such as Must have ( 1) , Should have ( 2) , Could have ( 3) , and Won't have ( 4) . This schem e is m nem onically referred t o as MoSCoW. [ 14] MoSCoW and sim ilar priorit y schem es work well for large num bers of sm all candidat es, such as bugs, or scenarios under considerat ion for fut ure it erat ions, where it is not wort h spending t he t im e t o exam ine each it em individually. St ack ranking, on t he ot her hand, is t he pract ice of assigning cardinal ranks t o a set of requirem ent s t o give t hem a unique order ( see Figure 2.3 ) . St ack ranking is well suit ed t o planning scenarios and QoS for near- t erm it erat ions. St ack ranking m it igat es t wo risks. One is t hat you sim ply end up wit h t oo m any " Must haves." The ot her is t hat your est im at es of effort are wrong. I f you t ackle scenarios and QoS according t o a linear rank order, t hen you know what 's next , regardless of whet her your planned work fit s your available t im e. I n fact , it helps you im prove your est im at es. DeMarco and List er describe t his effect well:

Figu r e 2 .3 .

I n M SF, sce n a r ios a r e st a ck r a n k e d, a s sh ow n on t h is w or k it e m for m . H e r e t h e sce n a r ios a r e sh ow n in t h e Te a m Ex plor e r of VSTS.

[View full size image]

Rank- ordering for all funct ions and feat ures is t he cure for t wo ugly proj ect m aladies: The first is t he assum pt ion t hat all part s of t he product are equally im port ant . This fict ion is preserved on m any proj ect s because it assures t hat no one has t o confront t he st akeholders who have added t heir favorit e bells and whist les as a price for t heir cooperat ion. The sam e fict ion facilit at es t he second m alady, piling on, in which feat ures are added wit h t he int ent ion of overloading t he proj ect and m aking it fail, a favorit e t act ic of t hose who oppose t he proj ect in t he first place but find it convenient t o present t hem selves as ent husiast ic proj ect cham pions rat her t han as proj ect adversaries. [ 15]

Adapting Process I t erat ions also give you a boundary at which t o m ake process changes. You can t une based on experience, and you can adj ust for cont ext . For exam ple, you m ight increase t he check- in requirem ent s for code review as your proj ect approaches product ion and use VSTS check- in policies and check- in not es t o enforce t hese requirem ent s. Alist air Cockburn has been one of t he clearest

proponent s of adapt ing process: There is one m ore device t o m ent ion in t he design of m et hodologies. That is t o adj ust t he m et hodology on t he fly. Once we underst and t hat every proj ect deserves it s own m et hodology, t hen it becom es clear t hat what ever we suggest t o st art wit h is t he opt im al m et hodology for som e ot her proj ect . This is where increm ent al delivery plays a large role. I f we int erview t he proj ect t eam bet ween, and in t he m iddle of, increm ent s, we can t ake t heir m ost recent experiences int o account . I n t he first increm ent , t he m id- increm ent t une- up address t he quest ion, " Can we deliver?" Every ot her int erview addresses, " Can we deliver m ore easily or bet t er?" [ 16] The process review at t he end of an it erat ion is called a ret rospect ive, a t erm wit h bet t er im plicat ions t han t he alt ernat ive post - m ort em .

Retrospectives and Lessons Learned Bot h inst ances of MSF provide guidance on ret rospect ives. MSF for CMMI Process I m provem ent capt ures t he inform at ion in a Lessons Learned docum ent t o facilit at e cont inuous im provem ent .

Risk Management The risks proj ect s face differ considerably. Tradit ional risk m anagem ent t ends t o be event - driven, in t he form of " What if X were t o happen?" MSF com plem ent s event - driven risk m anagem ent wit h a const it uency- based approach, in which t here are seven point s of view t hat need t o be represent ed in a proj ect t o prevent blind spot s. I n som e cases, for exam ple, t echnology is new and needs t o be proven before being adopt ed. Som et im es ext rem e QoS issues m ust be addressed, such as huge scalabilit y or t hroughput . At ot her t im es, t here are significant issues due t o t he assem bly of a new t eam wit h unknown skills and com m unicat ion st yles. VSTS, in t urn, t racks risks direct ly in t he work it em dat abase so t hat t he proj ect t eam can t ake t hem on direct ly.

Viewpoints of Risk: Advocacy Groups This is t he descript ion of Advocacy Groups in t he Team Model of MSF ( bot h inst ances) . Pr ogr a m M a n a ge m e n t Advoca t e s for Solu t ion D e live r y The focus of program m anagem ent is t o m eet t he goal of delivering t he solut ion wit hin proj ect const raint s. This group ensures t hat t he right solut ion is delivered at t he right t im e and t hat all st akeholders' expect at ions are underst ood, m anaged, and m et t hroughout t he proj ect . Ar ch it e ct u r e Advoca t e s for t h e Syst e m in t h e La r ge This includes t he services, t echnology, and st andards wit h which t he solut ion will int eroperat e, t he infrast ruct ure in which it will be deployed, it s place in t he business or product fam ily, and it s roadm ap of fut ure versions. The archit ect ure group has t o ensure t hat t he deployed solut ion m eet s all qualit ies of service as well as t he business obj ect ives and is viable in t he long t erm . D e ve lopm e n t Advoca t e s for t h e Te ch n ica l Solu t ion I n addit ion t o being t he prim ary solut ion builders, developm ent is responsible for t hought ful t echnical decisions, clean design, good bot t om - up est im at es, and high- qualit y m aint ainable code and unit t est s. Te st Advoca t e s for Solu t ion Qu a lit y fr om t h e Cu st om e r Pe r spe ct ive Test ant icipat es, looks for, and report s on any issues t hat dim inish t he solut ion qualit y in t he eyes of t he users or cust om ers. Re le a se / Ope r a t ion s Advoca t e s for t h e Sm oot h D e live r y a n d D e ploym e n t of t h e Solu t ion in t o t h e Appr opr ia t e I n fr a st r u ct u r e This group ensures t im ely readiness and com pat ibilit y of t he infrast ruct ure for t he solut ion.

Use r Ex pe r ie n ce Advoca t e s for t h e M ost Effe ct ive Solu t ion in t h e Eye s of t h e I n t e n de d Use r s User experience m ust underst and t he user's cont ext as a whole, appreciat e any subt let ies of t heir needs, and ensure t hat t he whole t eam is conscious of usabilit y from t heir eyes. Pr odu ct M a n a ge m e n t Advoca t e s for t h e Cu st om e r Bu sin e ss Product m anagem ent has t o underst and, com m unicat e, and ensure success from t he st andpoint of t he econom ic cust om er request ing t he solut ion.

Looking at risk from t he st andpoint of const it uencies changes t he approach t o m it igat ion in a very healt hy way. For exam ple, all proj ect s face a risk of poorly underst ood user requirem ent s. Paying at t ent ion t o user experience and QoS as cent ral concerns encourages ongoing usabilit y research, st oryboarding, prot ot yping, and usabilit y labs. The event - driven version of t his approach m ight be const rued as a risk t hat t he cust om er will refuse accept ance and t herefore t hat requirem ent s need t o be overly docum ent ed and signed off t o enforce a cont ract . One approach em braces necessary change, whereas t he ot her t ries t o prevent it . The const it uency- based approach follows t he value- up paradigm , whereas t he event - driven one, when used in isolat ion, follows t he work- down paradigm .

Fit the Process to the Project The Agile Proj ect Managem ent Declarat ion of I nt erdependence speaks of " sit uat ionally specific st rat egies" as a recognit ion t hat t he process needs t o fit t he proj ect . VSTS uses t he m echanism of process t em plat es t o im plem ent t his principle. But what cont ext ual differences should you consider in det erm ining what is right for your proj ect ? Som e of t he considerat ions are ext ernal business fact ors; ot hers have t o do wit h issues int ernal t o your organizat ion. Even if only cert ain st akeholders will choose t he process, every t eam m em ber should underst and t he rat ionale of t he choice and t he value of any pract ice t hat t he process prescribes. I f t he value can't be ident ified, it is very unlikely t hat it can be realized. Som et im es t he purpose m ay not be int uit ive, such as cert ain legal requirem ent s, but if underst ood can st ill be achieved. The prim ary dim ensions in which cont ext s differ are as follows: Adapt ive Versus Plan- Driven Tacit Knowledge Versus Required Docum ent at ion I m plicit Versus Explicit Sign- Off Gat es and Governance Model Level of Audit abilit y and Regulat ory Concerns Prescribed Organizat ion Versus Self- Organizat ion One Proj ect at a Tim e Versus Many Proj ect s at Once Geographic Dist ribut ion Cont ract ual Obligat ions The next sect ions describe each of t hese considerat ions.

Adaptive Versus Plan-Driven To what ext ent does your proj ect need t o pin down and design all of it s funct ionalit y in advance, or conversely, t o what ext ent can t he funct ionalit y and design evolve over t he course of t he proj ect ? There are advant ages and disadvant ages t o bot h approaches. The prim ary benefit of a plan- driven approach is st abilit y. I f you are assem bling a syst em t hat has m any part s t hat are being discret ely delivered, obviously int erfaces need t o be very st able t o allow int egrat ions from t he cont ribut ing t eam s. ( Consider a car, a web port al host ing product s from m any sellers, or a paym ent clearing syst em .) Of course, t he m ain drawback t o a plan- driven approach is t hat you oft en have t o m ake t he m ost im port ant decisions when you know t he least . On t he ot her hand, if your proj ect needs t o be highly responsive t o com pet it ive business pressure, usabilit y, or t echnical learning achieved during t he proj ect , you can't pin t he design down t ight ly in advance. ( Consider consum er devices and com m ercial web sit es.) The m ain risk wit h t he adapt ive approach is t hat you discover design issues t hat force significant rework or change of proj ect scope.

Adaptive or Plan-Driven MSF for Agile Soft ware Developm ent is m ore adapt ive. MSF for CMMI Process I m provem ent can be m ore plan- driven.

Required Documentation Versus Tacit Knowledge A huge difference in pract ice is t he am ount of docum ent at ion used in different processes. For t his process discussion, I int end " writ ing docum ent at ion" t o m ean preparing any art ifact sspecificat ions, visual m odels, m eet ing m inut es, present at ions, and so onnot direct ly relat ed t o t he execut able code, dat a, and t est s of t he soft ware being built . I am not including t he help files, m anuals, t raining, and ot her m at erials t hat are clearly part of t he product value being delivered t o t he end user or cust om er. Nor am I discount ing t he im port ance of t he educat ional m at erials. Any docum ent at ion t hat you writ e should have a clear purpose. Usually t he purpose concerns cont ract , consensus, archit ect ure, approval, audit , delegat ion, or work t racking. Som et im es t he docum ent at ion is not t he m ost effect ive way t o achieve t he int ended goal. For exam ple, when you have a sm all, cohesive, collocat ed t eam wit h st rong dom ain knowledge, it m ay be easy t o reach and m aint ain consensus. You m ay need lit t le m ore t han a vision st at em ent , a scenario list , and frequent face- t oface com m unicat ion wit h t he cust om er and wit h each ot her. ( This is part of t he eXt rem e Program m ing m odel.) On t he ot her hand, when part of your t eam is t welve t im e zones away, dom ain knowledge varies, and m any st akeholders need t o review designs, m uch m ore explicit docum ent at ion is needed. ( And of geographic necessit y, m any discussions m ust be elect ronic rat her t han face- t o- face.) The key point here is t o writ e t he docum ent at ion for it s audience and t o it s purpose and t hen t o st op writ ing. Aft er t he docum ent at ion serves it s purpose, m ore effort on it is wast e. I nst ead, channel t he t im e and resources int o work product s t hat bring value t o t he cust om er direct ly. I f you lat er discover t hat your docum ent at ion is inadequat e for it s purpose, updat e it t hen. I n ot her words, " fix it when it hurt s." [ 17]

Tacit Knowledge or Documentation MSF for Agile Soft ware Developm ent relies on t acit knowledge m ore heavily. MSF for CMMI Process I m provem ent requires m ore explicit docum ent at ion.

Implicit Versus Explicit Sign-Off Gates and Governance Model Appropriat ely, business st akeholders usually require point s of sign- off. Typically, t hese gat es release

funding and are, at least im plicit ly, point s at which a proj ect can be canceled. When developm ent is perform ed by out side com panies or consult ant s, cont ract ual form alit y usually prescribes t hese checkpoint s. Oft en, set t ing expect at ions and com m unicat ing wit h t he st akeholders is a m aj or part of t he proj ect m anager's responsibilit y. For correct com m unicat ion, perhaps t he m ost im port ant issue is whet her t he sign- offs are based on t im e ( how far did we get in t he allot t ed t im e?) or funct ionalit y ( how long did it t ake us t o im plem ent t he funct ionalit y for t his m ilest one?) . This alignm ent of I T wit h t he business st akeholders is t he dom ain of I T governance. I n MSF, unlike ot her processes, t he governance m odel is separat ed from t he operat ional process. Governance concerns t he alignm ent of t he proj ect wit h it s cust om ers, whereas t he operat ional act ivit ies are m anaged in m uch finer- grained it erat ions ( see Figure 2.4 ) . Not e t hat t he alignm ent checkpoint s do not preclude any work from being st art ed earlier, but t hey do enable t he delivery t eam and it s cust om er t o pass an agreem ent m ilest one wit hout a need t o revisit t he earlier t rack.

Figu r e 2 .4 .

Th e gove r n a n ce m ode l in M SF for CM M I Pr oce ss I m pr ove m e n t ca lls for e x plicit bu sin e ss a lign m e n t ch e ck poin t s a t t h e e n d of e a ch t r a ck , w h e r e a s M SF for Agile Soft w a r e D e ve lopm e n t a ssu m e s a m or e con t in u ou s a n d le ss for m a l e n ga ge m e n t .

[View full size image]

Governance Checkpoints MSF for CMMI Process I m provem ent has explicit governance checkpoint s t hat are organized at t he end of each t rack. MSF for Agile Soft ware Developm ent relies on a less form al m odel.

VSTS provides a web- host ed proj ect port al, based on Windows SharePoint Services, t o m ake t he proj ect m ore t ransparent t o all t he st akeholders out side t he im m ediat e proj ect t eam ( see Figure 2.5 ) . Of course, access t o t he port al can be cont rolled by perm ission and policy.

Figu r e 2 .5 .

Th e pr oj e ct por t a l pr ovide s a da ily vie w of t h e pr oj e ct st a t u s so t h a t ch e ck poin t s ca n a ct t o con fir m de cision s r a t h e r t h a n t o com pile n e w ly a va ila ble in for m a t ion .

[View full size image]

Auditability and Regulatory Concerns Relat ed t o sign- off gat es is t he need t o sat isfy act ual or pot ent ial audit ors. I n m any indust ries, t he risk of lit igat ion ( over product safet y, cust om er privacy, business pract ices, and so on) drives t he creat ion of cert ain docum ent at ion t o prove t he suit abilit y of t he developm ent process. At t he t op of t he list for all publicly t raded U.S. com panies is Sarbanes- Oxley ( SOX) com pliance, which requires execut ive oversight of corporat e finance and accordingly all I T developm ent proj ect s t hat concern m oney or resources. Som et im es t hese audit ors are t he well- known indust ry regulat ors, such as t he FDA ( pharm aceut icals

and m edical equipm ent ) , SEC ( securit ies) , FAA ( avionics) , or t heir count erpart s in t he European Union and ot her count ries. Usually, t hese regulat ors have very specific rules of com pliance.

Audit and Appraisal MSF for CMMI Process I m provem ent defines an Audit or role and has explicit act ivit ies, work product s, and report s t hat gat her evidence for t he audit . MSF for Agile Soft ware Developm ent also keeps a full audit t rail of changes but does not prescribe t he gat hering of evidence for an audit .

Prescribed Versus Self-Organization High- perform ance t eam s t end t o be self- direct ed t eam s of peers t hat t ake advant age of t he diverse personal st rengt hs of t heir m em bers. Such t eam s st rive t o organize t hem selves wit h flexible roles and enable individuals t o pick t heir own t asks. They follow t he principle t hat " responsibilit y cannot be assigned; it can only be accept ed." [ 18] I n ot her organizat ions, t his ideal is less pract ical. People have t o assum e roles t hat are assigned. Oft en a good coach m ay have a bet t er idea t han t he players of t heir capabilit ies and developm ent al opport unit ies. Cent ral assignm ent m ay also be necessary t o t ake advant age of specialist skills or t o balance resources. This const raint becom es anot her fact or in det erm ining t he right process. I n eit her case, it is im port ant t o recognize t he dist inct ion bet ween roles on a proj ect and j ob descript ions. MSF m akes a point of describing roles, which consist of relat ed act ivit ies t hat require sim ilar skills, and of em phasizing t hat one person can fill m ult iple roles. Most career ladders have condit ioned people t o t hink in t erm s of j ob descript ions. This is an unpleasant dichot om y. When st affing your proj ect , t hink about t he skills your t eam offers and t hen m ake sure t hat t he roles can be covered, but don't be slavish about part it ioning roles according t o j ob t it les.

Specificity of Roles MSF for CMMI Process I m provem ent assum es a m ore prescribed organizat ion and defines up t o eight een roles. MSF for Agile Soft ware Developm ent defines six roles and can be scaled down t o t hree.

One Project at a Time Versus Many Projects at Once One of t he m ost valuable planning act ions is t o ensure t hat your t eam m em bers can focus on t he proj ect at hand wit hout ot her com m it m ent s t hat drain t heir t im e and at t ent ion. Twent y years ago, Gerald Weinberg proposed a rule of t hum b t o com put e t he wast e caused by proj ect swit ching, shown

in Table 2.1 [ 19]

Ta ble 2 .1 . W a st e Ca u se d by Pr oj e ct Sw it ch in g N u m be r of Sim u lt a n e ou s Pr oj e ct s

Pe r ce n t of W or k in g Tim e Ava ila ble pe r Pr oj e ct

Loss t o Con t e x t Sw it ch in g

1

100%

0%

2

40%

20%

3

20%

40%

4

10%

60%

5

5%

75%

I n sit uat ions of cont ext swit ching, you need t o perform m uch m ore careful t racking, including effort planned and spent , so t hat you can correlat e result s by proj ect . A relat ed issue is t he ext ent t o which t eam m em bers need t o t rack t heir hours on a proj ect and use t hose num bers for est im at ion and act ual work com plet ion. I f your t eam m em bers are dedicat ed t o your proj ect , t hen you ought t o be able t o use calendar t im e as effect ively as hours of effort t o est im at e and t rack t ask com plet ion. However, if t heir t im e is split bet ween m ult iple proj ect s, t hen you need t o account for m ult it asking. This issue adds a subt le com plexit y t o m any m et rics concerning est im at ion and com plet ion of effort . Not only are t hey harder t o t rack and report , but also t he quest ion of est im at ing and t racking overhead for t ask swit ching becom es an issue. I n m any organizat ions, however, it is a fact of life t hat individuals have t o work on m ult iple proj ect s, and VSTS m akes t his possible. VSTS keeps all t he relevant proj ect s on a Team Foundat ion Server and displays t he act ive ones you have chosen on t he Team Explorer ( see Figure 2.6 ) .

For Detailed Instructions on Setting Up VSTS for Multiple Projects, See This MSDN Section Developm ent Tools and Technologies Visual St udio Team Syst em Team Foundat ion Team Foundat ion Proj ect Leads Creat ing and Managing Team Proj ect s

Figu r e 2 .6 .

VSTS m a k e s it possible t o sw it ch con t e x t be t w e e n m u lt iple pr oj e ct s ( su ch a s n e w de ve lopm e n t a n d m a in t e n a n ce ) dir e ct ly in t h e de ve lopm e n t e n vir on m e n t . Th e book k e e pin g of code ve r sion s, t e st s, docu m e n t s, a n d w or k it e m s is h a n dle d a u t om a t ica lly. Alt h ou gh t h is a ppr oa ch doe s n ot r e m ove t h e cogn it ive loa d, it doe s e lim in a t e m a n y e r r or s t h a t a r e ca u se d by t h e m e ch a n ica l com ple x it y of h a n dlin g m u lt iple pr oj e ct s.

[View full size image]

Geographic and Organizational Boundaries Team s are oft en dist ribut ed geographically, som et im es around t he world. The VSTS t eam , for exam ple, was dist ribut ed in Redm ond, Raleigh, Copenhagen, Hyderabad, and Beij ing. When you have t o work across locat ions and t im e zones, you t ypically need t o rely on explicit docum ent at ion over t acit knowledge and m ust be m ore of a prescribed organizat ion t han a self- organizing t eam . I t is im port ant t o have clear t echnical boundaries and int erfaces bet ween t he local t eam s and leave decisions wit hin t he boundaries of t he t eam s.

Service Oriented Architecture and Organizational Boundaries Service Orient ed Archit ect ure is a great t echnical m odel for designing a syst em so t hat different services can be im plem ent ed by t eam s in different locat ions. See Chapt er 5, " Archit ect ural Design"

A very large am ount of I T work is out sourced, t hat is, cont ract ed wit h a separat e com pany. The out sourcer m ight be local or on t he ot her side of t he planet . These cases obviously require business cont ract s, which t end t o specify deliverables in high det ail. Clearly, t hese cases also require explicit docum ent at ion and a m ore plan- driven t han adapt ive approach.

Explicit Documentation MSF for CMMI Process I m provem ent requires m ore explicit docum ent at ion and provides a m ore plan- driven approach.

Summary Chapt er 1, " A Value- Up Paradigm " , argued t he im port ance of t he value- up paradigm . This chapt er addressed t he issues of choosing and applying a suit able process in t he value- up cont ext . I n VSTS, t he t wo processes delivered by Microsoft are MSF for Agile Soft ware Developm ent and MSF for CMMI Process I m provem ent . Bot h are value- up processes, relying on it erat ive developm ent , it erat ive priorit izat ion, cont inuous im provem ent , const it uency- based risk m anagem ent , and sit uat ionally specific adapt at ion of t he process t o t he proj ect . MSF for CMMI Process I m provem ent st ret ches t he pract ices of it s m ore agile cousin t o fit cont ext s t hat require a m ore explicit governance m odel, docum ent at ion, prescribed organizat ion, and audit abilit y. I t support s appraisal t o CMMI Level 3, but m ore im port ant ly, it provides a low- overhead, agile m odel of CMMI .

Endnotes 1.

Alist air Cockburn coined t he phrase " St ret ch t o Fit " in his Cryst al fam ily of m et hodologies and largely pioneered t his discussion of cont ext wit h his paper " A Met hodology Per Proj ect ," available at ht t p: / / alist air.cockburn.us/ cryst al/ art icles/ m pp/ m et hodologyperproj ect .ht m l.

2.

For a com parison of t hirt een published processes, see Barry Boehm and Richard Turner, Balancing Agilit y wit h Discipline: A Guide for t he Perplexed ( Bost on: Addison- Wesley, 2004) , 168194.

3.

As Kent Beck pit hily put it : " All m et hodologies are based on fear." Kent Beck, Ext rem e Program m ing Explained: Em brace Change, First Edit ion ( Bost on: Addison- Wesley, 2000) , 165.

4.

Barry Boehm and Richard Turner, Balancing Agilit y wit h Discipline: A Guide for t he Perplexed ( Bost on: Addison- Wesley, 2004) , 152.

5.

www.agilealliance.org

6.

www.sei.cm u.edu

7.

See David J Anderson, " St ret ching Agile t o fit e CMMI Level 3," present ed at t he Agile Conference, Denver 2005, available from ht t p: / / www.agilem anagem ent .net / Art icles/ Papers/ St ret chingAgilet oFit CMMI L.ht m l.

8.

[ I EEE St d 610.12- 1990] , 39.

9.

Frederick P. Brooks, Jr., The Myt hical Man- Mont h: Essays on Soft ware Engineering, Twent iet h Anniversary Edit ion ( Reading, MA: Addison- Wesley, 1995) , 266.

1 0 . ht t p: / / www.csc.ncsu.edu/ facult y/ xie/ realopt ionse.ht m 1 1 . Tom DeMarco and Tim ot hy List er, Walt zing wit h Bears: Managing Risk on Soft ware Proj ect s ( New York: Dorset House, 2003) , 137.

1 2 . The convent ion of " it erat ion" for t he planning cycle and " increm ent " for t he coding/ t est ing one is com m on. See, for exam ple, Alist air Cockburn's discussion at ht t p: / / saloon.j avaranch.com / cgi- bin/ ubb/ ult im at ebb.cgi?ubb= get _t opic&f= 42&t = 000161.

1 3 . Anderson 2004, op. cit ., 89. 1 4 . MoSCoW scheduling is a t echnique of t he DSDM m et hodology. For m ore on t his, see ht t p: / / www.dsdm .org.

1 5 . DeMarco and List er, op. cit ., 130. 1 6 . Cockburn, op. cit . 1 7 . Scot t W. Am bler and Ron Jeffries, Agile Modeling: Effect ive Pract ices for Ext rem e Program m ing and t he Unified Process ( New York: Wiley, 2002) .

1 8 . Kent Beck wit h Cynt hia Andres, Ext rem e Program m ing Explained: Em brace Change, Second Edit ion ( Bost on: Addison- Wesley, 2005) , 34.

1 9 . Gerald M. Weinberg, Qualit y Soft ware Managem ent : Syst em s Thinking ( New York: Dorset House, 1992) , 284.

Chapter 3. Requirements

Business Analyst Product Manager User Experience " The single hardest part of building a soft ware syst em is deciding precisely what t o build." Frederick Brooks, Myt hical ManMont h

Figu r e 3 .1 .

Edison pa t e n t e d t h e ligh t bu lb in 1 8 8 0 , bu t it is st ill be in g im pr ove d t oda y.

[ 1]

United States Patent and Trademark Office I n t he previous chapt er, I discussed t he choice of soft ware process and t he cont ext ual issues t hat would favor one process over anot her. Regardless of your process choice, you need t o st art by envisioning your solut ion requirem ent s. Usually, t his responsibilit y is led by a business analyst or product m anager, and in pure XP, it is done by an onsit e cust om er. Obviously, you have t o do m uch of t he solut ion definit ion before you st art your proj ect . You won't get funding, recruit t eam m em bers, or ident ify t he rest of work unt il you know what you're t rying t o do.

Vision Statement, Personas, Scenarios, and Quality of Service Requirements I n VSTS, t he vision st at em ent and personas are capt ured in docum ent s accessible from t he Team Explorer and Proj ect Port al. Scenarios and qualit y of service requirem ent s are work it em s, and t hey are t racked and m aint ained like ot her work it em s in t he work it em dat abase. There are only m inor differences bet ween MSF for Agile Soft ware Developm ent and MSF for CMMI Process I m provem ent wit h regard t o envisioning, m ost ly in t he det ails of t heir work it em t ype definit ions.

MSF defines requirem ent s prim arily as scenarios and qualit ies of service ( QoS). VSTS uses specific work it em t ypes t o capt ure and t rack t hese requirem ent s so t hat t hey show up in t he proj ect backlog, ranked as an ordinary part of t he work. You cont inue t o refine your requirem ent s t hroughout your proj ect as you learn from each it erat ion. Because t hese requirem ent s are work it em s, when you m ake changes t o t hem , a full audit is capt ured aut om at ically, and t he changes flow direct ly int o t he m et rics warehouse.

What's Your Vision? Every proj ect should have a vision t hat every st akeholder can underst and. Oft en t he vision st at em ent is t he first st ep in get t ing a proj ect funded. Even when it 's not , t reat ing t he vision as t he m ot ivat ion for funding is helpful in clarifying t he core values of t he proj ect . Cert ainly it should capt ure why a cust om er or end user will want t he solut ion t hat t he proj ect im plem ent s. The short er t he vision st at em ent is, t he bet t er it is. A vision st at em ent can be one sent ence long, or one paragraph, or longer only if needed. I t should speak t o t he end user in t he language of t he user's dom ain. A sign of a successful vision st at em ent is t hat everyone on t he proj ect t eam can recit e it from m em ory and connect t heir daily work t o it .

Strategic Projects Proj ect s can be st rat egic or adapt ive. St rat egic proj ect s represent significant invest m ent s, based on a plan t o im prove significant ly over t heir predecessors. For exam ple, when st art up com panies form around a new product idea, or when new business unit s are launched, t he rat ionale is t ypically a bold st rat egic vision. A useful form at for t hinking about t he key values of such a st rat egic proj ect , and hence t he vision st at em ent , is t he "elevat or pit ch." [ 2 ] I t 's called an elevat or pit ch because you can recit e it t o a prospect ive cust om er or invest or in t he brief durat ion of an elevat or ride. I t 's useful because it capt ures t he vision crisply enough t hat t he cust om er or invest or can rem em ber it from t hat short encount er. Geoffrey Moore invent ed a succinct form for t he evaluat or pit ch.

Ta ble 3 .1 . Ele va t or Pit ch For

( t arget cust om er personas in ident ified segm ent only)

Who are dissat isfied with

( t he current ... alt ernat ive)

Our solut ion is a

( new product cat egory)

That provides

( key problem solving capabilit y)

Unlike

( t he product alt ernat ive)

We have assem bled

( key scenarios and QoS for your solut ion)

Adaptive Projects On t he ot her hand, m ost I T proj ect s are m ore adapt ive. They t ackle increm ent al changes t o exist ing syst em s. I t is oft en easiest and m ost appropriat e t o describe t he vision in t erm s of business process changes. I n t hese cases, if a business process m odel or descript ion is available, it is probably t he best st art ing point . Use t he asis m odel as t he current alt ernat ive and t he proposed one as t he solut ion.

Business Process Models I n t he current MSF process t em plat es, t here are work product s for t he vision st at em ent , scenarios, and qualit ies of service. However, current ly t here are no direct act ivit ies for business process m odeling. I f you need t o define a business process m odel, VSTS does include Microsoft Visio wit h useful st encils for diagram m ing business process flow.

When to Detail Requirements " Analysis paralysis" is a com m on condem nat ion of st alled proj ect s. This com plaint reflect s t he experience t hat t rying t oo hard t o pin down proj ect requirem ent s early can prevent any forward m ovem ent . This can be a significant risk in envisioning. There are t wo fact ors t o keep in m ind t o m ediat e t his risk: t he shelf life of requirem ent s and t he audience for t hem .

Requirements Are Perishable One of t he great insight s of t he valueup paradigm is t hat requirem ent s have a lim it ed shelf life, for four reasons: Th e bu sin e ss e n vir on m e n t or pr oble m spa ce ch a n ge s. Com pet it ors, regulat ors, cust om ers, users, t echnology, and m anagem ent all have a way of alt ering t he assum pt ions on which your requirem ent s are based. I f you let t oo m uch t im e lapse bet ween t he definit ion of requirem ent s and t heir im plem ent at ion, you risk discovering t hat you need t o redefine t he requirem ent s t o reflect a changed problem . Th e k n ow le dge be h in d t h e r e qu ir e m e n t s be com e s st a le . When an analyst or t eam is capt uring requirem ent s, knowledge of t he requirem ent s is at it s peak. The process of com m unicat ing t heir m eanings, exploring t heir im plicat ions, and underst anding t heir gaps should happen t hen. Docum ent at ion m ight capt ure som e of t his inform at ion for int erm ediat e purposes such as cont ract ual signoff, but ult im at ely what m at t ers is t he im plem ent at ion. The m ore closely you couple t he im plem ent at ion t o t he requirem ent s definit ion, t he less you allow t hat knowledge t o decay. Th e r e is a psych ologica l lim it t o t h e n u m be r of r e qu ir e m e n t s t h a t a t e a m ca n m e a n in gfu lly con side r in de t a il a t on e t im e . The sm aller you m ake t he bat ch of requirem ent s, t he m ore at t ent ion you get t o qualit y design and im plem ent at ion. Conversely, t he larger you allow t he requirem ent s scope of an it erat ion t o grow, t he m ore you risk confusion and carelessness. Th e im ple m e n t a t ion of on e it e r a t ion 's r e qu ir e m e n t s in flu e n ce s t h e de t a ile d de sign of t h e r e qu ir e m e n t s of t h e n e x t it e r a t ion . Design doesn't happen in a vacuum . What you learn from one it erat ion enables you t o com plet e t he design for t he next . I n t he discussion about capt uring requirem ent s t hat follows, I risk encouraging you t o do t oo m uch t oo early. I don't m ean t o. Think coarse and underst andable init ially, sufficient t o priorit ize requirem ent s int o it erat ions. The det ailed requirem ent s and design can follow.

Who Cares About Requirements For m ost proj ect s, t he requirem ent s t hat m at t er are t he ones t hat t he t eam m em bers and st akeholders underst and. On a successful proj ect , t hey all underst and t he sam e requirem ent s; on an unsuccessful one, t hey have divergent expect at ions ( see Figure 3.2 ) .

Figu r e 3 .2 .

Th e Xa x is sh ow s t h e de gr e e of a m bigu it y ( opposit e of spe cificit y) in t h e r e qu ir e m e n t s docu m e n t s; t h e Ya x is sh ow s t h e r e su lt in g u n de r st a n da bilit y.

Dean Leffingwell and Don Widrig, Managing Software Requirements (Boston, MA: AddisonWesley, 2000), 273 This t ension bet ween specificit y and underst andabilit y is an inherent ly hard issue. The m ost precise requirem ent s are not necessarily t he m ost int elligible, as illust rat ed in Figure 3.2 . A m at hem at ical form ulat ion or an execut able t est m ight be m uch harder t o underst and t han a m ore com m onplace pict ure wit h a prose descript ion. I t is im port ant t o t hink of requirem ent s at m ult iple levels of granularit y. For broad com m unicat ion, priorit izat ion, and cost ing t o a rough order of m agnit ude, you need a coarse underst anding t hat is easy t o grasp. On t he ot her hand, for im plem ent at ion of a requirem ent wit hin t he current it erat ion, you need a m uch m ore det ailed specificat ion. I ndeed, if you pract ice t est driven developm ent ( see Chapt er 6, " Developm ent " ) , t he ult im at e det ailed requirem ent definit ion is t he execut able t est . Think about t he audience for your requirem ent s work. The audience will largely det erm ine t he breadt h and precision you need t o apply in specifying requirem ent s. At one end of t he scale, in eXt rem e Program m ing, you m ay have a sm all, collocat ed t eam and an onsit e cust om er. You not e fut ure requirem ent s on a 3" x 5" card or as t he t it le on a work it em , and you m ove on unt il it is t im e t o im plem ent . When it is t im e t o do a requirem ent , wit hin t he sam e session you discuss t he requirem ent wit h t he cust om er and writ e t he t est t hat validat es it , j ust before im plem ent at ion. [ 3 ] At t he ot her ext rem e, your requirem ent s specificat ion m ay be subj ect t o regulat ory approval, audit , or com pet it ive bidding. At t his end, you m ay want every det ail spelled out so t hat every subsequent change can be individually cost ed and subm it t ed for approval. [ 4 ]

MSF recom m ends t hat you plan and det ail requirem ent s for one it erat ion at a t im e. Effect ively, t he it erat ion becom es a bat ch of requirem ent s t hat progress t hrough realizat ion. For m ost proj ect s, t his is a balanced m edium t hat m axim izes com m unicat ion, keeps requirem ent s fresh, avoids wast ed work, and keeps t he focus on t he delivery of cust om er value.

Personas and Scenarios I n a valueup paradigm , qualit y is m easured as value t o t he cust om er. When you don't have a live cust om er on sit e, or your cust om ers can't be represent ed by a single individual, personas and scenarios are t he t ools you use t o underst and and com m unicat e t hat value.

Start with Personas To get clear goals for a proj ect , you need t o know t he int ended users. St art wit h recognizable, realist ic, appealing personas. By now, personas have becom e com m onplace in usabilit y engineering lit erat ure, but before t he t erm personas becam e popular t here, t he t echnique had been ident ified as a m eans of product definit ion in m arket research. Moore described t he t echnique well: The place m ost . . . m arket ing segm ent at ion get s int o t rouble is at t he beginning, when t hey focus on a t arget m arket or t arget segm ent inst ead of on a t arget cust om er . . . We need som et hing t hat feels a lot m ore like real people . . . Then, once we have t heir im ages in m ind, we can let t hem guide us t o developing a t ruly responsive approach t o t heir needs. [ 5 ] I n t urn, personas are t he int ended users who are accom plishing t he scenarios. For proj ect s wit h ident ified users, who are paying cust om ers, you can nam e t he personas for represent at ive act ual users. For a broad m arket product , your personas will probably be com posit e fict ional charact ers, who don't bet ray any personal inform at ion. ( Not e t hat t here are ot her st akeholders, such as t he users' boss, who m ay be funding t he proj ect but are at least one st ep rem oved from t he personal experience of t he users.) Alan Cooper has described t he charact erist ics of good personas: Personas are user m odels t hat are represent ed as specific, individual hum ans. They are not act ual people, but are synt hesized direct ly from observat ions of real people. One of t he key elem ent s t hat allow personas t o be successful as user m odels is t hat t hey are personificat ions ( Const ant ine and Lockwood, 2000) . They are represent ed as specific individuals. This is appropriat e and effect ive because of t he unique aspect s of personas as user m odels: They engage t he em pat hy of t he developm ent t eam t owards t he hum an t arget of t he design. Em pat hy is crit ical for t he designers, who will be m aking t heir decisions for design fram eworks and det ails based on bot h t he cognit ive and em ot ional dim ensions of t he persona, as t ypified by t he persona's goals. [ 6 ] Good personas are m em orable, t hreedim ensional, and sufficient ly well described t o be decisive when t hey need t o be. I f you were m aking a m ovie, personas would be t he profiles you would give t o t he cast ing agent . They describe t he cat egories of users not j ust in t erm s of j ob t it les and funct ions but also in t erm s of personalit y, work or usage environm ent , educat ion, com put er experience, and any ot her charact erist ics t hat m ake t he im agined person com e t o life. Assign each persona a nam e and at t ach a pict ure so t hat it is m em orable. Som e t eam s in Microsoft put post ers on t he wall and hand out fridge m agnet s t o reinforce t he presence of t he personas. [ 7 ] Personas can also be useful for describing adversaries. For exam ple, if securit y and privacy are a concern, t hen you probably want t o profile t he hacker who t ries t o break int o your syst em . The vandal who t ries indiscrim inat ely t o bring down your web sit e is different from t he t hief who t ries t o st eal your ident it y wit hout det ect ion. Wit h disfavored personas, everyone can be m ade conscious of ant iscenariost hings t hat m ust be prevent ed.

Scenarios I n MSF, scenarios are t he prim ary form of funct ional requirem ent s. By " scenario," MSF m eans t he single int eract ion pat h t hat a user ( or group of collaborat ing users, represent ed by personas) perform s to achieve a known goal. The goal is t he key here, and it needs t o be st at ed in t he user's language, such as Place an order . As your solut ion definit ion evolves, you will evolve t he scenario int o a specific sequence of st eps wit h sam ple dat a, but t he goal of t he scenario should not change. There are m any reasons t o use scenarios t o define your solut ion: Scenarios com m unicat e t angible cust om er value in t he language of t he cust om er in a form t hat can be validat ed by cust om ers. Part icularly useful are quest ions in t he form of " What if you could [ accom plish goal] like [ t his scenario] ?" You can use t hese quest ions in focus groups, cust om er m eet ings, and usabilit y labs. As t he solut ion evolves, scenarios nat urally t urn int o prot ot ypes and working funct ionalit y. Scenarios enable t he t eam t o plan and m easure t he flow of cust om er value. This enables t he creat ion of a progressive m easurem ent of readiness and gives t he t eam a nat ural unit of scoping. Scenarios unit e t he narrow and ext ended t eam s. All st akeholders underst and scenarios in t he sam e way so t hat differences in assum pt ions can be exposed, biases overcom e, and requirem ent s discussions m ade t angible. A key is t o st at e goals in t he language and cont ext of t he user. The goalsuch as Place an order or Ret urn goodsdefines t he scenario, even t hough t he st eps t o accom plish it m ay change. Even t hough you m ay be m uch m ore fam iliar wit h t he language of possible solut ions t han wit h t he cust om ers' problem s, and even t hough you m ay know one part icular subset of users bet t er t han t he ot hers, you need t o st ick t o t he users' language. Alan Cooper describes t his well: Goals are not t he sam e as t asks. A goal is an end condit ion, whereas a t ask is an int erm ediat e st ep t hat helps t o reach a goal. Goals m ot ivat e people t o perform t asks . . . Goals are driven by hum an m ot ivat ions, which change very slowly, if at all, over t im e. [ 8 ] Oft en, t he goals are chosen t o address pain point s, t hat is, issues t hat t he user faces wit h current alt ernat ives, which t he new solut ion is int ended t o solve. When t his is t rue, you should capt ure t he pain point s wit h t he goals. At ot her t im es, t he goals are int ended t o capt ure excit ersbe sure t o t ag t hese as well so t hat you can t est how m uch delight t he scenario act ually brings.

Research Techniques We find t he personas by observing t arget users, preferably in t heir own environm ent . Several t echniques can be appliedt wo widely used ones are focus groups and cont ext ual inquiries. For all t hese t echniques, it is im port ant t o select a represent at ive sam ple of your users. A good rule of t hum b is not t o be swayed heavily by users who are t oo novice or t oo expert and inst ead t o look for int erm ediat es who represent t he bulk of your t arget populat ion for m ost of your solut ion's life. [ 9 ] Focus groups are essent ially direct ed discussions wit h a group of carefully chosen int erview subj ect s wit h a skilled facilit at or. When conduct ed professionally, focus groups usually are closed sessions wit h observers sequest ered behind a oneway m irror ( or on t he end of a video feed) . Open focus groups work t oo, where observers sit in t he back of t he room , provided t hat t hey sit quiet ly and let t he discussion unfold wit hout int erj ect ion. Focus groups are undoubt edly useful, especially when you are gat hering issues expansively. At t he

sam e t im e, you need t o be very careful t o dist inguish bet ween what people say and what t hey do. There are m any foibles t hat m ake t he dist inct ion im port ant : wishful t hinking, overloaded t erm inology, desires t o im press, and hum an difficult y at selfobservat ion, t o nam e a few. People oft en describe t heir ( or t heir colleagues') behavior different ly from how an out side observer would. Cont ext ual inquiry is a t echnique t hat at t em pt s t o overcom e t his problem by direct ly observing users in t heir own environm ent when t hey are doing t heir everyday t asks. [ 10] Cont ext ual inquiry is a t echnique borrowed from ant hropology. The observer wat ches, occasionally asks openended j ournalist ic quest ions ( Who, What , When, Where, Why, How ) , does not t ry t o bias t he user's act ivit ies, and does not t ry t o offer solut ions. I t is im port ant t o underst and pain point s and opport unit ieswhere t he user is annoyed, where t hings can be m ade bet t er, what would m ake a big difference. I f possible, gat her relevant docum ent s and ot her work product s and t ake pict ures of t he user's environm ent , especially reference m at erial post ed on t he wall. My favorit e anecdot e t o illust rat e t he problem of selfreport ing com es from a cust om er visit t o a m aj or bank in 2003 when Microsoft was t rying t o define Visual St udio Team Syst em requirem ent s. I n a m orning m eet ing, which was run as an open focus group, t he Microsoft t eam asked about developm ent pract ices, especially unit t est ing. They were t old t hat unit t est ing was a universal pract ice am ong t he bank's developers and indeed was a requirem ent before checking in code. One of t he Microsoft usabilit y engineers spent t hat aft ernoon wit h one of t he developers in his cube. ( That 's why it 's called a cont ext ual int erview.) The developer went t hrough several coding t asks and checked code in several t im es. Aft er a couple hours, t he usabilit y engineer asked, " I 've seen you do several checkins, and I 'm wondering, do you do unit t est ing, t oo?" " Of course," cam e t he reply, " I press F5 before checking int hat 's unit t est ing." ( I n case you're not fam iliar wit h Visual St udio, F5 is t he com pilat ion key and has not hing t o do wit h t est ing.) The report ed pract ice of unit t est ing was sim ply not found in pract ice. I n designing VSTS, we int erpret ed t his discrepancy as an opport unit y t o m ake unit t est ing easy and pract ical for developers in organizat ions like t his. ( See Chapt er 6, "Developm ent .") One user does not m ake a persona, and you m ust repeat t he observat ion unt il you feel t hat you have clear pat t erns of behavior t hat you can dist ill int o a persona descript ion.

Get Specific Early When you have defined personas, you can describe t he scenarios for t he specific personas. Scenarios, t oo, were first ident ified as a m arket research t echnique. Again, Geoffrey Moore: [ Charact erizing scenarios] is not a form al segm ent at ion surveyt hey t ake t oo long, and t heir out put is t oo dry. I nst ead, it is a t apping int o t he fund of anecdot es t hat act ually carries business knowledge in our cult ure. Like m uch t hat is anecdot al, t hese scenarios will incorporat e fict ions, falsehoods, prej udices, and t he like. Nonet heless, t hey are by far t he m ost useful and m ost accurat e form of dat a t o work wit h at t his st age in t he segm ent at ion process. [ 11] St art wit h t he scenario goal. Then break down t he goal int o a list of st eps, at what ever granularit y is m ost underst andable. Use act ion verbs t o enum erat e t he st eps. Keep t he scenarios in your user's language, not in t he language of your im plem ent at ion. I n ot her words, scenarios should be described first from t he problem space, not t he solut ion space. Don't t ry t o det ail t he alt ernat e and except ion pat hs init iallyt hat usually ham pers underst andabilit y and quickly becom es t oo com plicat ed t o m anagebut record j ust t heir nam es as ot her scenarios and save t hem for fut ure it erat ions. When you are com fort able wit h t he sequence, you can add t he second part t o t he st ep definit ion, t hat

is, how t he int ended solut ion responds. You now have a list in t he form of PersonaDoesSt ep, Solut ionShowsResult . Tag t he st eps t hat will elicit a " Wow! " from your user. ( These are t he excit ers, which are discussed lat er in t his chapt er.)

Figu r e 3 .3 .

W h e n w e w e r e de ve lopin g VSTS, w e u se d h igh le ve l sce n a r ios t o e n vision t h e pr odu ct scope . Th is is a ve r y e a r ly e x a m ple . Ove r t im e , t h e lin e s of t h is slide be ca m e sce n a r io w or k it e m s in ou r w or k it e m da t a ba se , a n d w e u se d t h is a s t h e pr im a r y list t o dr ive e x e cu t ion a n d t e st in g.

[View full size image]

Storyboards Alt hough scenarios st art as goals and t hen evolve int o list s of st eps, t hey don't st op t here. When you are com fort able t hat t he st eps and t heir sequence define t he experience you want , elaborat e t hem int o wirefram es, st oryboards, prot ot ypes, and working it erat ions. Alt hough t he goal of each scenario should rem ain const ant , t he det ails and st eps m ay evolve and change wit h every it erat ion. Wirefram es and st oryboards are oft en essent ial t o com m unicat e t he flow of int eract ion effect ively, especially when you have rich dat a and st at e in t he solut ion being designed. Microsoft Visio provides a st encil for drawing wirefram es of user int erfaces. For each t ask, wit h a t it le in t he form of " Persona does st ep," draw a wirefram e. Sequence t he wirefram es t oget her, paying at t ent ion t o t he flow and dat a, and you have a st oryboard. This can be a cut andpast e of t he Visio screens int o PowerPoint . I f

you need t o dist ribut e t he st oryboard, you can easily m ake a narrat ed m ovie of it using Producer for PowerPoint or by recording a LiveMeet ing.

Figu r e 3 .4 .

Th is is a ve r y e a r ly Visio w ir e fr a m e fr om t h e in it ia l sce n a r ios for VSTS. N ot a ll t h e w in dow s sh ow n w e r e im ple m e n t e d, a n d n ot a ll t h e w in dow s im ple m e n t e d w e r e de sign e d a s e n vision e d in t h e w ir e fr a m e . Th r e e su cce ssive r ou n ds of t e st in g in u sa bilit y la bs a n d su bse qu e n t r e vision s w e r e n e e de d be for e u se r s cou ld a ch ie ve t h e in t e n de d goa l e ffe ct ive ly.

[View full size image]

Breadth of Scenarios I t is useful t o t hink of t he end- t o- end st ory t hat your solut ion needs t o t ell and t o line up t he scenarios accordingly. This st ory should st art wit h t he cust om er deploying your solut ion and run t hrough t he cust om er realizing t he value. This is equally useful whet her you are building an int ernal, bespoke solut ion or a com m ercial product . I f you t ake t his approach, you m ust deal wit h bot h t he issues of discoverabilit yhow first t im e users find t he necessary feat uresand t he longit udinal issueshow int erm ediat e users, as t hey becom e m ore experienced, becom e product ive.

How m any scenarios do you need? Typically, you should have enough scenarios t o define t he m aj or flows of t he t arget syst em and few enough t o be easy t o rem em ber. Your scenarios should be sufficient t o cover t he end- t o- end flows of t he solut ion wit hout gaps and wit hout unnecessary duplicat ion. This m eans t hat your scenarios probably need t o decom pose int o int erm ediat e act ivit ies, which in t urn cont ain t he st eps. I f you are creat ing a m ult iperson scenario, t hen it is useful t o draw a flow wit h swim lanes t o show who does what before whom .

Figu r e 3 .5 .

Th is Pow e r Poin t slide is a n illu st r a t ion of t h e sce n a r ios in a n e n d- t o- e n d st or y. Th is pa r t icu la r slide w a s u se d in t h e de fin it ion of sce n a r ios for VSTS.

[View full size image]

Customer Validation A key benefit of scenarios and it erat ions is t hat you have m any opport unit ies for cust om ers t o review your solut ion as it evolves and t o validat e or correct your course before you incur expensive rework. Depending on t he business circum st ances, you m ight want up t o t hree kinds of signoff you want t o see for your solut ion. Users ( or t heir advocat es) need t o confirm t hat t he solut ion let s t hem product ively accom plish t heir goal. Econom ic buyers, som et im es t he users' boss, som et im es called business decision m akers, need t o see t hat t he financial value is being realized from t he proj ect . Technical evaluat ors need t o see t hat t he im plem ent at ion is following t he necessary archit ect ure t o fit int o t he required const raint s of t he cust om er organizat ion. Som et im es t hese roles are collapsed, but oft en t hey are not . Scenarios are useful in com m unicat ing progress t o all t hree groups.

Early in a proj ect , t his validat ion can be done in focus groups or int erviews wit h scenarios as list s and t hen as wirefram es. St oryboards and live funct ionalit y, as it becom es available, can also be t est ed in a usabilit y lab.

Figu r e 3 .6 .

Th is is a fr a m e fr om t h e st r e a m in g vide o of a u sa bilit y la b. Th e bu lk of t h e im a ge is t h e com pu t e r scr e e n a s t h e u se r se e s it . Th e u se r is su pe r im pose d in t h e low e r r igh t so t h a t obse r ve r s ca n w a t ch a n d list e n t o t h e u se r w or k in g t h r ou gh t h e la b a ssign m e n t s.

[View full size image]

Usabilit y labs are set t ings in which a t arget user is given a set of goals t o accom plish wit hout coaching in a set t ing t hat is as realist ic as possible. At Microsoft , we have room s perm anent ly out fit t ed wit h oneway m irrors and video recording, which enable spect at ors t o wat ch behind t he glass or over st ream ing video. Alt hough t his set up is convenient , it is not necessary. All you need is a basic video cam era on a t ripod or a service such as Microsoft LiveMeet ing wit h a webcam and deskt op recording. The t hree keys t o m aking a usabilit y lab effect ive are as follows: Creat e a t rust ing at m osphere in which t he user knows t hat t he soft ware is being t est ed, not t he user.

Have t he user t hink out loud cont inually so t hat you can hear t he unfilt ered int ernal m onologue. Don't " lead t he wit ness" t hat is, don't int erfere wit h t he user's explorat ion and discovery of t he soft ware under t est . Usabilit y labs, like focus groups and cont ext ual int erviews, are ways t o challenge your assum pt ions regularly. Rem em ber t hat requirem ent s are perishable and t hat you need t o revisit user expect at ions and sat isfact ion wit h t he pat h t hat your solut ion is t aking.

Evolving Scenarios How do you know when you've defined your scenarios well enough? When t he t eam has a com m on pict ure of what t o build and t est in t he next it erat ion. There are t hree reasonable ways t o t est whet her t he scenarios com m unicat e well enough: Cust om ers can review t he scenarios and provide m eaningful feedback on suit abilit y for achieving t he goal, ( in) com plet eness of st eps, and relevance of t he dat a. Test ers can underst and t he sequence sufficient ly t o define t est s, bot h posit ive and negat ive. They can ident ify how t o t est t he sequence, what dat a is necessary, and what variat ions need t o be explored. The developm ent or archit ect ure t eam can design t he services, com ponent s, or feat ures necessary t o deliver t he flow by ident ifying t he necessary int eract ions, int erfaces, and dependencies. On t he ot her hand, if you discover t hat your scenarios don't capt ure t he int ended funct ionalit y sufficient ly for t hese st akeholders, you can add m ore det ail or ext end t he scenarios furt her. Dem os of newly im plem ent ed scenarios during or at t he end of an it erat ion are very m ot ivat ing. They can energize t he t eam , creat e valuable peer int eract ion, and if cust om ers are available, generat e great dialog and ent husiasm . As it erat ions progress, you should add scenarios. I n early it erat ions, you t ypically focus on sat isfiers and excit ers, whereas in lat er it erat ions, you should ensure t he absence of dissat isfiers. This approach nat urally leads you t o scenarios t hat explore alt ernat e flows t hrough t he solut ion. The next chapt er shows exam ples of t urning scenario com plet ion int o a scorecard for t racking velocit y and progress.

Personas and Scenarios and Their Alternatives Alt hough MSF uses personas and scenarios t o capt ure t he vision of users and t heir requirem ent s, t here are alt ernat ive t erm s and t echniques used by ot her processes.

Actors and Use Cases I f you are fam iliar wit h m et hodologies such as Rat ional Unified Process, you probably know t he t erm s " act or" and " use case," [ 12] and you m ight be wondering whet her persona and scenario m ean t he sam e t hing. MSF int ent ionally dist inguishes scenarios from use cases wit h different t erm s because MSF advocat es capt uring a level of specificit y t hat is not ordinarily capt ured in use cases. Scenarios and use cases st art sim ilarly, but t hey evolve very different ly. One of t he MSF m indset s is Get Specific Early. For exam ple, give a persona a pict ure, a nam e, and a biography inst ead of using a st ick figure wit h an anonym ous role nam e. You m ight have several personas for different dem ographics, whereas in use case analysis, you would have j ust one act or. The personas described as m ovie charact ers will be m uch m ore m em orable and m uch easier t o discuss when you envision your int ended syst em behavior. Sim ilarly t o personas, m ake scenarios specific. I nclude wirefram es or sket ches for screens and show t he specific flow of dat a from one screen t o t he next . I n use case analysis, t his specificit y is usually deferred t o t he st ep of designing a " realizat ion." Don't defer it t est t he int ent wit h t he specifics of t he exam ple im m ediat ely. Also, unlike wit h use cases, don't get bogged down in alt ernat ive flows unt il you need t hem . Your goal is t o deliver working soft ware as quickly as possible and t o learn from it . I f you deliver a working flow, you will know m uch m ore about what t he alt ernat ives should be t han if you t ry t o define t hem all in t he abst ract . The single flow of a scenario is easier t o schedule, m onit or, and revise.

User Stories I f you are fam iliar wit h eXt rem e Program m ing, t hen you know about user st ories. [ 13] I n XP, user st ories are equivalent t o t he goals of scenarios t hat I described earlierone sent ence st at em ent s of int ent writ t en on 3" x 5" cards. XP does not elaborat e t he scenarios but requires t hat an onsit e cust om er be available t o describe t he int ended flow at t he t im e of im plem ent at ion. This is a good process if you have an onsit e cust om er, a collocat ed t eam , no need t o explore alt ernat ive designs, and no requirem ent t hat ot hers sign off on designs. Unfort unat ely, m any t eam s need m ore explicit com m unicat ion and signoff and cannot rely on a single onsit e cust om er. I n t hose cases, elaborat ing user st ories int o scenarios can be really helpful. Scenarios are also t ypically larger t han user st ories, especially when conceived around end- t o- end value.

Exciters, Satisfiers, and Dissatisfiers Scenarios t end t o focus on requirem ent s of a solut ion t hat m ake users enj oy t he solut ion and achieve t heir goals. You can t hink of t hese as sat isfiers. When t he goals are im port ant and t he scenario com pelling enough t o elicit a " Wow! " from t he cust om er, we can call t he scenario an excit er. At t he sam e t im e, t here are pot ent ial at t ribut es of a solut ion ( or absence of at t ribut es) t hat can really annoy users or disrupt t heir experience. These are dissat isfiers. [ 14] Bot h need t o be considered in gat hering a solut ion's requirem ent s. Dissat isfiers frequent ly occur because qualit ies of service haven't been considered fully. " The syst em 's t oo slow," " Spyware picked up m y I D," " I can no longer run m y favorit e app," and " I t doesn't scale" are all exam ples of dissat isfiers t hat result from t he corresponding qualit ies of service not being addressed. Cust om ers and users rarely describe t he dissat isfierst hey are assum ed t o be absent . Consider t he Mont y Pyt hon sket ch, The Pet Shoppe, in which a m an goes t o a pet st ore t o ret urn a parrot he bought . He did not specify t hat t he parrot m ust be alive, so t he pet dealer sells him a dead one. The sket ch is hilarious because it reveals such a frequent m isunderst anding. St at em ent s such as " You didn't t ell m e X," " The cust om er didn't specify X," and " X didn't show up in our research" are all sym pt om s of failing t o consider t he requirem ent s necessary t o elim inat e dissat isfiers.

Figu r e 3 .7 .

Th e Pe t Sh oppe

Customer: I w ish t o com pla in a bou t t h is pa r r ot w h a t I pu r ch a se d n ot h a lf a n h ou r a go fr om t h is ve r y bou t iqu e .

Owner: Oh ye s, t h e , u h , t h e N or w e gia n Blu e . . . W h a t 's, u h . . . W h a t 's w r on g w it h it ?

Customer: I 'll t e ll you w h a t 's w r on g w it h it , m y la d. 'E's de a d, t h a t 's w h a t 's w r on g w it h it ! [ 1 5 ]

Python (Monty) Pictures LTD

Qualities of Service Scenarios are ext rem ely valuable, but t hey are not t he only t ype of requirem ent . Scenarios need t o be underst ood in t he cont ext of qualit ies of service ( QoS) . ( Once upon a t im e, QoS were called " nonfunct ional requirem ent s," but because t hat t erm is nondescript ive, I won't use it here. Som et im es t hey're called 'ilit ies, which is a useful short hand.) Most dissat isfiers can be elim inat ed by appropriat ely defining qualit ies of service. QoS m ight define global at t ribut es of your syst em , or t hey m ight define specific const raint s on part icular scenarios. For exam ple, t he securit y requirem ent t hat " unaut horized users should not be able t o access t he syst em or any of it s dat a" is a global securit y QoS. The perform ance requirem ent t hat " for 95% of orders placed, confirm at ion m ust appear wit hin t hree seconds at 1,000user load" is a specific perform ance QoS about a scenario of placing an order. Not all qualit ies of service apply t o all syst em s, but you should know which ones apply t o yours. Oft en QoS im ply large archit ect ural requirem ent s or risk, so t hey should be negot iat ed wit h st akeholders early in a proj ect . There is no definit ive list of all t he qualit ies of service t hat you need t o consider. There have been several st andards, [ 16] but t hey t end t o becom e obsolet e as t echnology evolves. For exam ple, securit y and privacy issues are not covered in t he m aj or st andards, even t hough t hey are t he m ost im port ant ones in m any m odern syst em s. The following four sect ions list som e of t he m ost com m on QoS t o consider on a proj ect .

Security and Privacy Unfort unat ely, t he spread of t he I nt ernet has m ade securit y and privacy every com put er user's concern. These t wo QoS are im port ant for bot h applicat ion developm ent and operat ions, and cust om ers are now sophist icat ed enough t o dem and t o know what m easures you are t aking t o prot ect t hem . I ncreasingly, t hey are becom ing t he subj ect of governm ent regulat ion. Se cu r it y: The abilit y of t he soft ware t o prevent access and disrupt ion by unaut horized users, viruses, worm s, spyware, and ot her agent s. Pr iva cy: The abilit y of t he soft ware t o prevent unaut horized access or viewing of Personally I dent ifiable I nform at ion.

Performance Perform ance is m ost oft en not iced when it is poor. I n designing, developing, and t est ing for perform ance, it is im port ant t o different iat e t he QoS t hat influence t he end experience of overall perform ance. Re spon sive n e ss: The absence of delay when t he soft ware responds t o an act ion, call, or event . Con cu r r e n cy: The capabilit y of t he soft ware t o perform well when operat ed concurrent ly wit h

ot her soft ware. Efficie n cy: The capabilit y of t he soft ware t o provide appropriat e perform ance relat ive t o t he resources used under st at ed condit ions. [ 17] Fa u lt Tole r a n ce : The capabilit y of t he soft ware t o m aint ain a specified level of perform ance in cases of soft ware fault s or of infringem ent of it s specified int erface. [ 18] Sca la bilit y : The abilit y of t he soft ware t o handle sim ult aneous operat ional loads.

User Experience While " easy t o use" has becom e a cliché, a significant body of knowledge has grown around design for user experience. Acce ssibilit y : The ext ent t o which individuals wit h disabilit ies have access t o and use of inform at ion and dat a t hat is com parable t o t he access t o and use by individuals wit hout disabilit ies.[ 19] At t r a ct ive n e ss: The capabilit y of t he soft ware t o be at t ract ive t o t he user. [ 20] Com pa t ibilit y: The conform ance of t he soft ware t o convent ions and expect at ions. D iscove r a bilit y : The abilit y of t he user t o find and learn feat ures of t he soft ware. Ea se of Use: The cognit ive efficiency wit h which a t arget user can perform desired t asks wit h t he soft ware. Loca liza bilit y : The ext ent t o which t he soft ware can be adapt ed t o conform t o t he linguist ic, cult ural, and convent ional needs and expect at ions of a specific group of users.

Manageability Most m odern solut ions are m ult it ier, dist ribut ed, serviceorient ed or client server applicat ions. The cost of operat ing t hese applicat ions oft en exceeds t he cost of developing t hem by a large fact or, yet few developm ent t eam s know how t o design for operat ions. Appropriat e QoS t o consider are as follows: Ava ila bilit y: The degree t o which a syst em or com ponent is operat ional and accessible when required for use. Oft en expressed as a probabilit y.[ 21] This is frequent ly cit ed as " nines," as in " t hree nines," m eaning 99.9% availabilit y. Re lia bilit y: The capabilit y . . . t o m aint ain a specified level of perform ance when used under specified condit ions. [ 22] Frequent ly st at ed as Mean Tim e Bet ween Failures ( MTBF) . I n st a lla bilit y a n d Un in st a lla bilit y: The capabilit y . . . t o be inst alled in a specific environm ent [ 23] and uninst alled wit hout alt ering t he environm ent 's init ial st at e. M a in t a in a bilit y : The ease wit h which a soft ware syst em or com ponent can be m odified t o correct fault s, im prove perform ance or ot her at t ribut es, or adapt t o a changed environm ent . [ 24] M on it or a bilit y : The ext ent t o which healt h and operat ional dat a can be aut om at ically collect ed from t he soft ware in operat ion.

Ope r a bilit y: The ext ent t o which t he soft ware can be cont rolled aut om at ically in operat ion. Por t a bilit y: The capabilit y of t he soft ware t o be t ransferred from one environm ent t o anot her. [ 25] Re cove r a bilit y: The capabilit y of t he soft ware t o reest ablish a specified level of perform ance and recover t he dat a direct ly affect ed in t he case of a failure. [ 26] Te st a bilit y : The degree t o which a syst em or com ponent facilit at es t he est ablishm ent of t est crit eria and t he perform ance of t est s t o det erm ine whet her t hose crit eria have been m et . [ 27] Su ppor t a bilit y: The ext ent t o which operat ional problem s can be correct ed in t he soft ware. Con for m a n ce t o St a n da r ds: The ext ent t o which t he soft ware adheres t o applicable rules. I n t e r ope r a bilit y: The capabilit y of t he soft ware t o int eract wit h one or m ore specified syst em s. [ 28] What m akes a good QoS requirem ent ? As wit h scenarios, QoS requirem ent s need t o be explicit ly underst andable t o t heir st akeholder audiences, defined early, and when planned for an it erat ion, t hey need t o be t est able. You m ay st art wit h a general st at em ent about perform ance, for exam ple, but in an it erat ion set specific t arget s on specific t ransact ions at specific load. I f you cannot st at e how t o t est sat isfact ion of t he requirem ent when it becom es t im e t o assess it , t hen you can't m easure t he com plet ion.

Kano Analysis There is a useful t echnique, called " Kano Analysis" aft er it s invent or, which plot s excit ers, sat isfiers, and dissat isfiers on t he sam e axes. [ 29] The Xaxis ident ifies t he ext ent t o which a scenario or QoS is im plem ent ed, while t he Yaxis plot s t he result ing cust om er sat isfact ion. I n t he case of Mont y Pyt hon's parrot , life is a sat isfier, or in ot her words, a necessary QoS. ( Like m any ot her QoS, t he cust om er assum es it wit hout specifying it and walks out of t he st ore wit hout having t est ed it . Just as a dysfunct ional proj ect m anager m ight , t he shopkeeper exploit s t he lack of specificat ion.) Of course, rem oving dissat isfiers only get s you a neut ral cust om er. The qualit y of service t hat t he parrot is alive and healt hy get s you a cust om er who is willing t o look furt her. On t he ot her hand, sat isfying t he cust om er requires adding t he requirem ent s t hat m ake a cust om er want t o buy your solut ion or a user want t o use it . For t he parrot , t his m ight be t he scenario t hat t he cust om er t alks t o t he parrot and t he parrot sings back cheerfully. Healt h is a m ust - have requirem ent of t he parrot it s absence leaves t he cust om er disgust ed; it s presence m akes t he cust om er neut ral. The singing scenario is a sat isfier or " perform ance requirem ent " cust om ers will expect it , and will be increasingly sat isfied t o t he ext ent it is present . Excit ers are a t hird group of scenarios t hat delight t he cust om er disproport ionat ely when present . Som et im es t hey are not described or im agined because t hey m ay require t rue innovat ion t o achieve. On t he ot her hand, som et im es t hey are sim ple conveniences t hat m ake a huge perceived difference. For a brief period, m inivan m anufact urers com pet ed based on t he num ber of cupholders in t he back seat s, t hen on t he num ber of sliding doors, and t hen on backseat video equipm ent . All t hese pieces of product different iat ion were sm all, evolut ionary feat ures, which init ially cam e t o t he m arket as excit ers and over t im e cam e t o be recognized as m ust - haves.

Figu r e 3 .8 .

Th e Xa x is sh ow s t h e e x t e n t t o w h ich diffe r e n t solu t ion r e qu ir e m e n t s a r e im ple m e n t e d; t h e Ya x is sh ow s t h e r e su lt in g cu st om e r r e spon se .

[View full size image]

Ret urning t o t he parrot exam ple, im agine a parrot t hat could clean it s own cage. You can generalize t hese t hree groups of requirem ent s as m ust - haves ( absence of dissat isfiers) , sat isfiers, and excit ers, based on t he ext ent t o which t heir presence or absence sat isfies or dissat isfies a cust om er. Must - haves const rain t he int ended solut ion, excit ers opt im ize it , and sat isfiers are expect ed but not different iat ed. For com plet eness, t here is a fourt h setindifferent feat urest hat don't m ake a difference eit her way ( such as t he color of t he parrot 's cage) .

Figu r e 3 .9 .

Th e m ode l pr ovide s a n e x pr e ssive t ool t o e va lu a t e solu t ion r e qu ir e m e n t s. I t split s t h e m in t o fou r gr ou ps: m u st - h a ve s, sa t isfie r s, e x cit e r s, a n d in diffe r e n t .

[View full size image]

Technology Adoption Lifecycle One last fram ework is useful in underst anding t he cat egorizat ion of requirem ent s and t he differences am ong separat e user segm ent s. This fram ework is t he t echnology adopt ion lifecycle, shown in Figure 3.10 . I t is a widely used m odel t hat describes t he penet rat ion of a product or t echnology int o widespread use. [ 30]

Figu r e 3 .1 0 .

Th e Scu r ve is t h e st a n da r d r e pr e se n t a t ion of t h e Te ch n ology Adopt ion Life cycle , br ok e n in t o fou r se gm e n t s for e a r ly a dopt e r s, e a r ly m a j or it y, la t e m a j or it y, a n d la gga r ds.

The relat ive im port ance of dissat isfiers and excit ers has a lot t o do wit h where on t he t echnology adopt ion lifecycle your users lie. I f t hey are early adopt ers, who em brace a new solut ion out of ent husiasm for t he t echnology, t hen t hey are likely t o be highly t olerant of dissat isfiers and willing t o accept involved workarounds t o problem s in order t o realize t he benefit s of t he excit ers. On t he ot her hand, t he lat er in t he adopt ion lifecycle your users lie, t he less t olerant t hey will be of dissat isfiers, and t he m ore t hey will expect excit ers as com m onplace. This is a nat ural m arket evolut ion in which users becom e progressively m ore sensit ive t o t he ent iret y of t he solut ion.

Gathering the Data Alt hough t he Mont y Pyt hon parrot is a com ic exam ple, Kano Analysis is a very useful t ool for t hinking about your requirem ent s. The easiest way t o gat her dat a for Kano Analysis is t o ask quest ions in pairs, one about t he appropriat eness of a feat ure and t he ot her about t he im port ance, such as t he following: [ 31] To what ext ent is t he feat ure ( scenario) t hat we j ust showed you ( described t o you) on t arget ? How im port ant is it t hat you can do what we j ust showed ( or described) ? Whet her you are doing broad st at ist ical research or inform al focus groups, you can gain insight int o t he solut ion by separat ing t he scenarios and QoS, or t he feat ures as im plem ent ed, int o t he four groups. Because it 's easy t o t hink of requirem ent s, m ost proj ect s end up having far t oo m any of t hem ( and not always t he right ones) . The Kano classificat ion let s you see a t ot al pict ure of requirem ent s readily. This pract ice can shed light on different viewpoint s and hidden assum pt ions about t he relat ive im port ance of excit ers, sat isfiers, and dissat isfiers.

Kano Analysis provides a way of priorit izing scenarios and QoS. When planning an it erat ion, you can plot a set of requirem ent s and evaluat e t he likely cust om er sat isfact ion. As you evolve a proj ect across it erat ions, you will want t o m ove up t he Yaxis of cust om er sat isfact ion. This fram ework for different iat ing t he value of requirem ent s gives you a powerful t ool for assessing t he im pact of t radeoff decisions.

Summary This chapt er covered t he descript ion of a proj ect 's vision and requirem ent s. As discussed in t he previous chapt er, MSF advocat es it erat ive soft ware developm ent , in keeping wit h t he valueup paradigm discussed in Chapt er 1, " A Value- Up Paradigm ." This im plies t hat requirem ent s be det ailed an it erat ion at a t im e because t hey are perishable. I n MSF, t he prim ary form s of requirem ent s are scenarios and qualit ies of service ( QoS) . Scenarios are t he funct ional requirem ent s. A scenario addresses t he goals of a persona or group of personas. St at ed in t he user's language, scenarios describe t he sequence of st eps and t he int eract ion necessary for t he persona t o accom plish t he int ended goal. Scenarios can be elaborat ed in wirefram es and st oryboards. QoS describe at t ribut es of t he int ended solut ion in areas such as securit y, perform ance, user experience, and m anageabilit y. These are bot h global at t ribut es and specific const raint s on scenarios. QoS are oft en considered archit ect ural requirem ent s, bot h because t hey m ay have significant im plicat ions for archit ect ure and because t hey oft en require specialist archit ect ural knowledge t o enum erat e. Not all requirem ent s are creat ed equal. I t is im port ant t o underst and t he cust om er percept ion of t hese requirem ent s as sat isfiers, excit ers, and m ust - haves ( t he rem oval of dissat isfiers) . Scenarios alone rarely elim inat e all dissat isfiers, which oft en lurk in poorly planned QoS. And scenarios by t hem selves do not design t he archit ect ure, t he im plem ent at ion det ails, or all t he variant flows. Wit h requirem ent s in hand, t he next chapt er looks at t he process of m anaging a proj ect .

Endnotes 1.

Frederick P. Brooks, Jr., The Myt hical ManMont h: Essays on Soft ware Engineering, Anniversary Edit ion ( AddisonWesley, 1995) , 199.

2.

Adapt ed from Geoffrey A. Moore, Crossing t he Chasm : Market ing and Selling HighTech Product s t o Mainst ream Cust om ers ( New York: HarperCollins, 2002) , 154.

3.

Beck 2000, op. cit ., 60 61 .

4.

I EEE STD 830 is an exam ple set of guidelines for such specificat ions.

5.

Moore, Crossing t he Chasm , 93 4.

6.

Alan Cooper and Robert Reim ann, About Face 2.0: The Essent ials of I nt eract ion Design ( I ndianapolis: Wiley, 2003) , 59.

7.

For a closer look at how Microsoft uses personas, see John Pruit t , Persona Lifecycle: Keeping People in Mind Throughout Product Design ( Elsevier, 2005) .

8.

Cooper 2003, op. cit ., 12.

9.

I bid., 35.

1 0 . Hugh Beyer and Karen Holzblat t , Cont ext ual Design: Designing Cust om erCent ered Syst em s ( San Francisco: Morgan Kaufm ann, 1998) and Cooper op cit ., 44 ff.

1 1 . Moore, op. cit ., 98. 1 2 . Originally coined in I var Jacobson et al., Obj ect Orient ed Soft ware Engineering: A Use Case Driven Approach ( Reading: AddisonWesley/ ACM Press, 1992) , 159 ff., and elaborat ed by m any including Alist air Cockburn, Writ ing Effect ive Use Cases ( Pearson Educat ion, 2000) . The body of pract ice around use cases t ook a t urn away from t he int ent ions of t hese aut hors, and by using " scenarios," MSF connect s t o t he separat e t radit ion of t he User Experience com m unit y.

1 3 . For exam ple, Mike Cohn, User St ories Applied: For Agile Soft ware Developm ent ( Bost on: AddisonWesley, 2004)

1 4 . J.M. Juran, Juran on Planning for Qualit y ( Sim on & Schust er, 1988) . 1 5 . Mont y Pyt hon's Flying Circus, " The Dead Parrot Sket ch," available on The 16- Ton Mont y Pyt hon DVD Megaset , Disc 3 ( A&E Hom e Video, 2005) .

1 6 . For exam ple, I SO/ I EC 9126 and I EEE St d 610.121990. 1 7 . [ I SO/ I EC 91261/ 2001] 1 8 . [ I SO/ I EC 91261/ 2001] 1 9 . Sect ion 504 of t he Rehabilit at ion Act , 29 U.S.C. § 794d, available from ht t p: / / www.usdoj .gov/ crt / 508/ 508law.ht m l.

2 0 . [ I SO/ I EC 91261/ 2001] 2 1 . [ I EEE STD 610.121990] , 11

2 2 . [ I SO/ I EC 91261/ 2001] 2 3 . [ I SO/ I EC 91261/ 2001] 2 4 . [ I EEE STD 610.121990] , 46 2 5 . [ I SO/ I EC 91261/ 2001] 2 6 . [ I SO/ I EC 91261/ 2001] 2 7 . [ I EEE STD 610.121990] , 76 2 8 . [ I SO/ I EC 91261/ 2001] 2 9 . Kano, N., Seraku, N., Takahashi, F. and Tsuj i, S. ( 1996) , " At t ract ive qualit y and m ust - be qualit y," originally published in Hinshit su ( Qualit y) , The Journal of t he Japanese Societ y for Qualit y Cont rol, XI V: 2, 3948, April 1984, t ranslat ed in The Best On Qualit y, edit ed by John D. Hrom i. Volum e 7 of t he Book Series of t he I nt ernat ional Academ y for Qualit y ( Milwaukee: ASQC Qualit y Press, 1996) .

3 0 . For exam ple, Moore, op. cit . 3 1 . For well docum ent ed quest ionnaires on gat hering t he dat a, see for exam ple, Cent er for Qualit y Managem ent Journal, I I : 4, Fall 1993.

Chapter 4. Project Management

Program Manager Proj ect Manager " The deficiencies of t he t heory of t he proj ect and of t he t heory of m anagem ent reinforce each ot her and t heir det rim ent al effect s propagat e t hrough t he life cycle of a proj ect . Typically, cust om er requirem ent s are poorly invest igat ed at t he out set , and t he process of requirem ent clarificat ion and change leads disrupt ion in t he progress of t he proj ect . The act ual progress st art s t o drift from t he plan, t he updat ing of which is t oo cum bersom e t o be done regularly. Wit hout an up- t o- dat e plan, t he work aut horizat ion syst em t ransform s t o an approach of inform al m anagem ent . I ncreasingly, t asks are com m enced wit hout all input s and prerequisit es at hand, leading t o low efficiency or t ask int errupt ion and increased variabilit y downst ream . Correspondingly, cont rolling by m eans of a perform ance baseline t hat is not based on t he act ual st at us becom es ineffect ive or sim ply count erproduct ive. All in all, syst em at ic proj ect m anagem ent is t ransform ed t o a facade, behind which t he j ob act ually get s done, even if wit h reduced efficiency and lessened value t o t he cust om er." [ 1 ] L. Koskela and G. Howell, " The Underlying Theory of Proj ect Managem ent I s Obsolet e"

Figu r e 4 .1 .

W it h ou t t r a n spa r e n t da t a , pr oj e ct m a n a ge m e n t ca n de sce n d in t o a ga m e of diffe r in g be t s ba se d on pa r t ia l in for m a t ion a n d dive r ge n t pe r spe ct ive s of st a k e h olde r s. Pok e r is a good m e t a ph or for t h is pa t t e r n .

"A Friend in Need" by C.M. Coolidge, c. 1870 The previous chapt er addressed t he gat hering of requirem ent s. This chapt er focuses on m anaging a running proj ect t hat im plem ent s t hose requirem ent s. First , I 'll cover t hree concept s t hat are core t o t he value- up paradigm : Variance t o dist inguish in- and out - of- cont rol proj ect s Descript ive rat her t han prescript ive m et rics Mult iple dim ensions of proj ect healt h VSTS applies t hese concept s wit h it s work it em dat abase and m et rics warehouse t o give you a pract ical basis for value- up proj ect m anagem ent . To illust rat e t his aspect of VSTS, I work t hrough a large num ber of exam ples in t his chapt er using report s from t he m et rics warehouse. These are " happy day" exam ples, in cont rast t o t roubleshoot ing t he " unhappy" exam ples t hat I 'll use in Chapt er 9, " Troubleshoot ing t he Proj ect ." Finally, in t his chapt er I cover est im at ion and t riage from a value- up perspect ive. These t wo essent ial proj ect m anagem ent pract ices rely closely on t he m et rics and queries t hat VSTS enables.

Understanding Variation Fundam ent al t o t he Value- Up Paradigm is t he concept of nat ural variat ion. Variat ion and it s im pact on qualit y were originally st udied in m anufact uring engineering and well t aught by W. Edwards Dem ing: Com m on ca u se s a n d spe cia l ca u se s. [ Dr. Walt er A. Shewhart of Bell Labs] saw t wo kinds of variat ionvariat ion com ing from com m on causes and variat ion from special causes . . . . Com m on causes of variat ion st ay t he sam e day t o day, lot t o lot . A special cause of variat ion is som et hing special, not part of t he syst em of com m on causes . . . . Dr. Shewhart also saw t wo kinds of m ist akes . . . . M ist a k e # 1 . To react t o an out com e as if it cam e from a special cause, when it act ually cam e from com m on causes of variat ion. M ist a k e # 2 . To t reat an out com e as if it cam e from com m on causes of variat ion, when act ually it cam e from a special cause. [ 2 ] The sam e dist inct ion bet ween com m on cause and special cause variat ion applies t o soft ware proj ect s. Processes t hat exhibit com m on- cause variat ion are in cont rol; t hose wit h special- cause variat ion are out of cont rol. I n soft ware proj ect s, som e t hings t ake longer t han ant icipat ed, and som e t ake short er. Som e soft ware int egrat es perfect ly; in ot her cases, bugs appear and need t o be fixed. Som e scenarios delight cust om ers exact ly as hoped; ot hers need t o be refined wit h usabilit y t est ing and learnings from it erat ion t o it erat ion. These are usually com m on- cause variat ions. Mist ake # 1 is t am pering wit h an in- cont rol process showing t his nat ural variat ion. This t am pering only increases variance and quickly sends a process out of cont rol. Perhaps m ore im port ant ly, it leads t o m icrom anagem ent t hat dem oralizes t he t eam . Dem ing uses a beaut ifully sim ple experim ent t o illust rat e t he effect of t am pering. [ 3 ] Wit h a funnel, a m arble, and a t able, you point t he funnel at a const ant t arget and let t he m arble roll t hrough m any t im es ( say 50) , m arking t he spot s where t he m arble falls. Aft er each drop, you m ove t he funnel t o com pensat e for each error. Dem ing proposes t hree correct ion rules t hat , on t heir own, sound plausible. The first pass shows a const rained variance, while t he ot hers drift m uch m ore broadly. At t he sam e t im e, failing t o ident ify and correct t he special cause of a process going out of cont rol can be disast rous ( Mist ake # 2) . This m ist ake leads t o a vicious cycle of t he process spinning furt her away from t he desired t arget .

Special Causes as Issues in the Work Item Backlog I n VSTS, special causes are capt ured as issues. I n MSF for CMMI Process I m provem ent , use t he work it em t ype called I ssues and regularly work t he issue log. I n MSF for Agile Soft ware Developm ent , use t he I ssue field on ot her work it em t ypes. I n bot h processes, risks are a work it em t ype for capt uring pot ent ial issues t hat m ight creat e special causes. Not e also t hat t he dist inct ion bet ween com m on and special causes is built int o t he st aged " levels" of t he CMMI , as it was in t he Manufact uring Mat urit y Model from which it was derived. CMMI Levels 2- 4 deal wit h reducing or elim inat ing special cause variat ion, and Level 5 aim s t o reduce com m on cause variat ion.

Det erm ining variat ion has been t oo difficult for m ost soft ware proj ect s because t he dat a have been t oo hard t o collect . Even agile m et hods have suffered from t his oversight . For exam ple, t he m anagem ent pat t ern called " Yest erday's Weat her," com m on t o XP and SCRUM, specifies t hat " you can't sign up for m ore work in an it erat ion t han you did in t he previous it erat ion," [ 4 ] which pot ent ially leads t o lower est im at es wit h each it erat ion. This flies in t he face of a prim ary value of it erat ionst hat t hey allow for cont inuous learning, course correct ion, and im provem ent . So how do you know what norm al variat ion is? You observe and m easure t he running process wit hout t am pering. Fort unat ely, VSTS wit h it s inst rum ent at ion of t he process m akes t his observat ion easy, and an it erat ive approach wit h sm all bat ch sizes and short int ervals provides frequent opport unit ies for observat ion and, when necessary, correct ion. You can never dist inguish every com m on- cause variance from every special- cause one, but you can get very close by wat ching t he report s described lat er in t he chapt er and m anaging t he issues and risks in t he proj ect .

Using Descriptive Rather Than Prescriptive Metrics Oft en t here are t acit or even explicit assum pt ions about t he " right " answers t o quest ions of expect at ions. These expect at ions can det erm ine how individuals are recognized or not recognized for t heir perform ance. Developers are praised for com plet ing t asks on t im e. Test ers are praised for running lot s of t est s or finding lot s of bugs. Hot line specialist s are praised for handling lot s of calls and m arking t hem resolved. Everyone is praised for keeping billable hours up. And so on. Using m et rics t o evaluat e individual perform ance is horribly count erproduct ive, as Robert Aust in describes: When a m easurem ent syst em is put in place, perform ance m easures begin t o increase. At first , t he t rue value of an organizat ion's out put m ay also increase. This happens in part because workers do not underst and t he m easurem ent syst em very well early on, so t heir safest course is t o st rive t o fulfill t he spirit of t he syst em archit ect s' int ent ions. Real im provem ent m ay result as well, because early t arget s are m odest and do not drive workers int o t aking severe short cut s. Over t im e, however, as t he organizat ion dem ands ever great er perform ance m easurem ent s, by increasing explicit quot as or inducing com pet it ion bet ween coworkers, ways of increasing m easures t hat are not consist ent wit h t he spirit of int ent ions are used. Once one group of workers sees anot her group cut t ing corners, t he" slower" group feels pressure t o im it at e. Gradually, m easures fall ( or, m ore accurat ely, are pushed) out of synchronizat ion wit h t rue perform ance, as workers succum b t o pressures t o t ake short cut s. Measured perform ance t rends upward; t rue perform ance declines sharply. I n t his way, t he m easurem ent syst em becom es dysfunct ional. [ 5 ] These are prescript ive m et rics. They can have unforeseen side effect s. There is a well- ident ified pat t ern of organizat ional behavior adapt ing t o fit t he expect at ions of a prescript ive m easurem ent program , as shown in Figure 4.2 . Typically, a m et rics program produces an init ial boost in product ivit y, followed by a st eep ret urn t o t he st at us quo ant e but wit h different num bers. For exam ple, if bug find and fix rat es are crit ically m onit ored, t hen bug curves st art conform ing t o desirable expect at ions.

Figu r e 4 .2 .

Th is gr a ph su m m a r ize s t h e com m on e x pe r ie n ce w it h pr e scr ipt ive , on e - dim e n sion a l m e t r ics pr ogr a m s. Pe r for m a n ce sh oot s u p e a r ly in a ccor d w it h m a n a ge m e n t a spir a t ion s, a n d t h e n u m be r s ge t be t t e r a n d be t t e r , bu t t h e de sir e d e ffe ct t a pe r s off qu ick ly.

Robert D. Austin, Measuring and Managing Performance in Organizations (New York: Dorset House, 1996), 16 Consider som e exam ples of prescript ive m et ric m isuse: I m agine m easuring program m er product ivit y based on lines of code writ t en per day. An individual has a choice of calling a fram ework m et hod ( perhaps 5 lines wit h error handling) or of copying 200 lines of open- source exam ple code. Which one get s rewarded? Which one is easier t o m aint ain, t o code- review, t o securit y- review, t o t est , and t o int egrat e? Or sim ilarly, t he individual has t he chance t o refact or t hree overlapping m et hods int o one, reducing t he size of t he code base. ( Now ask t he sam e quest ions.) I m agine rewarding program m ers based on num ber of bugs fixed. This was once t he subj ect of a Dilbert cart oon, which ended wit h Wally saying, " I 'm going t o code m e up a Winnebago." I m agine rewarding t he t eam for creat ing t est s and code t o achieve 90% code coverage. Do t hey spend t heir t im e writ ing com plex t est set ups for every error condit ion, or easily com m ent out t he error handling code t hat t est s aren't able t o t rigger? Aft er all, if t he t est s can't invoke t hose condit ions, how im port ant can t hey be? ( Not very, unt il a cust om er encount ers t hem .) I m agine m easuring t est ers based on t he num ber of bugs found. Do t hey look for easy- t o- find, overlapping, sim ple bugs or go aft er significant ones t hat require set t ing up com plex cust om er dat a and configurat ions? Which approach get s rewarded? Which one yields m ore cust om er value? Each exam ple would lead t o obvious dysfunct iondiscouraging reuse and m aint ainabilit y, encouraging buggy check- ins, reducing error handling, and discouraging finding t he im port ant bugs. Ot her dysfunct ions from m et rics m isuse will be less obvious but equally severe. People who don't get t he best scores will be dem oralized and will be faced wit h t he choice of gam ing t he num bers or leaving t he t eam .

Preventing Distortion At t he root of t he dist ort ion is t he prescript ive, rat her t han descript ive, use of t hese m et rics. This problem has several facet s. First , t he m et rics are only approxim at ions of t he business obj ect ive, such as cust om er sat isfact ion or solut ion m arket abilit y. The t eam aim s t o deliver cust om er value, but t hat

can't be count ed easily on a daily basis. So t he available m et rics, such as t ask com plet ion, t est pass rat e, or bug count , are im perfect but easily count able proxies. A t enet of t he value- up paradigm is t o give credit only for com plet ed, cust om er- deliverable unit s of work at known qualit y. [ 6 ] Wit h it erat ions of t wo t o four weeks and assessm ent at t he end of t he it erat ion, t his pract ice allows for int ervals of proj ect m onit oring at it erat ion boundaries. Treat all int erim m easurem ent s as hypot het ical unt il you can assess delivery of working scenarios at known qualit ies of service. Second, t he m easurem ent s are m ade one dim ension at a t im e. The negat ive consequences of a onedim ensional view are dram at ic. I f you're only m easuring one dim ension at a t im e and are prescribing expect ed result s, behavioral dist ort ion is a nat ural consequence. Most experienced proj ect m anagers know t his. However, gat hering dat a from m ult iple sources at t he sam e t im e in a m anner t hat lends it self t o reasonable correlat ion is usually very difficult . Third, when applied t o individuals, m et rics creat e all sort s of disincent ives, as illust rat ed in t he previous exam ples. Keep t he observat ions, even descript ive ones, at t he t eam level. Fourt h, expect com m on- cause variat ions and t hink in ranges. Don't reward t he m ost prolific coder or highest - count bug finder. Expect t he num bers t o show variance and don't punish cases of in- cont rol variance. I nst ead, reward a t eam based on com plet ed cust om er- deliverable unit s of funct ionalit y and m ake t he it erat ion cycle for t hat assessm ent frequent .

Many Dimensions of Project Health Take a m inut e t o t hink of t he m ost com m on report s t hat you use in m onit oring your proj ect s. A t ypical list would include t he following: Tim e spent on t asks ( hopefully, planned and unplanned) Bug count ( and hopefully, find and fix rat es) Proj ect t ask com plet ion, as report ed by t he individual perform ers ( hopefully m easured by scenario, feat ure, QoS, or ot her cust om er- visible unit ) Resource ut ilizat ion, t ypically in account ing t erm s Each is t ypically m easured separat ely. Soft ware proj ect s have m any int eract ing dim ensions, and any of t hem can be relevant . Looking at t hese dim ensions provides an opport unit y for early discovery of except ions and bot t lenecks t hat need course correct ions. Yet having m ult iple dim ensions helps you m ake sure t hat t he m easurem ent s as a whole t ell a consist ent st ory. I n t he first chapt er, I describe how VSTS inst rum ent s t he soft ware process. A com m on proj ect backlog is m aint ained in t he single work it em dat abase. At t he sam e t im e, version cont rol, t est s, and builds are inst rum ent ed so t hat m any dim ensions of proj ect dat a can be gat hered aut om at ically and correlat ed in a m et rics warehouse. This m echanism enables daily assessm ent of t he proj ect while it is running. This idea of a m et rics warehouse is illust rat ed in Figure 4.3 .

Figu r e 4 .3 .

Un lik e in m a n y olde r pr oj e ct m ode ls, w h e r e pr oj e ct m e t r ics a r e ga t h e r e d in isola t ion , VSTS com bin e s t h e m e a su r e s in a m e t r ics w a r e h ou se u sin g a com m on se t of a n a lysis cu be s a n d r e por t s on t h e dim e n sion s t oge t h e r .

I n est im at ion and m onit oring, t he backlog also helps considerably. You can do t he sam e analyses on all work it em s regardless of t ype. Work it em changes becom e m et rics t hat aut om at ically t rack progress for you t o provide dat a for t he next est im at ion, and you can assess t he qualit y of your est im at es t o approve t hem next t im e. Qualit y can be m easured against t ask progress using m any sim ult aneous views.

Answering Everyday Questions Any proj ect m anager should be able t o answer t he following quest ions, hopefully in t he day- t o- day course of business: How m uch work is left and when will it be done? How product ive is t he t eam ? How m uch unplanned work do we have? What 's t he qualit y of t he soft ware we're producing? How effect ively are we finding, fixing, and closing bugs? How m any false t ask and bug resolut ions do we have? Are we finding and t riaging t he right bugs? How fast can we go before qualit y suffers? Each quest ion is relevant at m any scalesfor t he days wit hin an it erat ion, for t he it erat ions wit hin a proj ect , and for t he proj ect s wit hin a program . The quest ions are also relevant for all kinds of workscenarios and ot her requirem ent s, t asks, bugs, and so on. I n t he next few pages, I present graphs t hat help t o answer t hese quest ions. Not unlike t he way one develops scenarios, I show t hem first wit h " happy day" exam ples, and in Chapt er 9, I show t heir use for t roubleshoot ing unhealt hy proj ect s.

In VSTS, Reports Are Defined in the Process Template Except where not ed, all t he following exam ples were produced wit h t he st andard report s available in MSF for Agile Soft ware Developm ent . MSF for CMMI Soft ware I m provem ent has t he sam e m odels, usually wit h m ore det ail in t he dat a. Specifically, scenarios or ot her requirem ent s st art as Proposed and m ust be accept ed by explicit decision before t hey becom e Act ive, so t hat you would see four st at es in t he diagram rat her t han t hree.

Remaining Work Of t he several ways t o t rack work, one of t he m ost useful is a cum ulat ive flow diagram ( see Figure 4.4) . [ 7 ] This is m ost useful when looking at days wit hin an it erat ion or it erat ions wit hin a proj ect .

Figu r e 4 .4 .

H ow m u ch w or k is le ft a n d w h e n w ill it be don e ? Th is cu m u la t ive flow dia gr a m sh ow s w or k r e m a in in g m e a su r e d a s t h e Sce n a r io a n d QoS W or k I t e m s be in g r e solve d a n d close d in t h e it e r a t ion .

[View full size image]

Every work it em has st at e. I n MSF for Agile Soft ware Developm ent , scenarios, which are a t ype of work it em , have t hree st at es: Act ive ( t hat is, in t he hands of a developer and not yet coded) , Resolved ( t hat is, coded and ready for t est ) , and Closed ( t hat is, t est ed and verified) . Each dat a series is a colored band ( reproduced here as shading) , t hat represent s t he num ber of scenarios t hat have reached t he corresponding st at e as of t he given dat e. The t ot al height is t he t ot al am ount of work t o be done in t he it erat ion. I f t he t op line increases, it m eans t hat t ot al work is increasing. Typically, t he reason is t hat unplanned work is adding t o t he t ot al required. That m ay be expect ed if you've scheduled a buffer for unplanned work such as fixing newly discovered bugs. ( See Figure 4.6 .) I f t he t op line decreases, it m eans t hat t ot al work is decreasing, probably because work is being rescheduled out of t he it erat ion. Current st at us is m easured by height on a part icular dat e. The rem aining backlog is m easured by t he current height of t he left m ost area, Act ive in t his

case. The current com plet ions are shown by t he current height of t he right m ost area, Closed. The height of t he band in- bet ween indicat es t he work in progress, in t his case it em s Resolved but not Closed. Wat ch for variat ion in t he m iddle bands. An expansion can reveal a bot t leneck, for exam ple, if t oo m any it em s are wait ing t o be t est ed and t est ing resources are inadequat e. Alt ernat ively, a significant narrowing of t he band could indicat e spare capacit y. Visually, it 's easy t o ext rapolat e an end com plet ion invent ory or end dat e for t he backlog from a cum ulat ive flow diagram like Figure 4.4 . A sm all caut ion applies, however. Many proj ect s observe an S- curve pat t ern, where progress is st eepest in t he m iddle. [ 8 ] The com m on- sense explanat ion for t he slower st art ing and ending rat es is t hat st art up is always a lit t le difficult , and unforeseen t ough problem s need t o be handled before t he end of a cycle.

Project Velocity The rat e at which t he t eam is processing and closing work it em s is capt ured in t he Proj ect Velocit y graph ( see Figure 4.5 ) . Sim ilar t o Rem aining Work, t his is m ost useful when looking at days wit hin an it erat ion or it erat ions wit hin a proj ect .

Figu r e 4 .5 .

H ow pr odu ct ive is t h e t e a m ? Un lik e t h e cu m u la t ive flow dia gr a m , ve locit y sh ow s t h e cou n t of w or k it e m s r e solve d a n d close d on e a ch da y. I n t h is e x a m ple , t h e r a n ge is 4 ± 2 sce n a r ios/ da y a ft e r a se ve n - da y la g.

[View full size image]

This chart is one of t he key elem ent s for est im at ion. I t shows how quickly t he t eam is act ually com plet ing planned work and how m uch t he rat e varies from day t o day or it erat ion t o it erat ion. I n exam ining variance, be sure t o dist inguish bet ween com m on- cause and special- cause event s. Specialcause variances include t eam illness, power out age, an office m ove, and so on. I f you elim inat e t hem , you can look at t he act ual norm al variance t o get a feel for t he upper and lower bounds of your t eam 's norm al product ivit y. Use t his dat a t o plan a t arget range for t he next it erat ion in conj unct ion wit h t he qualit y m easures discussed in t he following.

Unplanned Work Very few proj ect t eam s know all t he it em izable work t o be done ahead of t im e, even wit hin t he it erat ion. This condit ion can be perfect ly accept able if you schedule a sufficient buffer for handling t he load of unplanned work ( for exam ple, m aint enance t asks and bugs) . On t he ot her hand, it can be a real problem if you have not scheduled t he capacit y and can force you t o cut back on t he planned work. The t op line of Figure 4.6 m at ches t he t op line of t he Rem aining Work graph in Figure 4.4 . The t ot al height is t he t ot al am ount of work t o be done in t he it erat ion.

Figu r e 4 .6 .

H ow m u ch u n pla n n e d w or k do w e h a ve ? Th is gr a ph br e a k s dow n t h e t ot a l w or k fr om t h e Re m a in in g W or k ch a r t in t o t h e pla n n e d a n d u n pla n n e d. I n t h is e x a m ple , pla n n e d w or k is de clin in g sligh t ly. Th is m igh t be du e t o t h e u n pla n n e d w or k a dde d la t e r , for cin g pla n n e d

w or k it e m s t o be t r ia ge d ou t of t h e it e r a t ion or du e t o ove r e st im a t ion , or it m igh t be occu r r in g sim ply be ca u se ce r t a in it e m s w e r e discove r e d t o be n o lon ge r n e ce ssa r y.

[View full size image]

The areas t hen divide t hat work int o t he planned and unplanned segm ent s, where " unplanned" m eans unscheduled as of t he beginning of t he it erat ion. For m onit oring, use t his graph t o det erm ine t he ext ent t o which unplanned work is forcing you t o cut int o planned work. For est im at ion, use t his t o det erm ine t he am ount of schedule buffer t o allow for unplanned work in fut ure it erat ions.

Quality Indicators Qualit y needs t o be seen from m any dim ensions. Figure 4.7 com bines t he t est result s, code coverage from t est ing, code churn, and bugs t o help you see m any perspect ives at once.

Figu r e 4 .7 .

W h a t 's t h e qu a lit y of t h e soft w a r e ? I de a lly, t e st r a t e s, bu gs, a n d code ch u r n w ou ld a ll pr odu ce t h e sa m e pict u r e , a s t h e y do in t h is e x a m ple . Te st pa sse s a n d code cove r a ge r ise , w h ile bu gs a n d code ch u r n fa ll. On t h e ot h e r h a n d, w h e n you fin d a discr e pa n cy in t h e se r e la t ion sh ips, you n e e d t o dr ill dow n in t o t h e a ppr opr ia t e bu ild a n d da t a se r ie s.

[View full size image]

The bars show you how m any t est s have been run and of t hose, how m any have ret urned Pass, Fail, and I nconclusive result s. The first series of point s is t he code coverage at t ained by t hose t est s ( specifically, t he ones run wit h code coverage enabled) . Ordinarily, as m ore t est s are run, m ore code should be covered. On t he ot her hand, if t est execut ion and t est pass rat es rise wit hout a corresponding increase in code coverage, t hen it can indicat e t hat t he increm ent al t est s are redundant . The second series of point s is code churn, t hat is, t he num ber of lines added and m odified in t he code under t est . High churn obviously indicat es a large am ount of change and t he corresponding risk t hat bugs will be int roduced as a side effect of t he changes. I n a perfect ly refact ored proj ect , you can see code churn wit h no change in code coverage or t est pass rat es. Ot herwise, high code churn can indicat e falling coverage and t he need t o rewrit e t est s. The t hird series is t he act ive bug count . Clearly, t here should be a correlat ion bet ween t he num ber of act ive bugs and t he num ber of t est failures. I f t he act ive bug count is rising and your t est s are not showing corresponding failures, t hen your t est s are probably not t est ing t he sam e funct ionalit y in t he sam e cont ext t hat t he bugs are report ing. Sim ilarly, if act ive bug count is falling and t est pass rat es are not increasing, t hen you m ay be at risk for a rising react ivat ion rat e. The different series are scaled by different powers of t en t o m ake t he graph legible wit h one Y- axis. This is com parable t o a Perform ance Monit or graph, which has different count ers at different orders of m agnit ude.

Bug Rates Bugs are, of course, a key part of t he t eam workload and a key risk area. To assess bugs, you need t o look at t hree t rends ( see Figure 4.8 ) :

Figu r e 4 .8 .

H ow e ffe ct ive ly a r e w e fin din g, fix in g, a n d closin g bu gs? W h e n t h e fix r a t e e x ce e ds t h e fin d r a t e , t h e a ct ive bu g cou n t fa lls.

[View full size image]

What is t he t ot al act ive bug count ( som et im es called " bug debt " ) ? How quickly are we finding new bugs?

How quickly are we fixing bugs already found? Bug rat es are best int erpret ed wit h your knowledge of all t he current proj ect act ivit ies and t he ot her m et rics on t he Qualit y I ndicat ors graph ( see Figure 4.7 ) . For exam ple, a high find rat e can be a sign of sloppy code ( a bad t hing) , newly int egrat ed code ( an expect ed t hing) , effect ive t est ing ( a good t hing) , or except ional event s, such as a bug bash ( an infrequent event , where large num bers of people t ry ad hoc t est ing for a day) . On t he ot her hand, a low find rat e can indicat e a high- qualit y solut ion or ineffect ive t est ing. Use code coverage, code churn, and t est rat es t o help you assess t he m eaning. Sim ilarly, a high resolve rat e is usually a good t hing, but check Work Progress t o m ake sure t hat t he resolved bugs are get t ing prom pt ly closed and check React ivat ions t o m ake sure t hat t he resolut ions are not prem at ure.

Reactivations React ivat ions occur when work it em s have been resolved or closed prem at urely. They are a huge warning sign of proj ect dysfunct ion. People som et im es record bugs as resolved when t he underlying problem has not been fixed. ( Sim ilarly, t hey can record scenarios resolved before t hey are working or developm ent t asks closed when t he work hasn't been finished.) Every t im e t his happens, it int roduces significant wast e int o t he process. Som eone has t o t est and reopen t he work it em , t he developer needs t o scrap and rework t he original code, and t hen t he code needs t o be ret est ed. At a m inim um , t he react ivat ion doubles t he num ber of handoffs and usually m ore t han doubles t he t ot al effort required t o com plet e t he corresponding work. Wat ching t he react ivat ion rat e ( also som et im es called t he Fault Feedback Rat io) is im port ant . [ 9 ] A sm all am ount of noise ( for exam ple, less t han 5% of t he work it em s resolved at any t im e) m ight be accept able, but a high or rising rat e of react ivat ion should warn t he proj ect m anger t o diagnose t he root cause and fix it . The t op line of Figure 4.9 shows t he num ber of t ot al work it em s of t he select ed t ypes ( for exam ple, bugs) resolved in t he build.

Figu r e 4 .9 .

H ow m a n y fa lse t a sk a n d bu g r e solu t ion s do w e h a ve ? Th e se fa lse r e solu t ion s sh ow u p a s r e a ct iva t ion sbu gs or t a sk s t h a t w e r e m a de a ct ive a ga in be ca u se t h e or igin a l w or k w a sn 't com ple t e d. Th e lin e sh ow s t h e cou n t of r e a ct iva t e d it e m s e a ch da y. Th e ba r s sh ow t h e cu m u la t ive a ct ive bu gs a s of t h a t da y, br ok e n in t o t h e r e a ct iva t ion s ( t op se gm e n t ) a n d n on - r e a ct iva t ion s ( bot t om se gm e n t ) .

[View full size image]

The height of t he t op area is t he num ber of react ivat ions, t hat is, work it em s previously resolved or closed t hat are now act ive again. The height of t he lower area is t he difference, t hat is, t he num ber of work it em s resolved less t he react ivat ions.

Bugs by Priority Bugs happen, and finding t hem is a good t hing. Oft en, however, t he easy- t o- find bugs aren't t he ones t hat will annoy cust om ers t he m ost . I f t he high- priorit y bugs are not being found and a disproport ionat e num ber of low- priorit y bugs are, t hen you need t o redirect t he t est ing effort s t o look for t he bugs t hat m at t er. I n t riage, it is easy t o over- priorit ize bugs beyond your capacit y t o resolve t hem or under- priorit ize t hem t o t he point where cust om ers will be highly dissat isfied. Figure 4.10 assesses t he effect iveness of t wo t hings: bug hunt ing and t riage, t hat is, t he process of priorit izing t he bugs t o fix, which is described lat er in t his chapt er.

Figu r e 4 .1 0 .

Ar e w e fin din g a n d t r ia gin g t h e r igh t bu gs? Th is ch a r t sh ow s for e a ch da y by pr ior it y t h e cu m u la t ive t ot a l a ct ive bu gs a n d t h e da ily su bt ot a ls for fou n d a n d fix e d. Th e cou n t s for fou n d in clu de bot h n e w a n d r e a ct iva t e d bu gs.

[View full size image]

I n Figure 4.10, t he t hree series of bars represent sim ilar dat a t o t he Bug Rat es graph ( Figure 4.8 ) . The series are as follows: Tot al act ive bugs at t he t im e of t he build Num ber found in build Num ber resolved in build Each series is furt her broken int o priorit y so t hat each bar st acks from highest t o lowest priorit y, wit h lowest on t op. I f you have t oo m any high- priorit y bugs act ive, you need t o be sure t hat you have capacit y t o address t hem . On t he ot her hand, a significant lack of low- priorit y bugs can also lead t o cust om er dissat isfact ion. ( See t he discussion of t he Broken Windows t heory in t he " Defining 'Good Enough'" sect ion of Chapt er 7.)

Actual Quality Versus Planned Velocity As m uch as t eam s believe t he saying, " Hast e m akes wast e," t here is usually a business incent ive t o go fast er. A proj ect m anager's goal should be t o find t he m axim um rat e of progress t hat does not m ake qualit y suffer. Figure 4.11 present s t he relat ionship for each it erat ion of est im at ed size t o overall qualit y.

Figu r e 4 .1 1 .

H ow fa st ca n w e go be for e qu a lit y su ffe r s?

[View full size image]

The X- axis is t he num ber of scenarios act ually closed ( com plet ed) in t he it erat ion. The Y- axis is t he t ot al num ber of bugs found divided by t he scenarios closed ( in ot her words, t he average num ber of bugs per scenario) . Each bubble is labeled according t o it s it erat ion. On screen, st oplight colors go from green for t he lowest bugs per it erat ion t o red for t he highest . Here you see t hem reproduced as light est t o darkest for green- am ber- red. I f hast e is in fact m aking wast e, posit ion of t he it erat ions t o t he right ( larger it erat ions) will be higher, indicat ing m ore bugs per requirem ent , and sm aller ( left ) ones will be lower, indicat ing fewer bugs. I f you are working at a com fort able pace and have not seen qualit y drop wit h planned it erat ion size, t hen all it erat ions will be at roughly t he sam e height on t he Y- axis.

Estimating an Iteration This sect ion describes a sim ple process for est im at ing proj ect s at t he level of person days using t he dat a from t he proj ect healt h report s shown previously. Wit h Visual St udio Team Syst em , all of t he backlog is one dat abase, and all t he hist orical m et rics are in one dat a warehouse, so it 's easy t o get t he num bers t hat you need here. Est im at ing t he work for an it erat ion is best done in m ult iple passes.

Top-Down Before t he Renaissance and t he advent of celest ial navigat ion, explorers ( such as Christ opher Colum bus) relied on a t echnique called dead reckoning. While im perfect , dead reckoning did enable t he discovery of t he New World and t he drawing of early m aps, which in t urn m ade furt her explorat ion and set t lem ent possible. Here's a st andard definit ion of dead reckoning: Dead Reckoning is t he process of est im at ing your posit ion by advancing a known posit ion using course, speed, t im e and dist ance t o be t raveled. I n ot her words figuring out where you will be at a cert ain t im e if you hold t he speed, t im e and course you plan t o t ravel. [ 10] Top- down est im at ion is like dead reckoning:

1 . Calculat e t he num ber of t eam - days available in t he it erat ion. This is your gross capacit y. 2 . Looking at t he act ual dat a in t he Unplanned Work chart , m ake a rough est im at e of t he buffer you will need for unplanned work. Subt ract t his from t he gross capacit y t o det erm ine your net capacit y for planned work. 3 . Looking at t he Proj ect Velocit y chart , det erm ine t he num ber of scenarios and QoS per day t hat your t eam can resolve and close. This is your gross velocit y. Consult t he Qualit y I ndicat ors and Act ual Qualit y versus Planned Velocit y chart s t o look for possible negat ive side effect s of your current pace. I f you see t hem , adj ust gross velocit y downward t o a sust ainable net velocit y. 4 . Mult iply net capacit y for planned work wit h sust ainable net velocit y t o det erm ine t he num ber of scenarios and QoS t hat you can im plem ent in t he it erat ion. On your ranked list of scenarios and QoS, count down from t he t op unt il you have reached t hat num ber. Draw a cut line t here. Everyt hing above t he cut line is your candidat e list . Dead reckoning m ight j ust be good enough, especially if you have low variance am ong your scenarios and QoS on t he Proj ect Velocit y chart ( see Figure 4.5 ) . I f you have high variance, t hen you can at t ack t he problem bot h by breaking down scenarios and QoS int o m ore consist ent ly sized work it em s and by est im at ing bot t om - up.

Bottom-Up Aft er you have a candidat e list of scenarios and QoS, you can perform bot t om - up est im at ion as well.

( You could do it earlier, but t his is not recom m ended. You'd spend a lot of t im e est im at ing work t hat won't be done yet , and you m ight never be done according t o t he current underst anding of t he requirem ent s.)

1 . I f necessary, t he business analyst and archit ect should refine t he scenarios, QoS, st oryboards ( refer t o Chapt er 3, " Requirem ent s" ) , and solut ion archit ect ure ( see Chapt er 5, " Archit ect ural Design" ) . These need t o be art iculat ed t o t he point where t hey com m unicat e sufficient ly t o all t eam m em bers t he det ails needed t o est im at e work for t he next it erat ion. 2 . Next , individual t eam m em bers decom pose t hese int o t asks, t ypically t o a granularit y of one t o t hree days. 3 . The t eam t riages bugs t o ident ify work needed t o fix bug backlogs for t he it erat ion. 4 . Working wit h t he t eam s, t he proj ect m anager rolls up t he t ot al volum e of t hese t ask est im at es t o see whet her t hey fit t he it erat ion capacit y for planned work. To t he ext ent t hat t he est im at es and capacit y don't m at ch, you should lat her, rinse, and repeat . I f you do bot t om - up est im at ion, don't forget t he feedback loop of t he ret rospect ive t o assess how act uals com pare t o est im at es.

Refinements Obviously, t here are som e refinem ent s t o t his m et hod you want t o m ake.

Variance in Size The way I described t he t echnique previously, I assum ed t hat all scenarios and QoS are roughly t he sam e size, wit h lit t le variance. Depending on your proj ect and your st yles of solut ion definit ion and archit ect ure, t his m ight be t rue. I f t he variance in t he Proj ect Velocit y graph is sm all, t hen it is a good assum pt ion. Alt ernat ively, it m ight be m ore useful t o cat egorize your scenarios and QoS int o " T- shirt size" groupings, t hat is, Large, Medium , and Sm all. You can plan, t rack, and graph t hese separat ely. Anot her alt ernat ive is t o use Rough Order of Magnit ude ( ROM) est im at es as part of t op- down cost ing. Bot h alt ernat ives let you draw t he cut line for t he it erat ion based on m ore careful init ial est im at ion. Again, depending on your proj ect , it m ay or m ay not be wort h t he ext ra effort because you will calibrat e t he t op- down est im at es based on bot t om - up cost ing.

Changes in Team I also sim plified t he previous t echnique t o assum e st at ic t eam size and com posit ion. Clearly, t hat 's oft en false. When your t eam grows or shrinks, capacit y changes. As skills and dom ain expert ise im prove wit h experience, your capacit y grows, t oo.

Differences in Iterations Not all QoS are alike. For exam ple, at Microsoft , it is t ypical t o single out blocks of t im e, usually a

m ont h or m ore, for a " Securit y Push." This is a period when all archit ect ure, designs, and code are reviewed for securit y considerat ions. This enables expert s t o assist t he t eam in an int ensive focus on a part icular QoS t hat requires specialized analysis and addresses specialized risks. Usabilit y, Manageabilit y, and Perform ance are ot her QoS t hat m ight , again depending on your proj ect , lend t hem selves t o specific em phasis in part icular it erat ions when specialist s are available. Sim ilarly, som e scenarios m ay be m ore im port ant t han ot hers for solut ion success, or t hey m ay require int egrat ion of several com ponent s and t herefore be possible only aft er a num ber of prerequisit es have been finished. I t is also com m on t o see an S- curve pat t ern in t he Rem aining Work chart , in which st art up and finish it erat ions ( or t asks wit hin an it erat ion) t ake longer t han t he m iddle ones. I f you ant icipat e t his pat t ern, you can plan capacit y around it .

Working on Multiple Projects Not e t hat in t he t echniques described previously, est im at ion and m onit oring do not require explicit t im e or effort t racking. Because work it em s t rack changes aut om at ically, and t he changes are all t im est am ped for audit abilit y, t he dat a for act ual elapsed t im e is available for free. This m akes it very easy t o do t he est im at es based on days planned and t rack on days act ually spent . The lim it at ion is t hat calendar t im e est im at ion assum es t hat t he t eam m em bers are working on only one proj ect at a t im e. I f t hat 's not t rue, or in ot her words, if som e of you are working on m ult iple proj ect s, t hen you do need t o int roduce effort t racking. Just as im port ant ly, you also need t o significant ly alt er your capacit y est im at es t o accom m odat e t he overhead cost of swit ching cont ext s.

Estimation Quality Twent y years ago, Tom DeMarco int roduced t he idea of an Est im at ing Qualit y Fact or ( EQF) as a way of t racking and t uning t he accuracy of est im at ion. [ 11] The idea is sim ple and powerful. Keep t rack of your est im at es, com pare t hem t o your act ual com plet ion t im e, com put e t he difference, and analyze t he reasons for t he difference. Use t his inform at ion t o t une your next est im at ion. Over m ult iple proj ect cycles, you can assess whet her you're get t ing bet t er and how m uch variance t here is in your est im at ion accuracy. Originally proposed for a proj ect as a whole, EQF is 1 /

t | ( est im at ed com plet ion - act ual com plet ion) |

You can also apply t his idea t o it erat ions. On a t im e- boxed it erat ion or proj ect , you can use t his t echnique against t he backlog size. On a feat ure- boxed one, you can use it against t im e.

Tracking Historical Estimates I n VSTS, you can t rack hist orical est im at es. Work it em dat a, as covered before, is roundt ripped bet ween t he Team Foundat ion Server and Microsoft Proj ect or Microsoft Excel. When you use Proj ect t o edit work it em s, Est im at ed Work, Rem aining Work, and Baseline Work fields are st ored in t he work it em s and pushed int o t he m et rics warehouse for report ing. Accordingly, if you use Proj ect t o creat e a proj ect baseline for your work it em s, t hat dat a is available t o you for report ing from t he m et rics warehouse.

EQF can also be a way of surfacing proj ect concerns am ong t he t eam m em bers. Survey t he t eam m em bers during t he ret rospect ives t o ask when t hey t hink t he proj ect will be " done" or when a cert ain m ilest one, such as live deploym ent , will be reached. Then ask follow- up quest ions t o drill int o t he percept ions. I f t here are changes, ask, " What did you see or hear t hat m ade you change your est im at e?" This approach can yield great inform at ion about risks, unplanned work, skills gaps, or ot her fact ors t hat need t o be addressed. [ 12] Alt ernat ively, if t he t eam is not worried about dat es and funct ionalit y, t hat is great inform at ion. Frequent ly, t eam s get opt im ist ic in t he m iddle of a proj ect as t hey becom e m ore proficient at closing a level of t asks and have not yet encount ered hard int egrat ion issues. This opt im ism will produce an Scurve in t he Rem aining Work chart and a hum p in t he EQF t racking. Of course, your m ileage m ay vary. Unless you are running a proj ect very m uch like one you've run before, wit h a very st able t eam , it is reasonable t o expect variat ion in t he est im at ion qualit y. Excluding special causes ( half t he t eam was divert ed t o an em ergency proj ect , t he com pany was reorganized, and so on) and exam ining t he residual variance will give you good guidelines for t he fact ors t o apply in t he next round of est im at es.

Retrospectives One of t he great benefit s of it erat ive developm ent is t hat you can learn and correct course frequent ly. Ret rospect ives are m eet ings in which you dist ill t he learning from t he it erat ion. The m ost im port ant prerequisit e for a ret rospect ive is a blam e- free environm ent . A great m indset t o bring is sum m arized by Norm an L. Kert h in his book and web sit e: Regardless of what we discover, we underst and and t ruly believe t hat everyone did t he best j ob t hey could, given what t hey knew at t he t im e, t heir skills and abilit ies, t he resources available, and t he sit uat ion at hand. [ 13] During t he ret rospect ive, you should st rive t o ident ify t he com m on- and special- cause variances. Look at t he chart s and ident ify unusual dips in Velocit y, bloat s in Rem aining Work, or inconsist encies in t he Qualit y I ndicat ors. Look for root causes of problem s, reusable lessons learned, and correct ive act ions. I n ot her words, find where t he real bot t lenecks are. For exam ple, t est ing m ay appear slow, but on furt her invest igat ion you find frequent unusable builds. Looking int o builds, you discover t hat services on which you depend or incom ing com ponent s t hat you int egrat e were not perform ing adequat ely. So t he appropriat e correct ive act ion m ight be t o provide bet t er com ponent accept ance t est s t hat m ust pass before t hose ext ernal services or ext ernal com ponent s are accept ed int o your lab and t o have t he providing t eam s run t hese as build verificat ion t est s ( BVTs) . The out put of t he ret rospect ive should inform t he next it erat ion plan. The scenarios, QoS, t asks, and

t heir priorit izat ion m ay all change based on t he discoveries you m ake in t he ret rospect ive.

Triage I n addit ion t o planning, m onit oring, and est im at ing, a good proj ect m anager needs t o priorit ize and schedule t he work t o fit available resources and t im e. This process is called t riage. The t erm was int roduced from m edical pract ice int o t he soft ware world. Originally, t riage only described t he handling of bugs, but now it is equally applicable for all work it em s. Triage is successful only if t he workload is cut t o t he point where it fit s t he available hours, skills, and schedule of t he t eam . I n ordinary, daily t riage sessions, t he resources and t im e dim ensions are fixed, so it is t he priorit izat ion of t he work t hat changes t he funct ionalit y and qualit y levels of t he planned solut ion.

A Triage Exercise To illust rat e t he point of t riage, here is a sim ple exercise: I 'll use bugs only, not all work it em t ypes, and I 'll assum e t hat t here is no ot her work on t he t eam . Suppose you are approaching t he end of an it erat ion. Along t he way, you have an it erat ion goal t o reduce your bug backlog t o zero. There are current ly 10 P1 ( " m ust fix" ) bugs out st anding, as shown in Figure 4.12. Your t eam of five developers can average t wo verified bug fixes per person per day. Your t eam of t hree t est ers is averaging four bugs found per t est er per day. Every four days, t heir find rat e decreases by one bug per t est er.

Figu r e 4 .1 2 .

I n VSTS, you ca n qu e r y w or k it e m s fr om t h e ba ck log, h e r e t h e Act ive Bu gs, for da ily t r ia ge m e e t in gs.

[View full size image]

When will you hit zero backlog at t hose rat es? ( Don't look aheadt ry t o solve t his by yourself first . Even wit h such sim ple num bers, it 's not easy.) Aft er you have solved t his, check your answer against Figure

4.13 .

Figu r e 4 .1 3 .

I n t h is ide a lize d ch a r t , t h e ca pa cit y of t h e t e a m is st a t ic a t 5 0 pe r da y, a n d t h e fin d r a t e dr ops fr om 6 0 t o 3 0 in e ve n in cr e m e n t s, bu t it is st ill n ot obviou s a t fir st w h e n you 'll e lim in a t e t h e ba ck log.

[View full size image]

I n pract ice, of course, t he rat es are not const ant and m ay not be reliable. There are several point s t o not e here: As long as t he find rat e is above t he fix rat e, backlog will increase. When t he find rat e is slight ly below t he fix rat e, convergence will be slow. I t 's easy t o cheat art ificially lowering t he find rat e, for exam ple, by depriorit izing bugs, can m ake

t he num bers look m uch bet t er t han t hey should look. These are only averages. ( Refer t o t he earlier Bug Rat es chart , Figure 4.8 , for a m ore realist ic exam ple of variance.) The point rem ains: Every day you should assess workload, progress, im pedim ent s, and probabilit y of success.

A History of Triage Triage was a m edical t erm long before it was adapt ed t o soft ware engineering: t r ia ge 1727, from Fr. t riage " a picking out , sort ing," from O.Fr. t rier " t o pick, cull" ( see t ry) , hence " act ion of assort ing according t o qualit y." There seem s t o be som e influence from or convergence wit h L. t ria " t hree" ( e.g., t riage for " coffee beans of t he t hird or lowest qualit y" ) . I n World War I , adopt ed for t he sort ing of wounded soldiers int o t hree groups according t o t he severit y of t heir inj uries. [ 14] Casualt ies in World War I far exceeded t he capacit y of t he available m edical st aff at t he front , so t he wounded were sort ed int o t hree groups: t hose who would die regardless of int ervent ion, t hose who would heal wit hout int ervent ion, and t hose who would recover only if t hey received prom pt care. The t erm is st ill used in m edicine- when you walk int o a hospit al em ergency room , t he first person you see is t he t riage nurse. Sim ilarly, t he first person t o handle a bug is t he proj ect m anager or com m it t ee perform ing t riage. Analogous t o t he m edical pract ice, t riage select s work it em s and bugs t hat require prom pt at t ent ion. Chapt er 8, " Report ing Bugs," describes how t riage heurist ics from t he m edical world can help classify bugs.

What Makes Triage Effective: The Red Line Averages, like in t he previous exercise, can be deceiving. Averages don't t ell you who has t he heavy backlog, where t he bugs are being found, and who's fixing t hem and how quickly t hey are doing it . I t is t ypical t o see big differences in all t hese areas. ( See t he Qualit y I ndicat ors chart , Figure 4.7 , t o relat e t he different m easures.) Take t he exercise a lit t le furt her. I m agine t hat t here are five com ponent s, wit h one developer each, and t hat 50% of t he backlog and found bugs are in Com ponent A. You'd see a breakdown on day four like Figure 4.14. The Red Line: ( Fix Rat e Find Rat e) x Days Available < Backlog

Figu r e 4 .1 4 .

Bu g r a t e s a m on g com pon e n t s ca n va r y w ide ly. I t is im por t a n t t o dr a w a " r e d lin e " t h a t t r a ck s ca pa cit y a n d t o loa d ba la n ce or r e pr ior it ize t h e e x ce ss be yon d a va ila ble ca pa cit y.

The red line on your car's t achom et er shows you t he speed at which your engine is likely t o fail. Sim ilarly, t he red line on your work it em dist ribut ion chart is t he level at which your proj ect or one of it s com ponent s will overheat . I t is t he level beyond which you are at risk of not hit t ing t he planned it erat ion exit wit h current find and fix rat es. Bear in m ind t hat som et im es you will let your queues clim b past t he red line, j ust as a racecar driver let s t he t achom et er needle cross t he red line for short periods. For exam ple, t o focus on increasing bug find rat es, you m ight hold a " bug bash" where everyone on t he t eam t est s for a period and t riage is deferred. Part of your plan j ust needs t o be bringing t he queues and red line back in balance. I t 's im port ant t o m anage your red line regularly and consciously.

What Happens During Triage Triage is your opport unit y t o m anage t he red line in real t im e. During t riage, you can use t he Bugs by Priorit y report in VSTS t o underst and t he current red line dist ribut ion and t he query All Act ive Work I t em s t o m anage priorit ies and ownership. Depending on your process, your t riage group m ay be a com m it t ee, wit h a represent at ive of each discipline, or it m ay be a select group. I t is im port ant t hat t he owner be clearly designat ed ( usually t he proj ect m anager or t est m anager) and have ult im at e responsibilit y for t he result ing priorit y list . The m echanics are very easy. VSTS let s you use a st andard query ( like t he previous one) or a bound Excel worksheet t o walk t hrough t he queues and edit priorit y, owner, and ot her appropriat e fields.

Escalating and Resolving Issues I t is im port ant not only t o plan t o capacit y but also t o rem ove any blocking issues t hat prevent t eam m em bers from com plet ing t heir t asks. I n MSF for Agile Soft ware Developm ent , any work it em can be escalat ed using t he I ssue field, and you can see t hese wit h a sim ple query called I ssues. I f it is not done elsewhere, t he t riage process should pick up all t hese issues and address t hem t o unblock t he corresponding work.

Iterations and Triage

One of t he prim ary benefit s of it erat ive developm ent ( described in Chapt er 2, " ValueUp Processes" ) is t hat you can plan for now and lat er, rat her t han now and never. Use t he it erat ions t o m anage t he quant it y of bugs in t he current queues and keep t hem off t he red line. Many proj ect m anagers, especially in agile proj ect s, believe t hat you should not allow any bugs t o be carried forward. According t o t his view, any backlog of bugs great er t han zero is bad, and t hose bugs should be fixed before any new feat ure work proceeds. This is cert ainly t rue of bugs caused by sim ple coding errors. These bugs are cheapest t o cat ch in t he developm ent process wit h unit t est s ( see Chapt er 6, " Developm ent " ) . Unless you have specific reasons ot herwise, t he unit t est s should pass before you check in code. Unfort unat ely, however, not all bugs are caused by coding errors, and not all bugs are caught before check- in. Many bugs appear for subt ler reasons of scenarios, QoS, archit ect ure, configurat ions, dat a, and so on, and will not be caught by unit t est s. I n t hese cases, t here m ay be good reasons t o post pone bug fixes t o t he next it erat ion, and t here will be not hing wrong wit h revisit ing t he priorit ies when you plan t hat it erat ion.

Triage Goals Early, Middle, and Late in the Iteration As you ent er an it erat ion, you need t o be clear about how m uch t im e will be spent on t he bug backlog and how m uch on ot her work it em s, and you need t o re- t riage bug priorit ies accordingly. As scenarios becom e available for t est ing, you m ay focus on finding bugs for a period, int ent ionally let t ing queues lengt hen. Wat ch t he find rat es. As t hey st abilize and fall, even aft er you vary t est ing t echniques ( see Chapt er 7) , you need t o be very clear about your fix capacit y and t riage t he queues accordingly.

Triage Daily, Unless You Have a Good Reason Otherwise Long list s are hard t o m anage. The easiest way t o keep t he incom ing list s for t riage short is t o t riage daily, especially during t he m iddle and end of an it erat ion.

Satisfying the Auditor I t is now com m on for soft ware t eam s t o face regulat ory requirem ent s. For exam ple, all financial syst em s of U.S. public com panies are subj ect t o audit under t he Sarbanes- Oxley Act of 2002. Fort unat ely, VSTS aut om at ically keeps an audit t rail of all changes t o work it em s and aut om at ically links work it em s t o all changes in source code and t est s. I f you are subj ect t o audit , you should enforce t he associat ion of work it em s t o check- ins in order t o m ake sure t hat t he changes t o source code are associat ed wit h work it em changes. For t he developer, t his m eans not hing m ore t han t icking a box at check- in. ( See Figure 6.17.) Wit h t he policy on, t he developer can't forget wit hout being warned and generat ing em ail not ificat ion t o an appropriat e t eam m em ber.

Figu r e 4 .1 5 .

Eve r y w or k it e m , in t h is ca se a sce n a r io, a ccu m u la t e s it s fu ll ch a n ge h ist or y e ve r y t im e it is e dit e d a n d sa ve d. Th is give s you a com ple t e a u dit t r a il.

[View full size image]

Setting CheckIn Policy to Require Associating Work Items to CheckIns I n VSTS, MSF for CMMI Process I m provem ent defines Audit or as one of t he Product Managem ent roles. The guidance for t he Audit or focuses on collect ing evidence for conform ance t o CMMI Level 3 process. This is t he sam e evidence needed for regulat ory audit s, and it is largely aut om at ed by VSTS. Be sure t hat you have configured check- in policies t o include t he requirem ent of associat ing work it em s on check- in. See t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Foundat ion Team Foundat ion Walkt hroughs Team Foundat ion Server Adm inist rat ion Walkt hroughs Walkt hrough: Cust om izing Check- in Policies and Not es

Figu r e 4 .1 6 .

Ch a n ge se t s, con t a in in g t h e code ch a n ge s for e a ch ch e ck - in , a r e a u t om a t ica lly lin k e d t o t h e sce n a r io or ot h e r w or k it e m so t h a t you ca n se e bot h w h a t ch a n ge d a n d w h y. Th e se ch a n ge s a n d a ssocia t ion s flow t h r ou gh t o t h e bu ild r e por t s, discu sse d in Ch a pt e r 6 .

[View full size image]

Summary I n t his chapt er, I covered t he basics of proj ect m anagem ent wit h VSTS. First , I covered t hree value- up concept s t hat inform t he act ivit ies: underst anding variance, using descript ive rat her t han prescript ive m et rics, and keeping m ult iple dim ensions of proj ect healt h in m ind. Then I reviewed exam ples of several of t he report s in VSTS and how t o use t hem . Finally, we looked at applying t hese report s and work it em queries t o est im at ion and t riage. I did not yet explore t roubleshoot ing wit h t hese report s, as t hat will be covered in Chapt er 9. I n t he next chapt er, I will look at t he archit ect ure and design of a solut ion.

Endnotes 1. L. Koskela and G. Howell, ( 2002) , " The Underlying Theory of Proj ect Managem ent is Obsolet e." Proceedings of t he PMI Research Conference, 2002, 293302, available at ht t p: / / www.leanconst ruct ion.org/ pdf/ Obsolet eTheory.pdf .

2. W. Edwards Dem ing, The New Econom ics: For I ndust ry, Governm ent , Educat ion , Second Edit ion ( Cam bridge: MI T Press, 1994) , 174.

3. I bid., 190 ff.

4. ht t p: / / www.c2.com / cgi/ wiki?Yest erdaysWeat her

5. Robert D. Aust in, Measuring and Managing Perform ance in Organizat ions ( New York: Dorset House, 1996) , 15.

6. For exam ple, Beck ( 2000) , op. cit ., 723.

7. Cum ulat ive flow diagram s were int roduced t o soft ware in Anderson 2004, op.cit ., 61.

8. I bid., 90 ff.

9. For exam ple: Johanna Rot hm an, " What 's Your Fault Feedback Rat io?," Com put erWorld ( Novem ber 04, 2002) , available from ht t p: / / www.com put erworld.com / developm ent t opics/ developm ent / st ory/ 0,10801,75654,00.ht m l .

10. ht t p: / / www.auxet rain.org/ Nav1.ht m l

11.

ht t p: / / www.auxet rain.org/ Nav1.ht m l

12. Tom DeMarco, 12. ht t p: / / www.j rot hm an.com / pragm at icm anager.ht m l

13. ht t p: / / www.j rot hm an.com / pragm at icm anager.ht m l

14. ht t p: / / www.ret rospect ives.com / pages/ ret roPrim eDirect ive.ht m l

Chapter 5. Architectural Design

Ar ch it e ct " Every syst em has an archit ect ure, encom passing t he key abst ract ions and m echanism s t hat define t hat syst em 's st ruct ure and behavior as seen from t he perspect ive of different st akeholders, each wit h a different set of concerns. I n every casefrom idiom s t o m echanism s t o archit ect urest hese pat t erns are eit her int ent ional or accident al, but insofar as t hey are visible, such pat t erns reflect t he st yle and inner beaut y of each syst em ." [ 1 ] Grady Booch, Handbook of Soft ware Archit ect ure

Figu r e 5 .1 .

Eve r y syst e m h a s a n a r ch it e ct u r e . Th e h on e ycom b of a be e h ive is a gr e a t e x a m ple of a n a r ch it e ct u r e w h ose con ce pt u a l in t e gr it y a llow s a n d a pplie s pa t t e r n s, con t in u a l e volu t ion , fle x ibilit y, sca lin g, a n d r e u se .

Archit ect ural design t ransform s t he requirem ent s, as capt ured in scenarios and qualit ies of service int o designs t hat can be built t o sat isfy and delight users. The analogy bet ween t he work of soft ware archit ect s and civil archit ect s has been m ade count less t im es t o illust rat e t he point . Civil archit ect ure, however, consist s not only of t he work of professional archit ect s but also of t housands of years of craft sm anship in t he applicat ion of pat t erns t hat work. [ 2 ] A good archit ect ure reflect s a concept ual int egrit y t hat m akes t he archit ect ure easy t o underst and,

use, m aint ain, and evolve. This concept ual int egrit y is t ypically achieved t hrough t he choice, applicat ion, and specializat ion of t he right pat t erns t o t he broad qualit ies of service and scenarios defined for a syst em . Alt hough t here is broad agreem ent on t he charact erist ics of a good archit ect ure, t here is cont roversy about t he process of archit ect ural design. The ext rem es range from t he concept t hat archit ect ural design should be pract iced independent ly of im plem ent at ion [ 3 ] t o t he idea t hat archit ect ure should em erge from im plem ent at ion. [ 4 ] Accordingly, archit ect ure is som et im es an explicit process, whereas at ot her t im es it is a set of act ivit ies com bined closely wit h developm ent .

A Value-Up View of Architecture A value- up approach assert s t hat archit ect ure needs t o be explicit but can em erge from t he im plem ent at ion, and it absolut ely needs t o be current wit h t he im plem ent at ion. VSTS follows t his view. I n VSTS, archit ect ural designs are views on source code at what ever st at e of im plem ent at ion t he code happens t o be. Archit ect urally significant updat es t o diagram s, source code, or configurat ion sim ult aneously updat e t he archit ect ure and t he im plem ent at ion. VSTS support s t he archit ect ural design process by enabling an archit ect t o design t he archit ect ure of soft ware t o be built , and it exposes t o t he archit ect aspect s of t he archit ect ure of soft ware t hat already exist s. VSTS focuses on t he deploym ent st ruct ure, which is expressed in t erm s of configurat ions of applicat ions, which are defined as discret e packages of resource t hat are deployed t oget her.

Service-Oriented Architecture Concept ual int egrit y is a key part of good archit ect ure. Service- orient ed archit ect ure ( SOA) , t he predom inant m odern concept ual fram ework for soft ware and syst em s archit ect ure, provides t hat int egrit y. The rise of SOA has been driven by t wo overriding business concernst he need for business agilit y and t he desire for m axim um reuse of exist ing asset s. Toget her, t hese t wo collapse int o t he t went y- first cent ury edict : " Do m ore wit h less." Business agilit y is all about " doing m ore" fast er. I T syst em s have hist orically suffered from m uch less flexibilit y t han t he business dem anded. As t he pace of business change has increased in t he last couple decades ( ironically, oft en spurred by I T) , t he gap bet ween t he evolving business needs and I T's abilit y t o adapt t o t hem has widened. The prom ise of SOA is t hat I T capabilit ies can be exposed as services t hat are closely aligned wit h business services, flexibly recom bined, and changed as needed in sm all increm ent s. A business process can be seen as a t hread of cont rol flowing t hrough t he capabilit ies of t he organizat ion. Exposing t he organizat ion's capabilit ies as discret e granular services allows flexibilit y in t he way t hey can be com bined int o business processes and increases t he ease wit h which t he processes can be changed and t he capabilit ies or t heir im plem ent at ion can be alt ered. This st yle cont rast s wit h t he hist oric approach of m onolit hic syst em s t hat at t em pt ed t o circum scribe large problem s, ant icipat e all requirem ent s in advance, and address t hem in very large proj ect s. I n m any ways, SOA is an archit ect ural paradigm aligned wit h cont inuous im provem ent . Maxim um reuse is about doing it " wit h less." First , alm ost every ent erprise has large sunk cost s in legacy syst em s. These syst em s were not oft en built for m aint ainabilit y or flexibilit y, and t he cost s and business risks of replacem ent are oft en high. SOA enables t hese legacy syst em s t o be wrapped wit h facades t hat expose t hem as web services so t hat t hey can be recom posed flexibly int o larger syst em s. Addit ionally, SOA enables int eroperabilit y am ong diverse operat ing syst em s and t echnical plat form s. The WS- * st andards m ake it possible t o orchest rat e dist ribut ed applicat ions across Windows, Linux, and m ainfram es, spanning .NET and J2EE im plem ent at ions. [ 5 ] Anot her business t rend favoring SOA is t he need for geographic and organizat ional flexibilit y in locat ing and com posing syst em s. I ncreasingly, what appears t o t he users as one syst em m ay be developed and m anaged as m any separat e services, and over t im e, t hose services m ay be m igrat ed or replaced as business needs warrant . Consider t he t ypical web experience of looking up direct ions and a m ap. There are only a handful of GI S m apping services, whose int erfaces are exposed t hrough web services t o hundreds of t housands of web sit es t hat provide t he appearance of cust om ized m aps. The visit ors t o t hose sit es couldn't care less whose engine is drawing t he m ap and direct ionst he web sit e owner m akes a sim ple business choice and can change t he choice as business condit ions evolve. The sam e flexibilit y applies t o propriet ary services. By using SOA and m aking services self- cont ained and granular, organizat ions can ret ain significant choices in how t o locat e, out source, or cont ract for t he developm ent and m anagem ent of t he business capabilit ies on which t hey rely. Sim ilarly, choices about m aint enance, upgrade, and t uning can be m ade at t he level of t he services, affording a flexibilit y not found wit h m ore m onolit hic syst em s.

Web Services and SOA

I n t heory, SOA does not require web services, but in pract ice, t he t echnology for im plem ent ing web services as been alm ost com plet ely aligned wit h t he WS- * st andards, if only for t he reason t hat t his enables t echnical int eroperabilit y . When developing wit h Visual St udio, t he Microsoft .NET Fram ework m akes it easy t o im plem ent t he web services. Of course, t here is m ore t o int eroperabilit y t han j ust t he st andards.

Contract-First Design The key t o int eroperabilit y is t hat services describe t hem selves in t erm s of int erface cont ract s. For web services, t hese int erfaces are expressed in t he Web Services Descript ion Language ( WSDL) . [ 6 ] A valuable SOA pract ice is cont ract - first design, or in ot her words, specifying t he m essage form at s and the WSDL am ong part icipat ing services before being concerned wit h im plem ent at ion det ails. Cont ract first design can ensure loose coupling and prevent decisions about how t o im plem ent services from creeping int o t he overall dist ribut ed syst em design. Pract ices for cont ract - first design are st ill evolving. WSDL and XSD do not yet describe m essage sequence or pre- and post - condit ions as exam ples. Fuller cont ract s need t o specify t he m essages t hat t he services publish and consum e, t he sequence of handling, and t he const raint s in order t o isolat e t he public cont ract s from t he privat e det ails of t he im plem ent at ion. One approach t o im proving cont ract s can be found in t he Windows Com m unicat ion Foundat ion Services ( WCF) , previously known as proj ect " I ndigo," part of Windows Vist a. [ 7 ]

Constraints with Degrees of Freedom A value- up approach requires an explicit archit ect ure, but it m ay em erge from t he im plem ent at ion . This creat es an apparent cont radict ion. On one hand, you m ust consider all t he QoS and deploym ent const raint s in t he cont ext of t he scenarios. On t he ot her hand, value- up considers running t est ed code t o be t he best m easure of syst em progress, and t he act of developing is t he act of designing, albeit at a very det ailed level. I t is exact ly t he act of developm ent , in sm all fixed- scope it erat ions, t hat enables design t o em erge from t he im plem ent at ion. The answer lies in t he creat ion of a baseline archit ect ure early in t he proj ect lifecycle.

Baseline Architecture A baseline archit ect ure is a execut able skelet al applicat ion designed expressly t o m it igat e t echnical risk and provide a solid, st able basis for proj ect it erat ions. Som e agile developers refer t o t his as " it erat ion zero" when plat form infrast ruct ure com ponent s are inst alled and a " t hin vert ical slice" of t he syst em is built , exercising a very sim ple scenario from end t o end ( for exam ple, from t he user int erface t o t he dat abase) . The goal here is t o flush out and validat e t he proj ect 's archit ect urally significant m echanism s unam biguously. Rem oving am biguit y m andat es t hat t he baseline archit ect ure be execut able. As an exam ple, consider a popular applicat ion st ylea web applicat ion using a relat ional dat abase. A num ber of fundam ent al quest ions need t o be answered early t o det erm ine t he best fit based on good cit izenship and all t he perspect ives of t he advocacy groups described in Chapt er 2 , " Value- Up Processes" . Should t his applicat ion be bought or built ? Should t he applicat ion be a sm art client or a web applicat ion? I f it is a web applicat ion, what processing should be done on t he client using t echnologies such as Asynchronous Java Script and XML ( AJAX) , and what processing should be done on t he server? As for persist ence ( assum ing a relat ional dat abase) , what are t he organizat ional st andards for t he choice of a relat ional dat abase? Should a new dat abase inst ance be creat ed, or should t ables be added t o an exist ing dat abase inst ance? What is t he source of reference dat a for t his applicat ion? How does t his applicat ion int egrat e wit h ot her applicat ions in t he ent erprise? How m any business t ransact ions per day m ust t his syst em be able t o execut e t o support t he proj ect 's econom ic j ust ificat ion? Will t his applicat ion be deployed, operat ed, and m aint ained in a corporat e dat acent er? I f so, what are t he t echnical const raint s of t he dat acent er? As t hese fundam ent al kinds of decisions are m ade early in t he proj ect lifecycle, t hey result in const raint s t hat guide and st abilize t he proj ect under developm ent . There is a delicat e balance t o st rike. As you work t hrough t hese t echnical issues, t ry t o defer decisions t o t he " last responsible m om ent ." The last responsible m om ent , however, will vary depending on proj ect com plexit y. Pull t oget her all t he advocacy groups from Chapt er 2 early in t he proj ect lifecycle t o agree on t he high- level decision policy. Oft en, you will let det ailed design issues, such as int erfaces and m et hod fact oring, evolve from im plem ent at ion. I n m ore com plex proj ect s wit h m any dependencies, you m ay need t o pin down int erfaces and m it igat e archit ect urally significant risks early t o avoid lat er rework. Then, wit hin t he agreed policy, build an execut able baseline t hat im plem ent s t hese high- level design decisions so t hat t hey can becom e archit ect ural const raint s on t he proj ect . Defining t his baseline early benefit s developers by providing st able plat form infrast ruct ure com ponent s and prot ocols for applicat ion developm ent . Defining t hese const raint s early also enables predict abilit y and planning for ot her st akeholders. For exam ple, Release/ Operat ions can plan for t he deploym ent

and m anagem ent of t he applicat ion in product ion and can ensure dat acent er readiness. Product m anagem ent in t he line- of- business can creat e m arket ing plans designed t o at t ract levels of usage at t ransact ion rat es t he syst em can support . Business analyst s can design business process m onit oring report s t o analyze t he applicat ion's ret urn on invest m ent , which will feed planning for t he next version. Ent erprise archit ect s can design and im plem ent reusable services across t he ent erprise, m igrat ing t oward an ent erprise- level SOA. Defining t hese const raint s early enables a st able, predict able basis for m ult iple st akeholders.

Validate Architectural Decisions This process is not j ust t op- down. You need t o validat e t hese const raint s bot t om - up by building an execut able skelet al archit ect ure . VSTS enables you t o m odel your " asis" dat acent er using t he Logical Dat acent er Designer ( shown in Figure 5.5 ) and your " t o- be" applicat ion using t he Syst em and Applicat ion Designers ( shown in Figures 5.3 and 5.4 ) and t hen t o run a validat ion report t o ident ify conflict s bet ween t he t wo . Then your Syst em and Applicat ion m odels can be used t o generat e skelet al code t hat can becom e part of your execut able baseline. Doing so provides m any benefit s. The running code is unam biguousshould areas of design seem unclear, t he source code can be consult ed. The running code can be perform ance t est ed t o validat e init ial t ransact ion rat es of t he syst em . Syst em set t ings can be verified against dat acent er policies t o ident ify pot ent ial conflict s. Also, building an execut able syst em forces lower- level design decisions t o becom e explicit , furt her refining t he baseline.

Refining the Baseline As you build out t he skelet al archit ect ure t o run end- t o- end, you are forced t o consider lower- level design decisions. How will you st ruct ure your applicat ion logic? What is your dat a access approach? Will you deploy t he business logic on t he sam e server as t he web server, or will you deploy t hem on separat e applicat ion servers? What aut hent icat ion and aut horizat ion m echanism s will you use? What prot ocols will you use bet ween servers? Don't t ry t o address all t he concerns yet . Choose t he areas of highest t echnical risk t o validat e and let t he ot her decisions em erge t hrough im plem ent at ion. Think about building t he sim plest skelet al applicat ion t hat will m eet your const raint s, m it igat e t echnical risks, and st ill provide a sound, st able basis for developm ent it erat ions.

Detailed Example of Refining the Baseline For a det ailed exam ple of t his t echnique, see t he MSDN Tech Not e: TN_1111: Top- Down Syst em Design ht t p: / / m sdn.m icrosoft .com / vst udio/ t eam syst em / reference/ t echnot es/ syst em _designer/ t opdown_sys.aspx

You should also leverage t he design knowledge and experience of ot hers who m ay have also solved sim ilar problem s before and capt ured t he learning in pat t erns. [ 8 ] Pat t erns also give you a language wit h which you can concisely describe solut ions, m it igat e risk by reusing known good designs, and enable fut ure m aint enance and reuse of your work.

.NET Framework-Specific Patterns For pat t erns specific t o t he .NET fram ework, see m sdn.m icrosoft .com / pract ices.

Reference Architectures I n m any cases, you can use a reference archit ect ure as a baseline. For exam ple, Microsoft has creat ed a reference applicat ion called Applied I nt egrat ion Baseline ( AI B ) , which uses a pat t ern- based approach t o build an execut able baseline ( see Figure 5.2 ) . This applicat ion ships wit h m ore t han t welve Visual St udio proj ect s and is based on ASP.NET, BizTalk Server, SQL Server, Host I nt egrat ion Server, and VSTS.

Figu r e 5 .2 .

Th e Applica t ion I n t e gr a t ion Ba se lin e pr ovide s a n in t e r a ct ive " N a r r a t or " t o visu a lize t h e ba se lin e a r ch it e ct u r e . Th e visu a l m ode ls in clu de VSTS Applica t ion D e sign e r a n d Logica l D a t a ce n t e r D e sign e r m ode ls a s w e ll a s a pa t t e r n - ba se d de scr ipt ion of t h e syst e m . H e r e t h e n a r r a t or is illu m in a t in g a sce n a r io on e st e p a t a t im e by r e n de r in g e a ch st e p on t h e Applica t ion D e sign e r dia gr a m .

[View full size image]

Windows Server System Reference Architecture For planning infrast ruct ure and dat acent ers, you can download t he Windows Server Syst em Reference Archit ect ure from ht t p: / / www.m icrosoft .com / t echnet / it solut ions/ wssra/ raguide/ default .m spx .

I n a sim ilar way, t he Windows Server Syst em Reference Archit ect ure ( WSSRA) offers t he infrast ruct ure archit ect a baseline for t he configurat ion of a dat acent er . Obviously, act ual circum st ances will vary, but t he WSSRA baseline provides a preferred m odel t o set up t he server environm ent s.

Download the Application Integration Baseline You can download t he Applicat ion I nt egrat ion Baseline from MSDN Library Servers and Ent erprise Developm ent Ent erprise Archit ect ure, Pat t erns, and Pract ices Microsoft Pat t erns & Pract ices Reference I m plem ent at ions

VSTS and Service-Oriented Architecture For all t he following reasons, VSTS focuses on t he pract ical aspect s of im plem ent ing a dist ribut ed syst em using a SOA. VSTS provides four designers t hat handle t he m aj or act ivit ies involved:

1 . The Applicat ion Designer ( see Figure 5.3 ) enables you t o design t he applicat ion com ponent s t hat expose and consum e web services.

Figu r e 5 .3 .

Use t h e Applica t ion D e sign e r t o de scr ibe t h e com pon e n t s t h a t w ill com m u n ica t e w it h w e b se r vice s. N ot e t h a t you ca n a dd you r ow n cu st om t ype s t o t h e t oolbox .

[View full size image]

2 . The Syst em Designer ( see Figure 5.4 ) enables you t o com pose and configure t he applicat ions int o syst em s and pot ent ially reusable subsyst em s.

Figu r e 5 .4 .

Use t h e Syst e m D e sign e r t o com pose t h e se a pplica t ion s in t o w h ole syst e m s or r e u sa ble su bsyst e m s.

[View full size image]

3 . The Logical Dat acent er Designer ( see Figure 5.5 ) capt ures servers ( such as I I S) , t heir configurat ions, and net work t rust zones as you use t hem in a dat acent er int o which one or m ore syst em s will be deployed.

Figu r e 5 .5 .

Use t h e Logica l D a t a ce n t e r D e sign e r t o ca pt u r e t h e con figu r a t ion of t h e se r ve r s a n d t h e n e t w or k t r u st zon e s t h a t a r e in ope r a t ion or pla n n e d for t h e da t a ce n t e r .

[View full size image]

4 . The Deploym ent Designer ( see Figure 5.6 ) enables you t o m ap each com ponent in a syst em t o t he servers in a logical dat acent er t o specify how t he dist ribut ed part s of t he syst em need t o be deployed.

Figu r e 5 .6 .

Use t h e D e ploym e n t D e sign e r t o m a p e a ch a pplica t ion com pon e n t t o t h e cor r e spon din g se r ve r a n d t h e r e by spe cify h ow t h e dist r ibu t e d a pplica t ion n e e ds t o be de ploye d.

[View full size image]

I n VSTS, t he designs generat e source code and XML files and t herefore becom e live views int o t he source code and configurat ion files. As archit ect urally significant changes are m ade in source code and configurat ion files, t hese diagram s updat e aut om at ically.

Quality of Service Mindset MSF describes a QoS m indset as follows: The idea is t hat qualit ies of service such as perform ance and securit y should not be considered lat e in t he proj ect but t hroughout it . When ignored, t hese qualit ies of service are ult im at ely consum er dissat isfiers. [ 9 ] Archit ect ural design needs t o reflect t his QoS m indset . Oft en t he archit ect is t he t eam m em ber m ost able t o consider t he im plicat ions of QoS requirem ent s, im plicit or explicit . I n t urn, t he decisions wit h t he great est im pact on QoS are t ypically m ade during design.

Security Securit y is a prim ary archit ect ural concern. The prim ary archit ect ural analysis t echnique for securit y is t hreat m odeling, which looks for pot ent ial vulnerabilit ies of t he planned syst em . [ 10] Fort unat ely for web services, t he securit y profiles are im plem ent ed in t he Web Services Enhancem ent s ( WSE) Microsoft .NET Fram ework. [ 11] WSE enables you t o sign and encrypt web services and set t rust dom ains.

Performance Perform ance is anot her leading archit ect ural concern. A t ypical t rap in archit ect ure is t hat perform ance problem s usually are not det ect ed unt il lat e in product developm ent , leading t o cost ly redesign or rework. VSTS m it igat es t his risk in t wo ways. Because t he SOA designers generat e code, you can creat e applicat ion skelet ons early t hat can be subm it t ed t o t rial deploym ent and perform ance t est ing, well before t he syst em is com plet e. And because t he syst em s are m ade of aut onom ous services, reconfigurat ion is st raight forward, and t uning can be opt im ized t o t he problem areas.

Citizenship Mindset The shift from self- cont ained applicat ions t o services requires a change in m indset , which MSF calls Cit izenship. As applicat ions m ove from being self- cont ained proj ect s t o consum ers and publishers of services, t he proj ect t eam needs t o t hink different ly about it s approach. Consum ers need t o look for services t o consum e and not bypass t he infrast ruct ure t hat is available t o t hem . Publishers need t o realize t hat m any of t heir consum ers are unknown. Published services need t o act reliably for any consum er adhering t o t he service cont ract and prot ect against m alicious hackers who are looking t o exploit t he service wit h ill- form ed m essages or ot her at t acks. This m indset presum es a t ransparency and t rust t hat are t ypical of high- perform ing organizat ions. Here's a descript ion from Philip Evans' and Bob Wolf's " Collaborat ion Rules" t hat plays equally t o t he services of an SOA as t o t he appropriat e cult ure for developing one: Where t rust is t he currency, reput at ion is a source of power. I n a dense net work . . . . t here is m ore power t o being an inform at ion source t han an inform at ion sink. Consequent ly, individuals are m ot ivat ed t o m axim ize bot h t he visibilit y of t heir work and t heir connect ions t o t hose who are t hem selves broadly connect ed. [ 12]

Design for Operations A t ypical problem t hat m ost organizat ions face is t he com plexit y of m oving a syst em as designed int o a dat acent er as configured. The developm ent and operat ions t eam s t ypically have different vocabulary, different st affs, and different physical locat ions ( som et im es on different cont inent s or in different com panies) . The inform at ion flow bet ween developm ent and operat ions can be very dissat isfying, oft en inaccurat e and incom plet e. Part icularly wit h dist ribut ed syst em s, t here are m any part s t o deploy on m any servers, each wit h it s own configurat ion requirem ent s. For exam ple, applicat ions t hat run in t est environm ent s don't run in product ion environm ent s because t he applicat ions violat e policies t hat developers were not aware of. I n t his sit uat ion, archit ect ural design changes are som et im es required t o bring applicat ions int o conform ance, or conversely, applicat ions need t o m ake requirem ent s of t he dat acent er, such as specific versions or service packs. These operat ional requirem ent s m ay conflict wit h t he exist ing deploym ent s of ot her applicat ions. The consequence of t hese com m unicat ion m ism at ches m ay be a delay of weeks or m ont hs bet ween t he t im e t hat archit ect s, developers, and t est ers announce t hat an applicat ion is ready t o deploy and t he readiness of operat ions t o act ually deploy it . I n t he value- up approach of em bracing change and enabling business agilit y, t hese delays are a huge barrier. VSTS st art s t o rem ove t hese disconnect s by enabling Design for Operat ions. The principle is sim ple. Rat her t han wait ing unt il an applicat ion is im plem ent ed t o at t em pt deploym ent , VSTS t est s t he deployabilit y of t he applicat ion during it s design. The solut ion archit ect and infrast ruct ure archit ect can t hen resolve any incom pat ibilit y bet ween t he applicat ion requirem ent s and t he dat acent er const raint s well ahead of act ual deploym ent and significant ly reduce t he bake t im e needed for successful deploym ent . Design for Operat ions is m ade possible by t he Syst em Definit ion Model ( SDM) : At t he heart of SDM is t he not ion of a syst em . I n it s m ost basic form , a syst em is an independent ly deployable configurat ion of resources. For soft ware syst em s, resources are ult im at ely direct ories and files, such as binaries, XML files, configurat ion files, SQL script files, and so on. For hardware syst em s, such as graphics syst em s, m ot herboards, net work int erface cards and power supplies, resources m ight include t he boards, processor chips, m em ory chips, fans, and ot her lower- level com ponent s. I f a syst em enables access t o it s resources, or if it s resources access services offered by resources in ot her syst em s, it exposes t hose resources via endpoint s. For exam ple, a m ot herboard m ight provide an I DE endpoint for peripherals; web service endpoint s provide a m eans for an applicat ion t o expose or consum e web services; HTTP endpoint s provide a m eans for a server t o enable access using t he HTTP prot ocol. [ 13] The SDM is visualized on t he Deploym ent Designer ( as shown in Figure 5.6 ) . I t m arries t he applicat ion design ( t ypically from t he solut ion archit ect ) wit h t he logical dat acent er design ( from t he infrast ruct ure archit ect ) . Direct ly on t his diagram , you can validat e t he archit ect uret hat is, you can det erm ine whet her t he applicat ion as designed will deploy in t he dat acent er as configured. Any except ions appear direct ly as warnings wit h glyphs on t he diagram as needed ( see Figure 5.7 ) .

Figu r e 5 .7 .

Va lida t ion of t h e D e ploym e n t D e sign e r ca n ide n t ify con flict s be t w e e n t h e a pplica t ion a s de sign e d a n d t h e da t a ce n t e r a s con figu r e d. I n t h is e x a m ple , t h e a pplica t ion r e qu ir e s W in dow s Se r ve r 2 0 0 3 , bu t it is in t e n de d for de ploym e n t on a se r ve r w it h a n e a r lie r OS.

[View full size image]

The SDM defines a m odel of t he syst em and a m odel of t he dat acent er and enables a deploym ent t o be defined and validat ed. Models allow validat ion of a design, even before coding has begun. Aft er t he applicat ion has been im plem ent ed, changes t o t he applicat ion's configurat ion t o resolve errors are direct ly synchronized wit h code and configurat ion files. The fact t hat m odels reflect real code is very im port ant for deploym ent validat ion. Models also describe how configurat ion can be overridden. I n addit ion t o enabling t he validat ion at design t im e, VSTS facilit at es t he verificat ion at t est t im e by enabling t est labs t o run in virt ual m achines ( which is discussed in Chapt er 7, " Test ing" ) .

Summary I n t his chapt er, I briefly discussed a value- up approach t o archit ect ural design, t he process of creat ing a solut ion archit ect ure. I t is highly it erat ive, working bot h t op- down and bot t om - up, em erging from choices about im plem ent at ion and driven by t he scenarios and qualit ies of service t hat define user value and cont ract boundaries. SOA is t he dom inant super- pat t ern t hat governs overall design of m odern dist ribut ed syst em s because it enables business alignm ent , agilit y, int eroperabilit y, and reuse. The MSF m indset s for Qualit ies of Service and Cit izenship bring essent ial point s of view t o SOAt he considerat ion of all aspect s of experience and creat ion or applicat ion of reuse wherever possible. Cont ract - first design is an im port ant pract ice for successful SOA. The web services st andards are t he m ain enabler of SOA. VSTS support s archit ect ural design of web services and com posit ion of syst em s direct ly based on t hese services. VSTS int roduces Design for Operat ions t o reduce t he com plexit y of preparing dist ribut ed syst em s for deploym ent . VSTS provides a Deploym ent Designer t hat m arries t he logical dat acent er design t o t he syst em design, enabling you t o see whet her t he applicat ion as designed will deploy in t he dat acent er as it is configured. The next chapt er looks at t he value- up pract ices of developm ent .

Endnotes 1.

Grady Booch, Handbook of Soft ware Archit ect ure, work in progress, available at ht t p: / / www.booch.com / archit ect ure/ index.j sp.

2.

Christ opher Alexander first int roduced t he not ion of pat t erns of building in civil archit ect ure, and t his spawned t he pat t erns com m unit y in soft ware archit ect ure. [ Alexander 1964] Christ opher W. Alexander, Not es on t he Synt hesis of Form ( Harvard Universit y Press, 1964) .

3.

For exam ple, Brooks, op. cit ., 45 .

4.

For exam ple, ht t p: / / xp.c2.com / TheSourceCodeI sTheDesign.ht m l.

5.

www.ws- i.org

6.

ht t p: / / www.w3.org/ TR/ wsdl

7.

ht t p: / / m sdn.m icrosoft .com / library/ default .asp?url= / library/ en- us/ dnlong/ ht m l/ int rot owcf.asp

8.

Three im port ant works here are Eric Evans, Dom ain- Driven Design: Tacking Com plexit y I n t he Heart of Soft ware ( Bost on: Addison- Wesley, 2003) ; Mart in Fowler et al., Pat t erns of Ent erprise Applicat ion Archit ect ure ( Bost on: Addison- Wesley, 2002) ; and Joshua Kerievsky, Refact oring t o Pat t erns ( Bost on: Addison- Wesley, 2004) .

9.

MSF, bot h inst ances

1 0 . ht t p: / / m sdn.m icrosoft .com / securit y/ 1 1 . ht t p: / / m sdn.m icrosoft .com / webservices/ webservices/ building/ wse/ default .aspx 1 2 . Philip Evans and Bob Wolf, " Collaborat ion Rules," Harvard Business Review, JulyAugust 2005. 1 3 . ht t p: / / m sdn.m icrosoft .com / vst udio/ t eam syst em / reference/ t echnot es/ apps_designer/ sdm .aspx

Chapter 6. Development

D e ve lope r D e ve lopm e n t M a n a ge r " Working soft ware over com prehensive docum ent at ion" The Agile Manifest o [ 1 ]

Figu r e 6 .1 .

N e w t on 's Cr a dle is a com m on de sk t op t oy. W h e n you a pply for ce fr om on e e n d, t h e ba lls sw in g in a pr e dict a ble r e gu la r m ot ion . W h e n you a dd a for ce fr om t h e opposit e e n d, t h e ba lls st a r t bou n cin g ch a ot ica lly a ga in st e a ch ot h e r . I t 's a m e t a ph or for de ve lopm e n t pr a ct ice . Sim ple , dir e ct ion a l for ce e n cou r a ge s pr e dict a bilit y, w h ile con t r a dict or y for ce s ca n cr e a t e ch a os.

This chapt er is not about program m ing languages or coding t echniques. These im port ant t opics are well covered in m any ot her books. Rat her, t his chapt er is about t he act ivit ies t hat VSTS enables a developer t o perform as a com plem ent t o writ ing product ion code t hat help t o m ake t hat code successful. There's no religion here. I com plet ely recognize t hat m any readers will t hink t hat t he pract ices I describe don't go far enough. Again, m any books cover m et hodologies or individual pract ices in considerable dept h, whereas I only have space t o t ouch t he surface.

Rat her, m y goal here is t o give you enough of t he reasoning behind and t ooling in VSTS t o do a bet t er j ob right away. I hope it m akes you want t o dig deeper fast er t hrough t he ot her sources, t oo.

More Detail on Development Techniques For a how- t o descript ion of t he developm ent t echniques discussed here, see Will St ot t and Jam es Newkirk, Visual St udio Team Syst em Bet t er Soft ware Developm ent for Agile Team s ( Addison- Wesley, 2006) .

A Value-Up View of Development The value- up approach m easures only deliverables t hat t he cust om er values. More t han anyt hing else, t his m eans working code of qualit y suit able for cust om er delivery. For t went y- five years, we've known t hat ensuring qualit y early is m uch cheaper t han rem oving bugs lat e. [ 2 ] Only in t he last t en years, however, have pract ices shift ed from not ions like " Code Com plet e," which t reat s bug rem oval as a deferrable act ivit y, t o " Test - Driven Developm ent ," which focuses on bug prevent ion wit h 100% of t est s passing at t he t im e of check- in. For purposes of t his chapt er, I 'm going t o assum e t hat you are a skilled developer. Also, I 'm going t o assum e t hat you, like virt ually every developer I 've ever m et , want t o do qualit y work and rem ove t he im pedim ent s t o delivering qualit y.

Quality from a Developer's Point of View I n t he vast m aj orit y of cases, t here are five broad process problem s t hat lead ( direct ly or indirect ly) t o qualit y problem s in a developer's work:

1 . Poor ly com m u n ica t e d, m isu n de r st ood, or ou t - of- da t e r e qu ir e m e n t s .Chapt er 3, " Requirem ent s," discusses how t o define and m anage scenarios and QoS requirem ent s t o cont ain problem s relat ed t o requirem ent m isunderst andings. However, as a developer, you can t ake responsibilit y for forcing good requirem ent s, as discussed lat er. 2 . Pr ogr a m m in g e r r or s . People writ e code, and people m ake m ist akes. I n part icular, it is oft en very hard t o writ e code t hat t akes all t he necessary qualit ies of service int o account , such as securit y, perform ance, localizat ion, and m aint ainabilit y. This is t rue for bot h m anaged and unm anaged code, alt hough unm anaged C and C+ + incur t he addit ional securit y hazards of buffer overruns, unallocat ed m em ory reads, and sim ilar m em ory violat ions. 3 . La ck of t e st in g fe e dba ck . Even developers who know t hat t hey should be writ ing unit t est s oft en find t hem selves facing t oo m any disincent ives. Unit t est ing t akes t im e; it 's oft en hard t o t ell how m uch code has been t est ed and under which condit ions, and t he t est s oft en do not get reused or not iced. Wit hout t horough unit t est ing and it s im m ediat e feedback t o a developer, t hough, it is very easy t o change code and not discover undesirable side effect s unt il m uch t oo lat e. 4 . Ve r sion sk e w s . Even developers who would never consider working wit hout source cont rol oft en find t hat t heir source cont rol does not deal adequat ely wit h t he m any non- source files on which t he proj ect depends, such as t he build syst em , t est s, and configurat ions. Differences in t hese files oft en lead t o t hose hard- t o- diagnose bugs t hat provoke from t he observat ion, " But it worked on m y m achine! " 5 . La ck of t r a n spa r e n cy . The developm ent infrast ruct ure, proj ect m anagem ent syst em , and bug/ change request t racking and m et rics ( if any) are t reat ed as disconnect ed black boxes. I n t his sit uat ion, t he t eam has lit t le visibilit y int o t he workings of t he act ual coding act ivit y ot her t han t he com m only hat ed and not always reliable st at us report s. Fort unat ely, VSTS addresses t hese five broad cat egories. The first is only part ially a developm ent responsibilit y. The rem aining four, however, can't be solved by im proving t he requirem ent s gat hering process and inst ead need t o be solved by focusing on developm ent and downst ream act ivit ies. These five are t he focus of t his chapt er.

Using Test-Driven Development to Ensure Requirements Clarity Test - Driven Developm ent ( TDD) is a pract ice in which you do not writ e a single line of code unt il you have writ t en a t est t hat fails in t he absence of t hat code. Next , you writ e j ust enough code t o pass t he t est , t hen writ e a new t est t hat fails, and keep repeat ing t he t ight loop. Advocat es of TDD docum ent t hat t he pract ice forces clear requirem ent s, cat ches m ist akes, enables refact oring, and rem oves st ress. [ 3 ] The st rongest argum ent in favor of TDD is t hat it uses t est s as t echnical product requirem ent s. Because you m ust writ e a t est before writ ing t he code under t est , you are forced t o underst and t he requirem ent s and wring out any am biguit y in order t o define t he t est . This process, in t urn, m akes you t hink in sm all increm ent s and in t erm s of reuse so t hat you do not writ e any unnecessary code. I n ot her words, TDD im poses a clear and at om ic design, which is easier t o grow and m aint ain. To facilit at e TDD, VSTS support s direct t est creat ion and execut ion, wit h code coverage, inside t he I DE ( see Figure 6.2 ) .

Figu r e 6 .2 .

VSTS su ppor t s u n it t e st in g dir e ct ly in t h e I D E. Th is is a vie w of t e st r u n r e su lt s fr om t h e la st r u n w it h t h e sou r ce code u n de r t e st in t h e u ppe r w in dow . Th e da r k sh a din g ( r e d on a color scr e e n ) in dica t e s a n u n cove r e d e x ce pt ion h a n dle r .

[View full size image]

The next argum ent is t hat TDD enables cont inual refact oring t o keep t he code lean ( see Figure 6.3 ) . I f you have t est s t hat cover 100% of t he code and im m ediat ely report failing result s when t here are any side effect s from refact oring, you have t he safet y net t o refact or wit h confidence. I ndeed, t he experience of TDD is t hat you do m uch less debugging of your code because your unit t est s pinpoint errors t hat you would ot herwise isolat e only by laboriously st epping t hrough t he execut ion wit h t he debugger.

Figu r e 6 .3 .

Re fa ct or in g is a lso su ppor t e d dir e ct ly, m a k in g VSTS a pow e r fu l I D E for TD D .

[View full size image]

Addressing Programming Errors with Code Reviews, Automated and Manual The m ost frequent prescript ion for cat ching program m ing errors is t he code review. Code review approaches include inform al walkt hroughs, form al inspect ions, and pair program m ing, which is a cont inuous review as t he code is being writ t en. Success wit h m anual code reviews varies according t o t he experience of t he reviewer and t he degree of safet y creat ed in t he review environm ent . Aut om at ed code analysis, oft en called st at ic analysis, is a t echnique t hat has received less at t ent ion because it depends on sophist icat ed t ools t hat can scan code for subt le errors, such as securit y risks. Microsoft developed code analysis t ools for it s own product t eam s ( FXCop for m anaged code and PreFAST for unm anaged code) t hat are now included as part of VSTS. They cover coding pract ices in t he areas of design, globalizat ion, int eroperabilit y, m aint ainabilit y, m obilit y, nam ing convent ions, perform ance, port abilit y, reliabilit y, and securit y. To encourage consist ent pract ices across t he t eam , VSTS enables you t o set a check- in policy t hat ensures t hat code analysis has been run before every check- in ( see Figure 6.4 ; m ore on check- in policies lat er) .

Figu r e 6 .4 .

Ch e ck - in policie s w a r n you w h e n you h a ve sk ippe d st e ps be for e ch e ck in g in you r sou r ce code . I n t h is e x a m ple , st a t ic code a n a lysis h a sn 't be e n r u n be for e t h e a t t e m pt e d ch e ck - in .

Automated Code Analysis VSTS enables aut om at ed code analysis as a set of opt ions on t he local build ( F5) and present s t he code analysis warnings and errors in t he sam e window as t he rest of t he build out put ( see Figure 6.5 ) .

Figu r e 6 .5 .

Th e w a r n in gs fr om code a n a lysis a ppe a r in t h e I D E in t h e sa m e w a y a s bu ild w a r n in gs. You ca n click on e a ch w a r n in g a n d j u m p t o t h e sou r ce for vie w in g a n d e dit in g.

[View full size image]

Managed and Unmanaged Code Analysis I n VSTS, t here are t wo different code analysis m echanism sone for C/ C+ + t hat works from t he source and one for m anaged code t hat works from t he m anaged assem blies. The st eps t hat you need t o follow vary depending on which you use. See t hese MSDN t opics: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Developers Writ ing Qualit y Code Det ect ing and Correct ing C/ C+ + Code Defect s Det ect ing and Correct ing Managed Code Defect s

Manual Code Reviews To facilit at e m anual code reviews, VSTS let s you shelve your code changes and share t hem privat ely wit h your reviewers prior t o check- in ( see Figure 6.6 ; m ore on shelving lat er.) I n ret urn, reviewers can give you suggest ions and com m ent s on t he code in a shelveset , and only when you're ready do you check it in for t he build.

Figu r e 6 .6 .

I n t h e ve r sion con t r ol da t a ba se , a sh e lve se t is a t e m por a r y se t of ch a n ge s t h a t m a y or m a y n ot be ch e ck e d in la t e r . On e u se of sh e lvin g is t o m a k e n e w sou r ce code a va ila ble for a code r e vie w be for e ch e ck - in .

Working With Shelvesets for Code Reviews and Other Uncommitted Changes To underst and how t o creat e and use shelveset s in VSTS, see t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Foundat ion Team Foundat ion Proj ect Mem bers Working wit h Team Foundat ion Source Cont rol Working wit h Source Cont rol Files and Folders Working wit h Source Cont rol Shelveset s

Providing Immediate Feedback with Unit Tests and Code Coverage Unit t est ing is probably t he single m ost im port ant qualit y pract ice for a developer. As a pract ice, unit t est ing has been advocat ed for at least t hirt y years. [ 4 ] However, in t he last t en years, sim ple t ools, such as NUnit and JUnit , have m ade t he pract ice m uch m ore widespread, oft en as part of TDD as previously described. Unit t est s support program m ing, as shown in Figure 6.2 . Their purpose is t o check t hat your code does what you int end it t o and does not do what you don't want it t o. Except in rare cases, t hey are not t est s from a cust om er viewpoint , t hey do not validat e scenarios, and t hey m ight not have a lot of m eaning t o users ot her t han t he owners of t he code under t est . The concept of a unit t est is st raight forward: I t is a t est of t he unit of funct ionalit y in which you writ e t he program , such as a m et hod in C# or a rout ine in VB. You can have m ult iple unit t est s for each m et hod, such as a posit ive t est , checking t hat t he m et hod behaves as expect ed wit h valid input , and a negat ive t est , validat ing t hat it does not behave as unexpect ed wit h invalid input . As a developer, you should usually writ e your own unit t est s. You know what you int end your code t o do. You know how t o validat e it . Writ ing t he unit t est s ensures t hat you have t hought t his subj ect t hrough, and it provides a necessary safet y net for t he fut ure when you have t o m aint ain t he code. Generally, whet her you pract ice TDD or not , you should not check in code wit hout having writ t en and run unit t est s t hat exercise all delivered funct ionalit y, including any part s of t he soft ware t hat are dependent on any of t his funct ionalit y. Norm ally in VSTS, a check- in policy ensures t hat t he newly delivered code passes unit t est ing prior t o check- in.

Test First or Code First? Everyone I have m et who pract ices TDD finds it a liberat ing pract ice. ( That 's not surprising; ot herwise, t hey wouldn't do it .) At t he sam e t im e, t here are accom plished developers who do qualit y work and achieve sim ilar result s by coding first and t hen writ ing unit t est s. I n eit her case, t wo principles hold:

1 . Unit t est s should be writ t en in t he sam e session as t he code and checked in t o source cont rol at t he sam e t im e in t he sam e changeset as t he code. 2 . Unit t est s should st rive for m axim um code coverage. The result of t hese t wo principles is t hat you do not check in code t hat does not have unit t est s t hat run and pass wit h it , and if som eone else ( or t he build syst em ) get s your code and runs your t est s, t hey should pass. I n t hat way, your unit t est s becom e a safet y net not j ust for yourself but also for t he whole t eam . VSTS m akes bot h approaches easy ( t est first and code first ) while support ing t hese principles. I f you writ e t est s first , you can right - click and refact or t o generat e t he code under t est , as shown in Figure 6.3. I f you writ e t he code first , you can right - click and generat e t est s for t hat code, as shown in Figure 6.7.

Figu r e 6 .7 .

I f you a r e w r it in g code be for e t e st s or w a n t t o e x t e n d t h e t e st s for a pa r t icu la r a r e a of t h e code ( by m e t h od, cla ss, or n a m e spa ce ) , you ca n r igh t - click a n d ge n e r a t e t e st s fr om t h e sou r ce .

[View full size image]

Code Coverage Regardless of whet her you writ e t he t est s or t he code first , when you run t he t est s, VSTS provides code coverage report ing wit h t he t est run result s ( see Figure 6.8 ) . Code coverage choices are st ored in t he Test Run Configurat ion set t ings ( m ore on t he rest of t hese lat er) . You need t o choose which assem blies t o inst rum ent for coverage because not all m ight be relevant t o t he t est ing at hand.

Figu r e 6 .8 .

W h e n you cr e a t e or e dit a t e st r u n con figu r a t ion , you ch oose t h e a sse m blie s for w h ich you w a n t t o colle ct code cove r a ge . On ly se le ct t h ose fr om t h e code u n de r t e st .

[View full size image]

At t he com plet ion of a t est run, you can use t he t oolbar of t he Test Result s Viewer t o show t he coverage in t he source t hat you j ust exercised. This let s you pinpoint any code t hat your t est s failed t o exerciseskipped code is paint ed red ( see Figure 6.9 ) . You can t hen right - click on t his code t o generat e a new t est for it , or you can ext end an exist ing t est t o cover it .

Figu r e 6 .9 .

At t h e com ple t ion of a t e st r u n , you ca n se e t h e sou r ce code u n de r t e st pa in t e d t o sh ow code cove r a ge . Th is le t s you ide n t ify a t a gla n ce block s of code t h a t a r e n ot be in g t e st e d. I n t h is m on och r om e r e n de r in g, t h e y a r e da r k e r ; in color , t h e y a ppe a r r e d. You ca n t h e n r igh t - click in t h e u n cove r e d code t o cr e a t e a n e w t e st t o e x e r cise t h is a r e a .

[View full size image]

Code coverage is an exquisit e t ool for showing you which blocks of code have not been exercised by your t est s. Use code coverage t o ident ify gaps where you need m ore unit t est s. Do not let good code coverage m ake you t oo confident , however. Code coverage t ells you neit her t hat your t est s are good nor t hat t he cust om er- visible scenarios and QoS have been m et . How m uch code coverage is enough? The ideal is obviously 100% , but t his is rarely obt ainable. Frequent ly, t here is som e error- handling or int egrat ion code for which it is im pract ically expensive t o writ e unit t est s. I n t hese cases, you need t o perform careful code reviews and exercise t he code in t he debugger. I n every case, when you check in code t hat does not have m at ching unit t est s passing, you should m ake t he choice consciously and docum ent it in t he check- in not es.

Making Unit Tests Better When you're t hinking about unit t est s, it 's key t hat you st art wit h a good t est list . [ 5 ] Consider sim ult aneously t he four variables: out put , m et hods invoked, code pat h, and error condit ions. Make sure t hat you have input s t hat m axim ize t he diversit y of t hose variables. I nclude negat ive t est s t hat broadly check for error condit ions. You m ay want a buddy or a t est er t o help brainst orm possible error condit ions t hat you haven't t hought of handling yet . [ 6 ]

Varying the Data and Configurations Used By Your Tests Think of using your unit t est s m ore broadly by varying t he dat a and by running t hem wit h m ult iple configurat ions. VSTS m akes t his st raight forward. See t hese MSDN t opics: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Developers Working wit h Unit Test s Creat ing Unit Test s Coding a Dat a- Driven Unit Test Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Configuring Test Execut ion How t o: Specify a Test Run Configurat ion

Using Data Usually when you writ e a unit t est , you st art it wit h a single set of values. One of t he best ways t o ext end a t est is t o drive t he t est wit h broader dat a, which exercises m ore pat hs t hrough t he code, including m ore error condit ions. Think about values t hat force different behavior and t hrow except ions. VSTS let s you do t his easily, as shown in Figure 6.10.

Figu r e 6 .1 0 .

You ca n dr ive you r u n it t e st s w it h va r ia ble se t s of da t a . D a t a se t s ca n be m a in t a in e d in OLED B pr ovide r s a n d spe cifie d a s pr ope r t ie s on t h e t e st .

[View full size image]

Configurations I f your program needs t o run in different configurat ions, you need t o t est against t hose configurat ions, t oo. There are usually at least t wo: one for developm ent and one for product ion. I f you dist ribut e your soft ware com m ercially, t o support diverse dat acent ers or deskt ops you will need m any m ore configurat ions. VSTS let s you m anage t hese as t est run configurat ions, which encapsulat e t he dat a needed t o run t he t est s in t he right environm ent ( see Figure 6.11) .

Figu r e 6 .1 1 .

Te st r u n con figu r a t ion s le t you t e st m u lt iple de ploym e n t con figu r a t ion s a n d t r a ck e a ch t e st r e su lt a ga in st t h e in t e n de d de ploym e n t t a r ge t .

[View full size image]

Component Integration Tests Depending on t he soft ware proj ect , it m ight be appropriat e t o supplem ent unit t est s wit h com ponent int egrat ion t est s. The purpose of t hese t est s is t o prot ect against unforeseen im pact s on ot her part s of a syst em . I n service- orient ed archit ect ures, for exam ple, each service has m any consum ers. These consum ers can provide com ponent int egrat ion t est s t hat act as com pliance t est s, ensuring t hat t he published service cont inues t o m eet t he consum ers' requirem ent s and cont ract s. When available, t hese com ponent int egrat ion t est s should be run before check- in as well.

Build Verification Tests Build verificat ion t est s ( BVTs) are t est s t hat run as part of t he aut om at ed build. The purpose of BVTs is t o look for unforeseen side effect s and errors due t o changes in t he new code. Any aut om at ed t est s t hat do not require m anual set up m ake good BVTs. This should include t he vast m aj orit y of unit t est s and com ponent int egrat ion t est s and t he m aj orit y of scenario t est s ( see Chapt er 7, " Test ing" ) . To achieve t his purpose, BVTs need t o achieve very high code coverage. I f t hey don't , you have no idea how m any unforeseen issues slipped by t he net . To set up BVTs in VSTS, you m ust creat e a t est list t hat ident it ies which t est s t o use ( see Figure 6.12) and t hen refer t o t he list on t he Build Type Creat ion Wizard, as shown in Figure 6.13.

Figu r e 6 .1 2 .

VSTS le t s you or ga n ize you r t e st s in t o t e st list s so t h a t you ca n gr ou p t h e m for e x e cu t ion . Typica lly, you a dd n e w t e st s t o t h e se list s a s t h e y be com e a va ila ble .

[View full size image]

Figu r e 6 .1 3 .

VSTS Te a m Bu ild in clu de s t h e de sign a t ion of t h e t e st list s t h a t you w a n t t o r u n a s t h e bu ild ve r ifica t ion t e st s.

[View full size image]

Specifying Tests for BVTs I n VSTS, BVTs are ordinary t est s t hat have been included on t he appropriat e t est list. You need t o creat e a t est list for your BVTs. See t he MSDN t opics: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Managing Test s Managing Large Num bers of Test s How t o: Creat e a Test List Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Managing Test s Managing Large Num bers of Test s How t o: Configure and Run Build Verificat ion Test s ( BVTs)

Tuning Performance Unit t est ing and code analysis are t echniques t hat you should apply before every check- in t o m ake sure t hat your code does t he right t hing in t he right way. Perform ance profiling is different . When you have perform ance problem s, it is usually a sm all port ion of your code t hat is culpable, so you should focus your t uning effort s t here. Frequent ly, t he problem s appear under load t est s ( discussed in Chapt er 7) ; som et im es, t hough, you can discover t hem t hrough rout ine funct ional t est ing or explorat ory walkt hroughs. To diagnose perform ance errors, you launch a profiling session and select from t he current solut ion t he code proj ect s on which you want t o collect dat a ( see Figure 6.14) .

Figu r e 6 .1 4 .

VSTS pr ovide s a w iza r d t h a t in st r u m e n t s t h e code u n de r t e st for pr ofilin g. W h e n you ch oose t h e sou r ce pr oj e ct t o pr ofile , VSTS a u t om a t ica lly ch oose s t h e in st r u m e n t a t ion m e ch a n ism for you .

Using Tests to Drive Performance Profiling You can use your unit t est s in VSTS t o drive perform ance profiling sessions. See t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Developers Working wit h Unit Test s Creat ing Unit Test s How t o: Creat e a Perform ance Session for a Test

You need t o choose bet ween t wo profiling t echniques. Sam pling enables you t o collect dat a wit hout perceivable overhead indicat ing how oft en a m et hod appears in t he call st ack, as shown in Figure 6.15 . Typically, you st art wit h sam pling. I nst rum ent ed profiling, on t he ot her hand, let s you walk t raced call sequences wit h m uch m ore dat a, at t he cost of som e overhead and expanded binary size. Use inst rum ent ed profiling t o drill int o t he hot spot s t hat sam pling reveals.

Figu r e 6 .1 5 .

Pr ofilin g da t a a ppe a r s in a pyr a m id of in for m a t ion , w it h t h e m ost im por t a n t da t a on t h e su m m a r y pa ge , a s sh ow n h e r e . Th is m igh t be a ll you n e e d. Fr om h e r e you ca n e it h e r dr ill dow n in t o fu r t h e r de t a il or j u m p t o t h e sou r ce of m e t h od sh ow n .

[View full size image]

Aft er you have select ed your t arget and t he t echnique, you can eit her drive t he applicat ion m anually or run a t est t hat exercises it . When you st op t he profiling session, you'll see a report t hat let s you drill down from a high- level sum m ary int o t he det ails of t he calls.

Preventing Version Skew Source code cont rol, also called version cont rol or configurat ion m anagem ent , has been recognized as a necessary pract ice for developm ent t eam s for t went y years or m ore. I t provides a dat abase of record for all source code, an abilit y t o reconst ruct hist orical versions of source, and a place of record for t he build syst em .

Checking In VSTS has built - in version cont rol t hat keeps t he full hist ory of changes t o t he t eam 's source base, all t he relat ed files open in your Visual St udio solut ion, and any ot hers t hat you include. When you st art t o edit a file, it is aut om at ically checked out for you, unless som eone else has it locked. ( By default , only one person can edit a file at a t im e, but you can enable m ult iple checkout from t he Team m enu.) The prim ary explicit way t hat you int eract wit h version cont rol is by checking in files. When you check in, you are in effect saying t hat your files are ready t o be used in t he t eam build ( see Figures 6.16 , 6.17 , and 6.18 ) .

Figu r e 6 .1 6 .

Th e Ch e ck I n dia log sh ow s you file s t h a t h a ve be e n ch a n ge d on t h e fir st pa n e so t h a t you ca n se le ct t h e on e s t o ch e ck in . N ot e t h e fou r t a b icon s on t h e le ft t h a t le t you flip a m on g sou r ce file s, w or k it e m s, ch e ck - in n ot e s, a n d policy w a r n in gs.

[View full size image]

Figu r e 6 .1 7 .

Th e se con d t a b sh ow s t h e w or k it e m s t h a t you w a n t t o a ssocia t e w it h t h e ch e ck - in . I f you r ch e ck - in com ple t e s t h e de live r y of t h e code for a t a sk or ot h e r w or k it e m , you se t t h e Ch e ck - I n Act ion t o Re solve . Th e r e solu t ion h a ppe n s on t h e n e x t su cce ssfu l t e a m bu ild. I f you a r e m a k in g a pa r t ia l de live r y a n d k e e pin g t h e w or k it e m a ct ive , t h e n ch a n ge t h e Ch e ck - in Act ion t o Associa t e .

[View full size image]

Figu r e 6 .1 8 .

On t h e t h ir d pa n e , e n t e r n ot e s for t h is ch e ck - in . Th e fie lds u se d for t h e n ot e s a r e de t e r m in e d by you r se t t in g for t h e t e a m pr oj e ct on t h e Te a m Fou n da t ion se r ve r .

[View full size image]

When you check in, VSTS prom pt s for t hree t hings: t he list of files, t he work it em s t hat you are resolving wit h t his check- in, and t he check- in not es t hat you are using t o describe t he changes. Toget her, t hese t hree it em s form a changeset , which t ies t oget her t he dat a of a check- in. The changeset includes t he newly added, m odified, or delet ed lines of code, t he work it em st at e changes associat ed wit h t hat code, and t he check- in not es.

A m aj or benefit of changeset s in VSTS is t hat t hey allow t he build syst em and t he m et rics warehouse t o t rack fine- grained relat ionships of work it em s changes wit h source code and t est changes. The build syst em can ident ify all t he changeset s t hat have been delivered t o t he build and t hereby ident ify all t he work it em t ransit ions and calculat e t he code churn. Report ssuch as t he Build Report in Figure 6.21 and t he Qualit y I ndicat ors shown in Figures 4.7, 9.12 , 9.15 , 9.18 , and 9.20 t hrough 9.22 rely on t he changeset st ruct ure.

Check-In Policies When you check in, VSTS also verifies t hat you have com plied wit h t he t eam 's check- in policies, as shown previously in Figure 6.4 . Three st andard check- in policies m ake sure t hat you have associat ed work it em s wit h your changes, have run unit t est s, and have perform ed st at ic code analysis. Your t eam m ay choose t o writ e ot her policies and have t hese evaluat ed at check- in, t oo.

Shelving Oft en you want t o back up, st ore, or share your code wit hout subm it t ing it for t he next build. Because changeset s delivered by check- in aut om at ically feed t he build syst em , you need a different m echanism . I n VSTS, you shelve your code in t hese cases, as shown previously in Figure 6.6 . When you shelve your code, it is st ored cent rally, and ot hers can view your changes ( assum ing you give t hem perm ission) , but not hing is subm it t ed for t he next build. When you subsequent ly unshelve your code, t here is no record of t he shelveset and correspondingly no hist ory t o clean up in t he source dat abase. Shelving is very useful for a num ber of cases. I f you need t o leave t he office when your code isn't ready for t he build, you can back it up. I f you need t o share your code wit h som eone else prior t o check- in, for exam ple for a code review or buddy t est , you can shelve your code and let som eone review it from t he shelveset . When you want t o experim ent wit h t wo solut ions t o a problem , you can t ry one, shelve it , t ry t he second, and swit ch bet ween t he shelveset s for local t est ing.

Branching I f you've used ot her source cont rol syst em s, you're probably fam iliar wit h t he concept of branching. Having branches let s you keep parallel versions of files t hat evolve separat ely. Probably t he m ost frequent use of branches is t o t rack m ult iple released versions of a solut ion. When releasing version 1, you can branch t o st art work on version 2. I f you subsequent ly need t o fix bugs or issue a service release ( perhaps for new environm ent s) for version 1, you can do so in it s branch wit hout having t o pull in any version 2 code. Use branches sparingly. Whenever you branch, you m ay be creat ing a fut ure need t o m erge. The bugs you fix for version 1 probably need t o be fixed in version 2 as well. I f you have m ult iple branches, t hen you will have m ult iple m erges t o consider.

What to Version Versioning is not j ust for source code. You should version all files associat ed wit h com piling, configuring, deploying, t est ing, and running your syst em ( see Figure 6.19) . By default , t he t est s and m ost of t he configurat ion files are part of your Visual St udio solut ion and appear as files t o check in when you look at t he Check I n dialog.

Figu r e 6 .1 9 .

" Sou r ce con t r ol" a ct u a lly t r a ck s a ll t h e file s in you r w or k spa ce , in clu din g t e st s, XM L file s, icon s, a n d so on . Ch e ck in you r t e st s w it h you r sou r ce .

[View full size image]

I f you expect t o m aint ain your solut ion for a long t im e, it is wort h creat ing a " t ools" t eam proj ect in which you keep t he versions of t he com piler and support program s ( for exam ple, m sbuild and m st est ) t hat you use t o recreat e and ret est t he solut ion. For exam ple, com m ercial soft ware com panies m ay have cont ract s t o support product s for t en years, and in m any governm ent sit uat ions, cont ract s are longer. I t won't t ake t en years for newer proj ect s t o ret ool, so having a record copy of t he build and t est environm ent is valuable.

Automating the Build Version cont rol is incom plet e wit hout an aut om at ed build syst em . The build syst em needs t o aut om at e not only com pilat ion but also t he t racking and t est ing of t he binaries against t he source versions. The build needs t o provide as m any qualit y checks as possible so t hat any errors can be correct ed before invest m ent of furt her t est ing. This approach ensures t hat t est ing t im e ( especially hum an t im e) is used appropriat ely. I n VSTS, you can configure an aut om at ed build from t he Team Explorer ( see Figure 6.20) . You can have different nam ed build t ypes, such as a daily build, a cont inuous int egrat ion build, and a branch build, each running separat e script s and t est s.

Configuring Team Builds For det ails of t eam build opt ions in VSTS, see t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Foundat ion Team Foundat ion Proj ect Leads Managing Builds wit h Team Foundat ion Build

Figu r e 6 .2 0 .

VSTS give s you a w iza r d t o cr e a t e a " t e a m bu ilds," t h a t is, t h e da ily bu ild a n d ot h e r r e gu la r bu ilds t h a t you a u t om a t e for t h e fu ll solu t ion .

[View full size image]

Daily Build The heart beat of your proj ect is t he daily build. At a m inim um , you should have a build configurat ion for daily builds t hat not only creat es t he binaries t hat you inst all but also runs all t he code analysis and BVTs and generat es t he m et rics t hat t rack t he healt h of your proj ect .

Build Report On com plet ion of t he daily build, you get a build report ( see Figure 6.21) . This shows you t he result s of build com plet ion, code analysis, and BVT runs. I n t he case of a failed build, it gives you a link t o t he work it em t hat was creat ed t o not ify t he appropriat e t eam m em ber t o fix and rest art t he build.

Figu r e 6 .2 1 .

Th e Bu ild Re por t bot h m on it or s t h e r e a l- t im e e x e cu t ion of a bu ild a n d a ggr e ga t e s t h e de t a ils of a bu ild on com ple t ion .

[View full size image]

Use t he Build Report , as shown in Figure 6.21, t o m onit or t he execut ion of a build and view t he det ails of a com plet ed build and t he work it em changes docum ent ing what went int o t he build. Opening t he t est result det ails shows you t he BVT result s and t he code coverage from BVTs. Build warnings include t he st at ic code analysis result s. I n t he case of a build failure, a work it em will have been aut om at ically creat ed t o t rack t he failure, and it is shown here, t oo. From t his report you can publish t he build qualit y st at us. Not e t hat you can navigat e t o t he changeset s t hat are included, t he code analysis warnings, and t he t est result s direct ly from t he build report . The dat a shown on t he build report is fed direct ly t o t he m et rics warehouse t o creat e hist orical dat a for t he proj ect .

Build Verification Tests (BVTs) Every build should go t hrough a consist ent series of BVTs, and on m any proj ect s, t hese are t he prim ary regression t est s perform ed. The obj ect ives of t he BVTs are t o

1 . I solat e any errors int roduced by check- ins or t he build process, including unant icipat ed int egrat ion errors. 2 . Det erm ine whet her t he soft ware is ready for furt her t est ing.

2. BVTs should include all unit t est s and com ponent int egrat ion t est s t hat are run prior t o check- in, plus any ot her t est s t hat are needed t o ensure t hat it is wort h spending t im e t est ing t he soft ware furt her. BVTs are aut om at ed. Typically, a t est er or designat ed developer " scout s" a build for t he t eam t hat is, he or she inst alls t he soft ware and runs a furt her series of t est s beyond t he BVTs, oft en m anually. For exam ple, scenario t est s m ay require using a new GUI t hat is st ill rapidly evolving, and aut om at ion m ay not be cost effect ive yet . For t his reason, t here is a Bu ild Qu a lit y field on t he report t hat can be m anually set . I t s init ial value on build com plet ion is Un e x a m in e d. You can set it aft er t hat t o Re j e ct e d, Un de r I n ve st iga t ion , Re a dy for I n it ia l Te st , La b Te st Pa sse d, I n it ia l Te st Pa sse d, UAT Pa sse d, Re a dy for D e ploym e n t , Re le a se d, or ot her values t hat you have cust om ized.

Continuous Integration Cont inuous int egrat ion refers t o t he pract ice of t riggering a build aft er every check- in. [ 7 ] I t has been proven very successful in eXt rem e Program m ing and ot her agile pract ices in t hat it delivers im m ediat e feedback on int egrat ion errors t o a developer who has j ust checked in. Wit h VSTS, you can set up a build t ype for cont inuous int egrat ion and t rigger t he build from check- in event s ( see Figure 6.22) . When you do t his, use a separat e build t ype from t he daily build so t hat you st ill have daily m et rics from t he daily build.

Figu r e 6 .2 2 .

Cr e a t e a se pa r a t e bu ild t ype t o pe r for m con t in u ou s in t e gr a t ion . Ke e pin g t h e da ily bu ild a s a se pa r a t e bu ild t ype w ill k e e p m e t r ics t r a ck e d t o t h e da ily bu ild.

[View full size image]

Making Work Transparent VSTS applies t he sam e t ransparency t o t he developer act ivit ies t hat it does t o t he work it em backlog and t he rest of t he t eam act ivit ies. I t t reat s all t eam m em bers as part of one int egrat ed workflow. Because all work it em s of all t ypes are st ored in t he com m on dat abase, when you check in you can ( and should) ident ify t he t asks and t he requirem ent s ( for exam ple, scenarios or QoS) t hat t he delivered code and t est s im plem ent . This creat es a link from t hose work it em s t hat t races t heir resolut ion t o t he corresponding changeset s. This t raceabilit y in t urn drives report s such as Rem aining Work and Velocit y, discussed in Chapt er 4. When it is t im e t o est im at e t he next it erat ion, you have a daily record available of t he current and prior it erat ions' hist ory. These m et rics are collect ed for you, and t hey t ake t he guesswork ( and grunt work) out of det erm ining t he act ual baseline t rends. Sim ilarly, t his t raceabilit y drives t he build report so t hat t he whole t eam ( not ably t est ers) can aut om at ically see what work is available in which build wit h what qualit y. There's no m yst ery of " Did feat ure X m ake it in?" or " Did it pass BVTs?" The build report provides a reliable, frict ion- free view t o t rigger t he t est ing cycle based on accept ed builds.

Summary A value- up approach developm ent is all about delivering working code of cust om er- ready qualit y. I n t his chapt er, I described t he m aj or im pedim ent s t o t hat delivery and how VSTS addresses t hem . The first issue is dealing wit h requirem ent s t hat m ay be inadequat e or hard t o underst and. VSTS support s Test - Driven Developm ent as a pract ice t o force clarificat ion of requirem ent s before you begin im plem ent at ion. The t est ing support direct ly inside VSTS m akes it easy t o creat e and run unit t est s and t o prom ot e t hese for reuse wit h every build. Second is t he considerat ion of qualit ies of service and t he check for program m ing errors t hat m ight not be caught in unit t est ing. VSTS support s aut om at ed code reviews wit h it s st at ic code analysis. Workflow for m anual reviews is support ed t hrough shelving, a feat ure of VSTS version cont rol. Third is t he issue of direct feedback from t est s and t he need t o see code execut ion and perform ance as a part of t est ing. VSTS let s you ext end unit t est ing wit h t est dat a and configurat ions and support s direct perform ance profiling from t he t est runs. Fourt h is t he com plexit y of version cont rol and t he t racking of as- built soft ware t o t he source code. VSTS int egrat es version cont rol and build aut om at ion and provides an audit t rail of source and work it em changes going int o every build. Check- in policies work as rem inders t o support hygienic pract ices for t he t eam . The last issue is t he difficult y of keeping t rack of all t he inform at ion sources. VSTS support s t ransparency of t he process wit h it s com m on work it em dat abase and m et rics warehouse and wit h t he int egrat ion of code and t est changes wit h work it em s and t he build syst em . I n t his way, VSTS let s you as a developer focus on t he subst ance of t he work, not t he cerem ony. The next chapt er looks at t he ext ended t est ing process and it s cont ribut ion t o value- up developm ent .

Endnotes 1 . ht t p: / / www.agilem anifest o.org/ 2 . Barry W. Boehm , Soft ware Engineering Econom ics ( Englewood Cliffs, NJ: Prent ice Hall, 1981) . 3 . For exam ple, K. Beck and E. Gam m a, " Test infect ed: Program m ers love writ ing t est s," Java Report, 3( 7) : 5156, 1998.

4 . Glenford J. Myers, The Art of Soft ware Test ing ( New York: John Wiley & Sons, 1979) . 5 . For exam ple, ht t p: / / www.t est ing.com / writ ings/ short - cat alog.pdf. 6 . Brian Marick, " Fault s of Om ission," first published in Soft ware Test ing and Qualit y Engineering Magazine, January 2000, available from ht t p: / / www.t est ing.com / writ ings/ om issions.ht m l.

7 . For exam ple, ht t p: / / www.m art infowler.com / art icles/ cont inuousI nt egrat ion.ht m l.

Chapter 7. Testing

Te st e r Te st M a n a ge r " Le sson 1 : You a r e t h e h e a dligh t s of t h e pr oj e ct . A proj ect is like a road t rip. Som e proj ect s are sim ple and rout ine, like driving t o t he st ore in broad daylight . But m ost proj ect s wort h doing are m ore like driving a t ruck off- road, in t he m ount ains, at night . Those proj ect s need headlight s. As t he t est er, you light t he way. You illum inat e t he road ahead so t hat t he program m ers and m anagers, however t hey bicker over t he m ap, can at least see where t hey are, what t hey're about t o run over, and how close t hey are t o t he cliff. The det ailed m ission of t he t est ing group varies from com pany t o com pany. Behind t hose det ails, t hough, is a com m on fact or. Test ing is done t o find inform at ion. Crit ical decisions about t he proj ect or t he product are m ade on t he basis of t hat inform at ion." [ 1 ] Cem Kaner, Jam es Bach, and Bret Pet t ichord, Lessons Learned in Soft ware Test ing

Figu r e 7 .1 .

Th in k in g of t e st in g a s t h e h e a dligh t s of a pr oj e ct ca n k e e p you r a ct ivit ie s focu se d on pr ovidin g u se fu l in for m a t ion .

For any professional soft ware applicat ion, t here are an infinit e num ber of possible t est s. Accordingly, m ost soft ware developm ent and I T organizat ions devot e a significant am ount of t heir budget t o t est ing, whet her done in- house or out sourced. Yet surprisingly, m ost t est m anagers and proj ect m anagers express m uch frust rat ion at t heir inabilit y t o answer fairly basic quest ions about t he effect iveness of t heir t est ing. Sim ilarly t o earlier chapt ers, I st art wit h a couple pages on t he value- up t enet s of t he discipline, in t his case, t est ing. The rest of t he chapt er drills int o t hose frust rat ing basic quest ions and illust rat es how VSTS helps you answer t hem .

A Value-Up View of Testing Probably no discipline act s as m ore of a light ning rod for confused discussions of soft ware process paradigm s t han t est ing. A great m any books discuss t est ing in isolat ion and assum e a work- down paradigm . [ 2 ] The negat ive effect of t hese works has been so great t hat in early days of t he Agile m ovem ent , it was unclear whet her t here was a role for t est ing at all because developers were responsible for t heir own unit t est ing.[ 3 ] Lot s of confusion rem ains. The first source of confusion is t hat t est ing act ivit ies have t wo purposes: To su ppor t de ve lopm e n t a ct ivit ie s. I n MSF, t hese t est s belong t o t he developm ent role, and I described t hem in t he previous chapt er. I 'm not going t o repeat t hem here. To a sse ss cu st om e r va lu e . I n MSF, t his is t he responsibilit y of t est ers, and it is t he subj ect of t his chapt er. The next source of confusion concerns t he appropriat e out put of t est ing act ivit y. Before cont inuing t o describe value- up t est ing, indulge m e in a sm all exercise t hat brings som e of t his confusion t o light .

What Makes a Good Tester? Consider t his exercise.[ 4 ] Caliban and Ariel are t est ers on a proj ect . They work independent ly. I t is close t o t he release dat e. Their proj ect has five subsyst em s, all equally im port ant , and all bugs t hat t hey m ight find are of equal priorit y. ( Of course, t his oversim plificat ion never happens, but for t he exercise, please suspend disbelief.) Prospero, t heir proj ect m anager, is reviewing t he st at e of t he proj ect . He sees t hat Caliban report ed 100 bugs, and Ariel report ed 74. Then, Prospero decides t o look at t he breakdown of t heir bug report s ( see Table 7.1) .

Ta ble 7 .1 . Ca liba n 's a n d Ar ie l's Fou n d Bu g Cou n t s Ca liba n Ar ie l Subsyst em 100 1

50

Subsyst em 0 2

6

Subsyst em 0 3

6

Subsyst em 0 4

6

Ca liba n Ar ie l Subsyst em 0 5

6

Tot a l

74

100

Here's what he sees: Caliban has found 100 bugs in one com ponent and none in t he ot hers. Ariel has found som e in all com ponent s. Prospero asks t he t wo t est ers t o explain t heir work. Ca liba n : I have t est ed Subsyst em 1 very t horoughly, and we believe we've found alm ost all of t he priorit y 1 bugs. Unfort unat ely, I didn't have t im e t o spend on t he rem aining five subsyst em s. Ar ie l: I 've t est ed all subsyst em s in m oderat e dept h. Subsyst em 1 is st ill very buggy. The ot her subsyst em s are about 1/ 10t h as buggy, t hough I 'm sure bugs rem ain. Now consider whose inform at ion is m ore useful. Many professional t est ers believe t hat t he only role of t est ing is t o find bugs, and t hat t he m ore bugs found, regardless of cont ext , t he bet t er t he t est ing. Where organizat ions ferociously t rack bug rat e curves, t his assum pt ion can be reinforced. Caliban behaves perfect ly for such a sit uat ion. He focused on t he buggiest com ponent and report ed t he highest num ber of bugs. On t he ot her hand, he only report ed 20 percent of t he inform at ion t hat Ariel report ed. Prospero, on t he ot her hand, m ust decide whet her it would be bet t er t o hold on t o t he solut ion for m ore work or t o ship it now. I f m ore work is needed, where should t he invest m ent be m ade? Prospero can m ake m uch m ore inform ed decisions from Ariel's work t han from Caliban's, even t hough she found far fewer bugs. I ndeed, Prospero m ight decide based on Ariel's invest igat ion t o scrap Subsyst em 1 and replace it . I n t hat case, 20 percent of her work will need t o redone, whereas all of Caliban's work will be t hrown away. Ariel also follows som e useful t enet s t hat are capt ured in t hree great value- up heurist ics: [ 5 ]

1 . I m por t a n t pr oble m s fa st . Test ing should be opt im ized t o find im port ant problem s fast rat her t han at t em pt ing t o find all problem s wit h equal urgency. 2 . Focu s on r isk . Test st rat egy should focus m ost effort on areas of pot ent ial t echnical risk while st ill put t ing som e effort int o low- risk areas j ust in case t he risk analysis is wrong. 3 . M a x im ize dive r sit y. Test st rat egy should be diversified in t erm s of t est t echniques and

Basic Questions What is t he inform at ion you need t o assess cust om er value? I t com es down t o som e pret t y basic quest ions: Are we delivering t he cust om er value? Are t he qualit ies of service, such as perform ance and securit y, fit for use? Have we t est ed t he changes? What haven't we t est ed? Does it work in product ion as well as in t he lab? Are we t est ing enough? When should we be running t he t est s? Which t est s should be aut om at ed? How efficient is our t eam , or our out sourced t eam ? These are fundam ent al quest ions t hat m em bers of a well- run proj ect should be able t o answer, and t hey are a good way t o t hink of t he t est ing t hat you do. I n t he rest of t his chapt er, I 'll show how VSTS helps wit h t he answers.

Are We Delivering the Customer Value? As discussed in Chapt er 3, " Requirem ent s," scenarios are a prim ary st at em ent of cust om er value. Good scenario t est ing requires put t ing yourself in t he role of cust om er advocat e. This m eans t est ing well beyond t he st at ed scenario requirem ent s. Cem Kaner has a good checklist t o put you in t he right m indset : [ 6 ] Designing scenario t est s is m uch like doing a requirem ent s analysis, but is not requirem ent s analysis. They rely on sim ilar inform at ion but use it different ly. The requirem ent s analyst t ries t o fost er agreem ent about t he syst em t o be built . The t est er exploit s disagreem ent s t o predict problem s wit h t he syst em . The t est er doesn't have t o reach conclusions or m ake recom m endat ions about how t he solut ion should work. Her t ask is t o expose credible concerns t o t he st akeholders. The t est er doesn't have t o m ake t he solut ion design t radeoffs. She exposes t he consequences of t hose t radeoffs, especially unant icipat ed or m ore serious consequences t han expect ed. The t est er doesn't have t o respect prior agreem ent s. ( Caut ion: t est ers who belabor t he wrong issues lose credibilit y.) The scenario t est er's work need not be exhaust ive, j ust useful. Scenario t est s need t o represent t he prim ary business flows you expect t o cover 80% of t he soft ware's usage. Typically, t hey use broad dat a set s, represent ing a realist ic m ix of business cases t hat you expect your solut ion t o handle. As m uch as t est ing should cover ident ified requirem ent s, such as known scenarios, good t est ing also discovers new ones. A value- up t enet is t hat your knowledge grows t hroughout t he proj ect , oft en em erging from use and discovery. Do not hesit at e t o recognize and capt ure t he new scenarios t hat you find in planning and execut ing t est s. Scenario t est s can be m anual or aut om at ed. I f you have experienced t est ers who can play cust om er advocat es, you m ay want t o docum ent t he m anual t est s only t o t he ext ent of cust om er goals and appropriat e dat a. I n t his case, you do not need t o enum erat e m anual t est s st ep by st ep, but can provide a m ore general t est ideas list . On t he ot her hand, if you delegat e m anual t est execut ion t o people who do not underst and t he cust om er well, you m ay need t o docum ent t he st eps in det ail ( see Figure 7.2 ) .

Figu r e 7 .2 .

I n VSTS, you ca n de scr ibe m a n u a l t e st s in docu m e n t s lik e t h is. Th e t e st r e su lt s a r e ca pt u r e d, t r a ck e d, a n d fe d t o t h e w a r e h ou se in t h e sa m e m a n n e r a s a u t om a t e d t e st s.

[View full size image]

Create a Test Project to Get Started To get st art ed wit h t est ing in VSTS, you will need t o creat e a t est proj ect . See t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Get t ing St art ed wit h Team Syst em Test ing Tools How t o: Creat e a Test Proj ect

Create a Manual Test To creat e a m anual t est in VSTS, see t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Creat ing and Edit ing Test s How t o: Creat e a Manual Test

Automated Scenario Tests The prim ary way t o aut om at e scenario t est s in VSTS is as web t est s ( see Figure 7.3 ) . These t est any applicat ion t hat present s it self t hrough a web browser. Whet her you creat e web t est s by recording or program m ing, you can enhance and m aint ain t hem in Visual Basic or C# .

Figu r e 7 .3 .

W h e n you a dd a w e b t e st in VSTS, you dr ive t h e sce n a r io a s a u se r w ou ldt h r ou gh a n in st r u m e n t e d w e b br ow se r . Th is ca pt u r e s t h e in t e r a ct ion n ot j u st a t t h e GUI le ve l bu t a lso a t t h e H TTP pr ot ocol le ve l a n d pr odu ce s a pa r a m e t e r ize d t e st .

[View full size image]

Alt hough web t est s are creat ed by recording, t hey are not dependent on t he browser UI for running because t hey exercise t he soft ware under t est where it count sat t he server level. During playback, you can see bot h t he browser int eract ion and t he HTTP or HTTPS t raffic ( see Figure 7.4 ) .

Figu r e 7 .4 .

Th e pla yba ck of t h e t e st sh ow s you bot h w h a t w a s r e n de r e d in t h e br ow se r a n d w h a t h a ppe n e d in t h e H TTP st r e a m so t h a t you ca n w a t ch t h e fu ll se r ve r in t e r a ct ion , in clu din g t h e in visible pa r t s of t h e t r a ffic.

[View full size image]

Create a Web Test To creat e a web t est in VSTS, see t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Creat ing and Edit ing Test s How t o: Record a Web Test

Using Test Data Varying t est dat a t o represent a m ix of realist ic input s is an im port ant part of scenario t est ing ( see Figure 7.5 ) . Seeing a t est pass for one dat a set does not m ean it will pass for all, especially when you've paid careful at t ent ion in designing your equivalence classes. Accordingly, you can have your web t est s access ext ernal t est dat a from any OLEDB dat a source, including .csv files, Excel, Access, and SQL Server dat abases.

Figu r e 7 .5 .

I n a lm ost a ll ca se s, you sh ou ld va r y t h e da t a u se d for t e st in g e it h e r t o cove r diffe r e n t com bin a t ion s ba se d on diffe r e n t e qu iva le n ce cla sse s or t o a pply u n iqu e va lu e s for e a ch t r a n sa ct ion pr e se n t in a m u lt iu se r w or k loa d.

[View full size image]

Because web t est s are easy t o creat e, t hey are also easy t o recreat e when a scenario changes.

Data Substitution for Web Tests To set up dat a subst it ut ion for web t est s, see t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Edit ion for Test ers Test ing Tools Tasks Test Types Working wit h Web Test s Dat a Binding in Web Test s

Testing Rich Clients You can record t est s in VSTS from t he UI of web applicat ions but not rich client applicat ions. Several part ners com panies, such as Com puware, provide product s t hat plug int o VSTS for rich- client t est aut om at ion. See ht t p: / / m sdn.m icrosoft .com / vst udio/ t eam syst em / downloads/ part ners/ default .aspx for a current list .

Insulating Your Tests from UI Changes There's an old j oke about Mr. Sm it h kneeling on a desert ed sidewalk at night under a st reet lam p. A policem an com es by and asks if t here's a problem .

Mr. Smith:

I lost m y car keys.

Policeman:

Where did you lose t hem ?

Mr. Smith:

Around t he corner in t he parking lot , near m y car.

Policeman:

Then why are you looking here?

Mr. Smith:

Because it 's dark t here and t he light 's so m uch bet t er over here!

Most soft ware is t est ed t hrough t he user int erface, and t he " light " is indeed bright est t here. However, in m odern dist ribut ed archit ect ures, m ost of t he program 's logic is act ually im plem ent ed on a server, and t he user int erface is a t hin veneer on t op of t hat logic. Accordingly, m ost of t he pot ent ial failures are on t he server side. The UI is also likely t o change considerably based on usabilit y t est ing and ot her cust om er feedback. You don't want t hose changes t o break t he t est s, or worse, you don't want t o use t he t est m aint enance cost as a reason not t o im prove t he UI . I f you lim it your t est s t o user int erface t est s, you alm ost cert ainly m ust rewrit e over t he course of t he proj ect as t he UI changes. Fort unat ely, VSTS web t est s exercise server- side API s ( see Figure 7.6 ) . They can be t urned int o C# or Visual Basic program s t hat enable you t o m aint ain t he t est s independent of t he UI changes. Of course, t he t est dat a t hat t hey use is already separat ely bound and m aint ained.

Figu r e 7 .6 .

An a lt e r n a t e vie w of t h e t e st sh ow n in Figu r e 7 .3 a s ge n e r a t e d code . N ot e t h a t t h e t e st code is r e a lly dr ivin g a se t of se r ve r in t e r a ct ion s, n ot click in g in t h e u se r in t e r fa ce .

[View full size image]

Are the Qualities of Service Fit for Use? As discussed in Chapt er 3, qualit ies of service ( QoS) are capt ured in t he work it em dat abase. Test s specialize according t o t he different qualit ies of service. For exam ple, load t est s, configurat ion t est s, securit y t est s, and usabilit y t est s are all radically different .

Load Testing Load t est ing aim s t o answer t wo prim ary quest ions:

1 . D oe s t h e soft w a r e r e spon d a ppr opr ia t e ly u n de r e x pe ct e d loa d con dit ion s? To answer t his, you com pose perform ance t est s t hat com bine reasonable scenario t est s, dat a, and workloads. 2 . Un de r w h a t st r e ss con dit ion s doe s t h e soft w a r e st op r e spon din g w e ll? For t his, you t ake t he sam e scenarios and dat a and crank up t he workload progressively, wat ching t he corresponding effect on perform ance and syst em indicat ors. All t he aut om at ed t est s m anaged by VSTSweb t est s, unit t est s, and any addit ional t est t ypes you creat ecan be used for load t est ing ( see Figures 7.7 t hrough 7.10 ) . Wit h VSTS, you can m odel t he workload t o represent a realist ic m ix of users, each running different t est s. Finally, VSTS aut om at ically collect s diagnost ic dat a from t he servers under t est t o highlight problem s for you.

Figu r e 7 .7 .

I n VSTS, a loa d t e st is a con t a in e r for a n y a r bit r a r y se t of t e st s w it h w or k loa d se t t in gs. Fir st , you ch oose h ow t o r a m p t h e loa d. Oft e n you w a n t t o obse r ve t h e syst e m w it h gr a du a lly in cr e a sin g u se r loa d so t h a t you ca n spot a n y " h ock e y st ick " e ffe ct in t h e r e spon se t im e a s t h e u se r loa d in cr e a se s.

[View full size image]

Figu r e 7 .8 .

N e x t , you ch oose t h e t e st s ( u n it , w e b, or ot h e r ) a n d t h e pe r ce n t a ge of loa d t o cr e a t e fr om e a ch of t h e a t om ic t e st s.

[View full size image]

Figu r e 7 .9 .

Th e n e x t st e ps a r e t o ch oose t h e br ow se r a n d n e t w or k m ix e s t h a t be st r e fle ct you r e n du se r popu la t ion .

[View full size image]

Figu r e 7 .1 0 .

Loa d t e st s ca n ge n e r a t e h u ge a m ou n t s of da t a fr om t h e se r ve r s u n de r t e st , a n d it 's oft e n h a r d t o k n ow w h a t 's r e le va n t . VSTS sim plifie s t h is by a sk in g you t o ch oose on ly w h ich se r vice s t o w a t ch on w h ich m a ch in e s a n d a u t om a t in g t h e r e st of t h e de cision s.

[View full size image]

Understanding the Output While a load t est runs, and aft er it com plet es, you need t o look at t wo levels of dat a ( see Figure 7.11) . Average response t im e shows you t he end- t o- end response t im e for a page t o finish loading, exact ly as a user would experience it . That 's st raight forward, and you can assess whet her t he range is wit hin accept able lim it s. At t he sam e t im e, while t he t est runs, all t he relevant perform ance dat a is collect ed from t he chosen servers, and t hese count ers give you clues as t o where t he bot t lenecks are in t he running syst em .

Figu r e 7 .1 1 .

Th is gr a ph sh ow s t w o k in ds of da t a t oge t h e r . Ave r a ge Re spon se Tim e is t h e pa ge loa d t im e a s a u se r w ou ld e x pe r ie n ce it . Re qu e st s/ Se c is a m e a su r e m e n t of t h e se r ve r u n de r t e st , in dica t in g a ca u se of t h e slow dow n . N ot e a ddit ion a lly t h e w a r n in g a n d e r r or icon s t h a t fla g pr oble m s a m on g t h e t r e e of cou n t e r s in t h e u ppe r le ft . Som e of t h e se m a y le a d you t o con figu r a t ion pr oble m s t h a t ca n be t u n e d in t h e se r ve r se t t in gs; ot h e r s m a y poin t t o a pplica t ion e r r or s t h a t n e e d t o be fix e d in code .

[View full size image]

Diagnosing When a load t est point s t o a likely applicat ion perform ance problem , t he developer of t he suspect code is usually t he best person t o diagnose t he problem . As a t est er, you can at t ach t he t est result t o a bug direct ly t o forward it t o an appropriat e t eam m at e, and when your t eam m at e opens t he bug, t he sam e graphs will be available for viewing. He or she can t hen use t he Perform ance Wizard t o inst rum ent t he applicat ion and rerun t he t est t hat you ran, as shown in Figure 7.12.

Figu r e 7 .1 2 .

I n a ddit ion t o t h e in for m a t ion offe r e d by t h e pe r fm on cou n t e r s, you ca n r e r u n t h e t e st w it h pr ofilin g ( or a t t a ch t h e t e st r e su lt t o a bu g a n d h a ve a colle a gu e ope n it a n d r e r u n w it h pr ofilin g) . Th is t a k e s you fr om t h e syst e m vie w t o t h e code vie w of t h e a pplica t ion a n d le t s you dr ill in t o t h e spe cific m e t h ods a n d ca ll se qu e n ce s t h a t m a y be in volve d du r in g t h e slow dow n.

[View full size image]

The profiling report can rank t he act ual suspect funct ions and lead you st raight t o t he code t hat m ay need opt im izing. This sequence of load t est ing t o profiling is a very efficient way t o det erm ine how t o t une an applicat ion. You can use it in any it erat ion as soon as enough of t he syst em is available t o drive under load.

Security Testing Securit y t est ing is a specialized t ype of negat ive t est ing. I n securit y t est ing, you are t rying t o prove t hat t he soft ware under t est is vulnerable t o at t ack in ways t hat it should not be. The essence of securit y t est ing is t o use a fault m odel, based on vulnerabilit ies observed on ot her syst em s, and a series of at t acks t o exploit t he vulnerabilit ies. There are m any published at t ack pat t erns t hat can ident ify t he vast m aj orit y of vulnerabilit ies. [ 7 ] Many com panies provide penet rat ion t est ing services, and m any com m unit y t ools are available t o facilit at e securit y t est ing. You can drive t he t ools from VSTS t est suit es, but t hey are not delivered as part of t he VSTS product it self.

Testing for Security Violations The only checks for securit y in VSTS are in t he code analysis described in t he previous chapt er, but part ners such as SPI Dynam ics offer solut ions t hat plug int o t he t est fram ework. See ht t p: / / m sdn.m icrosoft .com / vst udio/ t eam syst em / downloads/ part ners/ default .aspx for a current list .

Usability Testing I n Chapt er 3, I discussed usabilit y t est ing as part of t he requirem ent s process, so I won't repeat it here. I t is equally relevant , of course, t hroughout t he lifecycle.

Have We Tested the Changes? Throughout t he lifecycle, t he applicat ion changes. Regressions are bugs in t he soft ware under t est t hat did not appear in previous versions. Regression t est ing is t he t erm for t est ing a new version of t he soft ware t o find t hose bugs. Alm ost all t ypes of t est s can be used as regression t est s, but in keeping wit h t he t enet of " I m port ant problem s fast ," your regression t est ing st rat egy m ust be very efficient . I deally, you should t est t he m ost recent changes first . Not only does t his m it igat e t he risk of unforeseen side effect s of t he changes, but also if you do find bugs and report t hem , t he m ore recent changes are m ore current in everyone's m em ory. One of t he challenges in m ost t est t eam s is ident ifying what exact ly t he changes are. Fort unat ely, t he daily build report shows you exact ly what changeset s have m ade it int o t he build and what work it em s ( scenarios, QoS, t asks, and bugs) have been resolved, t hereby ident ifying t he funct ionalit y t hat should be t est ed first ( see Figure 7.13) . Moreover, if you have reasonable build verificat ion t est s ( BVTs) , t hen you can check t heir result s and code coverage.

Figu r e 7 .1 3 .

On e of t h e m a n y pu r pose s of t h e da ily bu ild r e por t is t o focu s t e st in g a ct ivit y on t h e n e w ly bu ilt fu n ct ion a lit y. Th is list of w or k it e m s r e solve d in t h e bu ild is lik e a n a u t om a t ic r e le a se n ot e , sh ow in g w h a t n e w fu n ct ion a lit y n e e ds t o be t e st e d.

[View full size image]

What Haven't We Tested? I dent ifying gaps in t est ing involves looking at t he t est result s from m ult iple dim ensions: Re qu ir e m e n t s. Have t he t est s dem onst rat ed t hat t he int ended funct ionalit y was im plem ent ed? Code . What code has been exercised during t est ing? Risk s. What blind spot s m ight we need t o guard against ? What event s do we need t o be prepared for?

Requirements One view of coverage in VSTS is t he result s of t est s against requirem ent s such as scenarios or QoS ( see Figure 7.14) . Test s and t heir result s can be t racked t o t hese specific work it em s.

Figu r e 7 .1 4 .

On e w a y of m e a su r in g t e st cove r a ge is by sce n a r io or ot h e r r e qu ir e m e n t , ba se d on t h e u se fu l disciplin e of cle a r ly ide n t ifyin g t h e sce n a r io t h a t a t e st t e st s. Th e a ggr e ga t ion is, by r e qu ir e m e n t , sh ow in g h ow m a n y r e qu ir e m e n t s a r e in w h ich st a t e of t e st in g.

[View full size image]

Tracking Tests Against Requirements The Requirem ent s Test Hist ory report shown in Figure 7.14 is st andard in t he MSF for CMMI Process I m provem ent but not in MSF for Agile Soft ware Developm ent . The requirem ent associat ed wit h a t est case is capt ured in t he Test Case Propert ies window.

Code A second view of coverage is against source code ( see Figure 7.15) . This is j ust a second, ort hogonal m easure t hat can be t aken from t he sam e t est runs t hat dem onst rat e t he t est ing against requirem ent s shown in Figure 7.14.

Figu r e 7 .1 5 .

Code cove r a ge is t h e se con d m a in m e a n in g of " cove r a ge ." H e r e you se e t h e t e st r e su lt s vie w e r e x pa n de d t o sh ow t h e cove r a ge st a t ist ics a ga in st t h e sou r ce code .

[View full size image]

From t his view, you can also paint t he source code under t est t o see exact ly which lines were and were not exercised, as shown previously in Figure 6.9 .

Tracking Code Coverage from Testing Code coverage set t ings in VSTS are specified in t he t est run configurat ion. Whenever you choose t o collect code coverage as part of t he t est run configurat ion, you see t he coverage st at ist ics at t he end of t he t est run. To prom ot e t he t est result s and t he coverage result s t o t he m et rics warehouse, you need t o " Publish" t he t est result s from t he Test Result s Viewer. BVT result s are published aut om at ically, but for ot her runs you choose whet her t o accum ulat e t he m et rics.

As discussed in Chapt er 4, " Proj ect Managem ent ," use t hese m et rics descript ively, not prescript ively. Coverage m et rics against code and requirem ent s are very useful, but t hey are only t wo dim ensions. Do not let coverage st at ist ics lull you int o a false sense of confidence. " 100% coverage" m erely m eans t hat t his m et ric does not reveal any m ore gaps. I t does not t ell you anyt hing about t he ext ent t o which you have t est ed condit ions for which no code or no requirem ent s have been writ t en. I ndeed, t he fast est way t o increase a code coverage m easure is usually t o rem ove error- handling source code, which is probably t he last behavior you want t o encourage. Assum e t hat t here are gaps in m ore dim ensions t hat t hese coverage m easures don't reveal.

Risks I n Chapt er 2, " Value- Up Processes," I described t he MSF views of const it uency- based and event driven risk m anagem ent . Bot h set s of risk need t o be considered for t est ing. Most risk t est ing is negat ive t est ing, t hat is, " t est s aim ed at showing t hat t he soft ware does not work." [ 8 ] These t est s at t em pt t o do t hings t hat should not be possible t o do, such as spending m oney beyond a credit lim it , revealing som eone else's credit card num ber, or raising an airplane's landing gear before t akeoff. Not e t hat coverage t est ing does not provide any clue about t he am ount of negat ive t est ing t hat has been done, and requirem ent s- based coverage helps only t o t he ext ent t hat QoS requirem ent s capt ure

error prevent ion, which is usually at m uch t oo cursory a level. I n t est ing for risks, you are t ypically looking for errors of om ission, such as an unwrit t en error handler ( no code t o cover) or an im plicit ( and hence unt raceable) requirem ent . To design effect ive negat ive t est s, you need a good idea of what could go wrong. This is som et im es called a "fault m odel." You can derive your fault m odel from any num ber of knowledge sources. Exam ple sources of a fault m odel illust rat ing const it uency- based knowledge are list ed in Table 7.2.

Ta ble 7 .2 . Typica l Sou r ce s a n d Ex a m ple s for a Fa u lt M ode l Sou r ce

Sa m ple Fa u lt t o Te st For

Business Rules

Cust om ers can't spend over t heir credit lim it s.

Technical Archit ect ure

The aut hent icat ion server could be down.

Dom ain Knowledge

This spending pat t ern, alt hough legal, could indicat e a st olen credit card.

User Underst anding

I f t here's no rapid confirm at ion, a user could push t he Subm it but t on m any t im es.

Bug Dat abases

Under t his pat t ern of usage, t he server t im es out .

VSTS let s you capt ure t hese pot ent ial fault s in t he work it em dat abase as risks. Typically, you would st art during early t est planning, and you would review and updat e t he risk list in planning every it erat ion and probably m ore frequent ly. The sam e t raceabilit y t hat t racks t est cases t o scenario work it em s enables you t o t race t est s t o risk work it em s so t hat you can report on t est ing against risks in t he sam e way ( see Figures 7.16 and 7.17 ) .

Figu r e 7 .1 6 .

Risk s a r e ca pt u r e d a s w or k it e m s so t h a t t h e y ca n be m a n a ge d in t h e sa m e ba ck log, t r a ck e d t o t e st ca se s, a n d r e por t e d in t h e sa m e w a y a s ot h e r w or k it e m t ype s.

[View full size image]

Figu r e 7 .1 7 .

Be ca u se r isk s a r e a t ype of w or k it e m , you ca n m e a su r e t e st cove r a ge a ga in st r isk s in a m a n n e r sim ila r t o t h e cove r a ge a ga in st sce n a r ios.

[View full size image]

Does It Work in Production as Well as in the Lab? Have you ever filed a bug and heard t he response, " But it works on m y m achine" ? Or have you ever heard t he dat acent er com plain about t he cost of st aging because t he soft ware as t est ed never works wit hout reconfigurat ion for t he product ion environm ent ? These are sym pt om s of inadequat e configurat ion t est ing. Configurat ion t est ing is crit ical in t hree cases:

1 . Dat acent ers lock down t heir servers wit h very specific set t ings and oft en have a defined num ber of m anaged configurat ions. I t 's essent ial t hat t he set t ings of t he t est environm ent m at ch t he dat acent er environm ent in all applicable ways. 2 . Soft ware vendors and ot her organizat ions t hat cannot precisely cont rol cust om ers' configurat ions need t o be able t o validat e t heir soft ware across t he breadt h of configurat ions t hat will act ually be used. 3 . Soft ware t hat is used int ernat ionally will encount er different operat ing syst em set t ings in different count ries, including different charact er set s, hardware, and input m et hods, which will require specific t est ing. Fort unat ely, VSTS support s explicit configurat ion t est ing in t wo ways: by enabling you t o set up t est labs wit h virt ual m achines and by explicit ly t racking t est run configurat ions and recording all t est result s against t hem .

Test Lab Setup When you have com binat ions t o t est , cycling t est lab m achines am ong t hem can be a huge drain on t im e. Norm ally, you m ust clean each m achine aft er a previous inst allat ion by rest oring it t o t he base operat ing syst em , inst all t he com ponent s, and t hen configure t hem . I f you are rot at ing m any configurat ions, t his preparat ion t im e can dwarf t he act ual t im e available for t est ing. An alt ernat ive is t o set up t he configurat ions on " virt ual m achines" using Microsoft Virt ual Server, included in VSTS ( see Figure 7.18) . Rat her t han inst alling and configuring physical m achines, you inst all and configure a virt ual m achine. When t he virt ual m achine is running, it appears t o t he soft ware and net work t o be ident ical t o a physical m achine, but you can save t he ent ire m achine im age as a disk file and reload it on com m and.

Figu r e 7 .1 8 .

You r solu t ion m a y n e e d t o r u n in diffe r e n t t a r ge t e n vir on m e n t s. Th e se m igh t be diffe r e n t loca lize d ve r sion s of t h e OS, diffe r e n t ve r sion s of su ppor t in g com pon e n t s, su ch a s da t a ba se s a n d w e b se r ve r s, or diffe r e n t con figu r a t ion s of you r solu t ion . Vir t u a l m a ch in e s a r e a low - ove r h e a d w a y of ca pt u r in g t h e e n vir on m e n t s in soft w a r e so t h a t you ca n r u n

t e st s in a se lf- con t a in e d im a ge for t h e spe cifie d con figu r a t ion .

[View full size image]

Set t ing up a library of virt ual m achines m eans t hat you will go t hrough t he set up and configurat ion t im e once, not wit h every t est cycle.

Reporting The ot her m aj or issue wit h t est ing configurat ions is t racking and report ing what has been t est ed so t hat you can ident ify gaps in configurat ion coverage and priorit ize your next t est ing appropriat ely. Fort unat ely, VSTS t racks t he configurat ion used on every t est run ( see Figure 7.19) . Report s m ake it easy t o t rack t he configurat ions t hat have been used and t hose t hat lack good t est coverage.

Figu r e 7 .1 9 .

Ru n con figu r a t ion s ca n ca pt u r e t h e r e pr e se n t a t ive t a r ge t e n vir on m e n t s of t h e syst e m s u n de r t e st . Th e m e t r ics w a r e h ou se a ccu m u la t e s t e st r e su lt s by con figu r a t ion so t h a t you ca n bu ild a pict u r e ove r t im e of t h e t e st cove r a ge a ga in st con figu r a t ion s.

[View full size image]

I t 's usually a good idea t o vary t he configurat ions wit h every round of t est ing so t hat you cycle t hrough t he different configurat ions as a m at t er of course. Because t he t est result s are always t racked against t he t est run configurat ion, you will also have t he inform at ion for reproducing any result s, and you will im prove your coverage of configurat ions t his way.

Are We Testing Enough? Defining "Good Enough" I n Chapt er 3, I present ed Kano Analysis as a t echnique for t hinking holist ically in t erm s of sat isfiers and dissat isfiers for a proj ect , and in Chapt er 4, I discussed planning an it erat ion. The it erat ion's obj ect ives should det erm ine t he t est obj ect ives. Alt hough it m ay be hard t o det erm ine what const it ut es " good enough" t est ing for a proj ect as a whole, it 's not t hat hard for t he it erat ion. The whole t eam should know what " good enough" m eans for t he current it erat ion, and t hat should be capt ured in t he scenarios, QoS t hat are planned t o be im plem ent ed, and risks t o be addressed in t he it erat ion. Test ing should verify t he accom plishm ent of t hat t arget t hrough t he course of t he it erat ion. [ 9 ] A value- up pract ice for planning " good enough" is t o keep t he bug backlog as close t o zero as possible. Every t im e you defer resolving or closing a bug, you im pose addit ional fut ure liabilit y on t he proj ect for t hree reasons: The bug it self will have t o be handled m ult iple t im es, som eone ( usually a developer) will have a longer lag before ret urning t o t he code for analysis and fixing, and you'll creat e a " Broken Windows" effect . The Broken Windows t heory holds t hat in neighborhoods where sm all det ails, such as broken windows, go unaddressed, ot her act s of crim e are m ore likely t o be ignored. Cem Kaner, a soft ware t est ing professor and form er public prosecut or, describes t his well: [ 10] The challenge wit h graffit i and broken windows is t hat t hey ident ify a com m unit y st andard. I f t he com m unit y can't even keep it self m oderat ely clean, t hen: ( 1) Problem s like t hese are not wort h report ing, and so cit izens will st op report ing t hem . ( We also see t he converse of t his, as a wellest ablished phenom enon. I n com m unit ies t hat st art act ually prosecut ing dom est ic violence or rape, t he report ed incidence of t hese crim es rises subst ant iallypresum ably, t he visible enforcem ent causes a higher probabilit y of a report of a crim e, rat her t han m ore crim e) . I n soft ware, m any bugs are kept off t he list s as not wort h report ing. ( 2) People will be less likely t o clean t hese bugs up on t heir own because t heir sm all effort won't m ake m uch of a difference. ( 3) Som e people will feel it is accept able ( socially t olerat ed in t his com m unit y) t o com m it m ore graffit i or t o break m ore windows. ( 4) Many people will feel t hat if t hese are t olerat ed, t here probably isn't m uch bandwidt h available t o enforce laws against m ore serious st reet crim es. Sim ilarly, in proj ect s wit h large bug backlogs, overall at t ent ion t o qualit y issues m ay decline. This is one of m any reasons t o keep t he bug backlog as close t o zero as possible.

Set Iteration Test Objectives by Assigning Work Items to the Iteration I n VSTS, all work it em s, including scenarios, QoS, bugs, and risks, can be assigned t o an it erat ion. This assignm ent creat es a t est t arget list for t hat it erat ion, or in ot her words, a visible bar defining good enough t est ing for t hat it erat ion. You can, of course, add m ore t o t hat list or reschedule it em s t o fut ure it erat ions, but t here is always a visible, agreed definit ion of t he it erat ion t est goals, and changes t o it are t racked in an audit able m anner.

Exploratory Testing Most t est ing I 've discussed so far is eit her aut om at ed or highly script ed m anual t est ing. These are good for finding t he t hings t hat you know t o look for but weak for finding bugs or issues where you don't know t o look. Explorat ory t est ing, also called ad hoc t est ing, is an im port ant m indset t o bring t o all of t he t est ing t hat you do. I n explorat ory t est ing, t he t est er assum es t he persona of t he user and exercises t he soft ware as t hat persona would. Kaner, Bach, and Pet t ichord describe explorat ory t est ing t his way: By explorat ion, we m ean purposeful wandering; navigat ing t hrough a space wit h a general m ission, but wit hout a prescript ed rout e. Explorat ion involves cont inuous learning and experim ent ing. There's a lot of backt racking, repet it ion, and ot her processes t hat look like wast e t o t he unt rained eye. [ 11] Explorat ory t est ing can be a very im port ant source of discovery, not j ust of bugs but also of unforeseen ( or not yet described) scenarios and QoS requirem ent s. Capt ure t hese in t he backlog of t he work it em dat abase so t hat you can use t hem in planning t he current and fut ure it erat ions. As a m anager, plan for a cert ain level of explorat ory t est ing in every it erat ion. Define chart ers for t hese t est ing sessions according t o t he goals of t he it erat ion. Tune t he chart ers and t he resource level according t o t he value you get from t hese sessions. I n short , plan capacit y for explorat ory t est ing.

Testing as Discovery Em brace t est ing t hat discovers new scenarios, QoS requirem ent s, and risks in addit ion of course t o finding bugs. Capt ure t he new scenarios, QoS, and risks as work it em s in t he product backlog. This is vit al inform at ion. I t m akes t he quant it at ive coverage m easurem ent a lit t le harder in t hat you're increasing t he denom inat or, but t hat 's a sm all price t o pay for helping t he proj ect deliver m ore cust om er value. A part icular t ype of scenario t est is t he " soap opera." Hans Buwalda describes t he t echnique in his art icle " Soap Opera Test ing" as follows: Soap operas are dram at ic dayt im e t elevision shows t hat were originally sponsored by soap vendors. They depict life in a way t hat viewers can relat e t o, but t he sit uat ions port rayed are t ypically condensed and exaggerat ed. I n one episode, m ore t hings happen t o t he charact ers t han m ost of us will experience in a lifet im e. Opinions m ay differ about whet her soap operas are fun t o wat ch, but it m ust be great fun t o writ e t hem . Soap opera t est ing is sim ilar t o a soap opera in t hat t est s are based on real life, exaggerat ed, and condensed. [ 12] Soap operas are harsh, com plex t est st hey t est m any feat ures using int ricat e and perhaps unforeseen

sequences. The essence of soap operas is t hat t hey present cases t hat are relevant t o t he dom ain and im port ant t o t he st akeholders but t hat cannot be t est ed in isolat ion. They are a good t est of robust ness in it erat ions where t he soft ware is m at ure enough t o handle long t est sequences.

False Confidence When you have aut om at ed or highly script ed t est ing, and you do not balance it wit h explorat ion, you run t he risk of what Boris Beizer coined as t he "Pest icide Paradox " : [ 13] Every m et hod you use t o prevent or find bugs leaves a residue of subt ler bugs against which t hose m et hods are ineffect ual. I n ot her words, you can m ake your soft ware im m une t o t he t est s t hat you already have. This pat t ern is especially a concern when t he only t est ing being done is regression t est ing, and t he t est pool is very st able. There are t hree ways t o m it igat e t he Pest icide Paradox:

1 . Make sure t hat t he t est s t he soft ware faces cont inually include fresh ones, including good negat ive t est s. 2 . Look at gaps in t est coverage against scenarios, QoS, risks, and code. Priorit ize t he gaps and t hink about t est s t hat can close t hem . 3 . Use progressively harsher t est s, not ably soap operas and explorat ory t est ing, t o confirm , from a knowledgeable dom ain expert 's perspect ive, t hat t he soft ware doesn't have undiscovered vulnerabilit ies. Explorat ory t est ing, soap operas, and risk ident ificat ion all m it igat e against a false sense of confidence.

When Should We Test? I n principle, it seem s obvious t hat at any t im e in developm ent proj ect , you should t est t he funct ionalit y t hat needs t o be t est ed at t hat t im e. I n pract ice, however, t eam s hist orically experience t rem endous churn because t est ers do not know when t o t est . I n t hese cases, t est ers claim t hey're blocked, developers feel t he t est ers are wast ing t heir t im e, and collaborat ion quickly disint egrat es. VSTS alleviat es t his issue in t hree ways:

1 . The it erat ion st ruct ure m akes clear what funct ionalit y is planned when, and course correct ions are frequent enough t o enable plans t o be fine- t uned ( refer t o Chapt er 4) . 2 . The com m on product backlog m akes clear what scenarios and QoS have act ually been resolved and are t herefore ready for t est at a part icular t im e. 3 . The build report m akes clear for every daily build what t asks and scenarios have act ually been delivered and int egrat ed in t he build and whet her t he build it self has passed BVTs and is ready for furt her t est ing ( see Figure 7.20) .

Figu r e 7 .2 0 .

Th e da ily bu ild pr odu ce s a n a u t om a t e d r e por t t h a t in clu de s t h e w or k it e m s t h a t h a ve be e n r e solve d in t h e bu ild. I n ot h e r w or ds, t h e r e por t a ccu m u la t e s a ll t h e it e m s t h a t w e r e m a r k e d r e solve d w h e n code w a s ch e ck e d in . Th is le t s you se e im m e dia t e ly w h e t h e r in t e n de d fu n ct ion a lit y ca n be e x pe ct e d t o w or k in t h e bu ild a n d cor r e spon din gly w h ich t e st s ca n n ow be r u n .

[View full size image]

I n addit ion t o using t he t ransparency provided by VSTS, a t est m anager should carefully t hink about which t est s are appropriat e for which cycles ( see Figure 7.21) .

Figu r e 7 .2 1 .

M SF u se s t h e con ce pt of in t e r lock in g cycle s t o gr ou p a ct ivit ie s t o t h e a ppr opr ia t e fr e qu e n cy a n d t o a llow t h e r igh t t h in gs t o h a ppe n fir st . Te st in g a ct ivit ie s va r y a ccor din g t o t h e se cycle s.

Check-In Cycle The at om ic program m ing cycle is t he set of act ivit ies leading t o checking in source code. I n VSTS, source should be delivered wit h unit t est s and resolut ion of t he corresponding work it em s. As a developer, before checking in, you should run t he unit t est s for any funct ionalit y t hat you deliver and any t hat is dependent on your delivery. Depending on t he soft ware proj ect , it m ight be appropriat e t o supplem ent unit t est s wit h com ponent int egrat ion t est s. ( You should also run st at ic code analysis.) Check- in policy can enforce t his pract ice.

Daily Build Cycle BVTs run wit h t he night ly build aut om at ion, as discussed in t he previous chapt er. For BVTs, it is best t o have lot s of t est s, each for relat ively fine- grained scenarios, rat her t han a few, m ore com plex t est s. Typically, t hese are a superset of t he pre- check- in unit t est s t hat t he individual t eam m em bers ran before delivering new source code, plus resilient scenario t est s t hat ot her t eam m em bers have delivered. Having sm all, self- cont ained t est s m akes it possible t o quickly isolat e t he failure cases and drill int o t hem . I t also m akes it possible t o updat e t he t est s when t he corresponding scenarios change. I t is im port ant t hat t he suit e of BVTs grow from it erat ion t o it erat ion. As t he soft ware funct ionalit y m at ures, com plet ed work from every prior it erat ion should be verified cont inually wit h BVTs.

Accepted Build Cycle Aft er a soft ware build has passed t he BVTs, it is ready for furt her t est ing. Now you can t est t he new scenarios and QoS t hat have been im plem ent ed in t he current it erat ion. This list should be obvious from bot h t he work it em s of t he product backlog and t he build report s. As scenarios and QoS first becom e available, you should run support ive t est s. As t hese easier t est s pass, you should run ever m ore challenging t est s. Which builds should be prom ot ed for furt her t est ing? The answer depends on t he goals of t he it erat ion and t he lengt h of t im e it t akes t o t est a build t horoughly. When t here is significant new funct ionalit y from one night ly build t o t he next , t he new work should be t est ed and feedback should be given t o t he developers as soon as pract ical. This priorit izat ion is a " Last I n, First Out " st ack, and it ensures t hat t he developer receives bugs on recent work while it is st ill fresh in m ind. On t he ot her hand, som e t est ing cycles for broad business funct ionalit y or m any configurat ions t ake days or ( unfort unat ely) weeks t o com plet e. Set up requirem ent s, such as configuring and loading a dat abase, can m ake changing builds disrupt ive during a t est cycle. I f ( and only if) t hat 's t rue, and t he part icular t est ing is not dependent on t he new work, it m ay be wort h delaying t he accept ance of t he new build int o t est . Even bet t er is t o find a way t o short en t he am ount of t im e needed t o run t he full t est pass.

Iteration Cycle The scenarios and QoS scheduled for an it erat ion should be considered exit crit eria t hat need t o be verified by t est ing. The work it em s t hat t rack t hese scenarios and QoS should not be closed unt il t he corresponding funct ionalit y has passed t est ing in t he soft ware as a whole. I f you reach t he end of an it erat ion and discover t hat you have not com plet ed t he planned scenarios and QoS, t his is essent ial inform at ion for planning t he next it erat ions. ( See t he discussion of proj ect velocit y in Chapt er 4.) However, you should not let an exit crit erion be declared sat isfied unt il t est ing has confirm ed t hat t he expect ed code works. This m akes t he it erat ion cycle t he m ost visible t est ing cycle. I n a well run proj ect , where t est ing happens concurrent ly wit h developm ent , t here will not be a large t est ing bot t leneck at t he end of t he it erat ion. On t he ot her hand, if t est ing is not kept in lockst ep wit h developm ent , t here m ay be a big bulge. This growing bulge will be visible during t he it erat ion in t he Rem aining Work chart . I t is undesirable, creat es a significant risk, and will disrupt t he rhyt hm of t he proj ect subst ant ially. As a t est m anager or proj ect m anager, you should keep your eye on t his chart daily t o avoid becom ing t rapped at it erat ion end ( see Figure 7.22) .

Figu r e 7 .2 2 .

Th e bu lge in t h is Re m a in in g W or k ch a r t in dica t e s t h a t a lt h ou gh t h e codin g of sce n a r ios is pr ogr e ssin g w e ll, t h e r e is a bot t le n e ck in t e st in g. Th is m a y be du e t o in a de qu a t e t e st in g r e sou r ce s or in a de qu a t e qu a lit y of sce n a r ios de live r e d in t h e bu ilds. ( You w ou ld ch e ck t h e Re a ct iva t ion s r e por t n e x t if you su spe ct e d poor qu a lit y.)

Project Cycle At t he end of an it erat ion, t he soft ware should be passing t he t est s planned for t hat it erat ion, and in subsequent it erat ions, it should cont inue t o pass t hem . When t he soft ware keeps passing t he t est s put t o it , you develop great m om ent um . The m ore aut om at ed t he t est s, t he m ore frequent ly t hey are run, and t he m ore frequent ly t his confidence is reinforced. Of course, t he m ore m at ure t he proj ect , t he harsher t he t est s should be. Cont inue t o use soap operas, negat ive t est s, and explorat ion t o check for blind spot s. Use configurat ion t est ing t o m ake sure t hat product ion const raint s have been addressed. Check t rends on coverage against code, requirem ent s, and risk.

Which Tests Should Be Automated? I n t he last t en years, a lot has been writ t en about t he pit falls and benefit s of t est aut om at ion. [ 14] I will sim plify t he argum ent s here. Aut om at ion is useful when it achieves high coverage and when t he t est s will be used m any t im es across changes in t he soft ware under t est ( SUT) . However, aut om at ion is expensive and hard t o m aint ain, especially if based on t he SUT's user int erface. Moreover, aut om at ion oft en leads t o a false sense of securit y, especially when it s result s are not balanced against views of coverage and when it s t est cases are not balanced wit h harsh explorat ory and negat ive t est ing. These considerat ions lead t o som e guidelines:

1 . Aut om at e t est s t hat support program m ing, such as unit t est s, com ponent / service int egrat ion t est s, and BVTs, and m ake sure t hat t hey achieve very high code coverage, as discussed in Chapt er 6, " Developm ent ." 2 . Aut om at e configurat ion t est s whenever you can. I f you expect your soft ware t o be long- lived, t hen

3 . Aut om at e scenario t est s when possible, but expect t hat t hey will need m aint enance. Where possible, do not rely on t he UI for t est ing and inst ead code m ore durable t est s against t he appropriat e API s. 4 . Aut om at e load t est s, but again, expect t hat t hey will need m aint enance. And. . .

5 . Guard against a false sense of confidence wit h explorat ory t est ing, negat ive t est s, soap operas, and t est s against risks. Most of t hese will be m anual because you will be m ore int erest ed in m axim izing t he diversit y of your t est ing t han repeat ing t he sam e t est s every t im e.

How Efficient Is Our Team, or Our Outsourced Team? Toget her, t he report s from t he m et rics warehouse answer t he quest ion of t eam effect iveness and efficiency. Specific pat t erns of problem s are covered in Chapt er 9, "Troubleshoot ing t he Proj ect ," but here is a general guide t o using t he report s t o answer t his quest ion. The Rem aining Work report , as illust rat ed previously in Figure 4.4 , shows you t he cum ulat ive flow of int ended work t hrough t est ing. The m iddle band, from Resolved t o Closed, is t he work in process of t est ing. I f you see a relat ively consist ent widt h, as in Figure 4.4 , t hen you know t hat you have a sm oot h flow. The sm oot h flow gives you a clear indicat ion of t he m at ch of your resources t o capacit y. ( This cont rast s wit h t he bot t leneck shown in Figure 7.22, indicat ing a resource m ism at ch or problem wit h qualit y at t he t im e of resolut ion.) Velocit y ( Figure 4.5 ) in t he Resolved series drills int o t he det ails of t he capacit y and it s variance. Requirem ent s Test Hist ory ( Figure 7.14) shows t he progression of t he t est ing against scenarios and QoS, while Qualit y I ndicat ors ( Figure 4.7 ) put s t est result s, bugs, and code coverage t oget her. By put t ing t hese series t oget her, you can m ake sure t hat t he independent dim ensions are progressing as expect ed. Of course, in j udging t est ing, you need t o look at upst ream qualit y t hat is being passed int o t est ing. The Build Hist ory report ( Figure 9.11) shows t he st at us of daily builds, which should be com plet ing successfully and passing t heir BVTs wit h rare except ions. Qualit y I ndicat ors provides t wo series t hat should be wat ched t oget hercode churn, t he indicat or of how m uch new code needs t o be t est ed, and code coverage, t he m easure of how m uch of it act ually is being t est ed. React ivat ions ( Figure 4.9 ) are bugs t hat have been reopened aft er being resolved, t hat is, report ed fixed. I f t hese are high or rising, it 's a clear indicat or of upst ream problem s.

Summary This chapt er has covered t est ing in a value- up paradigm . I st art ed wit h an exercise t o illust rat e t he im port ance of t est ing as a key source of inform at ion and it s t enet s of creat ing inform at ion fast , addressing risk, and m axim izing diversit y. Next , I went t hrough basic quest ions t hat t est ing should answer and illust rat ed how t o use VSTS t o help answer t hem : Are we delivering t he cust om er value? Are t he qualit ies of service, such as perform ance and securit y, fit for use? Have we t est ed t he changes? What haven't we t est ed? Does it work in product ion as well as in t he lab? Are we t est ing enough? When should we be running t he t est s? Which t est s should be aut om at ed? How efficient is our t eam , or our out sourced t eam ? These are t he sim ple value- up quest ions t hat t est ing needs t o address.

Endnotes 1.

Cem Kaner, Jam es Bach, and Bret Pet t ichord, Lessons Learned in Soft ware Test ing ( New York: Wiley, 2002) , 1.

2.

The root of m any of t hese is I EEE STD 8291983.

3.

For exam ple, Beck 2000, op. cit .

4.

Adapt ed from Brian Marick, " Classic Test ing Mist akes," 1997, available at ht t p: / / www.t est ing.com / writ ings/ classic/ m ist akes.pdf .

5.

Originally called cont ext - driven t est ing by Kaner, Bach, and Pet t ichord ( 257) , I 've included it as part of t he Value- Up Paradigm .

6.

[ Kaner 2003] " Cem Kaner on Scenario Test ing," STQE Magazine, Sept em ber/ Oct ober 2003, 22, available at www.st ickym inds.com . For a det ailed view of Kaner's course on t his subj ect , see ht t p: / / www.t est ingeducat ion.org/ BBST/ ScenarioTest ing.ht m l

7.

Jam es A. Whit t aker and Herbert H. Thom pson, How t o Break Soft ware Securit y: Effect ive Techniques for Securit y Test ing ( Bost on: Addison- Wesley, 2004) . Whit t aker and Thom pson have ident ified 19 at t ack pat t erns t hat are st andard approaches t o hacking syst em s.

8.

Boris Beizer, Soft ware Test ing Techniques ( Bost on: I nt ernat ional Thom son Com put er Press, 1990) , 535.

9.

Jam es Bach has writ t en ext ensively on t he heurist ics for defining when our soft ware is good enough for it s purpose. See ht t p: / / www.sat isfice.com / art icles.sht m l for a collect ion of his essays.

1 0 . Cem Kaner, privat e em ail. Malcolm Gladwell, The Tipping Point ( Lit t le Brown & Co., 2000) , 141, has popularized t he discussion, based on Mayor Giulini's use in New York Cit y. The st at ist ical evidence support ing t he t heory is disput able; see St even D. Levit t and St ephen J. Dubner, Freakonom ics: A Rogue Econom ist Explores t he Hidden Side of Everyt hing ( New York: HarperCollins, 2005) . Nonet heless, t he psychological argum ent t hat com m unit ies, including soft ware t eam s, can becom e habit uat ed t o condit ions of disrepair is widely consist ent wit h experience.

1 1 . Kaner, Bach, and Pet t ichord 2002, 18. 1 2 . Hans Buwalda, " Soap Opera Test ing," Bet t er Soft ware, February 2004, 3037, available at www.st ickym inds.com .

1 3 . Boris Beizer, Soft ware Test ing Techniques ( Bost on: I nt ernat ional Thom son Com put er Press, 1990) , 9.

1 4 . For a classic discussion of t he risks of bad aut om at ion, see Jam es Bach, " Test Aut om at ion Snake Oil," originally published in Windows Tech Journal ( Novem ber 1996) , available at ht t p: / / www.sat isfice.com / art icles/ t est _aut om at ion_snake_oil.pdf.

Chapter 8. Reporting Bugs

Everyone " A group of program m ers were present ing a report t o t he Em peror. " What was t he great est achievem ent of t he year?" t he Em peror asked. The program m ers spoke am ong t hem selves and t hen replied, " We fixed 50% m ore bugs t his year t han we fixed last year." The Em peror looked on t hem in confusion. I t was clear he did not know what a " bug" was. Aft er conferring in low undert ones wit h his chief m inist er, he t urned t o t he program m ers, his face red wit h anger. " You are guilt y of poor qualit y cont rol. Next year t here will be no bugs! " he dem anded. And sure enough, when t he program m ers present ed t heir report t o t he Em peror t he next year, t here was no m ent ion of bugs." [ 1 ] Geoffrey Jam es, The Zen of Program m ing

Figu r e 8 .1 .

Life cycle of a ( r e a l) bu g. Re a l bu gs h a ve a w e ll- de fin e d life cycle , a ccor din g t o t h e ir spe cie s. Th e ir m e t a m or ph osis is a good m e t a ph or for soft w a r e bu gs, w h ich oft e n st a r t w it h a sim ple obse r va t ion of a pr oba ble sou r ce of cu st om e r dissa t isfa ct ion ( t h e e gg) a n d pr oce e d t o a w e ll- de fin e d r e por t of t h e obse r ve d sym pt om s, st e ps t o r e pr odu ce , a n d a t e ch n ica l in ve st iga t ion ( t h e la r va ) , t o a fix for t h e pr oble m ( t h e ch r ysa lis) , a n d fin a lly t o a w or k in g bu ild w it h t h e fix ve r ifie d ( t h e a du lt ) . Lik e r e a l bu gs, n ot a ll soft w a r e bu gs su r vive t o a du lt h ood. Of cou r se , u n lik e r e a l bu gs, soft w a r e bu gs ca n r e t r e a t ba ck t o t h e ir la r va l st a t e w h e n t h e fix e s a r e u n su cce ssfu l. Th ose a r e t h e r e a ct iva t ion s.

I n a value- up paradigm , qualit y is everyone's business all t he t im e. Prevent ion of errors is one of t he m ost im port ant aspect s t o every act ivit y. So why do I have a separat e chapt er on bugsisn't t hat ant iquat ed t hinking? I n m ost organizat ional cult ures I have seen, bugs t ake on such an im port ant life t hat t hey should be addressed head on. And despit e t he best - known lifecycle and developm ent pract ices, bugs st ill occur, alt hough t heir nat ure changes. Hence t his chapt er. MSF for Agile Soft ware Developm ent defines t his principle: Qu a lit y is e ve r yon e 's j ob e ve r y da y. Qualit y requires bot h bug prevent ion and solut ion verificat ion. Ut ilize pract ices such as code analysis and peer reviews t o prevent bugs as well as m axim ize t he t est ing t o find bugs. All roles are responsible for prevent ion and verificat ion of bugs.[ 2 ] Consist ent wit h MSF, I call bugs " bugs" and not " defect s." " Defect " oft en has connot at ions of blam e and nonconform ance t o spec, bot h of which are red herrings. For value- up purposes, anyt hing t hat dim inishes perceived value is a bug and should be considered for t riage and pot ent ial change. For t he sam e reason, I do not dist inguish here bet ween bugs and change request s, alt hough in cert ain processes, t hey would be t reat ed different ly. Take t hem all t o t riage t he sam e way.

Implicit and Explicit Change Requests Consist ent wit h t he Agile value of em bracing change, MSF for Agile Soft ware Developm ent does not have a change request work it em t ype. Changes m ay be capt ured as bugs or new scenarios, depending on t he scope. On t he ot her hand, MSF for CMMI Process I m provem ent has an explicit Change Request work it em t ype t o support t he Configurat ion Managem ent Process Area of t he CMMI .

A Cautionary Tale Let 's revisit t he pract ices of t wo t est ers on our proj ect , Caliban and Ariel. At t he end of t he proj ect , Caliban had report ed 100 bugs; Ariel had report ed 74. ( Assum e t hat t hey were all equal priorit y and severit y.) Prospero, who is t heir m anager, m ust pick one of t he t wo t est ers for a key new proj ect . Whom should Prospero pick? Based on t his inform at ion alone, he'd probably pick Caliban. Before choosing, however, Prospero t ook a look at t he st at e of t heir bugs at t he end of t he proj ect ( see Table 8.1) .

Ta ble 8 .1 . Br e a k dow n of Re a son s Give n for Closu r e of t h e Bu gs Ca liba n

Ar ie l

Fixed and Validat ed

10

40

Works as Planned

15

2

No Plan t o 20 Fix

6

Duplicat e

15

6

Deferred

40

20

Prospero not ed t hat 80% of t he bug fixes cust om ers received were t he result of Ariel's work, and only 20% were t he result of Caliban's effort s. As a result , he picked Ariel.

A (Software) Bug's Life Not unlike biological bugs, soft ware bugs go t hrough a very precise set of st at es ( see Figure 8.2 ) . I n VSTS, t he default st at es and t ransit ions t hat are allowed depend on t he process t em plat e t hat you choose for t he proj ect . The select ion and t im ing of perm issible st at es, t he groups who aut horize advancing a bug t o t he next st at e, and t he allowable reasons for t he t ransit ion are governed by a set of rules in t he chosen process. I f you have t he perm ission, you can cust om ize t he rules furt her t o m at ch your t eam 's workflow.

Figu r e 8 .2 .

Th is st a t e ch a r t is t h e life cycle of a soft w a r e bu g in M SF for CM M I Pr oce ss I m pr ove m e n t .

With VSTS I n MSF for CMMI Process I m provem ent , bugs have four st at es, st art ing wit h Proposed. They need t o becom e accept ed before becom ing Act ive. I n MSF for Agile Soft ware Developm ent , bugs only have t hree st at es, st art ing wit h Act ive.

Anyone can propose a bug, t hat is, request t hat it be queued for fixing. A proj ect m anager can act ivat e t he bug and assign it t o a developer. Aft er invest igat ing and fixing t he code, t he developer m arks t he bug as resolved wit h t he next checkin ( see Figure 8.3 ) . The developer is not allowed t o close t he bug direct ly.

Figu r e 8 .3 .

I n VSTS, m e r e ly by ch e ck in g off t h e bu g in t h e ch e ck - in dia log, t h e de ve lope r a ssocia t e s t h e ch a n ge se t be in g de live r e d w it h t h e bu g be in g fix e d. Th e a ssocia t ion is pe r m a n e n t a n d is u se d t o popu la t e t h e bu ild r e por t a n d t h e m e t r ics w a r e h ou se .

[View full size image]

The check- in process aut om at ically resolves t he bug when t he corresponding source code is checkedin. A t est er can see t he resolved bug in t he corresponding query. Only t he t est er closes t he bug aft er verifying t hat it passes t he t est s t o prove t he fix. ( Of course, if t he t est fails, t hen t he t est er would react ivat e t he bug.) I n pract ice, workflows can be sim pler or m ore com plex. MSF Agile, for exam ple, does not require a proposed st at e before act ivat ion, does not enforce different perm issions, and inst ead relies on t rust . On t he ot her hand, som e processes, especially in lat e it erat ions, do not allow check- in unt il t he code has been reviewed. Work it em s can enforce t his rule t hrough anot her st at e or a separat e field, and check- in policy can enforce t he workflow. What ever t hey are, t hese rules are t he t angible form of t he process for your current proj ect .

Bug Reporting Is Like Journalism Caliban has an anecdot e about his favorit e bug. I t goes som et hing like t his: I found t his bug in t he lab m ont hs ago and report ed it . No one paid at t ent ion; so t hey t riaged it out and m arked it deferred. Last week our biggest cust om er found it t oo and all of a sudden it 's a crisis. Now we need t o ship a service release, which m eans t hat I 'll be in t he lab all weekend checking t he fix. Why couldn't t hey have paid at t ent ion when I report ed it t he first t im e? Why indeed? Very oft en t he answer is t hat t he bug report was not clear enough t o com m unicat e t he im pact t o t he t riage com m it t ee, and t he report ed bug was closed. Prospero, Caliban's m anager, has a different view of Caliban's st ory: " Why can't Caliban m ake his bug report s clear enough for t he t eam t o act on t hem ?" ( And, as before, Prospero did not choose Caliban for his next proj ect .) The lesson from Caliban's experience is well sum m arized in t his quot e from Kaner, Bach, and Pet t ichord: You r a dvoca cy dr ive s t h e r e pa ir of t h e bu gs you r e por t . Any bug report t hat you writ e is an advocacy docum ent t hat calls for t he repair of t he bug. Som e bugs will never be fixed. Your responsibilit y is not t o m ake sure t hat all t he bugs are fixed. Your responsibilit y is t o report bugs accurat ely and in a way t hat allows t he reader t o underst and t he full im pact of t he problem . How well you research and writ e t he report will oft en have a subst ant ial effect on t he probabilit y t hat t he bug will be fixed. [ 3 ] Writ ing up a bug well is like writ ing newspaper ( or web page) copy. I t is j ournalism . The t it le is like t he headline, and you should expect m ost readers never t o look " below t he fold." The j ournalist 's six basic quest ionswhat , where, how, who, why, and whenneed t o be covered in a sum m ary sent ence or t wo.

Know Your Audience The first audience for your bug report s is t he t riage com m it t ee. ( Triage is discussed in Chapt er 4, " Proj ect Managem ent ." ) They will see a query t hat looks som et hing like Figure 8.4 .

Figu r e 8 .4 .

Ke e p in m in d w h a t you r r e a de r s w ill se e du r in g t r ia ge . I t 's n ot a ll t h e de t a il, j u st w h a t fit s on t h e qu e r y r e su lt s list . You sh ou ld w r it e bu gs t o be vie w e d in t h is for m a t .

[View full size image]

The t riage com m it t ee includes t he busiest people on t he proj ect . I t is com m on for t hem t o spend less t han one m inut e reviewing each bug. ( An im port ant reason for daily t riage is t o keep t he lengt h of t he list down so t hat individual work it em s get t he at t ent ion t hey deserve.) I n t hat m inut e, t he fat e of your bug report will be decided, as was t he fat e of Caliban's bugs. I f Caliban had spent a lit t le m ore t im e on t he qualit y of his bug report s, t he value of his work ( and his perform ance rat ing) m ight have been considerably higher.

Learn from History Reading bug report s, especially aft er t hey have been t riaged, is a good way t o learn about your proj ect and your audience. Audit ing t riage m eet ings is anot heryou learn what is im port ant t o st akeholders in pract ice. Here are som e t ips on using your bug dat abase: Execut e t he repro st eps on int erest ing bugs. I t 's a good way t o learn t he soft ware under t est in it s m ore int erest ing cases ( and a good way t o learn how t o writ e clear st eps) . Read t he com m ent ary t o see how ot her t eam m em bers answer bug report s t o learn bot h what t hey t hink is im port ant and how t o im prove your report ing st yle. St udy closed bug report s t o see which bugs have and have not been fixed, based on t he way t hey were report ed. I f you want yours fixed, report t hem like t he ones t hat hist orically do get fixed. I f you're working on an exist ing solut ion, read t ech support logs for errors t o check and st ories t o support your bugs.

SOAP: An Analogy from Medicine The soft ware world borrowed t he t erm " t riage" from m edical pract ice, so it 's appropriat e t o look at t he cont ext of it s use in m edicine. Medical personnel learn t he acronym SOAP ( not hing t o do wit h web services) t o describe clinical report ing. Here is an excerpt from a m edical syllabus from t he Universit y of Maryland School of Pharm acy: [ 4 ]

For each pat ient ... care problem ( one or m ore) , writ e a progress not e ( in t he SOAP form at ) t o include: Subj ect ive inform at ion t hat support s t he problem ident ified; Obj ect ive inform at ion t hat support s t he problem ident ified; Assessm ent of t he problem including et iology, severit y, effect iveness/ t oxicit y of current t reat m ent , list of t herapeut ic alt ernat ives wit h discussion of advant ages and disadvant ages of each ( t herapeut ic and econom ic) , rat ionale for recom m endat ions; and Plan for m anaging t he ident ified problem including a specific out line for im plem ent at ion and cont inued m onit oring of t he problem and t herapeut ic response.

Figu r e 8 .5 .

Th e SOAP m n e m on ic is a u se fu l w a y t o t h in k of t h e fie lds of t h e for m a n d a h e a lt h y con ve r sa t ion h ist or y. Th e Tit le a n d Sym pt om a r e su bj e ct ive ; t h e fie lds su ch a s Fou n d I n [ Bu ild] , Fou n d- in En vir on m e n t , a n d St e ps t o Re pr odu ce a r e obj e ct ive ; Pr ior it y, Se ve r it y, Ar e a Pa t h , a n d Root Ca u se a r e a a sse ssm e n t , a n d I t e r a t ion a n d Fix a r e t h e pla n .

[View full size image]

Good craft sm en wat ch what ot her good craft sm en do, including t hose in ot her disciplines. SOAP is a good m odel t o use when you report bugs and review bug report s. Let 's t ake a look at t he st andard bug report ing form t hat VSTS provides wit h MSF for CMMI Process

I m provem ent .

Subjective Data The m ost im port ant quest ion for t he readers of your report is: What 's t he im pact on t he user? Most of t he t im e, t he answer requires som e t hought . Ask yourself quest ions like t he following:

•

What will t he cust om er say t o Product Support when she sees t his?

•

What will t he cust om er expect from experience wit h ot her soft ware ( including com pet it ors' product s) ?

•

What will sales reps or t rainers say if t hey hit t his in a dem o?

•

What scenarios will be affect ed by t his bug?

•

What is t he downst ream im pact of t his bug?

•

How m any users will care and how oft en?

•

I s t his an isolat ed occurrence?

•

I s t here a securit y risk here?

•

Are ot her qualit ies of service affect ed?

Writ e t his st ory first . How you answer t hese quest ions will influence bot h how m uch t im e you should spend researching t he bug and what dat a you should collect wit h t he report .

Titles Matter As you can see in Figure 8.4 , t he t it le is oft en t he only inform at ion t hat t he reader sees. I f t he t it le is unclear, when t runcat ed t o fit t he colum n widt h, t he rest of t he inform at ion m ay never be read.

Descriptions Stick The persist ent next level of inform at ion about t he bug is t he descript ion. Hist ory changes can get very long, but t he descript ion should be t he succinct det ail about t he bug.

One Report Per Bug I f you have t wo bugs, t hen writ e t wo report s. Keeping each observed bug in it s own report will m ake all t he report s easier t o underst and.

Objective Data Aft er you have writ t en t he subj ect ive dat a about t he bug, it is t im e t o gat her obj ect ive dat a.

Figu r e 8 .6 .

Th e bu g w or k it e m t ype , h e r e fr om M SF for CM M I Pr oce ss I m pr ove m e n t , ca pt u r e s t h e in for m a t ion t h a t a ll it s u se r s n e e d t o qu e r y, t r a ck , a n d a n a lyze t h e bu g.

[View full size image]

Quality of Service Be clear about t he t ype of issue t hat you are report ingwhat QoS is affect ed.

Related Feature or Code I f you can relat e t he bug t o a specified feat ure t hat you are t est ing or t o specific areas of t he code

under t est , t hen do so.

Test to Reproduce I f you have a t est t o reproduce t he bug, ident ify it .

Attachments What dat a can you at t ach t o t he bug report t o dem onst rat e t he condit ions in which t he bug occurs? All t hese can be useful, where applicable: Screenshot s Dat a files Configurat ion files Trace files Server logs Dr. Wat son logs

Reproduction Steps How do you reproduce t he bug in t he m inim um num ber of st eps wit h t he least set up? Eight y percent of t he developer's effort in working on a bug is oft en in t he reproduct ionhow well can you st ream line t hat process? Anyt hing unnecessary is clearly wast eful. Make t he repro st eps accurat e and clear. Check t hem before subm it t ing t he bug.

Preconditions I s special set up needed t o reproduce t he error, such as loading a part icular dat abase? I f t he problem can be dem onst rat ed wit h product ion dat a or cust om er dat a, it 's part icularly com pelling.

Assessment Data The assessm ent will change a lot t hrough t he bug's life as people refine t he diagnosis and observat ion, debug t he source, t ry and t est fixes, st udy relat ed bugs, and so on. I t is your j ob t o keep t he bug hist ory accurat e and t horough as you learn m ore about t he bug and it s condit ions. Som e of t he assessm ent you can do on t he init ial report , and m ore of it will evolve wit h t he bug hist ory and conversat ion.

Is It a Duplicate? Spend a lit t le bit of t im e looking t hrough t he dat abase t o det erm ine whet her t his is m ore inform at ion about an exist ing bug or som et hing new, but don't go crazy. I t 's easy enough for som eone t o m ark your bug as a duplicat e lat er. Obviously, if you have discovered m ore about an exist ing bug, updat e t he exist ing one wit h your findings.

How General Is It? Uncorner your corner cases. [ 5 ] Frequent ly, you will discover bugs using ext rem e values, unusual configurat ions, or long ent ry sequences. Essent ial t o underst anding t he im port ance of t he bug is t o know how specific or general t he condit ions m ust be. Here are som e t ips: Va r y t h e da t a . I f you discover t he bug wit h a part icular set of input values, t ry different ones. Especially t ry different equivalence classes. How dependent is t he bug's appearance on specific values or classes? Va r y t h e con figu r a t ion s. I f you find t he bug in a part icular configurat ion of hardware and soft ware com ponent s, t ry it on a very different set of m achines. Va r y t h e flow s. I f t he bug appeared under a part icular sequence of act ions ( especially if it is a long one) , see whet her t here are ot her ( preferably short er) pat hs t hat lead t o t he sam e result . Look for su spe ct in t e r a ct ion s. I f som et hing seem s t o be a st range side effect , but you're not sure, keep poking. For exam ple, aft er you use one program , anot her m ight suddenly seem slower. Perhaps t hey rely on som e com m on com ponent , dat a, or OS service. Ar e t h e r e m or e ? Hypot hesize where sim ilar bugs m ay be lurking. Look for t hem .

How Severe Is It? When you first find a bug, t here is a great t em pt at ion t o be sat isfied and t o report it . I n fact , you m ight be at t he t ip of a m uch worse problem . Not e t he init ial sym pt om but also keep using t he soft ware under t est . You are likely t o discover poorly handled error pat hs and except ions, unforeseen int eract ions, and ot her problem s t hat would baffle a user ( and t ech support ) .

Plan At init ial report ing, t he prim ary planning t ools you have are priorit y and ownership. Depending on your process, t hese m ay be set t able only by t he t riage com m it t ee, or in less form al organizat ions, t he subm it t er m ay fill t hese in direct ly. Priorit y should be handled in t he sam e way as it is for proj ect m anagem ent .

Iteration Think carefully about t he planned it erat ion. Oft en in early it erat ions, bugs are filed appropriat ely against funct ionalit y t hat won't work unt il lat er. A purpose of t riage is t o rem ove t his noise from developers' im m ediat e queues.

Summary This chapt er focused on bug report ing. Obviously, as all previous chapt ers discussed, it is far bet t er t o prevent bugs from occurring wit h good requirem ent s, archit ect ure, and developm ent pract ices t han t o discover t hem and incur t he rework necessary t o fix t hem . Nonet heless, t hey happen, and t heir det ails should be capt ured by everyone who finds t hem . When you writ e up a bug, be conscious of your audience. Most of your readers will only see t he t it le, perhaps in t runcat ed form . Work hard t o m ake t he t it le sum m arize t he im pact and im port ance of t he bug clearly; ot herwise, t he rest of t he inform at ion m ight be overlooked. A useful m nem onic for com plet ing t he rest of t he report is SOAP ( subj ect ive- obj ect ive- assessm ent plan) , borrowed from t he world of hospit als. I n part icular, t he m edical analogy can help you t hink hard about t he assessm ent and plan, and it can help m ove you away from t hinking of bugs as a num bers gam e.

Endnotes 1 . Geoffrey Jam es, The Zen of Program m ing ( Sant a Monica: I nfobooks, 1988) , 5960. Quot ed in Weinberg Syst em s Thinking 1992, 33.

2 . MSF for Agile Soft ware Developm ent . 3 . Kaner, Bach, and Pet t ichord, op. cit ., 656. 4 . ht t p: / / www.pharm acy.um aryland.edu/ courses/ syllabi/ PDF/ PHNT534.pdf 5 . Kaner, Bach, and Pet t ichord, op. cit ., 73.

Chapter 9. Troubleshooting the Project

Pr ogr a m M a n a ge r Pr oj e ct M a n a ge r " Happy fam ilies are all alike; every unhappy fam ily is unhappy in it s own way." Leo Tolst oy, Anna Karenina [ 1 ]

Figu r e 9 .1 .

Le o N ik ola ye vich Tolst oy.

Tolst oy's aphorism applies t o soft ware proj ect s, t oo. There are m any pat t erns of unhappiness in a soft ware proj ect , which are usually m anifest ed in a couple dozen sym pt om s. This chapt er focuses on t hese pat t erns, t he sym pt om s, and how t o recognize t hem . By now, I hope you are convinced of t he value- up paradigm , which assert s t hat we apply syst em s t o look cont inually for ways t o im prove t he flow of value. I n Chapt er 1, " A Value- Up Paradigm ," I cont rast ed t his t o t he " iron t riangle" view of t he work- down paradigm , which assum es fixed capacit y and reduces problem s t o t im e, resources, funct ionalit y, and qualit y. [ 2 ] The m et rics warehouse, like t he work it em dat abase, enables you t o run a proj ect based on t rust and t ransparency. Everyone on t he t eam get s t o see t he sam e dat a, ask t he sam e quest ions, look for t he answers, and be part of finding t he solut ion. This chapt er m ight be crit icized for being unnecessarily quant it at ive. By no m eans do I m ean t o suggest t hat you always need num bers t o grasp problem s or t hat solut ions and im provem ent s should be num eric. As discussed in Chapt er 4, " Proj ect Managem ent ," t he m et rics need t o be descript ive, not prescript ive. Pride of workm anship, one of t he MSF m indset s, is assum ed for everyone, and t he m et rics are a t ool t o reinforce t hat pride, not supplant it . You win at sport s by playing well and keeping your eye on t he ball, not on t he scoreboard. The scoreboard j ust keeps count so t hat you don't have t o be dist ract ed by arguing about whose num bers are right . Many sym pt om s require no m et rics warehouse. A well- known exam ple is t he I nverse Dilbert Correlat ion Fact or: The m ore Dilbert cart oons t here are past ed on office doors and bullet in boards, t he less well off t he proj ect is. [ 3 ] ( Of course, an absence of cart oons m ight be a warning sign of a cert ain com pany policy, t oo.) Anot her exam ple is t he m orale of t he t eam , which is visible in t he energy level and ent husiasm . I f t eam m em bers are losing sleep over proj ect problem s, t hey should be able t o t ell you what 's bot hering t hem . And if you aren't seeing it , t hen you're not spending enough t im e wit h t he rest of your t eam . On t he ot her hand, we are all suscept ible t o blind spot s. Trends and drilldowns are great for ident ifying possible risks and oversight s, and t he healt h chart s are a great st art ing point . They give you t he dat a t o ask t he right quest ions, but only t he t eam can find t he answers. The great est benefit of t he m et rics chart s is t heir capabilit y t o com plem ent what you sense or suspect from int eract ing wit h your fellow t eam m em bers, t rying current builds of t he soft ware, reviewing code, researching bugs, planning it erat ions, and so on. The chart s give you indicat ors of t he overall healt h of t he proj ect and, when you suspect a problem , t he abilit y t o look at t he dat a t o confirm or deny t he suspicion. For t he rest of t his chapt er, I cat alog a series of pot ent ial problem s, m any of which you m ay recognize from personal experience, and how t hey can show up in t he VSTS proj ect healt h chart s. The goal here is t o show how VSTS, wit h it s daily report ing, can provide early warnings and help you wit h cont inuous course correct ion t o im prove capacit y.

Using Excel with the Metrics Warehouse I n addit ion t o t he st andard report s t hat are inst alled wit h t he process t em plat e in VSTS, you can access all t he m et rics warehouse dat a from Microsoft Excel. See t he MSDN t opic: Developm ent Tools and Technologies Visual St udio Team Syst em Team Foundat ion Team Foundat ion Proj ect Leads Using Report ing and Met rics Team Foundat ion Server Report ing Using Team Report s Using Microsoft Excel for Team Foundat ion Server Report ing

Underestimating One of t he m ost frequent ly report ed problem sym pt om s is underest im at ion. When progress falls short of t he plan and effort is great er t han est im at ed, proj ect m em bers are underest im at ing t he resources, difficult y, t im e, or ot her fact ors ( see Figure 9.2 ) .

Figu r e 9 .2 .

Ba se d on t h e slope of t h e Close d lin e , t h e Close d lin e w ill h it t h e it e r a t ion e n d da t e w e ll be low t h e pla n n e d le ve l, m e a n in g t h a t n ot a ll t h e sce n a r ios pla n n e d for t h is it e r a t ion w ill be com ple t e d be for e it e r a t ion e x it .

I f you see t his pat t ern, you will, of course, want t o drill down int o t he root causes. There are several possible reasons for underest im at ing, covered in Figures 9.39.10 .

Uneven Task Decomposition

Check t he degree of variat ion in t he t ask definit ion and t he size range of t he t ask granularit y. You would hope t o see t asks planned t o t he scale of one t o t hree days ( see Figures 9.3 and 9.4) .

Figu r e 9 .3 .

Th is h ist ogr a m of n u m be r of t a sk s a ga in st size sh ow s t h a t t a sk size va r ie s sign ifica n t ly.

Figu r e 9 .4 .

Cor r e spon din gly, t h e Pr oj e ct Ve locit y ch a r t sh ow s a h igh va r ia n ce in t h e n u m be r of t a sk s close d pe r da y.

[View full size image]

Architectural Blind Spots Som et im es t he t eam discovers a need for an archit ect ural change. This could be t he need t o focus m ore heavily on int egrat ion, reconsider QoS, change t he com ponent st ruct ure, int roduce new com m on services, change t he planned deploym ent environm ent , or ot herwise m ake ext ensive archit ect ural changes. Exam ining t he Rem aining Work chart s in Figures 9.5 and 9.6 shows t he pat t ern. Scenarios and QoS requirem ent s are st aying act ive longer t han expect ed ( in fact , som e need t o be cut from t he it erat ion) , while developm ent t asks are rising significant ly. This lat e discovery of t asks m ay indicat e undiscovered archit ect ural work.

Figu r e 9 .5 .

Th is Re m a in in g W or k ch a r t , filt e r e d for t a sk s, sh ow s sign ifica n t gr ow t h in t a sk s be in g a dde d t o t h e ba ck log.

Figu r e 9 .6 .

Sim u lt a n e ou sly, w h e n filt e r e d for sce n a r ios, t h e Re m a in in g W or k ch a r t sh ow s t h e sce n a r ios st u ck in t h e a ct ive st a t e , a n d t h e t ot a l n u m be r pla n n e d for t h e it e r a t ion is de clin in g.

Scope Creep " Scope creep," depending on your perspect ive, is t he usual t erm for t he lat e piling on of requirem ent s or t he inabilit y of t he proj ect m anager t o keep t he current it erat ion focused on it s init ial t arget . I n ot her words, at t he st art of an it erat ion, you norm ally know which scenarios are chosen for t he it erat ion. I f t his list shift s, t he t eam can be seriously disrupt ed ( see Figure 9.7 ) .

Figu r e 9 .7 .

Th e Un pla n n e d W or k ch a r t , filt e r e d for sce n a r ios, sh ow s a sign ifica n t clim b in t h e Adde d La t e r a r e a .

On t he ot her hand, it is ent irely appropriat e t o ret hink t he fut ure bet ween it erat ions. A purpose of it erat ion chunking is t o give you t he opport unit y t o correct your course based on new learning and new inform at ion.

Inadequate Bug Allotment I f you plan t asks t hat consum e 100% of your available resources, t hen you have no capacit y left t o handle unplanned work. I t is necessary t o schedule t im e for handling bugs and ot her work t hat will arise but t hat m ight not be known at it erat ion st art . Som et im es, especially in feat ure- boxed planning, t his pat t ern is a sign of polit ical dysfunct ion. One exam ple is " schedule chicken," where com pet ing t eam s bluff t he schedule because t hey expect som eone else t o m iss t he dat es by a larger m argin t han t hey will ( see Figures 9.8 and 9.9) .

Figu r e 9 .8 .

Tr yin g t o blu ff t h e sch e du le sh ow s u p in a n a ccu m u la t in g list of issu e s t h a t a r e n 't be in g r e solve d.

[View full size image]

Figu r e 9 .9 .

Th is cu m u la t ive flow dia gr a m for bu gs sh ow s a st e e ply r isin g t op lin e be ca u se m ost bu gs a r e n ot k n ow n a t t h e st a r t of t h e it e r a t ion . Th e w ide n in g Act ive ba n d in dica t e s a fin d r a t e h igh e r t h a n t h e cu r r e n t fix r a t e , gr ow in g t h e bu g ba ck log. Th is cou ld h a ppe n for a va r ie t y of r e a son s, su ch a s n e w ly in t e gr a t e d fu n ct ion a lit y be in g a va ila ble for t e st , a bu g ba sh , n e w r e por t s fr om be t a u se r s, a n d so on . On t h e ot h e r h a n d, t h e bu bble in t h e Re solve d ba n d sh ow s a pe r iod w h e n t e st e r s ca n n ot ve r ify t h e n u m be r of bu g r e solu t ion s a s qu ick ly a s t h e y a r e be in g de live r e d.

Resource Leaks I n som e environm ent s, a t eam does not fully cont rol it s own t im e. For exam ple, developers m ight be pulled off new developm ent t o fix bugs on product ion releases, or people m ight be pulled int o

som eone's pet proj ect . ( " Can you set up t his dem o for m e?" ) These are resource leaks because t he resources you scheduled are being reallocat ed t o unplanned work t hat does not benefit your proj ect . To spot resource leaks, look for unusual variat ions in t he Proj ect Velocit y graph ( see Figure 9.10) or flat spot s in t he Rem aining Work chart .

Figu r e 9 .1 0 .

On t h is Pr oj e ct Ve locit y gr a ph , t h e n u m be r of de ve lopm e n t t a sk s close d t a k e s a su dde n dip in w e e k 4 . Th e r e m a y be a k n ow n r e a son ( va ca t ion , sch e du le d t r a in in g, illn e ss, a n d so on ) , or it m a y be a sign t h a t som e t h in g is in t e r fe r in g w it h t h e e x pe ct e d w or k . I t 's a good qu e st ion for you r st a n du p m e e t in gs.

Development Practices Too Loose I t is a developm ent responsibilit y t o deliver working soft ware for t est ing. I f t he soft ware isn't passing t he build verificat ion t est s ( BVTs) , or t he BVTs and unit t est s are inadequat e, t hen fix t he problem at it s source. Ordinarily, t he t eam will know if t hese are problem s, but if you are a m anager and t herefore are one st ep rem oved, you m ight first see signs in t hese report s.

Build Failures A night ly build is t he heart beat of your proj ect ( refer t o Chapt er 6, " Developm ent " ) . I f your builds are not com plet ing successfully or are not passing BVTs, as shown in Figure 9.11, t hen you need t o do what is necessary t o fix t he problem im m ediat ely. Usually t he t eam will self- correct and rest ore t he working build wit hout m uch m anagem ent int ervent ion.

Figu r e 9 .1 1 .

Th is bu ild su m m a r y sh ow s t h a t som e bu ilds a r e n ot com ple t in g a n d ot h e r s a r e fa ilin g BVTs.

[View full size image]

Inadequate Unit Testing Code should be delivered wit h adequat e unit t est st hat 's pret t y well accept ed nowadays. The best aut om at ed approxim at ion for t he breadt h of t he unit t est s is code coverage ( refer t o Chapt er 7, " Test ing," on t he use and m isuse of code coverage) . I f you are pract icing Test Driven Developm ent or sim ilar t echniques, t hen your code coverage should approach 100% , except where you have good reasons for exclusions. I f your unit t est s are reused as BVTs, t hen t he coverage should be visible wit h t he Qualit y I ndicat ors and Build report s ( see Figures 9.12 and 9.13 ) .

Figu r e 9 .1 2 .

Th is Qu a lit y I n dica t or s r e por t sh ow s a de cr e a se in code cove r a ge a n d a n in cr e a se in code ch u r n ove r t h e five da ys. Th is is a cle a r w a r n in g t h a t n e w code is be in g ch e ck e d in w it h ou t cor r e spon din g u n it t e st s t o cove r it .

[View full size image]

Figu r e 9 .1 3 .

Th is Bu ild D e t a ils r e por t sh ow s t h a t code cove r a ge va r ie s w ide ly a cr oss t h e diffe r e n t sou r ce pr oj e ct s. Th is in dica t e s u n e ve n u n it t e st in g or a t le a st u n e ve n cove r a ge fr om t h e BVTs.

[View full size image]

Reactivations Anot her warning sign is a high react ivat ion rat e, som et im es called a " fault feedback rat io." React ivat ion rat e count s t he num ber of supposedly fixed bugs whose fixes don't work ( see Figure 9.14 ) . These react ivat ions creat e a vicious cycle of rework t hat crowds out planned t asks.

Figu r e 9 .1 4 .

Th is Re a ct iva t ion s gr a ph sh ow s a h igh a n d r isin g r a t e of bu g fix e s t h a t w e r e r e j e ct e d in t e st in g. Th is t r e n d is cle a r ly diggin g a h ole t h a t w ill con su m e r e sou r ce s for n o visible cu st om e r be n e fit .

[View full size image]

Not e t hat if you encount er high react ivat ions, it 's wort h looking int o t he root causes. Alt hough sloppy developm ent pract ice is an obvious possibilit y, ot her pot ent ial causes include poor bug report ing, poor t est lab m anagem ent , and overly aggressive t riage.

Bluffing Bluffing, t hat is, report ing work as com plet e when it isn't , is hard t o achieve when you have t he t ransparency of a value- up process wit h VSTS. I f som eone is bluffing, you would expect t o see a com binat ion of t he pat t erns shown previouslybuild breaks, high react ivat ions, rising code churn, and decreasing code coverage from unit t est ing and BVTs. There will be enough anom alies t hat t he

behavior m ight correct it self t hrough pride of workm anship and peer pressure. Of course, if it doesn't , you need t o int ervene.

Tests Passing; Solution Doesn't Work One of t he m ore frust rat ing proj ect sit uat ions is t o find t hat t est s are report ed as passing but t he solut ion under t he t est st ill doesn't work for observers out side t he t est t eam . I n t hese cases, you want t o ident ify why t he t est s do not seem t o find t he sam e issues t hat ot her users do. Figures 9.15 9.18 are exam ples of t his case.

High Bug Find Rate Frequent ly you see a high t est pass rat e but st ill see a large incom ing bug rat e ( or worse, cust om ers or bet a users are report ing lot s of bugs t hat t est ing seem s t o be m issing) . This can occur for several reasons: The t est s m ight be t oo gent le for t his st age of t he solut ion. I n early it erat ions, gent le t est s are good, but as t he solut ion m at ures, t est s should exercise broader scenarios and int egrat ions. These t est s m ight be m issing. Test s m ight be st ale or be t est ing t he wrong funct ionalit y. I t m ight be t im e t o swit ch t est t echniques. ( See Chapt er 7.) Consider Figures 9.15 , 9.16 , and 9.17 .

Figu r e 9 .1 5 .

On t h e Qu a lit y I n dica t or s ch a r t , t h e t e st pa ss r a t e is h igh , bu t a ct ive bu gs a r e a lso h igh .

Figu r e 9 .1 6 .

Cor r e spon din gly, on t h e Bu g Ra t e s ch a r t , a ct ive bu gs a r e h igh be ca u se fin d r a t e st a ys h igh .

Figu r e 9 .1 7 .

Te st s a r e n 't fin din g t h e bu gs. On t h is r e por t , m a n y of t h e bu gs fou n d h a ve n o cor r e spon din g t e st . Th is m igh t be a sign t h a t t e st in g is look in g e lse w h e r e . An d, if you a r e e x pe ct in g r e gr e ssion t e st in g t o pr e ve n t t h e ir u n discove r e d r e cu r r e n ce , you w ill n e e d r e gr e ssion t e st s t h a t you don 't h a ve ye t .

[View full size image]

Tests Are Stale Test s do not necessarily evolve at t he sam e rat e as t he code under t est . This risk is present especially when t est s are heavily aut om at ed. I n t his sit uat ion, you see high t est pass rat es wit h ongoing code churn and dim inishing code coverage ( see Figure 9.18) .

Figu r e 9 .1 8 .

Th is Qu a lit y I n dica t or s ch a r t sh ow s a h igh r a t e of code ch u r n a n d a low r a t e of code cove r a ge fr om t e st in g, ye t t e st pa ss r a t e s r e m a in h igh . Th is su gge st s t h a t t h e t e st s be in g r u n a r e n ot e x e r cisin g t h e n e w code . D on 't be lu lle d by t h e h igh t e st pa ss r a t e t h e se t e st s a r e cle a r ly n ot t e st in g a ll t h e n e w de ve lopm e n t w or k .

[View full size image]

Solution Stuck in Testing Som et im es it appears t hat t here is a bot t leneck in t est ing, as indicat ed by t he Rem aining Work chart ( see Figure 9.19) .

Figu r e 9 .1 9 .

Th is Re m a in in g W or k ch a r t sh ow s a bu lgin g lin e in Re solve d, m e a n in g t h a t t h e de ve lope r s a r e r e por t in g w or k it e m s r e solve d, bu t t e st e r s h a ve n 't close d t h e m . Fu r t h e r dr illdow n is w a rra nt ed.

[View full size image]

This bulge in t est ing can happen for very different reasons. I t always m erit s furt her drilldown.

Tests Failing One case is t hat lot s of t est s are failing, requiring effort t o diagnose t he failures and report t he bugs.

This should prom pt you t o invest igat e why t he soft ware is failing so oft en. Not e t hat code churn is st uck at a high level as well, indicat ing t hat lot s of new code is being writ t en ( see Figure 9.20) . The next pat t erns t o look for are t he ones shown previously in t his chapt er under " Developm ent Pract ices Too Loose" ( see Figures 9.12 t hrough 9.14 ) .

Figu r e 9 .2 0 .

Th e Qu a lit y I n dica t or s ch a r t sh ow s t h a t lot s of t e st s a r e be in g r u n w it h r e a son a ble code cove r a ge , bu t t h e t e st s a r e fa ilin g. Th is is pr oba bly a n in dica t or of loose de ve lopm e n t pr a ct ice s, a lt h ou gh in e a r ly it e r a t ion s, it m igh t be a n in dica t or t h a t t h e t e st s a r e t oo h a r sh for t h is st a ge of t h e solu t ion .

[View full size image]

I t 's also possible t hat t est s are failing because t he t eam discovered an unexpect ed need t o refact or t he code. This m ight be ent irely healt hy and foreseenit is exact ly t he kind of answer t hat t he m et rics alone cannot t ell you.

Too Little Testing On t he ot her hand, t he problem m ight be t hat not enough t est ing is being done t o process t he incom ing work quickly enough ( see Figures 9.21 and 9.22 ) . This could be a lim it at ion of resources,

planning, or logist ics.

Figu r e 9 .2 1 .

Th is Qu a lit y I n dica t or s ch a r t sh ow s a low r a t e of t e st s be in g r u n . Th is w ou ld pr oba bly m e a n t h a t t oo lit t le t e st in g h a s be e n h a ppe n in g. Th is cou ld be du e t o in a de qu a t e r e sou r ce a lloca t ion for t e st in g, or it cou ld be t h a t t e st e r s a r e doin g som e t h in g e lse , su ch a s w r it in g t e st a u t om a t ion r a t h e r t h a n t e st in g t h e cu r r e n t fu n ct ion a lit y. I n e it h e r ca se , r e sou r ce ba la n cin g m a y be w a r r a n t e d.

[View full size image]

Figu r e 9 .2 2 .

A va r ia n t pa t t e r n t h a t occu r s w it h a u t om a t e d t e st ge n e r a t ion is t h a t t h e de ve lope r s h a ve ge n e r a t e d t e st s, bu t n o on e h a s fin ish e d t h e logic, so t h e t e st s a r e in con clu sive .

[View full size image]

Summary I n t his chapt er, I showed a large num ber of ant ipat t erns t hat proj ect s experience, using exam ples from t he VSTS m et rics warehouse. None of t hese are direct prescript ions for a cure, but t hey are illust rat ions of dat a t hat allow you t o ask t he right quest ions of t he t eam . When everyone on t he t eam can see t he sam e m et rics across all t he dim ensions, discussions shift from t rying t o det erm ine t he dat a t o t rying t o answer t he quest ions posed by t he dat a. Wit h t rust and t ransparency, it becom es possible for everyone t o part icipat e in ident ifying and execut ing solut ions.

Endnotes 1 . Tolst oy, Anna Karenina. 2 . Anderson 2005, op. cit . 3 . Edward Yourdon, Deat h March: The Com plet e Soft ware Developer's Guide t o Surviving 'Mission I m possible' Proj ect s ( Upper Saddle River, NJ: Prent ice Hall, 1997) .

Chapter 10. Conclusion " I t is t rue t hat we have really in Flat land a Third unrecognized Dim ension called 'height ,' j ust as it also is t rue t hat you have really in Spaceland a Fourt h unrecognized Dim ension, called by no nam e at present , but which I will call 'ext raheight .' But we can no m ore t ake cognizance of our 'height ' t han you can of your 'ext raheight .' Even I who have been in Spaceland, and have had t he privilege of underst anding for t went yfour hours t he m eaning of 'height 'even I cannot now com prehend it , nor realize it by t he sense of sight or by any process of reason; I can but apprehend it by fait h." Edwin Abbot t , Flat land: A Rom ance of Many Dim ensions[ 1 ]

Figu r e 1 0 .1 .

Th e se t w o figu r e s a r e bot h t w odim e n sion a l vie w s of t h e sa m e oct a h e dr on . I f you cove r t h e r igh t h a n d im a ge , h ow e ve r , you w ill pr oba bly se e t h e le ft im a ge a s a squ a r e w it h t w o dia gon a ls. On ly w h e n you se e t h e t w o t oge t h e r do t h e dia gon a ls a ppe a r a s e dge s of a t h r e e dim e n sion a l sh a pe .

I n t his book, I 've described how a t eam of peers can apply a valueup approach and apply Visual St udio Team Syst em ( VSTS) t o im prove t heir capacit y for creat ing cust om er value t hrough t heir work. I 've done t his from t he various qualit at ive viewpoint s of t he disciplines and from t he quant it at ive m et rics t hat get capt ured in t he process. Bot h are m ult idim ensional approaches. I have frequent ly used t he t wo inst ances of MSF ( Microsoft Solut ions Fram ework) as references because t hey are t he process guidance t hat ships wit h VSTS and because t hey are t he processes t hat , as far as I know, m ost closely represent valueup t hinking. I have t ried, wherever possible, t o give credit t o t he int ellect ual sources of t he ideas.

Expected Criticisms At t he t im e I am writ ing t his book, Team Syst em has j ust shipped. I 've m ade a num ber of claim s, and I expect a fair am ount of crit icism for t hem , so let m e plead guilt y t o five m aj or indict m ent s right now:

1 . Th is book is n ot Agile e n ou gh . I n part icular, I expect t o be dinged for not following t he Agile Manifest o t enet s closely enough. [ 2 ] For exam ple, I writ e m ore about processes and t ools t han individuals and int eract ions, and m ore about change in t he cont ext of a plan t han responding t o change wit hout a plan. Act ually, I t hink t hat a huge st rengt h of t he Agile com m unit y has been t he effervescent t ooling t hat has em erged for unit t est ing and change m anagem ent . And processes such as XP have dem onst rat ed t he great value of discipline. Wit h VSTS, we are t rying t o m ake t ools and pract ices like t hese easier and m ore approachable t o a m uch wider com m unit y t han t hey have been before. 2 . Th is book is n ot pr e scr ipt ive e n ou gh . A value- up t enet is t he im port ance of sit uat ionally specific st rat egies. I 've t ried t o focus on recognizing and underst anding t he sit uat ion, t rust ing t hat if t he whole t eam can see t he sam e dat a, t he t eam can work on t he prescript ions. There is cert ainly a book wait ing t o be writ t en on sit uat ionally specific m anagem ent pat t erns. 3 . Th e r e is n ot e n ou gh de pt h for e a ch of t h e disciplin e s. I 've writ t en t his as t he int roduct ory book t o a series. A book t arget ed specifically t o developm ent pract ices wit h VSTS will be available short ly aft er t his one is released. I hope m any aut hors will j oin m e over t he next few years. And I realize t hat I st opped short on t he vit al t opics of user experience, release m anagem ent , and operat ions. 4 . I don 't h a ve e n ou gh da t a t o su ppor t m y cla im s. Not yet . Microsoft is cert ainly going t o accum ulat e case st udies around VSTS, and you will be able t o find t hem on ht t p: / / m sdn.m icrosoft .com / t eam syst em . I hope t hat t hey reveal enough insight int o t he processes used and enough dat a t o illust rat e t he values t hat I discuss here. 5 . Th e sou r ce s a r e t oo r a n dom . Soft ware engineering is not new and does not occur independent ly of it s business cont ext . I 've t ried hard t o bring in t he t hreads of bot h t he valuable work of t he com m unit y and t he business environm ent of t he t went y- first cent ury. I oft en find soft ware debat es t o be very black- and- whit e, but I see t he sit uat ion as m uch m ore m ult icolored. I hope t hat you t oo m ay com e t o prefer t he color and gradat ion over t he absolut e black or whit e proj ect ion. I also m ight be accused of m aking t oo m uch of a product sales pit ch. I have t ried t o argue t he design ideas t hat led t o t he creat ion of VSTS wit h as m any exam ples as I could fit . I 've t ried wherever possible t o dist inguish t he ideas from t he im plem ent at ion, but I have used t he product t o illust rat e t hem . I hope you found t he present at ion balanced.

Value-Up, Again The core idea of t his book is t hat a new paradigm is em erging for soft ware developm ent , which I 've called t he value- up approach. The early int ellect ual root s of value- up lie in t he work of Dem ing, Weinberg, and Goldrat t , and t he Agile, Lean Manufact uring, and Theory of Const raint s com m unit ies. The cent ral t enet of value- up is t o m axim ize t he flow of cust om er value. Consist ent wit h t hese root s, value- up espouses t hese ideas:

1 . Em br a ce ch a n ge . I nvest in j ust enough planning and design t o underst and risk and t o m anage t he next sm all increm ent . 2 . M e a su r e on ly t h ose de live r a ble s t h a t t h e cu st om e r va lu e s. View all int erim m easures skept ically. 3 . Un de r st a n d qu a lit y a s va lu e t o t h e cu st om e r . The cust om er's percept ion m ay change, so keep opt ions open and plan for frequent delivery. 4 . Acce pt va r ia n ce a s a pa r t of a ll pr oce sse s. Dist inguish specialcause from com m oncause variance. Treat t hem different ly. 5 . Use in t e r m e dia t e w or k pr odu ct s on ly in sofa r a s t h e y im pr ove t h e flow of va lu e . Use t hem t o reduce uncert aint y and variat ion, not m easure progress. 6 . I n cr e a se ca pa cit y by a t t a ck in g bot t le n e ck s in t h e flow of va lu e . Tune resources and t im e only aft er rem oving t he bot t lenecks. 7 . Be t r a n spa r e n t a n d t r u st in g. Assum e t hat t he t eam t akes pride in it s workm anship and want s it t o be seen.

Figu r e 1 0 .2 .

Th e pr in ciple s a n d m in dse t s of M SF r e fle ct t h e va lu e - u p pa r a digm .

I f you consider t his list t o be m ot herhood and apple pie, t hen I 've achieved half m y purpose. The second half of m y purpose is t o convince you t hat t ooling can m ake a large, posit ive difference. VSTS is not t he first product t o address t he soft ware lifecycle, and it won't be t he last . However, I believe t hat it is t he first t hat at t em pt s t o offer a com prehensive value- up approach for t he whole t eam in a sim ple, product ive, int egrat ed, and ext ensible m anner. Free t rials of t he product are available, and you can j udge it s suit abilit y for yourself. Of course, VSTS is a new product , and t here will be a large num ber of request s t o t ake it furt her. I welcom e t he dialogue. VSTS, by inst rum ent ing t he soft ware process, enables t rust wort hy t ransparency. The m et rics warehouse gives t he whole t eam a com m on view of t he fact s and a com m on base of hist orical perform ance. This com m on view changes t he discussion from " Whose num bers are right ?" t o " What should we do next ?" I hope t hat VSTS spawns sim ilar innovat ions across t he indust ry. I m proving t he capacit y of I T and t he soft ware indust ry is one of t he great econom ic challenges and opport unit ies of t he com ing decades. I believe t hat t his requires bot h a value- up approach and t he t ooling t o support it .

Endnotes 1 . Edwin Abbot t , Flat land: A Rom ance of Many Dim ensions, by A Square ( Bost on, Robert s Brot hers, 1885) , Preface t o Second Edit ion.

2 . www.agilem anifest o.org

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W]

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] accepted build cycle, testing during accessibility, as QoS (quality of service) actors, personas versus Actual Quality versus Planned Velocity graph ad hoc testing [See exploratory testing.] adapting process (iterative development) adaptive approach, plan-driven approach versus adaptive projects advocacy groups, viewpoints of risk Agile Alliance 2nd Agile Manifesto agility AIB (Applied Integration Baseline) Anderson, David J. 2nd Application Designer Applied Integration Baseline (AIB) architecture baseline architecture citizenship Design for Operations QoS mindset and reference architectures SOA (service-oriented architecture) VSTS and troubleshooting validating value-up approach 2nd in VSTS assessment data in bug reports attractiveness, as QoS (quality of service) audit trails auditability auditors Austin, Robert automated build system automated code analysis automated scenario testing automated testing availability, as QoS (quality of service)

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] baseline architecture reference architectures refining batch sizes Beizer, Boris bluffing Boehm, Barry bottom-up estimation branching (source control) Broken Windows theory Brooks, Fred bug find rate Bug Rates graph bugs [See also prioritizing bugs; load testing.] capacity to handle lifecycle of reactivation rate, troubleshooting writing bug reports assessment data in objective data in plans in SOAP analogy subjective data in Bugs by Priority graph build failures, troubleshooting build reports build verification tests (BVTs) 2nd 3rd business process models Buwalda, Hans BVTs (build verification tests) 2nd 3rd

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] Capability Maturity Model Integration [See CMMI (Capability Maturity Model Integration).] change requests [See also bugs.] changesets check-in cycle, testing during check-ins, associating work items with checking in files (source control) citizenship CMMI (Capability Maturity Model Integration) MSF for CMMI Process Improvement Cockburn, Alistair code analysis [See automated code analysis.] code churn code coverage overlaying troubleshooting in unit testing code reviews commercial-off-the-shelf software (COTS) common cause variation compatibility, as QoS (quality of service) competition, global competition compliance component integration tests concurrency, as QoS (quality of service) configuration management [See source control, development quality and.] configuration testing 2nd conformance to standards, as QoS (quality of service) contextual inquiry continuous integration contract-first design control theory (iterative development) Cooper, Alan 2nd COTS (commercial-off-the-shelf software) coverage [See code coverage.] Csikszentmihalyi, Mihaly cumulative flow diagram customer validation of scenarios

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] daily activities, instrumenting daily build cycle, testing during daily builds data sets, unit testing data warehouse [See work item databases.] dead reckoning Declaration of Interdependence defects [See bugs.] DeMarco, Tom Deming, W. Edwards Deployment Designer descriptive metrics, prescriptive metrics versus Design for Operations development quality issues programming errors 2nd requirements 2nd transparency 2nd unit testing 2nd version skews 2nd troubleshooting value-up approach to discoverability, as QoS (quality of service) discovery, testing as dissatisfiers Kano Analysis documentation [See required documentation, tacit knowledge versus.] duplicate bugs

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] ease of use, as QoS (quality of service) economics (iterative development) efficiency, as QoS (quality of service) Einstein, Albert elevator pitch EQF (Estimating Quality Factor) errors [See bugs.] estimating iterations bottom-up estimation estimation quality refinements to retrospectives top-down estimation Estimating Quality Factor (EQF) estimating tasks value-up approach flow work-down approach versus work-down approach value-up approach versus Excel, accessing metrics warehouse from exciters Kano Analysis explicit sign-off gates, implicit sign-off gates versus exploratory testing

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] false confidence in testing fault feedback ratio, troubleshooting fault model fault tolerance, as QoS (quality of service) flow transparency work-down approach versus focus (iterative development) focus groups

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] gaps in testing geographic boundaries, fitting process to project global competition goals of scenarios "good enough" testing governance model granularity (iterative development)

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] historical estimates, tracking

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] implicit sign-off gates, explicit sign-off gates versus indifferent features (Kano Analysis) installability, as QoS (quality of service) instrumented profiling (performance profiling) instrumenting daily activities interoperability, as QoS (quality of service) iron triangle iteration cycle, testing during iterations estimating bottom-up estimation estimation quality refinements to retrospectives top-down estimation test objectives triage and iterative development adapting process control theory economics focus granularity length of motivation prioritization reasons for using risk management stakeholder involvement

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] job descriptions, project roles versus

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] Kaner, Cem 2nd Kano Analysis Kerth, Norman L.

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] late delivery length of iterations lifecycle of bugs load testing localizability, as QoS (quality of service) Logical Datacenter Designer

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] maintainability, as QoS (quality of service) manageability, as QoS (quality of service) managed code analysis managing projects [See project management.] manual code reviews McConnell, Steve metrics multidimensional metrics prescriptive metrics versus descriptive metrics team efficiency metrics warehouse accessing from Excel for project management troubleshooting with development practices testing bottleneck tests pass, solution doesn"t work underestimation usefulness of Microsoft Project monitorability, as QoS (quality of service) Moore, Geoffrey 2nd 3rd MoSCoW (prioritization scheme) motivation (iterative development) MSF for Agile Software Development change requests reports special cause variation MSF for CMMI Process Improvement audit trails change requests governance model special cause variation MSF for CMMI Software Improvement, reports multidimensional metrics must-haves (Kano Analysis)

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] objective data in bug reports operability, as QoS (quality of service) operations, Design for Operations organization, prescribed versus self-organization organizational boundaries, fitting process to project outsourcing/offshoring overlaying code coverage

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] pain points paradigm shifts paradigms [See value-up approach; work-down approach.] performance architecture and as QoS (quality of service) performance tuning personas actors versus researching Personify Design Teamlook Pesticide Paradox plan-driven approach, adaptive approach versus plans in bug reports Poincaré, Henri portability, as QoS (quality of service) post-mortems prescribed organization, self-organization versus prescriptive metrics, descriptive metrics versus prioritization (iterative development) prioritizing bugs triage example iterations and red line privacy, as QoS (quality of service) Process Template process, fitting to project 2nd adaptive approach versus plan-driven approach auditability and regulatory concerns geographic and organizational boundaries implicit versus explicit sign-off gates and governance model prescribed versus self-organization project switching required documentation versus tacit knowledge product backlog production environment, testing for profiling (performance tuning) programming errors, development quality and 2nd project cycle, testing during project management [See also projects.] audit trails estimating iterations bottom-up estimation estimation quality refinements to

retrospectives top-down estimation metrics warehouse prescriptive versus descriptive metrics questions to answer Actual Quality versus Planned Velocity graph Bug Rates graph Bugs by Priority graph Project Velocity graph Quality Indicators graph Reactivations graph Remaining Work graph Unplanned Work graph triage example iterations and red line variation project portal project smells project switching Project Velocity graph projects [See also project management.] adaptive projects fitting process to 2nd adaptive approach versus plan-driven approach auditability and regulatory concerns geographic and organizational boundaries implicit versus explicit sign-off gates and governance model prescribed versus self-organization project switching required documentation versus tacit knowledge requirements [See requirements.] strategic projects troubleshooting [See troubleshooting with metrics warehouse.] when to build

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] QoS (qualities of service) 2nd architecture and Kano Analysis testing quality [See also bugs.] in development programming errors 2nd requirements transparency 2nd unit testing 2nd version skews 2nd velocity versus Quality Indicators graph

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] ranking order reactivation rate, troubleshooting Reactivations graph recoverability, as QoS (quality of service) red line (triage) reference architectures regression testing regulatory compliance [See compliance.] regulatory concerns reliability, as QoS (quality of service) Remaining Work graph reporting bugs [See bugs.] reports available (project management) Actual Quality versus Planned Velocity graph Bug Rates graph Bugs by Priority graph Project Velocity graph Quality Indicators graph Reactivations graph Remaining Work graph Unplanned Work graph reports, defining required documentation, tacit knowledge versus requirements development quality and Kano Analysis personas actors versus researching QoS (qualities of service) scenarios customer validation of dissatisfiers in end-to-end story evolving exciters satisfiers storyboarding use cases versus user stories versus writing steps for specificity versus understandability tests against time frame for vision statements requirements analysis, scenario testing versus

researching personas resource leaks, troubleshooting responsiveness, as QoS (quality of service) retrospectives 2nd risk management 2nd risk testing risk, viewpoints of advocacy groups roles, job descriptions versus

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] sampling (performance profiling) Sarbanes-Oxley Act of 2002 (SOX) 2nd satisfiers Kano Analysis scalability, as QoS (quality of service) scenarios 2nd customer validation of dissatisfiers in end-to-end story evolving exciters Kano Analysis satisfiers states of storyboarding testing use cases versus user stories versus writing steps for scheduling time for unplanned work scope creep SCRUM, product backlog SDM (System Definition Model) security architecture and as QoS (quality of service) testing self-organization, prescribed organization versus service-oriented architecture [See SOA (service-oriented architecture).] shelvesets shelving (source control) smells SOA (service-oriented architecture) organizational boundaries and VSTS and soap opera testing SOAP, bug reporting analogy software [See projects.] source control, development quality and 2nd SOX (Sarbanes-Oxley Act of 2002) 2nd special cause variation specificity of requirements stack ranking stakeholders (iterative development) stale tests static analysis

storyboards strategic projects subjective data in bug reports supportability, as QoS (quality of service) System Definition Model (SDM) System Designer

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] tacit knowledge, required documentation versus tasks estimating [See estimating tasks.] troubleshooting TDD (Test-Driven Development) team builds team efficiency Team System, instrumenting daily activities technology adoption lifecycle templates, Process Template test lists (for BVTs), creating test run configurations, unit testing Test-Driven Development (TDD) testability, as QoS (quality of service) testing [See also bugs.] as discovery bottleneck in configuration testing exploratory testing false confidence in "good enough" testing load testing questions to answer with amount of testing to do automated testing change testing delivering customer value gaps in testing production environment qualities of service (QoS) team efficiency when to test regression testing risk testing scenario testing security testing TDD (Test-Driven Development) tests pass, solution doesn"t work unit testing development quality and 2nd troubleshooting value-up approach of Theory of Special Relativity (Einstein) time boxes top-down estimation tracking work

instrumenting daily activities work item databases transparency [See also tracking work.] development quality and 2nd of flow triage example iterations and red line triage committee, writing bug reports for troubleshooting with metrics warehouse development practices testing bottleneck tests pass, solution doesn"t work underestimation usefulness of Turner, Richard

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] underestimation understandability of requirements uninstallability, as QoS (quality of service) unit testing development quality and 2nd troubleshooting unmanaged code analysis Unplanned Work graph unplanned work, capacity to handle usability labs, validating scenarios in use cases, scenarios versus user experience, as QoS (quality of service) user interface tests user stories, scenarios versus

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] validating architecture scenarios value-up approach architecture and 2nd to development flow transparency work-down approach versus ideas in in testing work-down approach versus variable data sets, unit testing variance estimating tasks in project velocity graph variation velocity, quality versus version control [See source control, development quality and.] version skews, development quality and 2nd virtual machines, configuration testing on vision statements VSTS architecture in SOA (service-oriented architecture) and

Index [A] [B] [C] [D] [E] [F] [G] [H] [I] [J] [K] [L] [M] [O] [P] [Q] [R] [S] [T] [U] [V] [W] waterfall model web services, SOA and Web Services Description Language (WSDL) web tests, in scenario testing Weinberg, Gerald Windows Server System Reference Architecture (WSSRA) wireframes work item databases instrumenting daily activities work items 2nd associating with check-ins states of work-down approach flow versus value-up approach versus writing bug reports assessment data in objective data in plans in SOAP analogy subjective data in WSDL (Web Services Description Language) WSSRA (Windows Server System Reference Architecture)