Controlling Cost Growth of NASA Earth and Space Science Missions [1 ed.] 9780309157384, 9780309157377

Citation preview

Committee on Cost Growth in NASA Earth and Space Science Missions Space Studies Board

Copyright © 2010. National Academies Press. All rights reserved.

Division on Engineering and Physical Sciences

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

THE NATIONAL ACADEMIES PRESS 

500 Fifth Street, N.W. 

Washington, DC 20001

NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study is based on work supported by Contract NNH06CE15B between the National Academy of Sciences and the National Aeronautics and Space Administration. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the agency that provided support for the project. International Standard Book Number-13:  978-0-309-15737-7 International Standard Book Number-10:  0-309-15737-4 Cover: Cover design by Tim Warchocki. Copies of this report are available free of charge from: Space Studies Board National Research Council 500 Fifth Street, N.W. Washington, DC 20001 Additional copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Copyright 2010 by the National Academy of Sciences. All rights reserved.

Copyright © 2010. National Academies Press. All rights reserved.

Printed in the United States of America

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.

Copyright © 2010. National Academies Press. All rights reserved.

www.national-academies.org

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Other Reports of the Space Studies Board Defending Planet Earth: Near-Earth Object Surveys and Hazard Mitigation Strategies: Final Report (Space Studies Board [SSB] with the Aeronautics and Space Engineering Board [ASEB], 2010) An Enabling Foundation for NASA’s Space and Earth Science Missions (SSB, 2010) Revitalizing NASA’s Suborbital Program: Advancing Science, Driving Innovation, and Developing a Workforce (SSB, 2010) America’s Future in Space: Aligning the Civil Space Program with National Needs (SSB with ASEB, 2009) Approaches to Future Space Cooperation and Competition in a Globalizing World: Summary of a Workshop (SSB with ASEB, 2009) Assessment of Planetary Protection Requirements for Mars Sample Return Missions (SSB, 2009) Near-Earth Object Surveys and Hazard Mitigation Strategies: Interim Report (SSB, 2009) A Performance Assessment of NASA’s Heliophysics Program (SSB, 2009) Radioisotope Power Systems: An Imperative for Maintaining U.S. Leadership in Space Exploration (SSB with ASEB, 2009)

Copyright © 2010. National Academies Press. All rights reserved.

Ensuring the Climate Record from the NPOESS and GOES-R Spacecraft: Elements of a Strategy to Recover Measurement Capabilities Lost in Program Restructuring (SSB, 2008) Launching Science: Science Opportunities Provided by NASA’s Constellation System (SSB with ASEB, 2008) Opening New Frontiers in Space: Choices for the Next New Frontiers Announcement of Opportunity (SSB, 2008) Science Opportunities Enabled by NASA’s Constellation System: Interim Report (SSB with ASEB, 2008) Severe Space Weather Events—Understanding Societal and Economic Impacts: A Workshop Report (SSB, 2008) Space Science and the International Traffic in Arms Regulations: Summary of a Workshop (SSB, 2008) United States Civil Space Policy: Summary of a Workshop (SSB with ASEB, 2008) Assessment of the NASA Astrobiology Institute (SSB, 2007) An Astrobiology Strategy for the Exploration of Mars (SSB with the Board on Life Sciences [BLS], 2007) Building a Better NASA Workforce: Meeting the Workforce Needs for the National Vision for Space Exploration (SSB with ASEB, 2007) Decadal Science Strategy Surveys: Report of a Workshop (SSB, 2007) Earth Science and Applications from Space: National Imperatives for the Next Decade and Beyond (SSB, 2007) Exploring Organic Environments in the Solar System (SSB with the Board on Chemical Sciences and Technology, 2007) Grading NASA’s Solar System Exploration Program: A Midterm Review (SSB, 2007) The Limits of Organic Life in Planetary Systems (SSB with BLS, 2007) NASA’s Beyond Einstein Program: An Architecture for Implementation (SSB with the Board on Physics and Astronomy [BPA], 2007) Options to Ensure the Climate Record from the NPOESS and GOES-R Spacecraft: A Workshop Report (SSB, 2007) A Performance Assessment of NASA’s Astrophysics Program (SSB with BPA, 2007) Portals to the Universe: The NASA Astronomy Science Centers (SSB, 2007) The Scientific Context for Exploration of the Moon (SSB, 2007) Limited copies of SSB reports are available free of charge from: Space Studies Board National Research Council The Keck Center of the National Academies 500 Fifth Street, N.W., Washington, DC 20001 (202) 334-3477/[email protected] www.nationalacademies.org/ssb/ssb.html

iv

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

COMMITTEE ON COST GROWTH IN NASA EARTH AND SPACE SCIENCE MISSIONS RONALD M. SEGA, Colorado State University, Chair VASSILIS ANGELOPOULOS, University of California, Los Angeles ROBERT E. BITTEN,1 The Aerospace Corporation ALLAN V. BURMAN, Jefferson Consulting Group, LLC OLIVIER L. de WECK, Massachusetts Institute of Technology ROBERT E. DEEMER, Regis University LARRY W. ESPOSITO, University of Colorado, Boulder JOSEPH FULLER, JR., Futron Corporation JOSEPH W. HAMAKER, Science Applications International Corporation VICTORIA E. HAMILTON, Southwest Research Institute JOHN M. KLINEBERG, Aerospace Consultant ROBERT P. LIN,2 University of California, Berkeley BRUCE D. MARCUS, TRW Inc. (Retired) EMERY I. REEVES, Independent Consultant WILLIAM F. TOWNSEND, Independent Consultant Staff ALAN C. ANGLEMAN, Senior Program Officer, Study Director CATHERINE A. GRUBER, Editor ANDREA M. REBHOLZ, Program Associate LINDA WALKER, Senior Project Assistant

Copyright © 2010. National Academies Press. All rights reserved.

MICHAEL H. MOLONEY, Director

1Resigned 2Resigned

from committee on September 2, 2009. from committee on September 8, 2009.



National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

SPACE STUDIES BOARD CHARLES F. KENNEL, Scripps Institution of Oceanography, University of California, San Diego, Chair A. THOMAS YOUNG, Lockheed Martin Corporation (retired), Vice Chair DANIEL N. BAKER, University of Colorado STEVEN J. BATTEL, Battel Engineering CHARLES L. BENNETT, Johns Hopkins University YVONNE C. BRILL, Aerospace Consultant ELIZABETH R. CANTWELL, Oak Ridge National Laboratory ANDREW B. CHRISTENSEN, Dixie State College and Aerospace Corporation ALAN DRESSLER, The Observatories of the Carnegie Institution JACK D. FELLOWS, University Corporation for Atmospheric Research FIONA A. HARRISON, California Institute of Technology JOAN JOHNSON-FREESE, U.S. Naval War College KLAUS KEIL, University of Hawaii MOLLY K. MACAULEY, Resources for the Future BERRIEN MOORE III, Climate Central ROBERT T. PAPPALARDO, Jet Propulsion Laboratory, California Institute of Technology JAMES PAWELCZYK, Pennsylvania State University SOROOSH SOROOSHIAN, University of California, Irvine JOAN VERNIKOS, Thirdage LLC JOSEPH F. VEVERKA, Cornell University WARREN M. WASHINGTON, National Center for Atmospheric Research CHARLES E. WOODWARD, University of Minnesota ELLEN G. ZWEIBEL, University of Wisconsin

Copyright © 2010. National Academies Press. All rights reserved.

MICHAEL H. MOLONEY, Director (from April 1, 2010) RICHARD E. ROWBERG, Interim Director (until March 31, 2010) CARMELA J. CHAMBERLAIN, Administrative Coordinator TANJA PILZAK, Manager, Program Operations CELESTE A. NAYLOR, Information Management Associate CHRISTINA O. SHIPMAN, Financial Officer SANDRA WILSON, Financial Assistant

vi

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Preface

Cost and schedule growth is a problem experienced by many types of projects in many fields of endeavors. Based on prior studies of cost growth in National Aeronautics and Space Administration (NASA) and Department of Defense projects, this report identifies specific causes of cost growth associated with NASA Earth and space science missions and provides guidance on how NASA can overcome these specific problems. The study was prompted by the NASA Authorization Act of 2008 (P.L. 110-422), which directed the NASA administrator to sponsor an “independent external assessment to identify the primary causes of cost growth in the large-, medium-, and smallsized space and Earth science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.” The NASA Science Mission Directorate subsequently made arrangements with the National Research Council to conduct a study that would execute the following statement of task (see Appendix A): The National Research Council will assemble a committee to identify the primary causes of cost growth in NASA Earth and space science missions involving large, medium, and small spacecraft. The committee will recommend what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s Earth and space science programs. In particular, the committee will:

Copyright © 2010. National Academies Press. All rights reserved.

• Review existing cost growth studies related to NASA space and Earth science missions and identify their key

causes of cost growth and strategies for mitigating cost growth. • Assess whether those key causes remain applicable in the current environment and identify any new major causes. • Evaluate the effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities. In making this assessment and related recommendations, the committee should note relevant differences, if any, that exist between Earth and space science missions.

The recommendations in this report focus on changes in NASA policies that would directly reduce or eliminate the cost growth of Earth and space science missions. This report does not assess trends in the average cost of missions from year to year or decade to decade, nor does it explicitly address the broader issue of how missions are selected or how the baseline cost of NASA Earth and space science missions might be reduced, for example, by changes in general policies regarding export controls or management of NASA personnel, contracting, or center organization. The committee was not specifically tasked with addressing schedule growth but, as detailed vii

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

viii

PREFACE

Copyright © 2010. National Academies Press. All rights reserved.

in the report, schedule growth causes cost growth, and so schedule growth is addressed in some findings and recommendations. The Committee on Cost Growth in NASA Earth and Space Science Missions was established to conduct this study and has extensive experience in Earth science, space science, and space exploration, including management of industry and NASA centers, spacecraft operations, piloted and robotic spacecraft, spacecraft systems, NASA cost estimating, and federal procurement and acquisition processes (see Appendix B). The committee met four times, including a meeting in Washington, D.C., with extensive briefings from NASA and a meeting at the Jet Propulsion Laboratory in Pasadena, California, which included discussions with staff from the Applied Physics Laboratory of Johns Hopkins University, the Jet Propulsion Laboratory, NASA Goddard Space Flight Center, the U.S. Air Force, and industry (Ball Aerospace, Lockheed Martin, and Northrop Grumman). The findings and recommendations contained in this report are based as much on the experience and discernment of the individual committee members as on the contents of the earlier studies and other information collected and reviewed by the committee. Large cost growth is a concern for Earth and space science missions, and it can be a concern for other missions as well. If the cost growth is large enough, it can create liquidity problems for NASA’s Science Mission Directorate that in turn cause cost profile changes and development delays that amplify the overall cost growth for other concurrent and/or pending missions. Addressing cost growth through the allocation of artificially high reserves is an inefficient use of resources because it unnecessarily diminishes the portfolio of planned flights. The most efficient use of resources is to establish realistic budgets and reserves and effective management processes that maximize the likelihood that mission costs will not exceed reserves. NASA is already taking action to reduce cost growth; additional steps, as recommended herein, will help improve NASA’s mission planning process and achieve the goal of ensuring frequent mission opportunities for NASA Earth and space science.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Acknowledgment of Reviewers

This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council (NRC). The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:

Copyright © 2010. National Academies Press. All rights reserved.

James G. Anderson, Harvard University, Steven J. Battel, Battel Engineering, David A. Bearden, The Aerospace Corporation, Dale Jorgenson, Harvard University, Cato T. Laurencin, University of Connecticut, Marcia J. Rieke, University of Arizona, Christopher Russell, University of California, Los Angeles, James M. Russell III, Hampton University, and Gerald Joseph Wasserburg, California Institute of Technology (professor emeritus). Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by W. Carl Lineberger, University of Colorado, Boulder, and Edward F. Crawley, Massachusetts Institute of Technology. Appointed by the NRC, they were responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

ix

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Copyright © 2010. National Academies Press. All rights reserved. National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Contents

SUMMARY

1

1

SIZE AND HISTORIC CAUSES OF COST GROWTH Study Background, 8 Announcement of Opportunity and Directed Missions, 9 Size of Cost Growth, 10 Causes of Cost Growth, 25 Differences Between Earth and Space Science Missions, 28

8

2

KEY PROBLEMS AND SOLUTIONS Cost Realism, 30 Development Process, 33 Comprehensive, Integrated Strategy for Cost and Schedule Control, 40

REFERENCES

30

44

Copyright © 2010. National Academies Press. All rights reserved.

APPENDIXES A B C D

Statement of Task and Supporting Documents Biographies of Committee Members and Staff Findings and Recommendations from Primary References Acronyms and Abbreviations

xi

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

49 53 58 62

Copyright © 2010. National Academies Press. All rights reserved. National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Summary

Study background Cost growth in Earth and space science missions conducted by the Science Mission Directorate (SMD) of the National Aeronautics and Space Administration (NASA) is a longstanding problem with a wide variety of interrelated causes. To address this concern, the NASA Authorization Act of 2008 (P.L. 110-422) directed the NASA administrator to sponsor an “independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized Earth and space science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.” NASA subsequently requested that the National Research Council (NRC) conduct a study to:

Copyright © 2010. National Academies Press. All rights reserved.

• Review the body of existing studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth; • Assess whether those key causes remain applicable in the current environment and identify any new major causes; and • Evaluate effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities. As part of this effort, NASA also asked the NRC to “note what differences, if any, exist with regard to Earth science compared with space science missions.” Cost Growth—Magnitude and Causes NASA identified 10 cost studies and related analyses that this study uses as its primary references (listed in the References chapter and in Table 1.1). The committee generally concurs with the consensus viewpoints expressed in these studies as a whole, but in some areas, the studies reached different conclusions. For example, the prior studies calculated values for average cost growth ranging from 23 percent to 77 percent. Different studies reach different conclusions because they examine different sets of missions and calculate cost growth based on different criteria. By definition, cost growth is a relative measure reflecting comparison of an initial estimate of mission costs against costs actually incurred at a later time. But studies use initial estimates made at different points in 

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

mission life cycles (see Figure S.1), as well as cost estimates that cover different phases of mission life cycles. For example, some studies consider only development costs (up to but not including launch), but other studies consider all costs through the end of each mission. In general, the earlier the initial estimate, the more the cost will grow. In addition, including a larger share of the later phases of a mission (such as launch, operations, and analysis of data collected by a mission) increases the total cost assigned to each mission and the absolute value of the cost growth (in dollars). These differences make it very difficult to derive a single, reliable value for the average cost growth of NASA Earth and space science missions on the basis of previous studies. The primary references also indicate that most cost growth occurs after critical design review. This implies that the required level of cost reserves remains substantial, even late in the development process. In addition, a relatively small number of missions cause most of the total cost growth. For one large set of 40 missions, 92 percent of the total cost growth (in dollars) was caused by only 14 missions (one-third of the total number). Conversely, the 26 missions with the least cost growth (two-thirds of the total number) accounted for only 8 percent of the total cost growth (see Figure S.2). The primary references identify a wide range of factors that contribute to cost and schedule growth of NASA Earth and space science missions. The most commonly identified factors are the following: • • • •

Overly optimistic and unrealistic initial cost estimates, Project instability and funding issues, Problems with development of instruments and other spacecraft technology, and Launch service issues.

Additional factors identified in the primary references include schedule growth that leads to cost growth. Schedule growth and cost growth are well correlated because any problem that causes schedule growth contributes to and magnifies total mission cost growth. Furthermore, cost growth in one mission may induce organizational replanning that delays other missions in earlier stages of implementation, further amplifying overall cost growth. Effective implementation of a comprehensive, integrated cost containment strategy, as recommended herein, is the best way to address this problem.

Approval NASA LifeCycle Phases

Copyright © 2010. National Academies Press. All rights reserved.

Project LifeCycle Phases

Formulation Pre-Phase A: Concept Studies

Selected Mission Reviews

Phase A: Concept and Technology Development

System Requirements Review

Implementation Phase B: Preliminary Design and Technology Completion

Phase C: Final Design and Fabrication

Phase D: System Assembly, Integration, Test, and Launch

Phase E: Operations and Sustainment

Preliminary Critical Design Design Review Review

Phase F: Closeout

End of Mission

FIGURE S.1  NASA mission life cycle. SOURCE: Based on NASA Procedural Requirements 7120.5D (NASA, 2007).

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



SUMMARY

$950M $900M $850M $800M

These 14 missions together account for 92% of the total cost growth for all 40 missions in this figure

Initial cost—directed missions Initial cost—AO missions Cost growth—14 missions with most cost growth Cost growth—26 missions with least cost growth

Initial Cost Estimate / Absolute Cost Growth

$750M $700M $650M $600M $550M $500M $450M $400M $350M

These 26 missions together account for just 8% of the total cost growth for all 40 missions in this figure

$300M $250M $200M $150M $100M $50M $0M 2 00 ,2 ua Aq S- 997 EO , 1 996 E 1 AC R, 98 EA 19 7 N S, 99 96 G , 1 99 19 8 M M 19 er, 199 M st, d r, TR rdu hfin cto a at e St s P osp ar Pr 98 M ar 19 7 n , 0 Lu E 20 AC S, 0 TR MI 200 E , TH GE 01 A 0 00 IM P, 2 20 98 A II, 19 M E- L, 3 ET P 0 H /M 20 O C E, M RC 996 1 02 SO T, 20 S E, FA AC 999 R 1 G E, 98 IR 9 W 1, 1 999 S- 1 02 D E, 20 S I, 02 FU SS 20 1 E , H r 0 R tou , 20 on is 3 C es 200 en , 8 G EX 99 AL , 1 1 G AS 200 04 , 20 SW D , a E M ur TI -A 05 S 0 3 5 0 EO , 2 20 200 O R T, t, M SA pac 99 E m 19 IC p I -7, ee t 4 6 D sa 00 00 nd , 2 , 2 La IFT AT 6 S 0 D 0 SW U , 2 LO O C E 0 ER 200 ST 1, 03 004 - 0 2 2 EO R, er, 06 004 E g 0 2 M sen , 2 B, es SO e M LIP rob A P 3 C vity 00 ra , 2 G TF R SI

-$50M

FIGURE S.2 Ranking of 40 NASA science missions in terms of absolute cost growth in excess of reserves in millions of dollars, excluding launch, mission operations, and data analysis, with initial cost and launch date for each mission also shown. NOTE: Acronyms are defined in Appendix D. SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009.

Copyright © 2010. National Academies Press. All rights reserved.

Comprehensive, integrated Strategy for Cost and Schedule Control NASA sets the strategic direction of its Earth and space science programs using decadal surveys, the SMD science plan, and supporting road maps. A comprehensive, integrated approach to cost and schedule growth is also essential. The primary references identify dozens of specific causes, make dozens of specific recommendations, and include dozens of additional findings concerning cost growth. The primary references, as a whole, are generally consistent and comprehensive, and so the individual causes of cost growth and the necessary corrective actions are not a mystery. However, rather than simply picking and choosing from among the many suggested causes, findings, and recommendations, development of a comprehensive, integrated strategy offers the best chance that future actions will work in concert to minimize or eliminate cost and schedule growth. An effective strategy would substantially reduce cost growth (beyond reserves) on individual missions and programs so that whatever growth does occur is offset by other missions and programs completed for less than the budgeted amount. This approach would allow NASA to execute the Earth and space science mission portfolio for the appropriated budget. Achieving this goal will require NASA to address both internal and external factors. Internally, a comprehensive, integrated cost containment strategy would improve the definition of baseline

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

costs and enhance the utility of NASA’s independent cost-estimating capabilities. Early development of technologies and more effective program reviews would improve the ability to identify and effectively manage risks and uncertainties. Externally, NASA has the opportunity to collaborate with other federal agencies, the Office of Management and Budget, and Congress to sustain and improve critical capabilities and expertise in the industrial base and the nation’s science and engineering workforce; to address cost and schedule risk associated with launch vehicles; and to improve funding stability. Successful implementation of a comprehensive, integrated strategy to control cost and schedule growth of NASA Earth and space science missions would benefit both NASA and the nation, while enabling NASA to more efficiently and effectively carry out these critical missions.  inding. Comprehensive, Integrated Cost Containment Strategy. Recent changes by NASA in the developF ment and management of Earth and space science missions are promising. These changes include budgeting programs to the 70 percent confidence level and specifying that decadal surveys include independent cost estimates. However, it is too early to assess the effectiveness of these actions, and NASA has not taken the important step of developing a comprehensive, integrated strategy.  ecommendation. Comprehensive, Integrated Cost Containment Strategy. NASA should develop a comR prehensive, integrated strategy to contain cost and schedule growth and enable more frequent science opportunities. This strategy should include recent changes that NASA has already implemented as well as other actions recommended in this report. Key Problems In addition to developing a comprehensive, integrated cost containment strategy, and as detailed below, NASA should address specific issues related to cost realism and the development process for Earth and space science missions. Cost Realism

Copyright © 2010. National Academies Press. All rights reserved.

Cost Estimates NASA project staff generally estimate mission costs using detailed engineering analyses of labor and material requirements, vendor quotes, subcontractor bids, and the like. Non-advocate independent cost estimates in NASA are generally parametric cost estimates using statistical cost-estimating relationships based on historical relationships among cost and technical and programmatic variables (mass, power, complexity, and so on). In both cases, mission cost estimates are created by summing costs at lower levels of a project’s work breakdown structure to obtain total project costs. Parametric cost models rely on observations rather than opinion, are an excellent tool for answering “what-if” questions quickly, and provide statistically sound information about the confidence level of cost estimates. In contrast, the process used within NASA to generate cost estimates on the basis of detailed engineering assessments does not provide a statistical confidence level and, in retrospect, has generally been less accurate than parametric cost models in estimating the cost of NASA Earth and space science missions.  A project manager or principal investigator who is personally determined to control costs can be of great assistance in avoiding cost growth. People and organizations tend to optimize their behavior based on the environment in which they operate. Unfortunately, instead of motivating and rewarding vigilance in accurately predicting and controlling costs, the current system incentivizes overly optimistic expectations regarding cost and schedule.   If

programs are budgeted at the 70 percent confidence level, there is a 70 percent probability that all of the missions included in the program can be completed without exceeding mission and program reserves.   A detailed list of the strengths and weaknesses of various cost-estimating methods appears in 2008 NASA Cost Estimating Handbook. Washington, D.C.: NASA.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



SUMMARY

For example, competitive pressures encourage (overly) optimistic assessments of the cost and schedule impacts of addressing uncertainties and overcoming potential problems. As a result, initial cost estimates generally are quite optimistic, underestimating final costs by a sizable amount, and that optimism sometimes persists well into the development process.  ecommendation. Independent Cost Estimates. NASA should strengthen the role of its independent costR estimating function by • Expanding and improving NASA’s ability to conduct parametric cost estimates, and  • Obtaining independent parametric cost estimates at critical design review (in addition to system requirements review and preliminary design review), comparing them to other estimates available from the project and reconciling significant differences. Cost Growth Methodology The measurement of cost growth has been inconsistent across programs, NASA centers, and Congress. The Government Accountability Office and Congress generally consider the baseline to be the first time a mission appears as a budget line item in an appropriations bill, which is often before preliminary design review. The contents of NASA estimates also differ—some estimates include Phase A and B, some start with Phase C, some (but not all) include launch costs and/or mission operations, and some include NASA oversight and internal project management costs. These differences make it difficult to develop a clear understanding of trends in cost and schedule growth.  ecommendation. Measurement of Cost Growth. NASA, Congress, and the Office of Management and R Budget should consistently use the same method to quantify and report cost. In particular, they should use as the baseline a life-cycle cost estimate (that goes through the completion of prime mission operations) produced at preliminary design review. Development Process

Copyright © 2010. National Academies Press. All rights reserved.

Management of Announcement of Opportunity Missions and Directed Missions NASA implements two separate and distinct classes of Earth and space science missionsannouncement of opportunity (AO) missions and directed missions. NASA headquarters competitively selects AO missions from proposals submitted in response to periodic AOs by teams led by a principal investigator (PI), who is commonly affiliated with a university but may work in industry or for NASA. NASA headquarters determines the scientific goals and requirements for directed missions, which are sometimes referred to as facility class missions or flagship missions. Headquarters then directs a particular NASA center, usually Goddard Space Flight Center or the Jet Propulsion Laboratory, to implement the mission. The differing nature and goals of directed and AO missions call for different management approaches. AO missions are on average much smaller than directed missions are, and the impact of cost growth in AO missions, which are managed within a mission budget line (e.g., Discovery), is limited to other missions within the line. Flagship missions, however, are typically much larger than AO missions are, and so cost growth in these missions has a much greater potential to diminish NASA’s Earth and space science enterprise as a whole.  ecommendation. Management of Large, Directed Missions. NASA headquarters’ project oversight funcR tion should pay particular attention to the cost and schedule of its larger missions (total cost on the order of $500 million or more), especially directed missions (which form a single line item).

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

 ecommendation. Management of Announcement of Opportunity (AO) Missions. NASA should continue R to emphasize science in the AO mission selection process, while revising the AO mission selection process to allocate a larger percentage of project funds for risk reduction and improved cost estimation prior to final selection.  ecommendation. Incentives. NASA should ensure that proposal selection and project management processes R include incentives for program managers, project managers, and principal investigators to establish realistic cost estimates and minimize or avoid cost growth at every phase of the mission life cycle, for both directed missions and announcement of opportunity missions. Technology and Instrument Development NASA Procedural Requirements (NPR) 7120.5, NASA Space Flight Program and Project Management Requirements, requires that “during formulation, the project establishes performance metrics, explores the full range of implementation options, defines an affordable project concept to meet requirements specified in the Program Plan, develops needed technologies, and develops and documents the project plan” (NASA, 2007, Section 2.3.4). However, despite these requirements, the primary references identify an ongoing need to improve technical and programmatic definition at the beginning of a project. The limited time and resources typically available in phases A and B to mature new technology and solidify system design parameters contribute to cost growth through higher risk and unrealistic cost estimates. Instrument technology is particularly important because Earth and space science missions generally require special-purpose, one-of-a-kind components. Delays and cost increases for instrument development are pervasive and impact a large number of missions. This problem is exacerbated by shrinkage of the U.S. industrial base that supports space system development.  ecommendation. Technology Development. NASA should increase the emphasis in phases A and B on R technology development, risk reduction, and realism of cost estimates.  ecommendation. Instrument Development. NASA should initiate instrument development well in advance R of starting other project elements and establish a robust instrument technology development effort relevant to all classes of Earth and space science missions to strengthen and sustain the nation’s instrument development capability.  ecommendation. Decadal Surveys. NASA should ensure that guidance regarding the development of instruR ments and other technologies is included in decadal surveys and other strategic planning efforts. In particular, future decadal surveys should prioritize science mission areas that could be addressed by future announcements of opportunity and the instruments needed to carry out those missions.

Copyright © 2010. National Academies Press. All rights reserved.

Major Reviews NASA has increased the size and number of external project reviews to the point that some reviews are counterproductive and disruptive, especially for small missions. Large numbers of reviews diffuse responsibility and accountability, creating an environment where NASA senior managers can become dependent on review teams with many outside members who sometimes do not understand NASA, the field center in question, and/or the mission being reviewed. In addition, major reviews are sometimes conducted as scheduled even though a project may not have progressed as rapidly as expected and, as a result, cannot achieve the intended review criteria, programmatically and/or technologically.   General

preliminary design review and critical design review readiness criteria exist within NPR 7120.5D (NASA, 2007). More detailed criteria are provided in center directives such as Criteria for Flight Project Critical Milestone Reviews (NASA/GSFC, 2009).

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,



SUMMARY

 ecommendation. External Project Reviews. NASA should reassess its approach to external project reviews R to ensure that (1) the value added by each review outweighs the cost (in time and resources) that it places on projects; (2) the number and the size of reviews are appropriate given the size of the project; and (3) major reviews, such as preliminary design review and critical design review, occur only when specified success criteria are likely to be met. Launch Vehicles Problems with the procurement of launch vehicles and launch services are a significant source of cost growth. Specific factors include increases in the cost of expendable launch vehicles, vendor issues such as strikes, weatherrelated issues at the launch site, problems with launch-site-facility capabilities, and delays in the availability of a given launch vehicle. In addition, if a mission is required to change launch vehicles, the costs can be substantial.  ecommendation. Launch Vehicles. Prior to preliminary design review, NASA should minimize missionR unique launch site processing requirements. NASA should also select the launch vehicle with appropriate margins as early as possible and minimize changes in launch vehicles. Differences Between Earth and Space Science Missions

Copyright © 2010. National Academies Press. All rights reserved.

Different classes of missions face different challenges. Earth science missions typically have more complex, more costly, and more massive instruments than do space science missions, because Earth science missions also have more stringent requirements in terms of pointing accuracy, resolution, stability, and so on, although astro­ physics missions also have stringent pointing requirements, and planetary spacecraft and instrument technology must be able to survive long cruise phases and radiation environments that are sometimes quite extreme. Space science missions that leave Earth orbit have greater incentives to minimize spacecraft mass and power, and the average cost and average spacecraft mass of these missions are lower than those for Earth science missions. However, the size of the cost growth of Earth and space science missions has been comparable. Both Earth and space science missions have shown good correlation between (1) instrument schedule growth and instrument cost growth, (2) instrument cost/schedule growth and mission cost/schedule growth, and (3) the absolute costs of instruments and instrument complexity.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

1 Size and Historic Causes of Cost Growth

Study Background

Copyright © 2010. National Academies Press. All rights reserved.

The Science Mission Directorate (SMD) of the National Aeronautics and Space Administration (NASA) conducts both Earth and space science missions. The latter type encompasses missions in planetary sciences, heliophysics, and astrophysics. Cost growth in NASA Earth and space science missions is a longstanding problem with a wide variety of interrelated internal and external causes, both technical and programmatic. Many different organizations, both public and private, have examined the cost growth of NASA Earth and space science missions,  but the results of these studies have not been assessed as a whole. In response to this situation, the NASA Authorization Act of 2008 (P.L. 110-422) directed the NASA administrator to sponsor an “independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized Earth and space science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.” NASA subsequently requested that the National Research Council (NRC) conduct a study to: • Review the body of existing studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth; • Assess whether those key causes remain applicable in the current environment and identify any new major causes; and • Evaluate effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities. The Committee on Cost Growth in NASA Earth and Space Science Missions was established to conduct this study. As part of this effort, NASA also asked the NRC to “note what differences, if any, exist with regard to Earth science compared with space science missions.” NASA identified a list of relevant cost studies and related analyses to use as primary references for the study (see Table 1.1).

  In

general, this report refers to NASA flight projects as missions. NASA programs are generally collections of projects or missions with similar objectives, although in some cases a program has only one flight project/mission.



National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

SIZE AND HISTORIC CAUSES OF COST GROWTH



TABLE 1.1  Primary References Provided to the Study Committee NASA Cost Studies  1 R.E. Bitten, D.L. Emmons, and C.W. Freaner. 2006. Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines. IEEE Paper #1545. December.  2

NASA Langley Research Center (LaRC). 2007. Cost/Schedule Performance Study for the Science Mission Directorate. Final Report. Prepared by the NASA LaRC Science Support Office and Science Applications International Corporation. NASA LaRC, Hampton, Va. October.

 3

NASA. 2007. “Cost and Schedule Growth at NASA.” Presentation provided to the committee by Director of the Cost Analysis Division Tom Coonce, Office of Program Analysis and Evaluation, NASA, Washington, D.C. November.

 4

NASA. 2008. “SMD Cost/Schedule Performance StudySummary Overview.” Presentation by B. Perry and C. Bruno, NASA Science Support Office; M. Jacobs, M. Doyle, S. Hayes, M. Stancati, W. Richie, and J. Rogers, Science Applications International Corporation. January.

 5

C.W. Freaner, R.E. Bitten, D.A. Bearden, and D.L. Emmons. 2008. “An Assessment of the Inherent Optimism in Early Conceptual Designs and Its Effect on Cost and Schedule Growth.” Paper presented at the Space Systems Cost Analysis Group/Cost Analysis and Forecasting/European Aerospace Cost Engineering Working Group 2008 Joint International Conference, European Space Research and Technology Centre, Noordwijk, The Netherlands, May 15-16. European Space Agency, Paris, France.

 6

B. Mlynczak and B. Perry, Science Support Office, NASA. 2009. “SMD Earth and Space Mission Cost Driver Comparison Study. Final Report and Presentation.” March.

Copyright © 2010. National Academies Press. All rights reserved.

Related Analyses  7

General Accounting Office. 1992. Space Missions Require Substantially More Funding Than Initially Estimated. GAO/ NSIAD-93-97. Washington, D.C. December.

 8

National Research Council. 1997. Reducing the Costs of Space Science Research Missions: Proceedings of a Workshop. National Academy Press, Washington, D.C.

 9

J. McCrillis, Office of the Secretary of Defense, Cost Analysis Improvement Group. 2003. “Cost Growth of Major Defense Programs.” Presentation to the Annual Department of Defense Cost Analysis Symposium, January 30, Williamsburg, Va.

10

Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. 2003. Report of the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force on Acquisition of National Security Space Programs. Department of Defense, Washington, D.C. May.

The committee generally concurs with the consensus viewpoints expressed in the primary references as a whole. However, as detailed below, in some areas the prior studies reached different conclusions. This chapter describes the size and historic causes of cost growth in NASA Earth and space science missions, based primarily on the above references. As an illustrative example, this chapter also describes the results of a detailed assessment of cost and schedule data for the 40 missions included in Primary Reference 1.  Chapter 2 describes the committee’s own assessment of key problems, as they currently exist, and recommended solutions. A full copy of the study statement of task, the tasking letter from NASA, and the legislation that prompted this study appear in Appendix A. The expertise of the study committee is summarized in Appendix B. The findings and recommendations contained in the primary references are summarized in Appendix C. Announcement of Opportunity and Directed Missions NASA implements two separate and distinct classes of Earth and space science missions: announcement of opportunity (AO) missions, such as the Mars Exploration Rover mission, and directed missions, such as the Hubble Space Telescope mission. NASA headquarters competitively selects AO missions from proposals submitted in response to periodic AOs by teams led by a principal investigator, who is commonly affiliated with a university   Primary

Reference 1 was selected because it assesses more NASA missions than any of the other primary references. (See Table 1.2.)

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

10

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

but may work in industry or for NASA. These teams seek to address one or more scientific questions within the scope of a particular AO. Teams commonly include a NASA center, such as the Goddard Space Flight Center (GSFC) or the Jet Propulsion Laboratory (JPL), as well as an industrial partner. NASA then uses a moderately complex peer review process to evaluate these “Step 1” proposals. This typically leads to the selection of two or three proposals for further development in what is referred to as Phase A. Phase A concludes with the submission of “Step 2” proposals, from which NASA selects a single mission for implementation. AO missions may carry one or several instruments, and they typically cost from $100 million to several hundred million dollars. Only rarely does the cost exceed $1 billion. NASA headquarters determines the scientific goals and requirements for directed missions, which are sometimes referred to as facility class missions or flagship missions. Headquarters then directs a particular NASA center, usually GSFC or JPL, to implement the mission. The spacecraft for directed missions usually constitute large scientific facilities that produce a variety of data using multiple instruments, which may be quite large. Some of the instruments may be selected using an AO process. NASA headquarters establishes a science working group for directed missions to guide the science aspects of the missions as they move through each phase of development and into operations. Directed missions may cost one to several billion dollars. Size of Cost Growth Finding. Size of Cost Growth. Historic studies of cost growth indicate the following:  • Past studies of cost growth in NASA Earth and space science missions calculated values for average cost growth ranging from 23 percent to 77 percent.  • Different studies reach different conclusions because they examine different sets of missions and because of differences in how cost growth is calculated.  • Relatively little cost growth occurs between preliminary design review (PDR) and critical design review (CDR). A majority of cost growth occurs after CDR, with the rest occurring prior to PDR.  • For one large set of 40 missions, 80 percent of the total cost growth (in absolute dollar terms) was caused by only 11 missions. • The size of the cost growth of Earth and space science missions has been comparable.

Copyright © 2010. National Academies Press. All rights reserved.

Range of Cost Growth The 10 primary references examined by this study (listed in Table 1.1) address cost growth in both NASA science missions (Primary References 1 to 8) and Department of Defense (DOD) programs (Primary References 9 and 10). Table 1.2 compares the average cost and schedule growth reported by some of these studies. While the lowest reported average cost growth in the 10 primary references was 23 percent, the maximum was 77 percent. The average across all 10 was 51 percent. The primary references confirmed that extensive cost growth exists in many—but not all—NASA Earth and space science missions, but the extent of the cost growth differed significantly from one mission to the next, and from one study of cost growth to the next. However, regardless of the wide range in cost growth reported by the various studies, unwanted cost and schedule growth have certainly made it more difficult to accomplish NASA’s Earth and space science missions than was originally anticipated when these missions were authorized.

  Here

and in the sections that follow, each section begins with a finding and/or recommendation, which is then explained and justified by the text that follows.   Changes in technical or mission requirements increase costs, but this is not always a problem that needs to be fixed. In some cases a deliberate decision is made to increase mission capabilities or otherwise add value to a mission despite the extra cost involved. One way in which successful projects incur extra costs is through extended mission operations. However, unplanned cost and schedule growth, which unexpectedly increase costs without adding value, are always unwelcome.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

11

SIZE AND HISTORIC CAUSES OF COST GROWTH

TABLE 1.2  Average Reported Cost and Schedule Growth of Past NASA and Department of Defense Missions Primary Reference

Missions

Number of Missions or Programs

Average Cost Growth (%)

1 2, 4 3 5 7 9

NASA NASA NASA NASA NASA DOD

40 15 25 10 29 142



27 23 68 76 77a 32

Average Schedule Growth 22 13 56 36

% months % %

NOTE: Primary References 2 and 4 examined the same data. Primary References 6, 8, and 10 did not calculate average cost or schedule growth. Primary Reference 7 concluded that (1) cost increased by more than 50 percent for three-fourths of the missions examined and (2) cost increased by more than 100 percent for one-third of the missions examined. a The 77 percent cost growth in Primary Reference 7 represents the median cost growth observed.

Different Studies—Different Results It is difficult to reconcile fully the different values of cost growth identified in the primary references. The committee identified two primary reasons for these differences:

Copyright © 2010. National Academies Press. All rights reserved.

• Differences in how cost growth is calculated, and • Differences in the sets of missions and programs examined. By its very definition, cost growth is a relative measure comparing an initial estimate of mission costs against actually incurred costs at a later time. The times at which the initial and final cost estimates are made differ across studies. Figure 1.1 shows the sequence of program phases, key decision points (KDPs), and key mission reviews for a typical NASA mission. Figure 1.1 also shows how three different references calculate cost growth. Primary Reference 1 only considers growth in phases A through D, except for launch. Primary Reference 2 assesses cost growth from the beginning of Phase B (KDP B) through end of mission. Primary Reference 7, on the other hand, measures cost growth from the first time a mission appears as a line item in the NASA budget submitted to Congress until the completion of the mission (end of Phase F). In general, the earlier the initial estimate, the more cost growth will occur. In addition, including more of the later phases (such as launch, operations, and data analysis) in the cost growth assessment increases the total cost assigned to each mission and the absolute value of the cost growth (in dollars). In particular, a key reason that Primary References 1, 2, and 4 show an average cost growth of 23 to 27 percent, whereas Primary Reference 7 shows a 77 percent average cost growth, is that the start and end points of the cost growth measurements made by Primary Reference 7 are very different from those for Primary References 1, 2, and 4. These differences make it very difficult to derive a single, reliable value for the average cost growth of NASA Earth and space science missions based on previous studies. Cost growth estimates can also be distorted by missions with exceptionally long development schedules, by changes in mission scope, and by the cost of extended mission operations. For example, Gravity Probe B was under continuous development for 40 years before it was launched in 2004 (NRC, 1995). The Gravity Probe B cost and schedule growth data used by Primary Reference 1 assume that development began in fiscal year 1994, although NASA had previously expended $128 million developing mission concepts and technology.  Costs are reduced when a mission is descoped, but the value of the mission is also reduced. Conversely, the extra value obtained from an increase in mission scope or an extension of mission operations is presumably worth the cost, or else the broader scope or extended operations would not be approved. In both cases, programmatic changes obscure the extent of cost growth that the missions would have experienced if they had been implemented as originally planned. For example, Aqua, an Earth Observing System (EOS) mission formerly known as EOS-PM,   Tom

Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

12

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

Approval NASA LifeCycle Phases

Project LifeCycle Phases

Formulation Pre-Phase A: Concept Studies

Key Decision Points (KDPs)

Phase A: Concept and Technology Development

KDP A

Implementation

Phase B: Preliminary Design and Technology Completion

KDP B

Phase C: Final Design and Fabrication

KDP C

Phase D: System Assembly, Integration, Test, and Launch

KDP D

Phase E: Operations and Sustainment

KDP E

Phase F: Closeout

KDP F

Mission Reviews MCR Ref. 1 (Sep 2006) Ref. 2 (Oct 2007) Ref. 7 (Dec 1992)

SRR MDR

PDR

CDR

SIR

TRR

FRR

DR

Cost growth only for phases A through D, excluding launch

Initial budget submitted Start of Phase B Initial estimate provided in NASA Budget to Congress

MCR Mission Concept Review PDR Preliminary Design Review TRR Test Readiness Review

End of Mission Only missions with development cost > $200M included

End of Mission

SRR System Requirements Review MDR Mission Design Review CDR Critical Design Review SIR System Integration Review FRR Flight Readiness Review DR Decommissioning Review

Copyright © 2010. National Academies Press. All rights reserved.

FIGURE 1.1 Different studies calculate cost growth over different portions of a mission’s total life cycle. Based in part on NASA Procedural Requirements 7120.5D.

was significantly descoped in 1992 when the EOS budget was reduced by NASA. Primary Reference 2 reports that Aqua cost growth is 7 percent (based on changes from CDR through launch). Primary Reference 6, however, reports that Aqua cost growth is 23 percent (based on changes during phases B, C, and D). Similar discrepancies are reported for the Spitzer Space Telescope mission (formerly known as the Space Infrared Telescope Facility, SIRTF), which also was significantly descoped. SIRTF cost growth during phases B though D is reported as 68 percent in Primary Reference 2 and 52 percent in Primary Reference 6. Cost growth for Chandra (formerly known as the Advanced X-ray Astrophysics Facility) is reported as 5 percent in Primary Reference 2 (for phases B through D), as negative 2 percent in Primary Reference 6 (for phases B through D), and as 194 percent in Primary Reference 7 (from project initiation through the current state of development when that report was completed in 1992; Chandra was subsequently descoped). These examples suggest that extreme care must be taken when comparing cost growth as measured by different studies, and they highlight the need for common metrics. Another key complicating factor is that different studies assess different sets of missions and programs. As seen in Table 1.2, the primary references that assessed NASA’s performance examined between 10 and 40 missions. Primary Reference 9 examined 142 DOD projects covering a wide range of applications; the vast majority were not space launch missions. Table 1.3 lists the missions assessed by each of the primary references that examined NASA missions.  Note   Primary

Reference 8 focused on NASA, but it did not assess the cost growth of individual missions.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

SIZE AND HISTORIC CAUSES OF COST GROWTH

13

that Primary Reference 7 examined some missions, such as the Advanced Communications Technology Satellite, that are not Earth or space science missions. As shown in Table 1.3, some missions are only discussed in one study while others are included in multiple studies. For example, there is a core group of 12 missions that are included in Primary References 1, 2, 4, and 6. These missions are listed below and offer the best chance for a more in-depth comparison of cost growth calculations across different studies: • • • • • • • • • • • •

Messenger (Mercury Orbiter), Cloud Satellite (CLOUDSAT), Galaxy Evolution Explorer (GALEX), Swift Gamma-Ray Burst Mission, Spitzer Space Telescope (formerly SIRTF), Mars Reconnaissance Orbiter (MRO), Solar Terrestrial Relations Observer (STEREO), Comet Nucleus Tour (CONTOUR), Deep Impact, EOS-Aqua, Mars Exploration Rover (MER), and Reuven Ramaty High Energy Solar Spectral Imager (RHESSI).

Seven of these 12 are also included in Primary Reference 5. Accordingly, the cost growth experience of these 12 missions is overemphasized in any simple averaging of the results of the primary references. On the other hand, Primary References 1, 3, and 7 assess a large number of missions that are not included in any of the other primary references, and this undoubtedly contributes to the large variance in average cost growth determined by the primary references. Cost Growth Versus Schedule Growth To assess the overall cost growth of past missions, it is helpful to consider how missions cluster in terms of their relative cost and schedule growth. Figure 1.2 shows the distribution of the 40 missions examined by Primary Reference 1 in terms of development cost and schedule growth. Missions are grouped as Type I, II, and III based on the cost growth thresholds defined in the NASA Authorization Act of 2005: 

Copyright © 2010. National Academies Press. All rights reserved.

• Type I, minor or no cost growth (less than 15 percent), • Type II, significant cost growth (15 to 30 percent), and • Type III, excessive cost growth (more than 30 percent). Of the 40 missions in Primary Reference 1, Figure 1.2 shows that 19 missions (roughly half) are Type I, 8 missions (one-fifth) are Type 2, and 13 missions (one-third) are Type III.  In other words, half of the Earth and space science missions in this large sample were completed within the 15 percent cost growth target needed to avoid congressional notification.

  The NASA Authorization Act of 2005 requires congressional notifications for cost growth of 15 percent or more during phases B and C or schedule delays of 6 months or more. Reauthorization is required for cost growth of 30 percent or more or $1 billion or more during phases B and C. These reporting requirements do not apply to all of the missions plotted in Figure 1.2; per statute, they only apply to programs with an estimated life-cycle cost greater than $100 million, and some missions in Figure 1.2 occurred prior to fiscal year 2005.   Many of the figures and tables in this report depict data from Primary Reference 1, which were used for these illustrative examples because those data assessed more missions than any of the other studies that examined NASA missions, and because the committee was able to obtain the detailed data used by the authors of Primary Reference 1 in order to conduct the committee’s own supplemental analyses.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

14

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE 1.3  NASA Missions Examined by the Primary References

Copyright © 2010. National Academies Press. All rights reserved.

Primary Reference Launch Year

1

2

Discovery

2004

X

Earth System Science Pathfinder

2006

X

GALEX

Explorer

2003

Swift Gamma-Ray Burst Mission

Explorer/Medium-class Explorer

Spitzer Space Telescope (formerly SIRTF)

Great Observatory

MRO STEREO

Mission Name

Program/Class

Messenger CLOUDSAT

4

5

6

X

X

X

X

X

X

X

X

X

X

X

X

X

2004

X

X

X

X

X

2003

X

X

X

X

X

Mars

2005

X

X

X

X

X

Solar Terrestrial Probe

2006

X

X

X

X

X

CONTOUR

Discovery

2002a

X

X

X

X

Deep Impact

Discovery

2005

X

X

X

X

EOS-Aqua

Earth Observing System

2002

X

X

X

X

MER

Mars

2003

X

X

X

X

RHESSI

Small Explorer

2002

X

X

X

EO-1

New Millennium

2000

X

EOS-Aura

Earth Observing System

2004

X

X

CALIPSO

Earth System Science Pathfinder

2006

X

X

GRACE

Earth System Science Pathfinder

2002

X

X

THEMIS

Explorer/Medium-class Explorer

2007

X

X

Landsat-7

Landsat Program

1999

X

X

FAST

Small Explorer

1996

X

X

TRACE

Small Explorer

1998

X

X

TRMM

Mission to Planet Earth

1997

X

Gravity Probe B

Astrophysics

2004

X

Genesis

Discovery

2001

X

Lunar Prospector

Discovery

1998

X

NEAR

Discovery

1996

X

Stardust

Discovery

1999

X

Mars Pathfinder

Discovery - Mars

1996

X

ICESAT

Earth Observing System

2003

X

SORCE

Earth Observing System

2003

X

ACE

Explorer

1997

X

HETE-II

Explorer

2000

X

IMAGE

Explorer

2000

X

MAP

Explorer

2001

X

FUSE

Explorer (Origins)

1999

X

MCO

Mars

1998a

X

MGS

Mars

1998

X

MPL

Mars

1999a

X

DS-1

New Millennium

1998

X

SWAS

Small Explorer

1998

X

Small Explorer

1999a

X

WIRE

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

3

7

X X

X

X

15

SIZE AND HISTORIC CAUSES OF COST GROWTH

TABLE 1.3  Continued

Copyright © 2010. National Academies Press. All rights reserved.

Primary Reference Launch Year

1 X

Mission Name

Program/Class

2

TIMED

Solar Terrestrial Probe

2001

EOS-Terra

Earth Observing System

1999

b

b

New Horizons (Pluto)

New Frontiers

2006

X

X

Chandra X-ray Observatory (formerly AXAF)

Great Observatory

1999

X

X

ACRIMSAT

Solar Science

1999

X

Dawn

Discovery

2007

X

Fermi Gamma-ray Space Telescope (formerly GLAST)

NASA/DOE/Japan

2008

X

X

IBEX

Small Explorer

2008

X

X

SOFIA

Astrophysics (airborne)

2007

X

TWINS A/B

Heliophysics

2008

X

LCROSS

ESMD

2009

X

LRO

ESMD/SMD

2009

X

Kepler

Discovery

2009

X

Glory

Earth Observing - Directed

2010c

X

OCO

Earth System Science Pathfinder

2009a

X

Planck

ESA

2009

X

Herschel

ESA Horizon 2000 Program

2009

X

WISE

Explorer/Medium-class Explorer

2009

X

JWST

Great Observatory

2014c

X

Aquarius

Instrument

2010c

X

OSTM

Joint NASA/CNES/NOAA

2008

X

MSL

Mars

2011c

X

CINDI

Mission of Opportunity

2008

X

Space Technology 7

New Millennium

cancelled

X

Space Technology 8

New Millennium

cancelled

X

M3-Foton

Russian-ESA

2007

X

MMS

Solar Terrestrial Probe

2014c

X

NPOESS Preparatory Project

NASA/NOAA/USAF

2011c

X

Cassini-Huygens

NASA-ESA

1997

X

Phoenix

Mars Scout

2007

X

AIM

Small Explorer

2007

X

Magellan

Venus

1989

X

NSCAT

NASA/Japan

1996

X

ACTS

Communications

1993

X

GOES I-M

Operational weather satellites

1994-2001

X

EUVE

Explorer

1992

X

XTE

Explorer/Medium-class Explorer

1995

X

Galileo

Planetary

1989

X

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

3

4

5

6

7

X X

X X

X

X X

X

continued

16

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE 1.3  Continued Primary Reference Launch Year

Mission Name

Program/Class

1

2

3

4

5

6

7

CGRO

Great Observatory

1991

X

HST

Great Observatory

1990

X

Mars Observer

Mars

1992a

X

TDRS-7

NASA Communications

1995

X

TOPEX

NASA/CNES

1992

X

Ulysses

NASA/ESA

1990

X

ASRM

Space Shuttle

cancelled

X

ATP

Space Shuttle

2002

X

TSS-1

Space Shuttle

1992

X

FTS

Space Station

cancelled

X

Freedom (Space Station), now ISS

Space Station

1998-2011

X

AFE

Technology development

cancelled

X

Collaborative Solar Terrestrial Research

International Solar-Terrestrial Physics

1992

X

Global Geospace Science Program (Polar and Wind missions)

International Solar-Terrestrial Physics

1994/1996

X

Landsat-D

Earth Observing

1982

X

OMV

Technology development

cancelled

X

Copyright © 2010. National Academies Press. All rights reserved.

NOTE: Acronyms are defined in Appendix D. a Mission failed. b Initially part of the study, but then dropped from the analysis. c Planned launch date (has not yet occurred).

Figure 1.2 also shows a substantial linear correlation (R2 = 0.64) between cost growth and schedule growth. This correlation would be even stronger if not for the effect of missions with fixed launch windows. For example, launch windows for missions to Mars open every 780 days, and mission staff are highly motivated to be ready to go during the assigned window. If that window is missed, the only alternatives are to cancel the mission or sit on the ground for 2 years waiting for the next window (resulting in a large increase in both cost and schedule). Hence, in Figure 1.2, it is not surprising to see that the Mars Reconnaissance Orbiter (MRO) mission, the Mars Exploration Rover (MER) mission, and the Comet Nucleus Tour (CONTOUR) mission (which had a single, 25day launch window) had significant cost growth, even though schedule growth was zero or nearly so. In fact, the pressure that such missions experience to meet the launch window may drive cost growth higher than it would otherwise be if the mission had a more flexible launch window. For many missions, there is substantial correlation between cost and schedule growth. However, even when that is the case, schedule delays are not necessarily the primary cause of cost growth. In some cases, schedule delays are a secondary effect of other factors, such as a failure to adequately fund the program or problems with launch vehicle or launch pad availability. In these cases, additional effort is required to replan and manage the mission, and additional costs are incurred due to extension of supporting areas such as program review and cost accounting. In other cases, it may even be necessary to store the spacecraft and supporting systems and keep the launch and support crew trained and capable of supporting a launch. The costs associated with these efforts are sometimes referred to as “sustaining cost” or “capability retention cost,” and they represent real effort that is required for a successful program. In most cases, however, schedule growth and cost growth arise from development problems that require both time and money to solve. Earth and space science programs are very complex, with extensive interactions among

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

17

SIZE AND HISTORIC CAUSES OF COST GROWTH

tasks, so that schedule delays in one development area impact the schedule and scope of other tasks. Furthermore, delays in one task cause additional effort to replan and manage all of the interdependent project tasks. Thus, cost growth may occur in many areas of a project because of a schedule delay in one area or on one task. When that happens, the total cost growth might be attributed to schedule growth, but the root cause is a development problem in one or more areas. In any case, as discussed below, there are many causes of cost growth in addition to schedule growth. .

Copyright © 2010. National Academies Press. All rights reserved.

Absolute Cost Growth Additional insight can be gained by considering absolute cost growth (in dollars) in addition to relative cost growth (as a percentage of the initial estimate). Figures 1.3a and 1.3b show in decreasing order the absolute cost growth in excess of reserves for the 40 missions from Primary Reference 1. The two figures are identical, except that the first shows the distribution of Earth science and space science missions, and the second shows the distribution of AO and directed missions. Figure 1.3c also shows the initial cost estimate and launch date for each of these missions. The average development cost for each of these 40 missions is $215 million. This does not include the cost of launch, mission operations, data analysis, or inflation. If savings from missions that under run their budgets are allowed to partially offset the cost growth of other missions, the total net absolute cost growth for these 40 missions is $1.3 billion. This corresponds to the total development cost of about 6 missions of average size. As is often the case, about 20 percent of the population causes 80 percent of the problem. In particular, with regard to the 40 missions examined by Primary Reference 1, 11 of them (28 percent of the total) account for about 80 percent of the total cost growth (in absolute terms), and 14 missions (35 percent of the total number of missions) account for about 92 percent of the total cost growth; the other 26 missions (two-thirds of the total number) account for only 8 percent of the total cost growth (see Table 1.4). Based on data from 1970 through 1999, there is not a consistent trend for cost growth of NASA missions over time. As shown in Table 1.5, cost growth for NASA missions increased from the 1970s to the 1980s, but then decreased in the 1990s. More recently, Figure 1.3c and Table 1.4 indicate that, for this set of 40 missions, cost and cost growth for missions launched in 2003 and later are generally higher than for missions launched in 2002 and earlier. The 15 missions launched in 2003 and later had an average initial cost of $244 million, whereas the 25 missions launched in 2002 and earlier had an average initial cost of $146 million, and the 15 missions launched in 2003 and later, as a whole, have a much higher cost growth (29 percent/$72 million) than the 25 missions launched in 2002 and earlier (6 percent/$8 million). In addition, all but 2 of the 14 missions with the most cost growth (in dollars) were launched in 2003 or later, and all but 3 of the 26 missions with the least cost growth were launched in 2002 or earlier. A comparison of Figures 1.2 and 1.4 shows that the slopes of the linear trend lines are nearly identical (y/x = 1.22 in Figure 1.2 and 1.23 in Figure 1.4), indicating that the relationship between schedule growth and cost growth is virtually the same for both small and large missions. However, the impact of even modest cost growth in a large mission can still be substantial. Figure 1.3b shows that both AO and directed missions experience significant cost growth (in absolute terms). Similarly, Figures 1.3a and 1.4 show that both Earth and space science missions experience a similar mix of cost growth in both absolute terms and as a percentage of initial cost. The 10 Earth science missions in this set of 40 projects exhibited a total cost growth of 13 percent (an average of $37 million per mission), while the 30 space science missions showed a total cost growth of 20 percent (an average of $31 million per mission). Primary Reference 6 concurs that Earth science missions are not significantly different from other Science Mission Directorate (SMD) missions in terms of cost or cost growth (and this is supported by Figure 1.4). Primary Reference 6 concludes that cost growth is more closely associated with increases in spacecraft mass and higher levels of mission complexity rather than with mission type. In any case, as difficult as it is to develop an accurate and consistent view of mission costs and cost growth based on historic studies, it is even more difficult to determine with certainty the cause and effect relationship among the many factors involved.   For

missions with multiple launches, the date of the first launch is listed in Figure 1.3c.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

18

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

150% CALIPSO 140%

130%

120% EO-1

110%

100%

y = 1.22x + 0.01 2 R = 0.64 90%

Percent Cost Growth

80%

CLOUDSAT HETE-II

70% SWAS

SWIFT 60% GALEX

Messenger ICESAT

50%

40% Type III 30%

HESSI

Gravity Probe B STEREO

SIRTF TIMED

See inset

Type II 20%

Earth Science Missions Space Science Missions

Copyright © 2010. National Academies Press. All rights reserved.

Type10% I 0% 0% -10%

-20%

10%

20% 30% EOS-Aqua

TRMM MGS Stardust NEAR

40%

50%

60%

70%

80%

Percent Schedule Growth

ACE

-30%

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

90%

19

SIZE AND HISTORIC CAUSES OF COST GROWTH

30%

Deep Impact

WIRE 25%

CONTOUR

y = 1.22x + 0.01

Type II

2

R = 0.64

Percent Cost Growth

20%

FAST Genesis

Landsat-7

MER 15%

TRACE

MRO

DS-1 THEMIS

10%

Type I

GRACE FUSE

SORCE

IMAGE MCO/ MPL

EOS-Aura

MAP

5% Earth Science Missions Space Science Missions

Lunar Prospector Mars Pathfinder

0% 0%

5%

10%

15%

20%

25%

30%

Percent Schedule Growth

Copyright © 2010. National Academies Press. All rights reserved.

Enlarged portion of Figure 1.2 showing names of missions classified as Type I and Type II.

FIGURE 1.2 (opposite): Clustering of 40 NASA missions from Primary Reference 1 in terms of percent cost and schedule growth in excess of reserves: Type I, less than 15 percent; Type II, 15-30 percent; and Type III, more than 30 percent. The linear trend line shows the best fit to all the missions plotted in the figure. For the names of the missions classified as Type I and Type II, see the enlargement, above, of the lower left portion of the figure NOTE: Acronyms are defined in Appendix D. SOURCE: Cost and schedule data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

20

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

$200M

$150M

Earth Science Missions Space Science Missions

$50M

St

ua Aq S- CE EO A R EA N S G MM M TR ust d ar

Absolute Cost Growth

$100M

r de fin or th ct Pa pe s os ar r M rP na E Lu AC IS TREM TH GE A IM P I A -I M E L ET P H O/M C E M C R SOST E FA AC R G E IR W -1 S D SE I FU SS UR E H O R NT O is C nes e G LEX A G AS SWED ra M u TI S-A EO O R T ct M SA pa E m IC p I 7 ee at D ds n T La IFT SA SW UD LO EO C ER ST -1 EO R ger E n M se O B es S be M LIP ro A P C vity ra G TF R SI

$0M

–$50M

FIGURE 1.3a  Ranking of 40 NASA science missions from Primary Reference 1 in terms of absolute cost growth in excess of reserves -$100M in millions of dollars (starting at the beginning of Phase B and through the end of Phase D, excluding launch, mission operations, and data analysis), showing the distribution of Earth science and space science missions. NOTE: Acronyms are defined in Appendix D. SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

Copyright © 2010. National Academies Press. All rights reserved.

Cost Growth by Phase One of the clear and consistent findings in the primary references is that cost growth does not accumulate uniformly across mission phases; rather, the bulk of cost growth occurs post-CDR (see Figure 1.5). Some post-CDR cost growth is driven by external factors, such as delays in the availability of launch vehicles. However, most postCDR cost growth is due to internal project development issues, even though (1) CDR is intended to be the final milestone of the design phase and (2) spacecraft configuration should be frozen at CDR. CDR for many missions may be held prematurely—driven by schedule rather than driven by design maturity. CDR approval of an immature design can cause downstream problems during Phase D such as integration difficulties and late changes. Primary Reference 2 found that, excluding external impacts, cumulative average cost growth to CDR is only 7 percent, but this grows to 28 percent by launch. So 75 percent of cost growth occurs after CDR. Primary Reference 5 also analyzed the evolution of cost growth over time and concluded that “it is important to notice that, unlike the mass and power growth time trends, cost growth is typically not recognized until after CDR. This is counter to standard industry guidelines that recommend a decreasing percentage reserve on a reduced cost-to-go basis. The

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

21

SIZE AND HISTORIC CAUSES OF COST GROWTH

$200M

$150M

Directed Missions AO Missions

$50M

$0M

ua Aq S- CE EO A R r EA de N S fin r G th cto MM Pa spe M TR ust ars Pro d M r ar na E St Lu AC IS TREM TH GE A IM P I A -I M E L ET P H O/M C E M C R SOST E FA AC R G E IR W -1 S D SE I FU SS UR E H O R NT O is C nes e G LEX A G AS SW ED ra M u TI S-A EO O R T ct M SA pa E m IC p I 7 ee at D ds n T La IFT SA SW UD LO EO C ER ST -1 EO R ger E n M se O B es S be M LIP ro A P C vity ra G TF R SI

Absolute Cost Growth

$100M

–$50M

–$100M

Copyright © 2010. National Academies Press. All rights reserved.

FIGURE 1.3b  Ranking of 40 NASA science missions from Primary Reference 1 in terms of absolute cost growth in excess of reserves in millions of dollars (starting at the beginning of Phase B and through the end of Phase D, excluding launch, mission operations, and data analysis), showing the distribution of AO and directed missions. NOTE: Acronyms are defined in Appendix D. SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

substantial cost growth after CDR implies that a greater percentage reserve on cost to go should be held. Alternately, it may mean that cost growth is occurring earlier in the project lifecycle, but isn’t recognized until later” (p. 9).10 Similarly, Primary Reference 6 reported that the highest percentage of schedule growth occurs after the start of integration and testing, i.e., during Phase D, and that this phenomenon is consistent across Earth science, heliophysics, astrophysics, and planetary missions. If risk has not been sufficiently reduced by CDR, then cost and schedule uncertainty remains high, especially because the underlying causes of some post-CDR cost growth may have originated prior to CDR without being recognized. Figure 1.5 suggests that, for one set of 20 missions examined by the committee, there is a slight decrease in cost growth as we go from initial cost estimates at the start of the program to PDR from about 48 percent to about 35 percent cost growth. Thus about 10-15 percent cost growth can be attributed to a lack of design maturity pre-PDR. 10  This

conclusion, from Primary Reference 5, implies that the cost growth that occurs post-CDR is disproportionately large compared to the percent of project funds that are expended post-CDR. The committee did not have the data necessary to verify this conclusion.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

22

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

$950M $900M $850M $800M

These 14 missions together account for 92% of the total cost growth for all 40 missions in this figure

Initial cost—directed missions Initial cost—AO missions Cost growth—14 missions with most cost growth Cost growth—26 missions with least cost growth

Initial Cost Estimate / Absolute Cost Growth

$750M $700M $650M $600M $550M $500M $450M

These 26 missions together account for just 8% of the total cost growth for all 40 missions in this figure

$400M $350M $300M $250M $200M $150M $100M $50M $0M

2 00 ,2 ua Aq S- 997 EO , 1 996 E 1 AC R, 98 EA 19 7 N S, 99 96 G , 1 99 19 8 M M 19 er, 199 M st, d r, TR rdu hfin cto a at e St s P osp ar Pr 98 M ar 19 7 n , 0 Lu E 20 AC S, 0 TR MI 200 E , TH GE 01 A 0 00 IM P, 2 20 98 A II, 19 M E- L, 3 ET P 0 H /M 20 O C E, M RC 996 1 02 SO T, 20 S E, FA AC 999 R 1 G E, 98 IR 9 W 1, 1 999 S- 1 02 D E, 20 S I, 02 FU SS 20 1 E , H r 0 R tou , 20 on is 3 C es 200 en , 8 G EX 99 AL , 1 1 G AS 200 04 , 20 SW D , a E M ur TI -A 05 S 0 3 5 0 EO , 2 20 200 O R T, t, M SA pac 99 E m 19 IC p I -7, ee t 4 6 D sa 00 00 nd , 2 , 2 La IFT AT 6 S 0 D 0 SW U , 2 LO O C E 0 ER 200 ST 1, 03 004 - 0 2 2 EO R, er, 06 004 E g 0 2 M sen , 2 B, es SO e M LIP rob A P 3 C vity 00 ra , 2 G TF R SI

-$50M

FIGURE 1.3c  Ranking of 40 NASA science missions from Primary Reference 1 in terms of absolute cost growth in excess of reserves in millions of dollars (starting at the beginning of Phase B and through the end of Phase D, excluding launch, mission operations, and data analysis), with initial cost and launch date for each mission also shown. NOTE: Acronyms are defined in Appendix D. SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

TABLE 1.4 Breakdown of Cost and Cost Growth for the 40 missions from Primary Reference 1

Copyright © 2010. National Academies Press. All rights reserved.

Total Initial Cost

Total Cost Growth

(billion $)

(%)

(billion $)

(%)

14 missions with the most cost growth

3.9

  53

1.2

  92

26 missions with the least cost growth

3.4

  47

0.1

   8

Total for all 40 missions

7.3

100

1.3

100

Average Initial Cost

Average Cost Growth

(million $)

(million $)

(%)

15 missions launched in 2003 and later

244

72

29

25 missions launched in 2002 and earlier

146

 8

 6

SOURCE: Based on data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

23

SIZE AND HISTORIC CAUSES OF COST GROWTH

150%

CALIPSO 140%

130%

120%

EO-1

110%

100%

Percent Cost Growth

90%

CLOUDSAT 80%

y = 1.23x + 0.13 2 R = 0.63

70%

SWIFT 60%

Messenger 50%

ICESAT

Type III 40%

Gravity Probe B

SIRTF STEREO

Deep Impact

30% Type II 20%

MER

Earth Science Missions Space Science Missions

Copyright © 2010. National Academies Press. All rights reserved.

Landsat-7 10% Type I

MRO

EOSAura

0% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Percent Schedule Growth FIGURE 1.4 Clustering of the 14 NASA missions from Primary Reference 1 that accounted for 92 percent of the total cost growth in dollars, in terms of percent cost and schedule growth in excess of reserves. NOTE: Acronyms are defined in Appendix D. SOURCE: Cost and schedule data from Tom Coonce, NASA Headquarters, e-mail to committee member Joseph W. Hamaker, December 21, 2009, providing source data for Primary Reference 1.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

24

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE 1.5  Decadal Trends in Cost Growth for NASA Missions Cost Growth

1970s 1980sa 1990s

Average (%)

Median (%)

43 61, 83 36

26 50, 60 26



a The source cited two different values for cost growth of NASA missions in the 1980s, based on two different prior studies. SOURCE: Based on data from Schaffer, 2004.

200.0%

CDR to Launch PDR to CDR Start to PDR 150.0%

100.0%

50.0%

0.0%

s

Copyright © 2010. National Academies Press. All rights reserved.

E ct

ST

on

iz

T

pa

Im

or

ll ra ve S O I ES EN G O LR

O

R

p

H

N

AC

R

M

G

ee

D

ew

N

AW

D

O SA

LA

r M AI nge se es M EX IB EX AL G T IF SW O E ER

ST

C D

I/G

M

er

er

U

LO

O

C

R

pl

FE

Ke

SO

IP

itz

Sp

-1

AL

C

EO -50.0%

FIGURE 1.5 Cost growth of missions by program phase, showing that most cost growth occurs after critical design review. NOTE: Acronyms are defined in Appendix D. SOURCE: Based on data from Claude Freaner, NASA, e-mail to committee member Joseph W. Hamaker, January 6, 2010, providing source data for Freaner et al. (2010).

Surprisingly, the cost growth between PDR and CDR is not very large: only about 5 percent for AO missions and about 12 percent for directed missions. Therefore the majority of cost growth, typically 20 to 30 percent of the initial estimate, occurs post-CDR. This is an important finding, which is supported by many of the primary references. This finding is symptomatic that a number of projects experience “surprises” post-CDR that significantly increase mission cost, even though the design of the instruments and spacecraft is supposed to be “frozen” at CDR. The reasons for the post-CDR cost growth will be discussed in the following sections.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

SIZE AND HISTORIC CAUSES OF COST GROWTH

25

Causes of Cost Growth  inding. Causes of Cost Growth. Past studies identify a wide range of factors that contribute to cost and F schedule growth of NASA Earth and space science missions. The most commonly identified factors are as follows:

• • • •

Overly optimistic and unrealistic initial cost estimates, Project instability and funding issues, Problems with development of instruments and other spacecraft technology, and Launch service issues.

Copyright © 2010. National Academies Press. All rights reserved.

I n addition, any problem that causes schedule growth contributes to and magnifies total mission cost growth, and cost growth in one mission may induce organizational replanning that delays other missions in earlier stages of implementation, further amplifying overall cost growth. Effective implementation of a comprehensive, integrated cost containment strategy, as recommended herein, is the best way to address this problem. The historic causes of cost growth in the primary references are discussed below and summarized in Table 1.6.11 In addition to the four most common causes listed in the above finding, the primary references have identified a wide range of additional factors that contribute to cost growth, such as poor contractor performance and a tendency to over-engineer. Primary Reference 10, which examined DOD projects, also noted the erosion of capabilities to lead and manage the space acquisition process. The committee generally relied on the assessments made in the primary references, rather than begin its assessment using raw data collected by the authors of the earlier studies, in part because the raw data were not available to the committee. However, spot checks of data for key missions supported the conclusions reached by the primary references and this study. For example, the Spitzer Space Telescope mission (formerly known as the Space Infrared Telescope Facility, SIRTF) had the largest absolute cost growth of the 40 missions assessed by Primary Reference 1. Cost growth problems encountered by Spitzer included many of the factors cited in the above finding and in Table 1.6: early planning deficiencies; problems with development, integration, and/or testing of the spacecraft as well as all three major instruments; launch vehicle problems; schedule delays associated with all of the above; and cost growth of project-level management functions (Mlynczak and Perry, 2009). The primary references are generally consistent in their conclusions, although, in some cases, different studies disagree about the significance of the impact of some factors on cost growth. For example, Primary Reference 6 uses data-driven correlation analyses to assert that there is little or no correlation between mission cost growth and planned cost reserves or the percent of funds spent in Phase B. Primary References 2 and 4 (which, for the most part, examined the same missions as Primary Reference 6) concur that there is no correlation between cost growth and planned cost reserves. However, several committee members have observed that cost reserves are often “manufactured” during the early stages of a program by unrealistically reducing baseline budgets (without reserves), so that the total budget can include a cost reserve of the expected magnitude without descoping the mission. This manufactured reserve does not represent a true contingency fund in excess of likely project costs. The committee concludes that the lack of correlation in some studies between cost reserve and cost growth (in excess of the cost reserve) indicates that more care is needed in establishing cost reserves in excess of realistic cost estimates. Primary References 2, 4, and 6 also point out a lack of correlation between funds spent in Phase B and cost growth. Primary Reference 2, however, recommends including a level of cost reserves that is commensurate with mission implementation risk. In addition, during Phase B, many projects are still in a competitive mode, and competitive pressures encourage (overly) optimistic assessments of the cost and schedule impacts of addressing uncertainties and overcoming potential problems. One reason to designate a larger percentage of mission budgets for Phase B technology development is to better 11  The committee’s assessment of historic causes of cost growth is consistent with the conclusions documented in Primary Reference 2, which identifies four key drivers of cost growth: over-optimism early in formulation, unstable or inadequate initial funding profile, instrument development complexity, and launch service issues.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

26

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE 1.6  Causes of Cost Growth Based on the Primary References Causes of Cost Growth

  Page Number in Primary Reference

Initial Cost Estimates

1

2

3

4

5

6

7

9

10

Inherent optimism in initial design and estimates (in part because of the emphasis that is placed on science, not cost; in part because mission complexity is underestimated), which leads to unrealistic proposals and estimates

9

38

4 6

15

1 5

 

12

58

2

Incomplete initial budget estimates (e.g., cost of launch services, mission operations, and/or data analysis not included)

 

 

 

 

 

 

17

58

 

Cost has replaced mission success as the primary driver in managing acquisition processes, resulting in excessive technical and schedule risk.

 

 

 

 

 

 

 

iii

 

Assumptions about heritage hardware, software, and commercial-offthe-shelf equipment did not materialize.

 

 

6

 

 

 

 

 

 

The space acquisition system is strongly biased to produce unrealistically low cost estimates throughout the acquisition process. These estimates lead to unrealistic budgets and unexecutable programs. Government budget space acquisition programs should use a most probable (80/20) cost, with a 20-25 percent management reserve for development programs included within this cost.

 

 

 

 

 

 

 

 

iii

Inflation

 

 

 

 

 

 

17

 

 

Program instability/changes (to mission requirements, spacecraft, instruments, launch vehicles, upper stage propulsion systems, trajectories, and/or operations)

 

25

6 12

 

 

 

11 15

58

2

Technical and programmatic uncertainty at the beginning of a project (not enough time or resources available in phases A and B; TRL lower than claimed; risk identification and mitigation prior to CDR needs attention; uncertainty in spacecraft mass, power, pointing accuracy, data transmission rates, and so on)

15

36

6 12 19

 

12 13

 

 

 

 

Budget constraints, lack of stable funding, and/or inadequate initial funding profile

 

25

6

15

 

 

11 15

 

 

Weak independent validation of cost and schedule

 

 

6

 

 

 

 

 

 

Inadequate definition of technical and management aspects of projects prior to NASA and OMB approval

 

 

 

 

 

 

11

 

 

Inadequate cost and schedule reserves

12

 

6

 

 

 

 

 

 

Technical complexity

 

 

 

 

 

 

15

 

 

Adverse impacts of financial problems experienced by other NASA missions

 

 

6

 

 

 

 

 

 

Instrument development problems caused, for example, by instrument designs that lack technical details and/or fail to identify technical challenges

11

29

6 15

8

12 13

117

 

 

 

The primary contributor to internal cost growth (i.e., growth caused by factors within the control of a given project) is instrument development problems.

4 11

 

 

 

 

 

 

 

 

Increase in spacecraft development costs, especially integration and test costs

 

31

 

 

 

114

 

 

 

Increases in spacecraft mass, power requirements, and complexity

 

 

 

 

4

 

 

 

 

Project Instability and Funding Issues

Copyright © 2010. National Academies Press. All rights reserved.

Development of Instruments and Other Spacecraft Technology

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

27

SIZE AND HISTORIC CAUSES OF COST GROWTH

TABLE 1.6  Continued Causes of Cost Growth

  Page Number in Primary Reference

Increases in payload cost (except for planetary missions)

 

 

 

 

 

114 117

 

 

 

Launch service issues (primarily higher launch vehicle prices or selection issues)

 

25

6

15

 

 

 

 

 

Launch vehicle delays resulting in overall schedule delays

 

25

6

 

 

 

17

 

 

The primary contributor to external cost growth (i.e., growth caused by factors beyond the control of a given project) is problems with the readiness of the launch vehicle.

4

 

 

 

 

 

 

 

 

Schedule growth that leads to cost growth

11

38

12

 

 

 

19

59

 

Poor contractor performance

 

 

 

 

 

 

15

 

 

Industry has failed to implement proven practices on some programs.

 

 

 

 

 

 

 

 

4

Industry guidelines do not in general adequately predict the uncertainty in the initial physical and programmatic parameters claimed in proposals.

 

 

 

 

13

 

 

 

 

Tendency to over-engineer

 

 

6

 

 

 

 

 

 

Increase in project-level management costs

 

27

 

8

 

 

 

 

 

Failure to achieve anticipated cost and/or schedule savings from mission descopes

 

30

6

 

 

 

 

 

 

Increases in mission operations and data analysis costs for science enhancements and/or extended missions

 

44

 

 

 

 

 

 

 

Hardware from foreign partners (late delivery or increased costs)

 

36

6 17

 

 

 

 

 

 

ITAR requirements

 

44

 

 

 

 

 

 

 

Department of Defense capabilities to lead and manage the space acquisition process have seriously eroded over time.

 

 

 

 

 

 

 

 

3

Launch Service Issues

Other Factors

NOTE: Primary Reference 8 is focused on reducing the absolute costs of NASA space science missions. It does not directly address cost growth, and so its results are not included in this table. Primary References 9 and 10 focus on Department of Defense systems.

Copyright © 2010. National Academies Press. All rights reserved.

define system designs, reconcile mission goals and budgets, retire risk prior to PDR, and reduce uncertainties in the project plan. If the baseline cost is established after PDR (that is, after the end of Phase B), increased costs identified in Phase B do not show up as cost growth. Primary References 2, 4, and 6, however, define baseline mission costs at the beginning of Phase B. As a result, technology investments made during Phase B cannot improve the understanding of the technical baseline, which is necessary to improve the accuracy of the initial cost estimate. Initial Cost Estimates The primary references have concluded that mission requirements are typically formulated on the basis of science return on the particular mission much more than cost. In addition, it is easy to underestimate the complexity of mission hardware and software and/or make unrealistically optimistic assumptions about how easy it will be to develop and manufacture systems for a new mission based on similarities to systems used in past missions. In a competitive environment, bidders are motivated to argue that science goals can be achieved as inexpensively as possible in order to maximize the science return for a given assumed budget. In addition, initial cost estimates may not include all costs, such as the cost of launch services, mission operations, and/or data analysis. As a result, initial cost estimates and mission proposals tend to be overly optimistic and unrealistic to the point that it becomes very difficult to implement them within the initial budgets. Yet, once projects get under way, cost constraints

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

28

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

may become so important that efforts to minimize cost growth may result in excessive technical and schedule risk. Inflation also becomes a significant factor for missions with a long life cycle, either by design or because of schedule growth. Project Instability and Funding Issues The primary references have concluded that project instability (associated with changes to mission requirements, spacecraft, instruments, launch vehicles, upper stage propulsion systems, trajectories, and/or operations) are a major cost growth factor. Budget constraints, lack of stable funding, inadequate initial funding profiles, insufficient cost and schedule reserves, and weak independent validation of cost and schedule likewise contribute to cost growth. In many cases, project instability and funding issues are intertwined. At the beginning of a project, actual technology readiness levels may be lower than anticipated, leading to uncertainty in mission requirements in terms of spacecraft mass, power, pointing accuracy, data transmission rates, and overall mission complexity. Technical and programmatic uncertainty tends to persist if the funding in phases A and B is insufficient to resolve risks adequately. As a result, missions are sometimes approved by NASA and OMB before cost and schedule requirements are well understood. Even when a particular mission is well understood and well planned, changes and delays are sometimes imposed when, in a constrained cost environment, cost growth experienced by one NASA mission may require NASA to delay or curtail other missions. Other sources of external changes, which are beyond the control of a given project manager, are changes in overall requirements or processes that may be imposed, for example, by an evolving view of what constitutes acceptable risk. As unanticipated problems and changes arise, either internally or externally, the resulting instability tends to redirect the attention of the project management team from the technical challenges of implementing the mission to work on replanning efforts. These replanning activities themselves generally result in increased cost and schedule. Project instability and funding issues can be especially difficult to avoid for large missions that rely on annual funding appropriations for a mission life cycle that may span a decade or more. Development of Instruments and Other Spacecraft Technology Earth and space science satellites are highly customized, and their design is often driven by the nature of their instruments and their requirements in terms of power, mass, pointing accuracy, thermal control, and so on. The primary references conclude that instrument development problems may be the largest element of mission cost growth that can be attributed to factors within the control of a given project. Common problems are instrument designs that lack technical details and/or fail to identify technical challenges. Cost growth can also arise from problems in the development of other spacecraft technologies and systems, especially in the area of integration and testing and as a result of increases in spacecraft mass, power requirements, and/or complexity.

Copyright © 2010. National Academies Press. All rights reserved.

Launch Service Issues Launch service issues include higher launch vehicle prices and/or launch vehicle selection issues. In addition, launch vehicle delays are a common cause of delays in the overall schedule. In fact, Primary Reference 1 concluded that problems with the readiness of the launch vehicle are the primary contributor to external cost growth (i.e., growth caused by factors beyond the control of a given project). Differences Between Earth and Space Science Missions A casual look at partial cost data or personal experience may indicate different cost growth potential for missions from different disciplines, because different classes of missions face different challenges. For example, Earth science missions typically have more complex, more costly, and more massive instruments than space science missions do. Earth science missions also have more stringent requirements in terms of pointing accuracy, resolution, stability, and so on, although astrophysics missions also have stringent pointing requirements, and planetary

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

SIZE AND HISTORIC CAUSES OF COST GROWTH

29

Copyright © 2010. National Academies Press. All rights reserved.

spacecraft and instrument technology must be able to survive long cruise phases and radiation environments that are sometimes quite extreme. Earth science missions also dedicate a higher fraction of mission costs to instruments than do other missions, their instruments cost the most, and their spacecraft are typically more complex than most space science spacecraft are. Space science missions that leave Earth orbit have greater incentives to minimize spacecraft mass and power, and the average cost and the average spacecraft mass for space science missions are less than those for Earth science missions. However, as specifically addressed by Primary Reference 6, Earth ­science missions have not shown a systematic difference in cost or cost growth compared to other SMD missions.12 Both Earth and space science missions have shown good correlation between (1) instrument schedule growth and instrument cost growth, (2) instrument cost/schedule growth and mission cost/schedule growth, and (3) the absolute costs of instruments and instrument complexity. Primary Reference 1 and other studies (Bitten, 2008; Bearden, 2008) confirm that the complexity of Earth science missions results in long development cycles (and associated large development costs as a fraction of total mission costs). Because Earth science missions rarely have highly restricted launch windows, it is easier to postpone launch (and slip the mission schedule) as necessary to solve technical problems. Restricted launch windows motivate mission teams to avoid schedule delays if at all possible. For some missions, this has involved descoping. Schedule-constrained missions (which are mostly planetary missions) typically respond to late-breaking technical problems not by increasing schedule but by increasing the workforce (and costs). In some cases, despite the best efforts of the mission staff, risk created by late-breaking problems is not fully resolved prior to launch. As a result, schedule-constrained missions tend to have less cost growth—and higher failure rates—than do other missions.

12  For the 40 missions analyzed in Reference 1, the average of the cost growth values for the 10 Earth science missions was 43 percent, almost twice as high as the average cost growth of the 30 space science missions (22 percent). On the other hand, the cost growth of the 10 Earth science missions, as a whole, was just 13 percent, compared to 20 percent for the space science missions. Overall comparisons are potentially misleading because one very large Earth science mission, EOS Aqua, had a final cost much less than the initial estimate because of a mission descope. In addition, although the average cost of the Earth science missions was almost twice the average cost of the space science missions ($274 million versus $152 million), the median initial cost of the Earth science missions was actually less than the median cost of the space science missions ($101 million versus $128 million). Regardless, in looking at the real source of the cost growth problem—which is to say, the 14 missions in this set of 40 that accounted for 92 percent of the total cost growth—6 were Earth science missions and 8 were space science missions.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

2 Key Problems and Solutions

In its detailed examination of the primary references, other related studies, and presentations by NASA and industry personnel, the committee did not identify any major new individual causes of cost growth. However, the committee has identified three key problem areas and suggests action to address them. First, there is the issue of cost realism, including the tendency to underestimate program difficulty; uncertainty in the definition/establishment of cost and schedule baselines; and the use and capability of NASA cost-estimating methodologies. Second, there are a series of process-based issues, including project selection and formulation, risk identification and mitigation, and the review process. The committee also believes that NASA needs to invigorate the technology base, particularly technology supporting instrument development. Third, a principal factor external to programs is launch vehicle selection and cost. In addition to the many specific causes of cost growth identified in the primary references, the integrated effects of these causes also leads to cost and schedule growth. Therefore, the committee recommends that NASA develop an overall strategy for dealing with the issues that cause cost and schedule growth. Cost Realism

Copyright © 2010. National Academies Press. All rights reserved.

Cost Estimates  inding. Unrealistic Initial Cost and Schedule Estimates. People and organizations tend to optimize their F behavior based on the environment in which they operate. The current system incentivizes overly optimistic expectations regarding cost and schedule. As a result, initial cost estimates generally underestimate final costs by a sizable amount.  ecommendation. Independent Cost Estimates. NASA should strengthen the role of its independent costR estimating function as follows:  • Expanding and improving NASA’s ability to conduct parametric cost estimates, and  • Obtaining independent parametric cost estimates at critical design review (in addition to system requirements review and preliminary design review), comparing them to other estimates available from the project and reconciling significant differences. 30

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Copyright © 2010. National Academies Press. All rights reserved.

KEY PROBLEMS AND SOLUTIONS

31

NASA project staff generally estimate mission costs using detailed engineering analyses of labor and material requirements, vendor quotes, subcontractor bids, and the like. Non-advocate independent cost estimates in NASA are generally parametric cost estimates using statistical cost-estimating relationships based on historical relationships among cost and technical and programmatic variables (mass, power, complexity, and so on). In both cases, mission cost estimates are created by summing costs at lower levels of a project’s work breakdown structure (WBS) to obtain total project costs. Parametric cost models rely on observations rather than opinion, are an excellent tool for answering what-if questions quickly, and provide statistically sound information about the confidence level of cost estimates. In contrast, the process used within NASA to generate cost estimates based on detailed engineering assessments does not provide a statistical confidence level and, in retrospect, has generally been less accurate than parametric cost models have been. Parametric cost models implicitly assume that the cost of current projects will be influenced by the same factors that influenced the cost of past missions, which generally seems to be the case with NASA Earth and space science missions. Parametric cost models automatically include allowances for many of the technical and programmatic problems that NASA development projects encounter (e.g., schedule delays, requirements changes, unforeseen technology challenges, and so on) because the historic database used by the models is comprised of past projects that had these types of problems. Parametric cost models should be used as the primary source of cost estimates in Phase A, and they remain applicable in phases B, C, and D (NASA, 2008). Parametric cost models are particularly useful when the parameters used to describe the missions are relatively accurate representations of the as-flown mission. The accuracy of parametric cost models produced by NASA for Earth and space science missions could be improved, for example, by better processes for (1) generating realistic system data early in the process and (2) validating models to improve their accuracy. Uncertainty in the lower-level WBS cost elements is inevitable. The uncertainties can be thought of as ranges or probability distributions. Most WBS element cost probability distributions are skewed to the high side—in cost, there is more room for cost growth than cost reduction. In such positively skewed distributions, the cost associated with the peak of the cost probability distribution (the “most likely value” or mode) is less than the median value (where there is a 50 percent expectation that actual costs will be less than the estimate). Arithmetically summing the most likely values—which has been a common practice throughout NASA history—leads to a total mission cost estimate that is less than the median. In practice, summing the most likely values generally leads to a total cost estimate that has only a 20 to 30 percent chance of being sufficient; meaning that there is a 70 to 80 percent probability that actual costs will exceed the estimate. This is a situation that pretty well describes the history of NASA cost outcomes. For example, the final cost of 83 percent of the 40 missions assessed in Primary Reference 1 exceeded their initial costs (including reserves). Beginning with the implementation of NASA Procedural Requirements (NPR) 7120.5C in 2005 (NASA, 2005), NASA required projects to statistically sum WBS costs using Monte Carlo or other mathematical techniques in order to obtain total project costs with higher confidence levels. This policy was further amplified with NPR 7120.5D (NASA, 2007) and NASA Policy Directive (NPD) 1000.5 (NASA, 2009). For NASA science missions, NPD 1000.5 requires cost estimates for individual projects to be calculated at a confidence level of 50 percent or higher, as necessary to ensure that programs are budgeted so that there is at least a 70 percent confidence level that the program budget will be sufficient for all of the missions included therein.  NPD 1000.5 also stipulates that programs and projects should perform statistical analyses necessary to ensure that the joint cost and schedule confidence levels achieve or exceed these thresholds. The committee endorses the current practice of funding programs at the 70 percent confidence level (thereby allowing sufficient reserves to be held at both the project and program levels). However, only projects that have

  A

detailed list of the strengths and weaknesses of various cost-estimating methods appears in NASA, 2008 NASA Cost Estimating Handbook, Washington, D.C.   For programs with a single large mission, this means that the mission must be budgeted to the 70 percent confidence level. However, for programs with multiple missions, mathematical analysis of current cost-estimating methods demonstrates that a program can achieve a 70 confidence level even though budgets for the missions included in that program have a (slightly) lower confidence level. The more missions in the program, the larger the difference in confidence levels can be.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

32

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

recently entered implementation comply with all of these requirements. If the policy works well, projects completing development in the coming years will have much improved cost outcomes. The primary references have shown that missions often expand beyond their initial scope (as defined at SRR and PDR). For example, the 10 missions assessed by Primary Reference 5 demonstrated average increases in spacecraft mass and power requirements of 43 percent and 42 percent, respectively. NASA’s cost models are kept up to date with cost and technical data from the latest projects that have flown. However, there may still be substantial uncertainty in the design of a new project in terms of the readiness level of proposed technologies, realistic mass and power requirements, design heritage, and so on. Early in a mission, it is common to make optimistic assumptions, which means that missions are often proposed and accepted with initial design margins that are significantly smaller than historical experience would dictate. For example, Primary Reference 8 notes that most small spacecraft have very small design margins to save cost. NASA could reduce cost and schedule growth by establishing design rules and other guidelines with minimum required levels of reserves in mass, power, and other key design parameters. Reserves should reflect a more realistic assessment of software and hardware technical heritage (or lack thereof) and other historical experience. Cost and schedule estimates should be based on these risk adjusted designs. Inputs to cost models could be improved by using the NASA Cost Analysis Data Requirement (CADRe) database to help define model parameters at various project milestones (e.g., how much growth should be expected after PDR, in terms of mass, power, complexity, and so on). In addition, parametric cost estimators should have access to independent and experienced systems engineers to obtain more realistic inputs to the models. Cost Growth Methodology  inding. Measurement of Cost Growth. The measurement of cost growth has been inconsistent across proF grams, NASA centers, and congressional appropriation and oversight bodies.

Copyright © 2010. National Academies Press. All rights reserved.

 ecommendation. Measurement of Cost Growth. NASA, Congress, and the Office of Management and R Budget should consistently use the same method to quantify and report cost. In particular, they should use as the baseline a life-cycle cost estimate (that goes through the completion of prime mission operations) produced at preliminary design review. Although NASA Procedural Requirements (NPR) 7120.5D (NASA, 2007) delineates KDP C (PDR) as the first official cost commitment to stakeholders, there continues to be considerable confusion regarding what cost and schedule estimate should be considered the baseline from which future cost and schedule growth should be measured. As detailed in Chapter 1, different reports and studies reach different conclusions regarding the magnitude of cost growth experienced by NASA Earth and space science missions, in part because they define baseline costs differently. The Government Accountability Office (GAO) and Congress generally considers the baseline to be the first time a mission appears as a budget line item in an appropriations bill. This is often before PDR. In addition to the project advocacy cost estimate presented at PDR, cost estimates are presented to various Program Management Councils and independent estimates are generated by subject matter experts and/or the Program Analysis and Evaluation Office. The content of these various estimates also differs—some include Phase A and B, some start with Phase C, some (but not all) include launch costs and/or mission operations, and some include NASA oversight and internal project management costs. These differences make it difficult to develop a clear understanding of trends in cost and schedule growth. However, as noted in Chapter 1, despite the inconsistencies noted in historical cost and schedule growth data, cost and schedule growth are clearly a problem, and urgent action should be initiated to reduce cost and schedule growth in parallel with efforts to avoid similar inconsistencies in the future.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

33

KEY PROBLEMS AND SOLUTIONS

Development Process Management of Announcement of Opportunity Missions and Directed Missions  inding. Differences between announcement of opportunity (AO) missions and directed missions are as F follows:  • The impact of cost growth in AO missions, which are managed within a mission budget line (e.g., Discovery), is limited to other missions within the line. To a much larger degree, cost growth in larger, “flagship” missions has the potential to diminish NASA’s Earth and space science enterprise as a whole.  • AO missions generally are selected at a very low level of system maturity and are very vulnerable to technology and instrument development problems.  ecommendation. Management of Large, Directed Missions. NASA headquarters’ project oversight funcR tion should pay particular attention to the cost and schedule of its larger missions (total cost on the order of $500 million or more), especially directed missions (which form a single line item).  ecommendation. Management of Announcement of Opportunity (AO) Missions. NASA should continue R to emphasize science in the AO mission selection process, while revising the AO mission selection process to allocate a larger percentage of project funds for risk reduction and improved cost estimation prior to final selection.

Copyright © 2010. National Academies Press. All rights reserved.

 ecommendation. Incentives. NASA should ensure that proposal selection and project management processes R include incentives for program managers, project managers, and principal investigators to establish realistic cost estimates and minimize or avoid cost growth at every phase of the mission life cycle, for both directed missions and announcement of opportunity missions. As noted in Chapter 1, the primary references indicate that extensive cost growth exists in many—but not all—NASA Earth and space science missions. In addition, cost and schedule growth have made it more difficult to accomplish those missions. For a variety of reasons (e.g., inherent optimism, excessive technical and schedule risk, and faulty assumptions about the ease of adapting heritage technologies and systems), initial estimates are typically too low and highly uncertain. Neither the primary references nor this committee conclude that deficiencies in the qualification or expertise of NASA project managers or PIs have been a significant cause of cost growth of NASA Earth and space science missions as a whole. Even so, a project manager or PI who is personally determined to control costs can be of great assistance in avoiding cost growth. People and organizations tend to optimize their behavior based on the environment in which they operate, and so it is important to motivate and reward vigilance in accurately predicting and controlling costs. However, the differing nature and goals of directed and AO missions calls for different management approaches. In particular, the impact of cost growth in AO missions that are managed within a mission budget line (e.g., Discovery) is limited to other missions within the line. To a much larger degree, cost growth in larger, “flagship” missions has the potential to diminish NASA’s Earth and space science enterprise as a whole. Prospective PIs for AO missions compete with other PIs proposing different missions within selected science areas with cost caps and schedule constraints defined by the AO. These constraints are intended to limit the scope and complexity of the proposed missions. However, experience to date has shown that the potential science return is the predominant factor in selection of AO missions; cost realism is given relatively little weight as long as the proposed cost is within the cost cap. AO missions tend to have a smaller scope, less complexity, and less spacecraft mass than directed missions do. As a result, the average cost and schedule for AO missions is less than for directed missions. However, the selection process for AO missions is designed to solicit mission concepts with a much lower level of technological and system maturity for mission instruments and spacecraft than is the case for directed missions. This increases

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

34

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE 2.1  Average Cost and Schedule Growth (in Excess of Reserves) for Missions from Primary Reference 1

Number of Missions

Initial Cost (million $)

Copyright © 2010. National Academies Press. All rights reserved.

All 40 missions   Directed 18 296   AO 22   90 14 missions with largest cost growth   Directed  9 362   AO  5 127

Cost Growth (%)

Initial Schedule (months)

Average Schedule Growth (months)

Schedule Growth (%)

44 23

15 25

60 44

13 10

21 23

89 77

25 60

64 49

14 22

22 45

Average Cost Growth (million $)

technological risks that must be overcome during the developmental process. As a result, AO missions tend to have higher percentage cost and schedule growth than do directed missions, but this is offset by the lower cost and shorter schedules they begin with. For example, Primary Reference 1 analyzed the cost growth (in excess of reserves) for 40 missions, and the results are charted in Figures 1.3a, b, and c and summarized in Table 2.1. As shown, on average, cost growth for AO missions was higher than for directed missions (AO, 25 percent; directed, 15 percent). However, this is more than offset by the higher average cost of directed missions, so that directed missions, on average, had a higher absolute cost growth (AO, $23 million; directed, $44 million). Among the 14 missions with the most cost growth (which accounted for 92 percent of the total cost growth, in absolute terms, for these 40 missions), the 5 AO missions had an average cost growth that was much higher than for 9 directed missions (AO, 60 percent; directed, 25 percent). Even so, because the average cost for the high-growth directed missions was almost three times the average cost for the high-growth AO missions, the absolute value of the cost growth of the high-growth directed missions remained higher than for the high-growth AO missions (AO, $77 million; directed, $89 million). Similarly, directed missions, on average, had longer schedules than AO missions had. For the full set of 40 missions, however, AO and directed missions had approximately the same schedule growth, both in terms of percentage and in absolute terms. For the missions with high cost growth, on the other hand, AO missions had more schedule growth than directed missions had in terms of percentage (AO, 45 percent; directed, 22 percent) and in absolute terms (AO, 22 months; directed, 14 months). The primary references have identified key reasons for cost and schedule growth that go beyond initial underestimates. Prior to CDR, important factors are unstable budgets, design evolution leading to increased complexity, excessive reviews, and difficulty in maturing new technologies. Following CDR, where most cost and schedule growth occurs, the most troublesome factors are associated with late delivery of instruments, hardware and software integration and testing, and launch vehicle availability and cost. These issues are more readily addressed with directed missions where the strategic and project planning processes are more likely to assure that risks associated with technology development, instrument development, and design maturity are sufficiently retired prior to PDR. AO missions are selected at a very low level of system maturity (typically after expending only 1 to 2 percent of total mission costs), and they are more susceptible to technology and instrument development problems. NASA manages the Discovery, Explorer, Mars Scout, New Frontiers, and other similar science programs as separate line items at NASA headquarters and at the various centers. As a result, the impact of cost growth in individual projects is accommodated within the line item, typically by delaying less mature projects. This is a reasonable approach as long as the cost growth of each individual AO mission can be accommodated within the overall budget for its respective science program; the alternative of attempting to force each AO mission to stay within its original, often highly optimistic budget estimate is more problematic. Managing AO missions primarily by program line item rather than by mission can thereby accomplish the overall goal of limiting cost growth in the NASA science budget while still achieving valuable science missions. A similar argument does not hold for the directed missions, which are much more expensive and each of which forms a single line item. In particular, the impact of a large cost overrun in these missions tends to be much more severe, with the potential to impact many other missions within the SMD, and other directorates as well.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

KEY PROBLEMS AND SOLUTIONS

35

Copyright © 2010. National Academies Press. All rights reserved.

Cost growth of large (expensive) missions clearly creates a much larger budgetary problem and has a much larger impact on other missions. Of the 40 missions in Primary Reference 1, 92 percent of the total cost growth was accounted for by just 14 missions. In addition, 8 of the 10 missions with the highest initial cost estimates were among the set of 14 with the most cost growth (in absolute terms). Directed missions come in all sizes. In Primary Reference 1, 4 of the 10 smallest missions were directed. However, most large missions are directed missions. In Primary Reference 1, the 9 largest missions (in terms of initial cost estimates) were all directed; the most expensive AO mission comes in at number 10. Therefore, one way to help reduce overall cost growth would be for NASA to monitor the most expensive directed missions more closely than other missions. When assessing trends for large missions compared to small, the results are similar to the above comparison of directed and AO missions, perhaps because almost all of the most expensive missions are directed. Of the 40 missions in Primary Reference 1, the 10 with the highest initial cost had a cost growth of 15 percent (in excess of reserves), compared to 48 percent for the 10 smallest missions. Nonetheless, the 10 missions with the largest initial costs had total cost growth of $649 million, dwarfing the total cost growth of the 10 smallest missions ($210 million). In fact, the 10 largest missions accounted for more than half of the total cost growth. The extent to which a relatively small number of missions run up the total cost growth of NASA Earth and space science missions would be even worse if some missions currently in development, such as the Mars Science Laboratory (cost growth of approximately $660 million) and the James Webb Space Telescope (cost growth of approximately $1.5 billion), were included (GAO, 2010.) Given the dominant role that large missions play in determining total cost growth of NASA’s Earth and space science missions, it would be prudent to pay special attention to missions whose planned cost exceeds a specified threshold. The average initial cost of the 9 most expensive directed missions in Primary Reference 1 was $474 million and the initial cost for number 9 (the TRMM mission) was $253 million. However, these costs do not include the cost of launch, mission operations, data analysis, or inflation (these missions were developed and launched mostly in the 1990s). For directed missions, cost realism should be emphasized from the start, with special emphasis given to missions with costs on the order of $500 million or more. This should begin with the decadal survey process and continue with disciplined program and project management: technology development and validation should precede the commitment to a given subsystem; instruments should be thoroughly tested before defining the spacecraft interfaces; PDR and CDR should be rigorous; and viable descope options should be identified in all project phases (and implemented as necessary to hold the line on cost and schedule growth). Thus, for NASA to be an effective, cost-conscious agency, every effort should be made to complete directed missions within cost, on schedule, and with the expected performance. For AO missions, cost growth could be addressed by changes to the AO selection process. Currently, the NASA AO selection process has two steps. During Step 1, an AO is released to solicit proposals for new mission concepts addressing questions of fundamental importance to the science community. At this point, proposing institutions (with no funding from NASA) will spend less than 1 percent of the stated cost cap on defining requirements, developing the concept, and maturing the required technologies to prepare a proposal. During Step 1, the emphasis is on science, because the initial selection of two or three proposals for Step 2 is primarily on the basis of value  The GAO recently released a report assessing the cost and schedule growth of selected NASA Earth and Space Science projects currently in implementation (GAO, 2010). The average development cost growth of 10 projects that had been in the implementation phase for several years was $121 million, corresponding to 18.7 percent of their total initial budgets. However, one of these projects, a directed mission known as the Mars Science Laboratory (MSL), has had a cost growth of $662.4 million, corresponding to 68.4 percent of the initial cost.   Interestingly, the summary table in the GAO study credits the James Webb Space Telescope (JWST) with zero cost growth, because in 2009 the mission was rebaselined, and the mission remains within the new baseline cost that was established at that time. However, the GAO study also reports that the JWST was replanned in 2006 after a cost growth of $1 billion, and costs increased another $500 million when the program was rebaselined in 2009. This reaffirms the potential for directed missions to have absolute cost growth (in dollars) that far exceeds the cost growth of AO missions even in situations where the average cost growth of directed and AO missions is similar. The cost growth of the MSL and JWST missions (totaling more than $2 billion) could have entirely funded a great many AO missions, and the accompanying slip in MSL’s launch date has severely impacted other Mars missions already in development as well as planned missions in the next decade. Thus, while it is certainly important to try to control cost growth on all missions, it is absolutely crucial to control cost growth on large directed missions, because when their costs get out of control, the potential impact is enormous.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

36

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

Copyright © 2010. National Academies Press. All rights reserved.

and quality of the science being proposed. During Step 2, NASA provides supplemental funding to the initial winners to further define the mission concept, technology, and so on. Still, even by the point of final downselect to a single mission concept, less than 2 percent of the stated cost cap will have been spent. To be selected for an AO mission, a PI must propose a mission that will generate as much new scientific information as possible, and there is little or no penalty (in terms of the proposal evaluation process) for unrealistic cost estimates as long as the estimate falls within the cost “cap.” In addition, NASA makes only a small financial investment in helping PIs who reach Step 2 to refine their requirements, costs, and schedule as well as to reduce risk by maturing key technologies. This lack of funds early on in the process leads to overly optimistic cost estimates because little effort can be expended to identify and retire risks. Furthermore, the cost-capped mission model encourages PIs and their contractors to provide unrealistic, over-optimistic cost estimates and incentives to control costs after selection are ineffective. The result is poor cost performance. In addition, ongoing and planned missions suffer as they are slowed down to make funds available to cover overruns of earlier projects. These problems could be alleviated by a revised AO selection process that postpones final commitment to execute a selected mission until a reliable cost estimate can be prepared as follows: 1. Continue to select missions based primarily on science return but recognize that the “cost cap” of the mission is better described as a cost target. This change in philosophical approach recognizes that AO missions seldom come in at or below their cost “cap” because of the competitive nature of the process, inadequate funds early in the process for technology maturation and risk reduction, and the difficulty of estimating the cost of immature system concepts for new one-of-a-kind missions. Why call it a cost “cap” when initial cost estimates are so uncertain and the cost growth of AO missions is generally accommodated, one way or the other? The Vegetation Canopy Lidar mission and the Full-sky Astrometric Mapping Explorer mission were cancelled because of cost increases, but this rarely occurs. 2. After downselecting a single mission, instead of proceeding with Phase B, as is the current practice, continue with an extended Phase A. Phase A is the best opportunity to improve the accuracy of the mission concept and cost estimates (NRC, 2006). During this extended Phase A, NASA should provide sufficient funding (up to 5 percent of target cost) to reduce risk, improve cost estimates and associated cost risks, and identify potential descopes. Risk reduction should focus on technology development of the highest risk elements of the proposed approach, including development of test hardware as appropriate. Descopes would be exercised at the end of the risk reduction phase to lower the proposed cost closer to the cost target, if necessary. The amount allocated to this task should be enough to reduce risk by a significant amount, but it should be small enough that the effort could be terminated without undue concern about wasting funds. 3. At the end of the extended Phase A, develop an independent cost estimate and assess the technological maturity of the high-risk elements of the proposed approach. This would ensure that there is a good understanding of the residual risk and realism of the cost estimate prior to the final confirmation of the selected mission. Before confirming the proposed mission for development, the PI should demonstrate—and NASA should concur—that (1) the mission is affordable (relative to the cost target); (2) it has a realistic cost estimate; (3) the technology needed by the high-risk elements is adequately mature; and (4) the budget includes sufficient cost reserves. If a mission fails to meet these four criteria, then necessary corrective action should be taken (e.g., by continuing the extended Phase A to address residual questions and concerns) or the proposed mission should be terminated; in no case should NASA allow AO missions to proceed into Phase B without meeting these criteria. Concurrent with efforts to improve management processes for AO and directed missions, NASA should establish incentives for PIs, project managers, and program managers to minimize costs and avoid cost growth throughout mission life, while still meeting or exceeding minimum mission requirements, including science return and mission duration. For example, NASA might encourage mission managers to minimize cost growth by establishing a policy that some percentage of unexpended reserves would routinely be made available for extended mission operations.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

37

KEY PROBLEMS AND SOLUTIONS

Technology and Instrument Development Technology Development  inding. Technology Development. The limited time and resources available in phases A and B to mature F new technology and solidify system design parameters contribute to cost growth through higher risk and unrealistic cost estimates.  ecommendation. Technology Development. NASA should increase the emphasis in phases A and B on R technology development, risk reduction, and realism of cost estimates. A recurring theme in the primary references is technical and programmatic uncertainty at the beginning of a project. Not enough time or resources are available in phases A and B to mature new technologies or to solidify principal system design parameters. NASA NPR 7120.5D requires that “during formulation, the project establishes performance metrics, explores the full range of implementation options, defines an affordable project concept to meet requirements specified in the Program Plan, develops needed technologies, and develops and documents the project plan” (NASA 2007, Section 2.3.4). Despite these procedural requirements, the primary references identify an ongoing need to improve technical and programmatic definition at the beginning of a project. This would require more time and funding for Phase A and Phase B, especially for complex projects, to allow more time for development of technology, baseline costs, funding profiles, and the overall implementation plan. Instrument Development  inding. Instrument Development. Delays and cost increases for instrument development are pervasive and F impact a large number of missions.  ecommendation. Instrument Development. NASA should initiate instrument development well in advance R of starting other project elements and establish a robust instrument technology development effort relevant to all classes of Earth and space science missions to strengthen and sustain the nation’s instrument development capability.

Copyright © 2010. National Academies Press. All rights reserved.

 ecommendation. Decadal Surveys. NASA should ensure that guidance regarding the development of instruR ments and other technologies is included in decadal surveys and other strategic planning efforts. In particular, future decadal surveys should prioritize science mission areas that could be addressed by future announcements of opportunity and the instruments needed to carry out those missions. Under NASA sponsorship, the National Research Council now performs decadal surveys for all SMD scientific disciplines, providing retrospective and forward-looking assessments of status and opportunities as well as recommendations for scientific and programmatic priorities for future investments. These surveys are broadly based and widely respected. To further improve the quality and utility of the surveys, a workshop on decadal science strategy surveys was held in November 2006 “… to review lessons learned from the most recent surveys, and to identify potential approaches for future surveys that can enhance their realism, utility, and endurance” (NRC, 2007). The workshop addressed several issues that are germane to the current discussion: • When a large, flagship mission encounters significant cost growth, it has dramatic impact on SMD’s balanced portfolio of large, medium, and small projects. • Estimates of program cost and technology maturity have often proved to be overly optimistic.

  Program

phases are described in Chapter 1. See Figure 1.1.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

38

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

• Instrument development is a leading contributor to cost risk, so that programs for reducing risk related to instruments are especially important. Instrument development problems were also identified as a major cause of cost growth in many of the primary references. Of particular importance are instrument development problems caused, for example, by instrument designs that lack technical details and/or fail to identify technical challenges. Shrinkage of the U.S. industrial base supporting space system development is exacerbating problems with the development of new instruments, particularly because Earth and space science missions generally require special purpose, one-of-a-kind components. It is not necessary to complete the development of new instruments before the missions that would use such instruments are initiated. However, the committee concurs with Primary References 1 and 3, which recommend increasing support for early instrument development to increase technical maturity and reduce risk. The committee concurs that NASA should (1) establish a robust instrument technology development program and (2) initiate instrument development well in advance of starting other program elements, especially in cases where there is a distinct separation between the payload and spacecraft bus with well understood interfaces. Currently, NASA supports several programs for early instrument development in selected program areas: the Mars Instrument Development Program, the Planetary Instrument Development Program, and the Instrument Incubator Program for Earth science. These programs seem to be quite successful and could serve as models for additional and more robust efforts to meet the needs of all Earth and space science mission areas for new instruments and instrument technologies. Cost growth caused by problems with the development of instruments and other advanced spacecraft technologies could also be addressed by expanding the scope of future decadal surveys to include instrument technology development. The statements of task for future surveys (and other strategic planning efforts) should ask for recommendations concerning the development of future instrument concepts and other technologies of particular importance to the next generation of science missions. The scope of these efforts should not be limited to directed missions. In addition, decadal surveys should also prioritize science mission areas that could be addressed by future AOs and the instruments needed to carry out these missions. As noted above, AO mission concepts are generally selected with a much lower level of instrument technological maturity than for directed missions, and development of instrument technology has been a particular problem for some AO missions. Identifying key priorities and investing in the technological needs of both directed and AO missions would make the best use of limited instrument development funds. Major Reviews

Copyright © 2010. National Academies Press. All rights reserved.

 inding. External Project Reviews. NASA has increased the size and number of external project reviews to F the point that some reviews are counterproductive. The large number of non-consensus reviews exacerbates this problem.  ecommendation. External Project Reviews. NASA should reassess its approach to external project reviews R to ensure that (1) the value added by each review outweighs the cost (in time and resources) that it places on projects; (2) the number and the size of reviews are appropriate given the size of the project; and (3) major reviews, such as preliminary design review and critical design review, occur only when specified success criteria are likely to be met. The primary references, other reports (see Box 2.1), as well as presentations to the committee by industrial representatives and NASA program personnel, alluded to the disruptive effect of an excessive number of external reviews. Primary Reference 2 noted that “large review teams and frequent reviews are too much—particularly for small missions” (p. 70). For example, the current (external) Standing Review Board (SRB) approach has been extended downward to all missions having a baseline cost of $250 million or more; this encompasses most missions. In addition, apparently because of conflict of interest issues that arose in NASA’s Constellation program, many boards now conduct non-consensus reviews. Simply stated, if a board includes any members who are not govern-

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

KEY PROBLEMS AND SOLUTIONS

39

BOX 2.1 Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs The National Research Council’s 2009 report Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs1 addressed many problems and issues similar to those facing NASA project managers. This study focused on concerns about the growing number of program and technical reviews of Air Force programs. The benefit of these reviews was sometimes outweighed by the time that program staff spent supporting them instead of tending to program execution. The resulting report did not single out specific reviews for modification or elimination. Rather, it focused on ways in which to improve the overall management of the review process as follows: • Improving front-end planning. The Air Force Senior Acquisition Executive should establish a process to plan, coordinate, and execute reviews at all levels of the organization. Lack of effective coordination and a multiplicity of informal pre-reviews resulted in significant costs in both time and money for program managers. In addition, to ensure that the review remains well focused, the report recommended that objectives, metrics, and success criteria be provided to the program manager well in advance of the review, with a follow-up report addressing each of these areas. • Better synchronizing of the reviews. By focusing more on milestones and decision points and seeing that reviews are properly timed based on the readiness of the program to move forward, there should be a reduction in pre-reviews, cutbacks in costs, and fewer schedule delays. • Ensuring a clear basis of need. Before any new review is put in place, the senior acquisition executive should compare its objectives with existing reviews to see if those might be modified, perhaps by broadening the review group. • Seeing that appropriate subject matter experts participate. The absence of reviewers with critical skill sets significantly reduces their effectiveness. • Clearly documenting and disseminating the output of the review. Documenting the outcomes of reviews and providing lessons learned should support a focus on results and foster an environment of continuous improvement. Implementing the above recommendations was expected to afford the Air Force Senior Acquisition Executive greater control over the review process and reduce the costs and time associated with oversight. Inasmuch as NASA project and management officials encounter similar issues, NASA should consider taking similar action, as appropriate and relevant to NASA’s operations. 1

Copyright © 2010. National Academies Press. All rights reserved.

NRC, Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs, The National Academies Press, Washington, D.C., 2009.

ment employees, the chair is not allowed to drive the board to a consensus position. Rather, the chair is expected to assimilate the various points of view of the board members in a single document that represents his/her own opinion, augmented by highlighting each member’s individual views. This can be a difficult task, because deliberations intended to achieve a consensus viewpoint are not allowed. Additionally, each member of a non-consensus board writes an individual report that is appended to the chair’s report. Project staff are then expected to respond to the comments of all board members, even though the comments may be inconsistent or even contradictory. It may be difficult to meet the requirements associated with holding consensus reviews, but holding a small number of consensus reviews is likely to be more valuable—and more cost-effective—than is the current practice of holding a large number of non-consensus reviews. Another option for reducing the number of reviews would be to make greater use of subject matter experts as in-plant monitors in lieu of some reviews.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

40

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

Other troubling aspects of NASA’s current approach to conducting reviews include the following: • Review boards have grown in size, sometimes to the point of outgrowing existing conference facilities, and program offices have difficulty finding new facilities large enough to accommodate the large number of participants. With some relatively small projects, the number of reviewers sometimes exceeds the size of the project staff. • The number of reviews has increased also. For example, SRBs are conducted at the mission level. In preparation for each SRB, a series of Integrated Independent Review Team Reviews is conducted for each major element of the mission (e.g., spacecraft, ground system, and each instrument). Currently, a relatively simple mission with a spacecraft, a ground system, and three instruments can expect to have more than 30 external independent reviews during its life cycle, in addition to the internal reviews conducted by NASA staff. This large number of reviews constitutes a major distraction that adds significant cost. • Given the importance of cost and schedule growth, SRBs are now charged with assessing programmatic as well as technical issues. However, it is difficult for one body to accomplish both of these tasks efficiently and effectively. • Large numbers of reviews can diffuse responsibility and accountability, creating an environment where NASA senior managers can become dependent on review teams with many outside members who sometimes do not understand NASA, the field center in question, and/or the mission being reviewed. NPR 1720.5 is quite specific about the composition, scheduling, and intent of project reviews, and it calls for cost and schedule to be reviewed as part of PDR and CDR. Well-conducted reviews allow program personnel and the sponsoring organization to identify problem areas and required mitigation steps. However, the involvement of large numbers of reviewers who are unfamiliar with the mission is not conducive to project progress. Also, if a mission has not progressed far enough for a review to accomplish its intended purpose, the review is a wasted effort. Major reviews would be more effective in reducing cost and schedule growth if NASA consistently exercised greater care in the conduct of major reviews by assuring that missions are ready for review, programmatically and technologically, and that the review board is appropriately selected and well prepared to conduct a given review. Proceeding with a review when a project is not ready or the review board is poorly constituted is likely to increase cost and schedule growth in subsequent project phases. Launch Vehicles  inding. Launch Vehicles. Problems with the procurement of launch vehicles and launch services are a sigF nificant source of cost growth.

Copyright © 2010. National Academies Press. All rights reserved.

 ecommendation. Launch Vehicles. Prior to preliminary design review, NASA should minimize missionR unique launch site processing requirements. NASA should also select the launch vehicle with appropriate margins as early as possible and minimize changes in launch vehicles. One of the primary references identifies launch service issues and delays in launch readiness as the primary external factor influencing cost growth of NASA missions. Specific factors include increases in the cost of expendable launch vehicles, vendor issues such as strikes, weather-related issues at the launch site, problems with launch site facility capabilities or range availability, and delays in the availability of a given launch vehicle. In addition, if a mission is required to change launch vehicles, the costs can be substantial. Differences in launch vehicle interface requirements can require substantial changes to spacecraft design and testing requirements. These problems could be minimized by taking a more disciplined approach to launch that fully recognizes the potential cost of setting unique requirements for launch site processing and/or changing launch vehicles. Comprehensive, Integrated Strategy for Cost and Schedule Control  inding. Comprehensive, Integrated Cost Containment Strategy. Recent changes by NASA in the developF ment and management of Earth and space science missions are promising. These changes include budgeting

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

41

KEY PROBLEMS AND SOLUTIONS

programs to the 70 percent confidence level and specifying that decadal surveys include independent cost estimates. However, it is too early to assess the effectiveness of these actions, and NASA has not taken the important step of developing a comprehensive, integrated strategy.  ecommendation. Comprehensive, Integrated Cost Containment Strategy. NASA should develop an inteR grated, comprehensive strategy to contain cost and schedule growth and enable more frequent science opportunities than would be possible in an environment with large cost growth. This strategy should include recent changes that NASA has already implemented as well as other actions recommended in this report. Internal Factors A consistent approach for defining cost and schedule growth and applying rigorous independent parametric cost and schedule estimates is necessary for a clear and consistent understanding of how individual projects are performing. Cost estimates become more accurate as risks and uncertainties are reduced through the maturation of critical technologies for instruments and other key systems, subsystems, and components. An increased emphasis on technology development in phases A and B will help missions avoid the cost and schedule growth—and the false expectations—that arise from sometimes optimistic assumptions about how easy it will be to develop new technology for a particular mission. The selective application of reviews that are tailored to add value to each mission and rigorously comply with NASA Space Flight Program and Project Management Requirements (NPR 7120.5D; NASA, 2007) would provide the necessary insight into mission status and highlight issues that may require special attention. In addition, larger missions (with costs on the order of $500 million or more) deserve special attention because cost and schedule growth of these missions can have a major impact on the stability of NASA’s overall Earth and space science mission portfolio and schedule. Although cost growth in smaller missions only rarely has the same kind of effect, a proactive effort to improve the cost and schedule performance of smaller missions is also encouraged to conserve resources and improve confidence in NASA’s plans and project management capabilities. As discussed in this study and summarized in the recommendations and supporting text above, the elements of the recommended cost and schedule strategy should include the following internal agency actions:

Copyright © 2010. National Academies Press. All rights reserved.

• Standardize cost baseline definition. • Improve independent cost estimating capability. Adopt and report using parametrically developed and experience-based cost estimates. • Increase the emphasis on early development of technologies for instruments and other spacecraft systems, within existing missions and as independent research activities. • Tighten use of the provisions of NPR 7120.5 regarding reviews and planning documentation. • Provide concrete incentives for program managers, project managers, and PIs to establish realistic cost estimates and minimize or avoid cost growth at every phase of mission life for both directed and AO missions. • Focus NASA headquarters’ project oversight function on the cost and schedule of its larger missions (costs on the order $500 million or more), particularly those which form a single line item. In addition to the issues discussed previously in this study, the effectiveness of the recommended comprehensive cost containment strategy would be enhanced by addressing key workforce, infrastructural, and organizational issues. To be successful, NASA must have access to a pool of talented personnel and key infrastructure, internally and externally, in national laboratories, universities, and industry. With constrained financial resources and ambitious plans for the future, optimizing the use of these assets is increasingly important. NASA may wish to consider opportunities to improve the effectiveness of its center structure by enhancing centers of excellence and sustaining redundant activities and capabilities only when necessary, for example, to maintain a competition of ideas or an assured national capability in critical areas. The recommendations contained in this report, especially the call for a comprehensive, integrated strategy for controlling cost and schedule growth, would enable NASA to conduct more frequent Earth and space science mis-

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

42

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

sions, to enhance accountability, and to build confidence in NASA’s Earth and space science endeavors, internally and within its external constituencies. External Factors

Copyright © 2010. National Academies Press. All rights reserved.

The primary references conclude that most cost and schedule growth is caused by internal factors that NASA has the ability to control. However, external factors also contribute to cost and schedule growth. The recommended comprehensive strategy to control cost and schedule growth should address key external factors, such as the following: • Industrial base. Consolidation within the shrinking U.S. space industrial base, which is largely dependent on the U.S. national security and NASA budgets, has reduced the number of competitors for NASA (and DOD) missions (CSIS, 2008). The limited, occasional demand for some technologies and systems that are critical to Earth and space science missions can result in a shortage of suppliers that makes it more difficult and costly for NASA to develop advanced instruments and subsystems for new missions. To improve the nation’s space technology capability, particularly in instrument development, NASA and other government technology organizations could leverage technology development across U.S. government space activities while also engaging in the national discussion on industrial base issues, especially with regard to export controls that are limiting the global competitiveness of the U.S. space industry and promoting the development of foreign competitors in areas traditionally dominated by U.S. industry (AFRL, 2007). • Workforce. As noted in the report Rising Above the Gathering Storm, “an educated, innovative, motivated workforce—human capital—is the most precious resource of any country” in the world (NAS-NAE-IOM, 2007, p. 30). That study recommended making the United States the most attractive setting in the world in which to study and perform research. Earth and space science missions clearly need many critical skills that can only be developed through a combination of advanced education and work experience. Critical shortages are already developing in some areas, such as program management, systems engineering, and software development capabilities (AFRL, 2007). Any reduction in key workforce skill areas would be a potentially serious problem. NASA should continue and, as appropriate, increase its support and development of the United States’ future science and engineering workforce through scholarships, fellowships, and internships. • Launch vehicles. Implementation of a standard set of launch-vehicle-to-payload interfaces for use by all U.S. government agencies, as well as more standard launch vehicle production lines, would likely reduce mission cost and schedule and increase the probability of mission success. Greater standardization would also help avoid the sometimes substantial cost of modifying systems when launch vehicles are changed during the development of a particular mission. Collaborative efforts by NASA and other agencies to develop a coordinated strategy for access to space, as well as related technologies and vehicles, would help assure reliable, timely, and affordable access to space. • Funding Stability. As program execution becomes more disciplined and cost and schedule growth is reduced, it will become increasingly important to maintain a funding profile that is consistent with the established, baselined program. A concerted effort by individual NASA missions, NASA’s Earth and space science programs, OMB, and Congress to improve budget stability will maximize the programmatic efficiency and scientific results produced by NASA. Strategic Benefits NASA sets the strategic direction of its Earth and space science programs using decadal surveys, the SMD science plan, and supporting road maps. A comprehensive, integrated approach to control cost and schedule growth is also essential. The primary references include dozens of specific causes, dozens of specific recommendations, and dozens of findings concerning this problem (see Table 1.6 and Appendix C). The primary references are generally consistent and comprehensive, and so the individual causes of cost growth and the necessary correction action are

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

KEY PROBLEMS AND SOLUTIONS

43

Copyright © 2010. National Academies Press. All rights reserved.

not a mystery. However, rather than simply picking and choosing from among these long lists of causes, findings, and recommendations, development of an integrated strategy offers the best chance that future actions will work in concert to minimize or eliminate cost and schedule growth. Internally, an integrated cost containment strategy would improve the definition of baseline costs and enhance the utility of NASA’s independent cost-estimating capabilities. Early development of technologies and more effective program reviews would improve the ability to identify and effectively manage risks and uncertainties. Externally, NASA has the opportunity to collaborate with other federal agencies, OMB, and Congress to sustain and improve critical capabilities and expertise in the industrial base and the nation’s science and engineering workforce, to address cost and schedule risk associated with launch vehicles, and to improve funding stability. Successful implementation of a comprehensive, integrated strategy for cost and schedule growth of NASA Earth and space science missions will benefit both NASA and the nation, while enabling NASA to more efficiently and effectively carry out these critical missions.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

References

primary References

Copyright © 2010. National Academies Press. All rights reserved.

NASA Cost Studies   1. Bitten, R.E., D.L. Emmons, and C.W. Freaner. 2006. Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines. IEEE Paper #1545. December.   2. NASA Langley Research Center (LaRC). 2007. Cost/Schedule Performance Study for the Science Mission Directorate. Final Report. Prepared by the NASA LaRC Science Support Office and Science Applications International Corporation. NASA LaRC, Hampton, Va. October.   3. NASA. 2007. “Cost and Schedule Growth at NASA.” Presentation provided to the committee by Director of the Cost Analysis Division Tom Coonce, Office of Program Analysis and Evaluation, NASA, Washington, D.C. November.   4. NASA. 2008. “SMD Cost/Schedule Performance StudySummary Overview.” Presentation by B. Perry and C. Bruno, NASA Science Support Office; M. Jacobs, M. Doyle, S. Hayes, M. Stancati, W. Richie, and J. Rogers, Science Applications International Corporation. January.   5. Freaner, C.W., R.E. Bitten, D.A. Bearden, and D.L. Emmons. 2008. “An Assessment of the Inherent Optimism in Early Conceptual Designs and Its Effect on Cost and Schedule Growth.” Paper presented at the Space Systems Cost Analysis Group/Cost Analysis and Forecasting/European Aerospace Cost Engineering Working Group 2008 Joint International Conference, European Space Research and Technology Centre, Noordwijk, The Netherlands, May 15-16. European Space Agency, Paris, France.   6. Mlynczak, B., and B. Perry, Science Support Office, NASA. 2009. “SMD Earth and Space Mission Cost Driver Comparison Study. Final Report and Presentation.” March. Related Analyses   7. General Accounting Office. 1992. Space Missions Require Substantially More Funding Than Initially Estimated. GAO/NSIAD-93-97. Washington, D.C. December.   8. National Research Council. 1997. Reducing the Costs of Space Science Research Missions: Proceedings of a Workshop. National Academy Press, Washington, D.C. 44

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

45

REFERENCES

  9. McCrillis, J., Office of the Secretary of Defense, Cost Analysis Improvement Group. 2003. “Cost Growth of Major Defense Programs.” Presentation to the Annual Department of Defense Cost Analysis Symposium, January 30, Williamsburg, Va. 10. Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. 2003. Report of the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force on Acquisition of National Security Space Programs. Department of Defense, Washington, D.C. May.

Copyright © 2010. National Academies Press. All rights reserved.

ADDITIONAL References AFRL (Air Force Research Laboratory). 2007. Defense Industrial Base Assessment: U.S. Space Industry. Final Report. Dayton, Ohio: AFRL. August 31. Available at http://www.space.commerce.gov/library/reports/. Bearden, D. 2008. Perspectives on NASA Mission Cost and Schedule Performance. Presentation at GFC Symposium, June 3. Bitten, R. 2008. Perspectives on NASA Mission Cost and Schedule Performance Trends. Presentation for the Future In-Space Operations Colloquium, July 2. CSIS (Center for Strategic and International Studies). 2008. Health of the U.S. Space Industrial Base and the Impact of Export Controls. Report Released February 19. Available at http://csis.org/files/ media/csis/pubs/021908_ csis_spaceindustryitar_final.pdf. Freaner, C., R. Bitten, and D. Emmons. 2010. Inherent optimism in early conceptual designs and its effect on cost and schedule growth: An update. Paper presented at the 2010 NASA Program Management Challenge, February 9-10, 2010, Houston, Texas. GAO (Government Accountability Office). 2010. NASA: Assessments of Selected Large-Scale Projects. GAO, Washington, D.C. Mlynczak, B., and Perry, B. 2009. SMD Cost/Schedule Performance Study Final Report Overview. Presentation to the Committee on Cost Growth in NASA Earth and Space Science Missions. Washington, D.C. September 1. NAS-NAE-IOM (National Academy of Sciences, National Academy of Engineering, and Institute of Medicine). 2007. Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future. Committee on Science, Engineering, and Public Policy. The National Academies Press, Washington, D.C. NASA (National Aeronautics and Space Administration). 2008 NASA Cost Estimating Handbook. NASA Headquarters Cost Analysis Division. NASA, Washington, D.C. NASA. 2005. NASA Procedural Requirements (NPR) 7120.5C. NASA Space Flight Program and Project Management Requirements. NASA, Washington, D.C. NASA. 2007. NASA Procedural Requirements (NPR) 7120.5D. NASA Space Flight Program and Project Management Requirements. NASA, Washington, D.C. NASA. 2009. NASA Policy Directive (NPD) 1000.5. NASA, Washington, D.C. January. NASA/GSFC (NASA Goddard Space Flight Center). 2009. Criteria for Flight Project Critical Milestone Reviews. GSFC-STD-1001. NASA Goddard Space Flight Center, Greenbelt, Md. Available at http://standards.gsfc. nasa.gov/gsfc-stds.html. NRC (National Research Council). 1995. Review of Gravity Probe B. National Academy Press, Washington, D.C. NRC. 2006. Principal-Investigator-Led Missions in the Space Sciences. The National Academies Press, Washington, D.C. NRC. 2007. Decadal Science Strategy Surveys: Report of a Workshop. The National Academies Press, Washington, D.C. NRC. 2009. Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs. The National Academies Press, Washington, D.C. Schaffer, M. 2004. NASA Cost Growth: A Look at Recent Performance. NASA, Washington, D.C.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Copyright © 2010. National Academies Press. All rights reserved. National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Copyright © 2010. National Academies Press. All rights reserved.

Appendixes

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

Copyright © 2010. National Academies Press. All rights reserved. National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

A Statement of Task and Supporting Documents

The full text of the study statement of task appears below, along with two other earlier documents that helped define the framework for conducting this study: • Tasking letter from Edward J. Weiler, Associate Administrator for Science Mission Directorate, NASA Headquarters, to Dr. Charles F. Kennel, Chair, Space Studies Board, National Research Council, dated February 19, 2009, • Legislation: NASA Authorization Act of 2008 (P.L. 110-422), Section 508. Assessment of Cost Growth. Study Statement of Task After receiving the tasking letter, the National Research Council and NASA agreed to the following statement of task for this study:

Copyright © 2010. National Academies Press. All rights reserved.

The National Research Council will assemble a committee to identify the primary causes of cost growth in NASA Earth and space science missions involving large, medium, and small spacecraft. The committee will recommend what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s Earth and space science programs. In particular, the committee will: • Review existing cost growth studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth. • Assess whether those key causes remain applicable in the current environment and identify any new major causes. • Evaluate the effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities. In making this assessment and related recommendations, the committee should note relevant differences, if any, that exist between Earth and space science missions.

49

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

50

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

Tasking letter The following letter was sent from Edward J. Weiler, associate administrator for NASA’s Science Mission Directorate, to Charles F. Kennel, chair of the National Research Council’s Space Studies Board, including the enclosure, “Provisional Bibliography for Science Mission Cost Growth External Independent Assessment.” The NASA Authorization Act of 2008 (Section 508) directs the Administrator to arrange for “an independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized space and Earth science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs.” While a significant amount of effort has been expended in recent years, by various parties, to collect and analyze cost growth in national security and NASA science space missions, the results of these studies have not been uniformly and critically appraised, nor their implications integrated into potentially actionable modifications to mission formulation and/or development practices. It would be valuable to leverage the results of these data collection and analysis efforts, in addition to assessing any other factors that may affect cost growth in NASA space and Earth science missions and making recommendations on cost containment. This new study should: • Review the body of existing studies related to NASA space and Earth science missions and identify their key causes of cost growth and strategies for mitigating cost growth; • Assess whether those key causes remain applicable in the current environment and identify any new major causes; and • Evaluate effectiveness of current and planned NASA cost growth mitigation strategies and, as appropriate, recommend new strategies to ensure frequent mission opportunities.

Copyright © 2010. National Academies Press. All rights reserved.

In making this assessment and related recommendations, the study should note what differences, if any, exist with regard to Earth science compared with space science missions. A provisional list of documents to be furnished by NASA to the panel is provided in the enclosure. NASA does not anticipate that subcontracting with an independent cost estimator will be necessary to complete the task and asks that the National Research Council’s (NRC’s) planning and budget for the study reflect this approach. If the NRC committee established to conduct the study determines that such a subcontractor is needed, a suitable augmentation of resources for the task will be provided. I would like to request that the NRC submit a plan to NASA for conduct of this study along these lines. To respond to the Congressional delivery date, we will need the findings and recommendations of the NRC review by January 31, 2010. Once agreement with the NRC on the scope and cost of the proposed study has been achieved, the NASA Contracting officer will issue a task order for implementation. Mr. Claude Freaner will be the NASA technical point of contact for this effort and may be reached at (202) 358-2522 or [email protected].

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

51

APPENDIX A

Provisional Bibliography for Science Mission Cost Growth External Independent Assessment NASA Cost Studies “Using Historical NASA Cost and Schedule Growth to Set Future Program and Project Reserve Guidelines,” Bitten, Emmons, and Freaner, September 2006 “SMD Cost/Schedule Performance Study—Final Report Overview,” Perry, Bruno, Jacobs, Doyle, Hayes, Stancati, Richie, and Rogers, November 2007 “Cost and Schedule Growth at NASA,” Coonce, November 2007 “SMD Cost/Schedule Performance Study—Summary Overview,” Perry, Bruno, Jacobs, Doyle, Hayes, Stancati, Richie, and Rogers, January 2008 “An Assessment of the Inherent Optimism in Early Conceptual Designs and Its Effect on Cost and Schedule Growth,” Freaner, Bitten, Bearden, and Emmons, May 2008 “SMD Earth and Space Mission Cost Driver Comparison Study,” Mlynczak and Perry, March 2009 (projected) Related Analyses “Space Missions Require Substantially More Funding Than Initially Estimated,” General Accounting Office, December 1992 “Reducing the Costs of Space Science Research Missions,” NRC Joint Committee on Technology for Space Science and Applications, 1997 “Cost Growth of Major Defense Programs,” McCrillis, January 2003 “Report of the Defense Science Board/Air Force Scientific Advisory Board Joint Task Force on Acquisition of National Security Space Programs,” A. Thomas Young (chair), May 2003

Copyright © 2010. National Academies Press. All rights reserved.

Additional sources may be identified

  The tasking letter noted that NASA might identify additional sources of cost growth information. NASA did not formally notify the NRC that it should examine additional studies. However, as noted in the main body of this report, some other documents were examined in the course of the study, some of them identified by NASA staff who participated in the open sessions of committee meetings.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

52

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

Legislation The study that produced this report was initially prompted by congressional legislation: NASA Authorization Act of 2008 (P.L. 110-422) Section 508. Assessment of Cost Growth

Copyright © 2010. National Academies Press. All rights reserved.

(a) Study—The Administrator shall enter into an arrangement for an independent external assessment to identify the primary causes of cost growth in the large-, medium-, and small-sized space and Earth science spacecraft mission classes, and make recommendations as to what changes, if any, should be made to contain costs and ensure frequent mission opportunities in NASA’s science spacecraft mission programs. (b) Report—The report of the assessment conducted under subsection (a) shall be submitted to Congress not later than 15 months after the date of enactment of this Act.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

B Biographies of Committee Members and Staff

Copyright © 2010. National Academies Press. All rights reserved.

RONALD M. SEGA, Chair, is the Woodward Professor of Systems Engineering at Colorado State University and the vice president for energy, environment, and applied research with Colorado State University Research Foundation. Dr. Sega was a faculty member in the College of Engineering and Applied Science at Colorado University at Colorado Springs, also serving as dean from 1996 to 2001. He served as technical director of the Laser and Aerospace Mechanics Directorate at F.J. Seiler Research Laboratory at the U.S. Air Force Academy and as assistant director of the Space Vacuum Epitaxy Center at University of Houston. He served as director of defense research and engineering, the chief technology officer for the Department of Defense, from 2001 to 2005. He retired from the Air Force Reserve in 2005 as a major general in the position of reserve assistant to the chairman of the Joint Chiefs of Staff after 31 years in the Air Force, having served in various assignments at Air Force Space Command and as a pilot. He most recently was the undersecretary of the Air Force from 2005 to 2007. A former astronaut, he flew aboard space shuttles Discovery (1994) and Atlantis (1996). He also led the Air Force team that won the overall Presidential Award for Leadership in Federal Energy Management for 2006. Dr. Sega holds a B.S. in math and physics from the U.S. Air Force Academy in Colorado Springs, an M.S. in physics from the Ohio State University, and a Ph.D. in electrical engineering from the University of Colorado. VASSILIS ANGELOPOULOS is professor of Earth and space sciences and a member of the Institute of ­Geophysics and Space Physics at the University of California, Los Angeles. He has 15 years of experience in space physics with emphasis in magnetospheric processes. His research interests include plasma sheet transport, electromagnetic instabilities in the plasma sheet and its boundary, beam-induced ionospheric low-frequency waves, substorm ­physics, turbulence, and self-organized criticality. Dr. Angelopoulos is also a member of the Space Sciences Laboratory at the University of California, Berkeley, and a distinguished visiting scientist at the Jet Propulsion Laboratory. He has authored and coauthored more than 100 publications in refereed journals on data analysis, plasma theory and space plasma phenomenology, space technology, space instrumentation, and mission analysis and design. He has been awarded the Macelwane Medal by the American Geophysical Union in recognition of significant contributions to the geophysical sciences by young scientists and the Zeldovich Medal by the Russian Academy of Sciences and COSPAR on young scientists for excellence and achievement. He served on the National Research Council (NRC) Committee on Solar and Space Physics between 2002 and 2005.

53

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

54

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

ALLAN V. BURMAN is president of Jefferson Solutions, a division of the Jefferson Consulting Group, LLC, a firm that provides change management services and acquisition reform training to many federal departments and agencies. Dr. Burman provides strategic consulting services to private sector firms doing business with the federal government as well as to federal agencies and other government entities. He also has advised firms, congressional committees, and federal and state agencies on a variety of management and acquisition reform matters. Prior to joining the Jefferson Consulting Group, Dr. Burman had a long career in the federal government, including serving as administrator for federal procurement policy in the Office of Management and Budget (OMB), where he testified before Congress over 40 times on management, acquisition, and budget matters. Dr. Burman authored the 1991 policy letter that established performance-based contracting and greater reliance, where appropriate, on fixed-price contracting as the favored approach for contract reform. As a member of the Senior Executive Service, Dr. Burman served as chief of the Air Force Branch in OMB’s National Security Division and was the first OMB branch chief to receive a Presidential Rank Award. Dr. Burman is a fellow of the National Academy of Public Administration, a fellow and member of the Executive Advisory Council of the National Contract Management Association, a director of the Procurement Round Table, and an honorary member of the National Defense Industrial Association. He is also a former contributing editor and writer for Government Executive magazine. Dr. Burman obtained a B.A. from Wesleyan University, was a Fulbright Scholar at the Institute of Political Studies, University of Bordeaux, France, and has a graduate degree from Harvard University and a Ph.D. from the George Washington University. Dr. Burman has been a member of four prior NRC study committees, most recently the Committee on Optimizing U.S. Air Force and Department of Defense Review of Air Force Acquisition Programs.

Copyright © 2010. National Academies Press. All rights reserved.

OLIVIER L. de WECK is the associate professor of aeronautics and astronautics and engineering systems and the associate director of the Engineering Systems Division at the Massachusetts Institute of Technology (MIT). Before joining MIT he was a liaison engineer and later engineering program manager on the F/A-18 aircraft program at McDonnell Douglas. His research interests, teaching emphasis, and professional experience include systems engineering for changeability and commonality and space exploration logistics. His research is helping to establish principles, methods, and tools to plan, simulate, and visualize the future as an interplanetary supply chain. In such a supply chain, innovative strategies (pre-positioning, carry-along, resupply, and orbital depots) are carefully matched with new technologies (space tugs, reconfigurable spares, and RFID-enabled asset management) to maximize scientific exploration, while minimizing the cost and risk of future exploration campaigns. He has written or ­cowritten more than 100 journal and conference publications in the area of systems engineering and space systems design for exploration and communications. His changeability research traces the evolution and change over time of existing technical systems, formalizes patterns of change propagation, and develops methods and tools for finding where and how to embed flexibility in design and how to value such flexibility. His commonality research investigates another strategic aspect in engineering design where systems and products are no longer designed as individuals, but the need for customization and efficiency drives considerations of commonality, reuse, and platform architectures. Dr. de Weck is an associate fellow of AIAA, winner of the 2006 Frank E. Perkins award for excellence in graduate advising, and recipient of the 2007 AIAA MDO TC outstanding service award. He has a degree in industrial engineering from ETH Zurich, Switzerland, and a Ph.D. in aerospace systems engineering from MIT. ROBERT E. DEEMER is an assistant professor at Regis University. He has been teaching for the last 7 years in business and emerging technologies after 28 years in industry with Litton Industries and Lockheed Martin. He has been involved from an operations management perspective on several space exploration programs, including the Mars Lander, Stardust, and Cassini. He also spent several years at the Advanced Spacecraft Technology Center as operations director while working for Lockheed Martin in Denver, Colorado. He has an M.S. in computer science from Colorado Technical College, an M.B.A. from Pepperdine University, an M.S. in management science from the University of Redlands, and an M.A. in philosophy and humanities from California State University; he has three B.S. and four B.A. degrees in various fields, and he is a professionally certified project manager through Villanova University. He has chaired two NRC study committees, including the Committee to Review NASA’s Space Communications Program, and served as a member of two others. He is a national associate of the National Academies.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

APPENDIX B

55

LARRY W. ESPOSITO is a professor at the Laboratory for Atmospheric and Space Physics at the University of Colorado, Boulder. He is the principal investigator of the Ultraviolet Imaging Spectrograph experiment on the Cassini space mission to Saturn. He was chair of the Voyager Rings Working Group and, as a member of the Pioneer Saturn Imaging Team, he discovered Saturn’s F ring. His research focuses on the nature and history of planetary rings. Dr. Esposito has been a participant in numerous American, Russian, and European space missions and used the Hubble Space Telescope for its first observations of Venus. He was awarded the Harold C. Urey Prize from the American Astronomical Society, the Medal for Exceptional Scientific Achievement from NASA, and the Richtmyer Lecture Award from the American Association of Physics Teachers and the American Physical Society. Dr. Esposito has chaired two NRC committees, including the Committee on Planetary and Lunar Exploration, and he served as a member of six others, as well as the Space Studies Board. Dr. Esposito has an S.B. in mathematics from MIT and a Ph.D. in astronomy from the University of Massachusetts, Amherst. JOSEPH FULLER, JR. is the founder, president, and CEO of Futron Corporation, a leader in providing decision management solutions to aerospace, telecommunications, and other technology enterprises. Before founding Futron, Mr. Fuller spent 20 years at NASA as an aerospace systems engineer, project manager, and senior executive. He is experienced in the design, development, and operations of both human-piloted and robotic spacecraft. Mr. Fuller is a recipient of the NASA Exceptional Service Medal and a former member of the Aeronautics and Space Engineering Board of the NRC. He currently serves on NOAA’s Advisory Committee on Commercial Remote Sensing, NASA’s Project Management (APPEL) Mission Operations Working Group, and on the board of the Challenger Center for Science Education. He has a B.S. in physics from Texas Southern University and an M.B.A. from the University of Houston. He was a member of the NRC Committee that produced NASA’s Beyond Einstein Program: An Architecture for Implementation (2007).

Copyright © 2010. National Academies Press. All rights reserved.

JOSEPH W. HAMAKER is a senior cost analyst with Science Applications International Corporation. He performs cost and schedule estimating, risk/uncertainty analysis and related economic assessments, cost-estimating tool development, and related work for new and ongoing space projects. Previously, Dr. Hamaker was the director of the Cost Analysis Division of NASA headquarters. In this position, Dr. Hamaker led the development of independent cost estimates for NASA programs, as well as the establishment of strategic management and policy for the overall NASA cost-estimating process. Prior to assuming the lead cost position at NASA headquarters in September 2002, he spent 29 years at NASA’s Marshall Space Flight Center, the last 16 as manager of the Engineering Cost Office there. Dr. Hamaker has a B.S. in industrial engineering from Tennessee Technological University and three degrees from the University of Alabama, Huntsville: a B.A. in economics, an M.S. in engineering management, and a Ph.D. in industrial and systems engineering. He is a Society of Cost Estimating and Analysis certified cost analyst and an International Society of Parametric Analysts certified parametric practitioner. He has supported two NRC studies, including the Review of Near-Earth Object Surveys and Hazard Mitigation Strategies. VICTORIA E. HAMILTON is a principal scientist at the Southwest Research Institute. Formerly, she was an associate researcher in the Hawaii Institute of Geophysics and Planetology at the University of Hawaii. She is a planetary geologist interested in the mineralogy and igneous processes/histories of planetary bodies. The main focus of her research is investigating the spectral features of minerals and rocks in the thermal infrared portion of the electromagnetic spectrum and using this knowledge to identify and/or characterize the rocks and minerals on planetary surfaces. Most of her recent work has focused on analysis of data from ongoing NASA spacecraft missions at Mars. She has served as a panel chair, NASA Mars Fundamental Research Program Review Panel (2007), and panel chair, NASA Mars Data Analysis Program Review Panel (2008). Dr. Hamilton is on the editorial board of the journal Minerals; she is a science team member and deputy instrument scientist on the OSIRIS-REx New Frontiers mission; she is a participating scientist on the 2001 Mars Odyssey mission; she served as a member of the Universities Space Research Association Lunar and Planetary Institute Science Council; she served as associate editor of JGR-Planets; and she was a member of the NASA Advisory Council Planetary Science Subcommittee (2006 to 2009). Dr. Hamilton has a B.A. in geology from Occidental College and a Ph.D. in geology from Arizona State University. She was a member of the NRC Committee for the Review of the Next Decade Mars Architecture.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

56

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

JOHN M. KLINEBERG is an aerospace consultant, the former CEO of Swales Aerospace, and the retired president of Space Systems/Loral (SS/L). Before assuming the presidency of SS/L, Dr. Klineberg served as executive vice president for Loral’s Globalstar program where he successfully led the development, production, and deployment of the Globalstar satellite constellation used for telephone services. Prior to joining Loral in 1995, Dr. Klineberg spent 25 years at NASA where he served in a variety of management and technical positions. He was the director of the Goddard Space Flight Center (GSFC), director of the Lewis (now Glenn) Research Center, deputy associate administrator for Aeronautics and Space Technology at NASA headquarters, and a research scientist at the Ames Research Center. Before beginning his career at NASA, he conducted fundamental studies in fluid dynamics at the California Institute of Technology and worked at the Douglas Aircraft Company and the Grumman Aircraft Company. Dr. Klineberg has a B.S. in engineering from Princeton University and an M.S. and Ph.D. from the California Institute of Technology. He is the former chair of two NRC study committees, including the Committee to Review the NASA Astrobiology Institute, a former member of one other committee, and a former member of the Aeronautics and Space Engineering Board. BRUCE D. MARCUS is retired from TRW Inc., where he was chief scientist and manager of advanced programs for the Space and Laser Programs Division. Dr. Marcus’s professional interests include Earth and space sciences. His research background includes heat and mass transfer, heat pipes, thermosiphons, spacecraft thermal control, and thermo-mechanical design of telescopes. Dr. Marcus has investigated technology issues related to the potential use of the NPOESS weather satellite for climate research, and his background also includes extensive experience in space systems program management. Dr. Marcus has a BME in thermal processes, an MME in nuclear and thermal engineering, and a Ph.D. in thermal processes, mathematics, and engineering physics, all from Cornell University. He was on the committee that authored the NRC report The Role of Small Satellites in NASA and NOAA Earth Observation Programs; he has served on six other NRC study committeesp; he is a former member of the Space Studies Board; and he has been designated a national associate of the National Academies.

Copyright © 2010. National Academies Press. All rights reserved.

EMERY I. REEVES is an independent consultant. He is the retired Schriever Chair Professor of Space Systems Engineering at the U.S. Air Force Academy. In 1985, he retired from TRW as vice president and general manager of the Spacecraft Engineering Division. He was a naval officer from 1955 to 1958. He has also served as a consulting professor at Stanford University from 1985 to 1992. Mr. Reeves was elected as a member of the National Academy of Engineering for pioneering contributions to the design and development of spacecraft and ballistic missile attitude-control and navigation systems. He has a B.E. from Yale University and an M.S.E.E. from MIT. He served on the NRC Panel on Critical Materials-Phase I of the Committee on Critical Materials and more recently on the NAE Aerospace Engineering Peer Committee. WILLIAM F. TOWNSEND is an independent aerospace consultant. Previously, he was with the Ball Aerospace and Technologies Corporation as vice president, exploration systems, and as vice president and general manager of the Civil Space Systems strategic business unit. Mr. Townsend also has more than 40 years of experience in program and project management with NASA, from aircraft and spacecraft instrument development to international space mission development. As deputy center director and chair of the program management council at GSFC from 1998 to 2004, Mr. Townsend oversaw the development, launch, and operation of all GSFC instruments, spacecraft, and missions. Mr. Townsend also spent 17 years in the Earth Science Enterprise area at NASA headquarters, culminating with almost 2 years as the acting associate administrator. Honors and awards received by Mr. Townsend include two Presidential Rank Meritorious Executive awards; two NASA Distinguished Service Medals; the NASA Outstanding Leadership Medal; the NASA Exceptional Service Medal; the French space agency’s (CNES’s) Bronze Medal; and the NASA GSFC’s Robert C. Baumann Memorial Award for Mission Success. He has a B.S.E.E. from Virginia Polytechnic Institute.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

APPENDIX B

57

Staff ALAN C. ANGLEMAN, Study Director, has been a senior program officer for the Aeronautical and Space Engineering Board (ASEB) since 1993, directing studies on a wide variety of aerospace issues. Previously, Mr. Angleman worked for consulting firms in the Washington, D.C., area, providing engineering support services to the Department of Defense and NASA headquarters. His professional career began with the U.S. Navy, where he served for 9 years as a nuclear-trained submarine officer. He has a B.S. in engineering physics from the U.S. Naval Academy and an M.S. in applied physics from the Johns Hopkins University. CATHERINE A. GRUBER, editor, joined the Space Studies Board (SSB) as a senior program assistant in 1995. Ms. Gruber first came to the National Research Council (NRC) in 1988 as a senior secretary for the Computer Science and Telecommunications Board and also worked as an outreach assistant for the National Science Resources Center. She was a research assistant (chemist) in the National Institute of Mental Health’s Laboratory of Cell Biology for 2 years. She has a B.A. in natural science from St. Mary’s College of Maryland. ANDREA M. REBHOLZ, program associate, joined the ASEB in January 2009. She began her career at the National Academies in October 2005 as a senior program assistant for the Institute of Medicine’s Forum on Drug Discovery, Development, and Translation. Prior to the Academies, she worked in the communications department of a D.C.-based think tank. Ms. Rebholz graduated from George Mason University’s New Century College in 2003 with a B.A. in integrative studies–event management and has more than 8 years of experience in event planning. LINDA WALKER, senior project assistant, has been with the National Academies since September 2007. Before her assignment with the SSB, she was on assignment with the National Academies Press. Prior to her working at the National Academies, she was with the Association for Healthcare Philanthropy in Falls Church, Virginia. Ms. Walker has 28 years of administrative experience.

Copyright © 2010. National Academies Press. All rights reserved.

MICHAEL H. MOLONEY is the director of the SSB and the ASEB at the NRC. Since joining the NRC in 2001, Dr. Moloney has served as a study director at the National Materials Advisory Board, the Board on Physics and Astronomy, the Board on Manufacturing and Engineering Design, and the Center for Economic, Governance, and International Studies. Before joining the SSB and ASEB in April 2010, he was associate director of the BPA and study director for the Astro2010 decadal survey for astronomy and astrophysics. In addition to his professional experience at the NRC, Dr. Moloney has more than 7 years experience as a foreign-service officer for the Irish government and served in that capacity at the Embassy of Ireland in Washington, D.C., the Mission of Ireland to the United Nations in New York, and the Department of Foreign Affairs in Dublin, Ireland. A physicist, Dr. Moloney did his graduate Ph.D. work at Trinity College Dublin in Ireland. He received his undergraduate degree in experimental physics at University College Dublin, where he was awarded the Nevin Medal for Physics.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

C Findings and Recommendations from Primary References

In addition to identifying the causes of cost growth, the primary references (see the References chapter) have made dozens of specific findings and recommendations. This appendix summarizes the findings and recommendations contained in these historic studies. In some cases the findings and recommendations listed are quoted from other prior studies.

TABLE C.1  Cost Growth Findings from the Primary References

Copyright © 2010. National Academies Press. All rights reserved.

  Page Number Primary Reference Finding

1

2

3

4

5

6

7

9

10

Cost growth and schedule slips are nearly universal among the projects studied.

 

 2

 

 

 

 

 

 

 

The highest percentage schedule growth tends to occur after the start of spacecraft integration and test.

 

 

 

 

 

71

 

 

 

There is no discernable correlation between actual cost performance and planned cost reserve level.

 

33

 

10

 

67

 

 

 

There is no discernable correlation between actual cost performance and percent of funds spent during Phase B formulation.

 

34

 

10

 

63

 

 

 

For the projects in this study, there is no discernable correlation between actual cost performance and percent of funds spent up to CDR.

 

35

 

 

 

 

 

 

 

There is a possible correlation between completing substantial activity prior to CDR and lower cost growth for the total development effort.

 

43

 

 

 

 

 

 

 

There appears to be little-to-no correlation between total flight system dry mass growth and Phase BCD cost growth or between instrument mass growth and instrument Phase BCD cost growth for the 30 SMD missions and 100+ individual instruments included in this study.

 

 

 

 

 

110

 

 

 

58

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

59

APPENDIX C

TABLE C.1  Continued   Page Number Primary Reference Finding

1

2

3

4

5

6

7

9

10

Earth Science missions do not show a systemic difference in cost or cost growth as compared to other SMD missions. Missions from each SMD division experience cost growth; total growth in dollars is greater for missions that have greater baseline costs.

 

 

 

 

 

 5 37

 

 

 

There is no correlation between mission cost growth and SMD division, acquisition mode, contractor type, Phase B investment or cost reserve.

 

 

 

 

 

 5

 

 

 

Directed missions cost more than AO-acquired missions do and are, in general, more complex and more massive.

 

 

 

 

 

47

 

 

 

AO-acquired and directed missions have comparable schedule growth in months.

 

 

 

 

 

47

 

 

 

In general, cost growth is of a similar percentage for AO-acquired and directed missions, although it may be a larger dollar increase for directed missions since it is against a larger base cost.

 

 

 

 

 

47

 

 

 

Mission cost growth correlates strongly with payload cost growth. Payloads from all four SMD divisions experience significant cost growth.

 

 

 

 

 

 5 79

 

 

 

Instrument cost shows good correlation to a multivariable instrument Level-of-Difficulty measure.

 

 

 

 

 

 5

 

 

 

Earth science instruments cost less per unit mass but are more massive, have more stringent requirements and higher levels of difficulty, and therefore are more costly overall than are instruments in the other SMD divisions.

 

 

 

 

 

 5

 

 

 

There is no systemic difference in spacecraft cost regimes between Earth and space science missions.

 

 

 

 

 

 5

 

 

 

Department of Defense capabilities to lead and manage the acquisition process have seriously eroded. The government should address acquisition staffing, reporting integrity, systems engineering capabilities, and program manager authority.

 

 

 

 

 

 

 

 

iii

While the space industrial base is adequate to support current programs, long-term concerns exist. A continuous flow of new programs—cautiously selected—is required to maintain a robust space industry. Without such a flow, the workforce, as well as critical national capabilities in the payload and sensor areas, are at risk.

 

 

 

 

 

 

 

 

iv 4

Copyright © 2010. National Academies Press. All rights reserved.

NOTE: The numbers in each cell of the table indicate the page number(s) in the respective report where the item is discussed. Primary Reference 8 is focused on reducing the absolute costs of NASA space science missions; it does not directly address cost growth, and so its results are not included in this table. Primary References 9 and 10 focus on Department of Defense systems. AO, announcement of opportunity; CDR, critical design review; SMD, Science Mission Directorate.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

60

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

TABLE C.2  Cost Growth Recommendations from Prior Studies

Copyright © 2010. National Academies Press. All rights reserved.

  Page Number in Primary Reference Recommendation

1

2

3

4

5

6

7

9

10

NASA should be realistic with itself, Congress, and the public in terms of the goals, capabilities, costs, schedule, and technical risks of a new project.

 

 

 

 

 

 

12

 

 

Improve technical and programmatic definition at the beginning of a project (increase time and funding for Phase A and Phase B and extend them as necessary for complex projects) to allow more time for development of technology, baseline costs, funding profiles, and the overall implementation plan before making significant investments in other mission elements.

15

38

7

11

12

 

11

 

 

SMD should work with projects beginning at the start of Phase B, or earlier if possible, to establish a credible baseline plan that fits within the available funding with sufficient margin instead of waiting for projects to present a plan at the end of Phase B.

 

20 

 

 

 

 

 

 

 

Require more robust initial cost and schedule estimates, including project-level management costs.

17

38

7

11

12

 

 

 

 

Do a better job of independently validating costs and schedule (this includes improving cost and schedule estimating tools).

 

 

7

 

12

 

 

 

5

SMD should perform independent cost estimates on all decadal planning and similar exercises.

 

 

 

 

 

12

 

 

 

Independently validate instrument resources and resulting spacecraft resources needed to meet mission requirements (cost estimators cannot be expected to validate system designs).

 

 

 

 

12

 

 

 

 

Give more attention to risk identification and mitigation prior to CDR.

 

 

12

 

 

 

 

 

 

Select AO missions with lower risk.

 

 

7

 

 

 

 

 

 

Remove funding constraints from AOs for more credible funding profiles for initial planning.

 

39

7

12

 

 

 

 

 

For AO missions, consider funding profiles, mission-specific launch date constraints, and program funding availability when making selections.

 

39

 

12

 

 

 

 

 

For AO missions, consider alternates to down-selecting to two finalists, delay setting the cost cap until PDR, and require proposes to address cost and schedule feasibility.

 

 

7

 

 

 

 

 

 

Direct that source selections evaluate contractor cost credibility and use the estimate as a measure of their technical understanding.

 

 

 

 

 

 

 

 

5

Hold basis-of-estimate discussions at the start of Phase A.

 

 

7

 

 

 

 

 

 

Spend more money on research and development programs to mature technology readiness levels.

 

 

7

 

 

 

 

 

 

Support early instrument development to reduce risk (phased development approach).

15

 

7

 

 

 

 

 

 

Carefully evaluate design heritage credits.

 

 

7

 

 

 

 

 

 

Improve tools for early estimation of science instrument costs.

 

41

 

13

 

12

 

 

 

Increase cost reserves.

15

 

7

 

 

 

 

 

 

Minimize or eliminate blanket requirements on the level of cost reserves. Instead, match reserves to implementation risk.

 

 

 

12 14

 

 

 

 

 

Hold a budget reserve at the program level at headquarters, in part to address impacts from changes external to the projects (such as changes in launch costs).

 

43

20

12

 

 

 

 

 

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

61

APPENDIX C

TABLE C.2  Continued

Copyright © 2010. National Academies Press. All rights reserved.

  Page Number in Primary Reference Recommendation

1

2

3

4

5

6

7

9

10

Ensure adequate funds (and reserves) to cover cost of spacecraft integration and test.

 

42

 

 

 

 

 

 

 

Increase cost reserves for missions relying on foreign hardware and identify backup options.

 

43

 

 

 

 

 

 

 

Increase funded schedule reserves.

16

 

7

 

 

 

 

 

 

Establish and maintain appropriate funding profiles and stable funding.

 

 

7

 

 

 

12

 

 

Ensure mission are properly scoped.

13

 

 

 

 

 

 

 

 

Calculate and report estimates to complete monthly.

17

 

 

 

 

 

 

 

 

Manage to schedule.

17

 

 

 

 

 

 

 

 

Make more effective use of Earned Value Management, beginning as early in the development cycle as possible.

17

40

 

12

 

 

 

 

 

Identify and disseminate project management best practices.

 

41

 

 

 

 

 

 

 

Establish a handbook or memorandum of understanding that details the relationship between PIs and project managers.

 

 

7

 

 

 

 

 

 

Program managers should establish early warning metrics and report problems up the management chain for timely corrective action.

 

 

 

 

 

 

 

 

6

Avoid changes and redirection, especially after PDR.

 

39

7

12

 

 

 

 

 

Use four-party agreements (among project manager, principal investigator, NASA headquarters, and prime contractor) or some other process to control requirements.

 

 

7

 

 

 

 

 

33

Include cost and schedule status details at CDR, Assembly Readiness Review, Pre-Environmental Review, and Mission Readiness Review.

 

 

 

11

 

 

 

 

 

Assess launch site capabilities before start of Phase B.

 

39

7

 

 

 

 

 

 

Select the expandable launch vehicle as early as possible and minimize changes.

 

39

7 20

 

 

 

 

 

 

Maintain a cost reserve at headquarters to cover unforeseen issues affecting launch vehicle price and launch site costs.

 

39

7

 

 

 

 

 

 

Develop estimates of cost and schedule savings from descopes earlier and with more rigor.

 

42

7

 

 

 

 

 

 

Establish a single source of cost data, with routine data collection from missions.

 

44

 

 

 

 

 

 

 

Improve NASA’s Cost Analysis Data Requirement (CADRe) system by including new missions and expanding the instrument subsystem to identify which instrument types have had the highest historical resource growth.

 

 

 

 

13

12

 

 

 

Cancel missions for poor performance.

 

 

7

 

 

 

 

 

 

Conduct mission postmortem reviews.

 

 

7

 

 

 

 

 

 

NOTE: The numbers in each cell of the table indicate the page number(s) in the respective report where the item is discussed. Primary Reference 8 is focused on reducing the absolute costs of NASA space science missions; it does not directly address cost growth, and so its results are not included in this table. Primary References 9 and 10 focus on Department of Defense systems. AO, announcement of opportunity; CDR, critical design review; PDR, preliminary design review; SMD, Science Mission Directorate.

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

D

Copyright © 2010. National Academies Press. All rights reserved.

Acronyms and Abbreviations

ACE ACRIMSAT ACTS AFE AIM AO ASRM ATP AXAF

Advanced Composition Explorer Active Cavity Radiometer Irradiance Monitor Satellite Advanced Communications Technology Satellite Aeroassist Flight Experiment Aeronomy of Ice in Mesosphere announcement of opportunity Advanced Solid Rocket Motor Alternate Turbopumps Advanced X-ray Astrophysics Facility

CADRe CALIPSO CDR CGRO CINDI CLOUDSAT CNES CONTOUR

Cost Analysis Data Requirement (NASA) Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observation critical design review Compton Gamma-Ray Observatory Coupled Ion Neutral Dynamics Cloud Satellite Centre National d’Etudes Spatiales Comet Nucleus Tour

DOD DOE DS

Department of Defense Department of Energy Deep Space

EO EOS ESA EUVE

Earth Observing Earth Observing System European Space Agency Extreme Ultraviolet Explorer

FAST

Fast Auroral Snapshot 62

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

63

Copyright © 2010. National Academies Press. All rights reserved.

APPENDIX D

FTS FUSE

Flight Telerobotic Servicer Far Ultraviolet Spectroscopic Explorer

GALEX GAO GLAST GOES GRACE GSFC

Galaxy Evolution Explorer Government Accountability Office Gamma-ray Large Area Space Telescope Geostationary Operational Environmental Satellite Gravity Recovery and Climate Experiment Goddard Space Flight Center

HESSI HETE HST

High Energy Solar Spectroscopic Imager (also referred to as RHESSI) High Energy Transient Explorer Hubble Space Telescope

IBEX ICESAT IMAGE ISS

Interstellar Boundary Explorer Ice, Cloud, and Land Elevation Satellite Imager for Magnetopause-to-Aurora Global Exploration International Space Station

JPL JWST

Jet Propulsion Laboratory James Webb Space Telescope

KPD

key decision point

LCROSS LRO

Lunar Crater Observation and Sensing Satellite Lunar Reconnaissance Orbiter

MAP MCO MER MESSENGER MGS MMS MPL MRO MSL

Microwave Anisotropy Probe Mars Climate Orbiter Mars Exploration Rover Mercury Surface, Space Environment, Geochemistry, and Ranging Mars Global Surveyor Magnetospheric Multiscale Mission Mars Polar Lander Mars Reconnaissance Orbiter Mars Science Laboratory

NASA NEAR NOAA NPOESS NPD NPR NRC NSCAT

National Aeronautics and Space Administration Near Earth Asteroid Rendezvous National Oceanic and Atmospheric Administration National Polar-orbiting Operational Environmental Satellite System NASA Policy Directive NASA Procedural Requirements National Research Council NASA Scatterometer

OCO OMB OMV OSTM

Orbiting Carbon Observatory Office of Management and Budget Orbital Maneuvering Vehicle Ocean Surface Topography Mission

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,

64

CONTROLLING COST GROWTH OF NASA EARTH AND SPACE SCIENCE MISSIONS

preliminary design review principal investigator

RHESSI

Reuven Ramaty High Energy Solar Spectroscopic Imager

SIRTF SMD SOFIA SORCE SRB STEREO SWAS

Space Infrared Telescope Facility (now known as the Spitzer Space Telescope) Science Mission Directorate Stratospheric Observatory for Infrared Astronomy Solar Radiation and Climate Experiment Standing Review Board Solar Terrestrial Relations Observatory Submillimeter Wave Astronomy Satellite

TDRS THEMIS TIMED TOPEX TRACE TRMM TSS TWINS

Tracking and Data Relay Satellite Time History of Events and Macroscale Interactions during Substorms Thermosphere Ionosphere Mesosphere Energetics and Dynamics Ocean Topography Experiment Transition Region and Coronal Explorer Tropical Rainfall Measuring Mission Tethered Satellite System Two Wide-angle Imaging Neutral-atom Spectrometers

USAF

U.S. Air Force

WBS WIRE WISE

work breakdown structure Wide-Field Infrared Explorer Wide-Field Infrared Survey Explorer

XTE

X-Ray Timing Explorer

Copyright © 2010. National Academies Press. All rights reserved.

PDR PI

National, Research Council, et al. Controlling Cost Growth of NASA Earth and Space Science Missions, National Academies Press, 2010. ProQuest Ebook Central,