Interpretation, Critical Review and Reporting in Life Cycle Assessment (LCA Compendium – The Complete World of Life Cycle Assessment) 3031357264, 9783031357268

Citation preview

LCA Compendium – The Complete World of Life Cycle Assessment Series Editors: Walter Klöpffer · Mary Ann Curran

Mary Ann Curran   Editor

Interpretation, Critical Review and Reporting in Life Cycle Assessment

LCA Compendium – The Complete World of Life Cycle Assessment Series Editors Walter Klöpffer, LCA Consult & Review, Frankfurt am Main, Germany Mary Ann Curran, LCA & Sustainability Consultant, BAMAC Ltd., York, SC, USA

Life Cycle Assessment (LCA) has become the recognized instrument to assess the ecological burdens and human health impacts connected with the complete life cycle (creation, use, end-of-life) of products, processes, and activities, enabling the assessor to model the entire system from which products are derived or in which processes and activities operate. Due to the steady, worldwide growth of the field of LCA, the wealth of information produced in journals, reports, books, and electronic media has made it difficult for readers to stay abreast of activities and recent developments in the field. This led to the realization of the need for a comprehensive and authoritative publication. The LCA Compendium book series will discuss the main drivers in LCA (SETAC, ISO, UNEP/SETAC Life Cycle Initiative, etc.), the strengths and limitations of LCA, the LCA phases as defined by ISO standards, specific applications of LCA, Life Cycle Management (LCM) and Life Cycle Sustainability Assessment (LCSA). Further volumes, which are closely related to these themes, will cover examples of exemplary LCA studies ordered according to the importance of the fields of application. They will also present new insights and new developments and will keep the whole work current. The aim of the series is to provide a wellstructured treatise of the field of LCA to give orientation and guidance through detailed descriptions on all steps necessary to conduct an LCA study according to the state of the art and in full agreement with the standards.

Mary Ann Curran Editor

Interpretation, Critical Review and Reporting in Life Cycle Assessment

Editor Mary Ann Curran LCA & Sustainability Consultant BAMAC Ltd. York, SC, USA

ISSN 2214-3505     ISSN 2214-3513 (electronic) LCA Compendium – The Complete World of Life Cycle Assessment ISBN 978-3-031-35726-8    ISBN 978-3-031-35727-5 (eBook) https://doi.org/10.1007/978-3-031-35727-5 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Paper in this product is recyclable.

Contents

1

Why Do We Cover Interpretation As Well As Critical Review and Reporting in One Volume? ������������������������������������������������    1 Birgit Grahl and Carl-Otto Gensch

2

Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation��������������������������������������������������������    7 Mary Ann Curran

3

 Data Quality Analysis as Part of Interpretation ����������������������������������   23 Daniela Kölsch and Sönke Giebeler

4

Quality Assurance by International Standards: The ‘Critical Review’������������������������������������������������������������������������������   51 Walter Klöpffer and Matthias Finkbeiner

5

The Terms “Critical Review” and “Verification” Addressing Quality Assurance in Life Cycle Assessment and ISO Type III Declarations����������������������������������������������������������������   83 Birgit Grahl and Eva Schmincke

6

 Benefits from Critical Review and Communication ����������������������������   95 Hans-Jürgen Garvens

7

Cost-Benefit Analysis of Critical Reviews: Learning from Practice ��������������������������������������������������������������������������  107 Christian Bauer and Andreas Detzel

8

Reporting and Communication��������������������������������������������������������������  123 Pere Fullana i Palmer

Index������������������������������������������������������������������������������������������������������������������  137

v

Contributors

Christian Bauer  Group Environment, Health & Safety, SIG International Services GmbH, Linnich, Germany Mary  Ann  Curran  LCA & Sustainability Consultant, BAMAC Ltd., York, SC, USA Andreas Detzel  IFEU GmbH, Heidelberg, Germany Matthias Finkbeiner  Chair of Sustainable Engineering, Institute of Environmental Technology, Technische Universität Berlin, Berlin, Germany Pere  Fullana i Palmer  UNESCO Chair in Life Cycle and Climate Change ESCI-UPF, Passeig Pujades 1, Barcelona, Spain Hans-Jürgen Garvens  LCA Consultant and Review, Wandlitz, Germany Carl-Otto Gensch  Öko-Institut e.V., Freiburg, Germany Sönke Giebeler  OEKO-TEX® Association, Zurich, Switzerland Birgit Grahl  Industrielle Ökologie, Heidekamp, Germany Walter Klöpffer  LCA Consult & Review, Frankfurt am Main, Germany Daniela Kölsch  Bayer Technology Services GmbH, Leverkusen, Germany Eva Schmincke  Tübingen, Germany

vii

Chapter 1

Why Do We Cover Interpretation As Well As Critical Review and Reporting in One Volume? Birgit Grahl and Carl-Otto Gensch

Abstract  The phase “Interpretation” is discussed along with its overarching capacity in the LCA framework. It is closely interlinked with goal and scope, inventory analysis, and impact assessment. Besides, one genuine function of the interpretation phase is providing relevant input for sound reporting. Keywords  Data quality · ISO 14040 · ISO/NP TS 14074 · LCA framework · Quality assurance

1.1 Relevance of Interpretation 1.1.1 Interpretation and Reporting “Interpretation” is the phase of any Life Cycle Assessment (LCA) study where conclusions are drawn from the results of the inventory analysis and the impact assessment (LCIA) and where recommendations are made according to the objective of the study. This means interpretation is directly related to goal and scope settings. The original editors of this book are Birgit Grahl, Industrial Ecology, Heidekamp, Germany, and Carl-Otto Gensch, Öko-Institut, Freiburg, Germany. They developed the structure, invited the authors and reviewed and revised the articles. The final editor, Mary Ann Curran, the editor of this book series, has decisively reviewed the articles and brought them to production. She hopes that this volume will become a valued contribution to the successful series and invites professionals, practitioners and students to dive into the neverending development of Life Cycle Assessment. B. Grahl (*) Industrial Ecology, Heidekamp, Germany C.-O. Gensch Öko-Institut e.V., Freiburg, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_1

1

2

B. Grahl and C.-O. Gensch

Likewise to be considered are the defined framework conditions, the reasons for carrying out the study as well as the context of the intended applications and the target groups of the results (see Curran 2017). One genuine function of the interpretation phase is providing relevant input for sound reporting that needs clear conclusions derived from the results.

1.1.2 Interpretation and Critical Review ISO 14040 (2006) clearly defines interpretation not as a fourth step after goal and scope definition, inventory and impact assessment but as a cross-phase approach considering the whole framework of the respective LCA study. The interpretation phase is an analysis of the reliability and capacity of the results. Hence, another genuine function of the interpretation phase is to ensure appropriate and maintainable deductions of conclusions and to correctly explain the results in the context of both, the chosen methodological framework as well as the used data basis. Depending on the goal of the study, different kinds of critical review may be chosen: external or internal, one expert or panel, with or without interested parties affected by the conclusions. In addition to the more general description of the critical review tasks (ISO 14040/44) in ISO/TS 14071,1 specifications concerning the necessary working steps in critical review processes as well as requirements like reviewer(s) competencies are given. The common characteristic of a review is the critical reflection at a meta-level of definitions, methodological approaches and data sources being used in the other stages of a specific LCA. In contrast to audits of annual financial statements, a critical review in LCA does not include the verification of the total database. Rather the data management as well as those data which affect the results significantly should be subject of a typical review process. For the critical review as an independent view on the study, interpretation is, on the one hand, an important information source as to how the practitioners handled the LCA methodology in the specific project, and on the other hand, it can be analyzed in a differentiated way, namely, whether the conclusions have been drawn consistently from the data under the respective boundary conditions. The relationship between interpretation and critical review can thus also be seen in the fact that interpretation is a kind of structured preparation of a critical review; in practice, it can be stated that the performance of a critical review can be made much easier if the preparers of a life cycle assessment very carefully follow the requirements that are specifically placed on the interpretation.

 PD ISO/TS 14074 (draft start date 2022) Environmental management – Life cycle assessment – Principles, requirements, and guidelines for normalization, weighting, and interpretation. 1

1  Why Do We Cover Interpretation As Well As Critical Review and Reporting in One…

3

1.1.3 Critical Review and Reporting Because critical review is the independent quality control of an LCA, the results will improve the credibility of reporting. Experience shows that the critical review helps to avoid text weaknesses and potential misunderstandings because these aspects will easier be realized by independent readers from different viewpoints. The reviewers thus also represent the first readership of a study and can help to ensure that the specific requirements for good and clear reporting of life cycle assessments are met. As outlined above, interpretation, critical review, and reporting are closely interlinked; it, therefore, seems obvious to discuss all three elements in one volume.

1.2 Structure of This Volume Chapter 1: Why Do We Cover Interpretation As Well As Critical Review and Reporting in One Volume? The authors of this chapter, Birgit Grahl and Carl-Otto Gensch, pursue the question why only little methodological guidance, literature, or best practice examples are provided on how to promote the interpretation in contrast to life cycle inventory and life cycle impact assessment. Indeed, this aspect needs more attention. Although ISO 14044 (2006) (clause 4.5) specifies requirements on interpretation, a subcommittee of the ISO Technical Committee TC 207 is (4/2019) working on a technical specification in which essential aspects of interpretation are to be addressed. The contributions in this volume highlight important aspects that have been shown to be relevant in practice. Furthermore, it remains to be seen which focal points will be addressed in PD ISO/TS 14074 (2022), that is, whether the standardization work will focus more on quantification and weighting or whether the more process-related requirements of interpretation will be considered. Chapter 2: Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation Mary Ann Curran elaborates the role of interpretation as a building block in any scientific work and thus as a centerpiece in any LCA. This discussion is particularly important because some software tools include a function called “interpretation” that solely aggregates the results of LCIA. Obviously, such approaches do not meet the scientific requirement of an appropriate approach in the context of LCA according to ISO 14040/44 (2006). Instead there are some useful techniques to provide good results from interpretation, especially with the help of contribution analysis and sensitivity analysis. Most important thinking when conducting the interpretation (but the same is true as well for critical review): continuously switching from details to the whole framework and back again in order to judge credibility, plausibility, and reliability. Nevertheless, in order to apply these techniques in an optimal way, it will be necessary to get a good picture of the product system at hand. In this context, it should be taken into account that LCA methodology necessarily

4

B. Grahl and C.-O. Gensch

includes reducing the complexity of the real word in more or less appropriate models. In this context, sensitivity analysis as well as scenario analysis are both methodological approaches which help to prepare the LCA results for reporting. Against this background, Chap. 2 describes iteration as a crucial part of the interpretation phase, which requires the practitioner to return to goal and scope, to the inventory as well as to the impact assessment. Chapter 3: Data Quality Analysis focuses on Data Quality Analysis (DQA) as part of interpretation. As a starting point of their considerations, Sönke Giebeler and Daniela Kölsch clarify the relation between data quality and data uncertainty, as these terms are closely interwoven with each other. While uncertainty refers to lack of knowledge, data quality should be considered as an umbrella term which includes criteria such as uncertainty and variability. Against this background, Giebeler and Kölsch define DQA as an approach to explore to what extent specific data quality requirements are fulfilled. The authors set forth the broad range of data which has to be managed in LCA and refer to several attempts to structure these data. Furthermore, the several types of data quality indicators and the operationalization are described in detail. Based on this theoretical framework, more insight could be provided by analyzing best practice examples, taking into account the type of data used, the quality indicators proposed, and the scoring system applied. Chapter 4: Quality Assurance by International Standards: The “Critical Review” Already during the discussion on the results of the early life cycle assessments, it was recognized that the method could be abused and the credibility of the results questioned (see green washing). Walter Klöpffer and Matthias Finkbeiner present a historical outline as to how continuous developments of requirements regarding the LCA method and the inclusion of these requirements in the international standards can be recognized as a generally accepted set of rules. They represent the character of the critical review of life cycle assessments as a consistency assessment, which should ensure the quality and credibility of LCA studies, especially in cases where the studies are specified for comparative assertions intended to be disclosed to the public. PDISO/TS 14074 (2022) plays a major role in the explanations, as these technical specifications describe in detail the requirements for the critical review. These requirements include the type of critical review, the requisites for reporting as well as the competencies of chairpersons and reviewers. Chapter 5:  The Terms “Critical Review” and “Verification” Addressing Quality Assurance in Life Cycle Assessment and ISO Type III Declarations In the context of life cycle assessments, two terms are used in relation to quality assurance: critical review and verification. Experience has shown that no clear distinction has been made between these terms, and thus often they are not used correctly outside professional circles. In Chap. 5, Birgit Grahl and Eva Schmincke clearly classify these terms, the overarching objective being quality assurance in LCAs. To this end, the critical review is described, quality assurance by means of verification, and the PCR (Product Category Rules) review according to the requirements of ISO 14025 (2006), in order to describe similarities but also specific differences.

1  Why Do We Cover Interpretation As Well As Critical Review and Reporting in One…

5

Chapter 6: Benefits from Critical Review and Communication Hans J. Garvens describes in Chap. 6 the benefits of the critical review, thereby covering both the benefits generated within the LCA projects that are not visible from the outside as well as the benefits taking effect in the communication of LCA results. Two different types of target group communication are considered: business to authorities as well as business to public/stakeholders. In addition, it is shown that – in the sense of a value-based approach – critical review must not only be seen as a binding task according to the standards but also as a benefit for communication. Chapter 7: Cost-benefit Analysis of Critical Review: Learning from Practice Christian Bauer and Andreas Detzel deepen this issue on the basis of practical experience. They differentiate between direct and indirect advantages and also give magnitudes of the financial costs of a review with three external experts in relation to the total costs of a life cycle assessment. Interestingly, regardless of the objective and scope of individual studies, there are recurring issues that can be conflicting in the review process. Against this background, the authors give practical advice on how to ensure a favorable relationship between expenditure and the benefits of the review. Chapter 8: Reporting and Communication Pere Fullana focuses on reporting and communication of an LCA.  The ISO standards and other guidelines deliver extensive provisions and requirements of reporting; however, they provide less guidance on how the content may be reported and the philosophy behind communication. Therefore, this chapter includes two main approaches: first, a reflection on how LCAs should be reported following the rules of science and, second, how they should be shaped depending on the object to be reported and the subject to receive the report. Furthermore, some implicit aspects of LCA nature are dealt with, such as being iterative, comparative, or weighted. We thank all authors contributing to this volume, bringing their long-term experience based on theory and praxis to the benefit of the readers.

References Curran MA(ed) (2017) Goal and scope definition in life cycle assessment. In: Klöpffer W, Curran MA (series eds) LCA compendium – the complete world of life cycle assessment. Springer, Dordrecht, p 170 ISO 14025 (2006) Environmental labels and declarations – type III environmental declarations – principles and procedures ISO 14040 (2006) Environmental management  – life cycle assessment  – principles and framework. ISO/TC 207/SC 5ISO/TC 207/SC 5. Life cycle assessment ISO 14044 (2006) Environmental management – life cycle assessment − requirements and guidelines. ISO/TC 207/SC 5. Life cycle assessment PD ISO/TS 14074 (2022) Environmental management – life cycle assessment – principles, requirements and guidelines for normalization, weighting and interpretation

Chapter 2

Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation Mary Ann Curran

Abstract This chapter describes the scientific background of the phase “Interpretation” in Life Cycle Assessment (LCA). In any scientific LCA work, interpretation is integral to explaining the patterns and trends uncovered through analysis of the data, bringing all of the background knowledge, experience, and skills of the analyzer to bear on the question and relaying their observations about the data. Interpreting the results of an LCA attempts to answer the question posed in the goal statement and reflects on and conveys the life cycle inventory and results of the impact assessment to the decision maker in an easily understandable manner. Interpretation is not only conducted after the data collection and impact-modeling tasks have been completed. Rather than following a linear path, interpretation is iterative. Iteration is a crucial part of the interpretation phase and in conducting a reliable, defensible LCA. The practitioner is required to return to the goal and scope in order to ensure the study is moving in line with the original goal and scope and return to the inventory and the impact assessment to ensure results of the impact indicators and the underlying inventory data are acceptable for drawing conclusions and making recommendations. The text in the ISO standard for LCA (14044) establishes a clear framework for the essential elements to be included in the interpretation phase. However, the sections in the standard on interpretation are brief, giving no details on step-by-step procedures or techniques to be employed. The same applies to most guidebooks and textbooks on LCA which may mention conducting uncertainty analysis but give no clear guidance on how this should be done. Detailed guidance on how to interpret LCA results is needed. Keywords  Goal and scope definition · Interpretation · LCA · LCI · LCIA · Life cycle assessment · Life cycle impact assessment · Life cycle inventory analysis · MCDA · Multi-criteria decision analysis

M. A. Curran (*) LCA & Sustainability Consultant, BAMAC Ltd., York, SC, USA © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_2

7

8

M. A. Curran

2.1 Interpretation Is a Basic Building Block in Any Scientific Work Whether measured, collected, or estimated, data generated for a scientific study – hereon Life Cycle Assessment (LCA) – only become useful for making decisions once they have been deliberated on and further presented in graphic form, through images, or via other analytical tools. Data (the plural form of the word datum) are scientific observations and measurements that, once fully interpreted, can be developed into evidence to address a question (Egger and Carpi 2008). In general, when scientists interpret data, they attempt to explain the patterns and trends by bringing their background knowledge, experience, and skills to bear on the question and relate their data to existing scientific ideas. Knowledge is derived from extensive amounts of experience dealing with information on a subject. Given the personal nature of the knowledge they draw upon, this step can be subjective, but that subjectivity is scrutinized through the peer review process. “Interpretation” involves determining what the results of the experiment show and deciding on the next action or actions to take. Evidence from other scientists and experience are frequently incorporated at this stage in the process. Depending on the complexity of the experiment or study, many iterations may be required to gather sufficient evidence to answer a question with confidence, or to build up many answers to highly specific questions in order to answer a single broader question.

2.2 Interpretation in Life Cycle Assessment Studies Life Cycle Assessment (LCA) relies heavily on both data and software. Reliable data is the driving force behind LCA as large amounts of process and production data are needed. These input and output data are then converted into a series of impact indicators through models that are also based on data. Early attempts to define LCA methodology in the 1990s focused on describing and advancing the data generation phases (inventory analysis and impact assessment). Instead of an interpretation phase, the initial framework included “Improvement Analysis” (Fava et al. 1991). Life Cycle Improvement Analysis – A systematic evaluation of the needs and opportunities to reduce the environmental burden associated with energy and raw materials use and environmental releases throughout the whole life cycle of the product, process, or activity. Analysis may include both quantitative and qualitative measures of improvements such as changes in product or process design, reductions in raw material or energy usage, or reduction in outputs with an adverse impact on the environment. (Fava et al. 1991).

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

9

Further development of the ISO standards on LCA led to the well-known four-­phase framework (goal and scope definition phase was introduced in 1992). The “improvement analysis” phase was seen as overly ambitious, requiring manufacturers to take action. Instead the new, and less demanding, title “Interpretation” was introduced into the standard (Klöpffer and Grahl 2014). The evolution of the LCA framework is shown in Fig. 2.1. In the ISO framework for LCA methodology, the Interpretation phase is where conclusions are drawn from the findings of the life cycle inventory analysis and the impact assessment. Recommendations are made according to the established goal of the study. The revised framework (shown on the right-hand side in Fig. 2.1) is notable in that it shows iterations between the different phases and does not suggest addressing one after the other in a linear, sequential approach (ISO 14040: 2006). The ISO framework shows direct links between interpretation and the other three phases. That is, interpretation is not something that is done at the end after the data collection and impact-modeling tasks have been completed. An LCA results in multiple tables of inventory data and impact indicators which can be difficult for an individual to comprehend because of the vast amounts of data, diversity of physical units, use of value judgments, and uncertainty in the parameters. These factors limit users from being able to directly, and transparently, interpret information for use in decision-making. As a result, many comparative LCA studies stop the assessment after calculation of potential impact indicators (characterized data), leaving the decision makers on their own to confront multi-criteria, multi-­ stakeholder problems (Prado et al. 2012).

Life cycle assessment framework Goal and scope definition Inventory analysis Impact Analysis

Life-cycle Assessment Inventory

(SETAC 1990)

Improvement Analysis

Impact Assessment

Goal Definition and Scoping

Improvement Assessment

Interpretation

Impact assessment

Inventory (SETAC 1993)

(ISO 1997/2006)

Fig. 2.1  Evolution of the life cycle assessment framework. (This figure was published in LCA Compendium, volume “Background and Future Prospects in Life Cycle Assessment,” p.  193 (Curran 2014))

10

M. A. Curran

2.3 Interpretation Is Different from Assessment and Valuation While the number of interventions included in an LCA is typically in the hundreds, the results are transformed into fewer, more easily manageable impact categories. The ISO standard allows for two additional, optional steps to be performed, normalization and weighting, to facilitate the understanding of the LCA results and communicate them to decision- and policy-makers. In the weighting step, the normalized results are multiplied by a factor representing the relative importance of the impact category to the other impact categories. Therefore, this step is the application of value judgment. The magnitude of the different impact categories can directly be compared, and it is possible to point out the most significant impact categories. Sometimes the normalization step and the weighting step are carried out as one single step. Since normalization and weighting imply that additional factors are multiplied by the characterized results, these results will be more uncertain. Therefore, these results will not be used for presenting the results of the LCA. However, the weighted results can be used for identifying the most significant impact categories. Many different weighting methodologies have been proposed and several are used regularly. Most existing studies apply the average of the responses obtained from a group of people representing the decision maker. Others believe that weighting factors should be based on the preferences of society as a whole so that LCA practitioners can successfully apply them to products and services everywhere. It is important to note that although assessment and valuation of results through weighting and normalization are often described as a way to interpret LCA results, these steps are different from the type of interpretation suggested by the LCA framework in the ISO standard (ISO 14044: 2006a). The standard defines interpretation as the “phase of life cycle assessment in which the findings of either the inventory analysis or the impact assessment, or both, are evaluated in relation to the defined goal and scope in order to reach conclusions and recommendations.” In contrast, decision-making is the act of choosing between multiple options or solutions. This process can be supported by decision analysis methods, such as Multi-Criteria Decision Analysis (MCDA). MCDA refers to a suite of methods that can be used by decision makers (individuals or organizations acting as an individual) to help them organize and synthesize information and, in the end, select the best (“sustainable”, “environmentally friendly”, etc.) alternative among competing options, capable of handling complex decision problems with multiple, conflicting criteria with incommensurate units. MCDA methods are not decision-making tools in themselves, that is, they are not intended to make actual decisions. Instead, they guide the decision-making process in a dynamic and iterative manner. Typically, there is no single optimal solution for such complex problems, so it is necessary to use decision maker’s preferences and values to differentiate between solutions.

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

11

The remainder of this chapter focuses on Interpretation without going further into how to conduct valuation.

2.4 Interpretation According to ISO As LCA methodology has been developed, the Interpretation phase has not received the same level of attention as inventory analysis and impact assessment, leaving users with little to go on regarding how to conduct interpretation. The text in the ISO standard on interpretation is very brief, giving no details on step-by-step procedures or techniques to be employed. The same applies to most guidebooks and textbooks on LCA which may mention conducting uncertainty analysis but give no clear guidance on how this should be done. However, the ISO standard establishes a clear framework for the essential elements to be included in the interpretation phase. The ISO standard defines two objectives of life cycle interpretation: 1. Analyze results, reach conclusions, explain limitations, and provide recommendations based on the findings of the preceding phases of the LCA and report the results of the life cycle interpretation in a transparent manner. 2. Provide a readily understandable, complete, and consistent presentation of the results of an LCA study, in accordance with the goal and scope of the study. In ISO 14044, the life cycle interpretation phase of an LCA, or a life cycle inventory (LCI), study contains several elements, as depicted in Fig. 2.2:

Fig. 2.2  The Interpretation phase is directly interconnected with the other phases of the LCA framework. (Modified according to ISO 14044: 2006a, Fig. 4)

12

M. A. Curran

• Identification of the significant issues based on the results of the LCI and LCIA phases • An evaluation that considers: –– Completeness check  – To ensure that all relevant information and data are available and complete –– Sensitivity analysis  – To assess the reliability of the final results and conclusions –– Consistency check  – To determine whether the assumptions, methods and data are consistent with the goal and scope • Conclusions, limitations, and recommendations Other important elements of interpretation noted in the ISO standard include: • Appropriateness of the definitions of the system functions, the functional unit and system boundary • Limitations identified by the data quality assessment and the sensitivity analysis The iterative nature of the ISO framework shows up in this context. It is especially important to determine that if the results of the impact assessment and the underlying inventory data are incomplete or unacceptable for drawing conclusions and making recommendations, that is, the uncertainties are too high, then those steps must be repeated until the results can support the original goals of the study. Whenever sensitivity analysis shows that some decisions are crucial, a more refined analysis may be in order.

2.5 The Problem of Impact Assessment Results The life cycle inventory results are presented in tables, as are the resultant impact indicators. Impacts are also presented in graphic form, most often in pie charts and bar charts. Because of the frequency, some people may erroneously see LCA as resulting in only a series of charts and graphs. However, they are the means, not the end (ACLCA 2014). It is necessary to relate not only to the results of the impact assessment. Results of the Impact Assessment Depend on the Characterization Model Chosen  Although the ISO guidelines on LCA offer some standardization to a general framework, they do not provide a single, technically detailed approach. LCIA methods that are readily available for use in LCA software include CML-IA, TRACI, IMPACT WORLD+, EDIP, ReCiPe, and ILCD/PEF (Rosenbaum 2016). The existence of several different LCIA methods, i.e., the availability of different characterization models, can create confusion partly due to differing results,

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

13

depending on the methodology chosen. The UNEP-SETAC Life Cycle Initiative aided further developments toward consensus and a recommended best practice, and this work has since been complemented by the activities of other organizations, such as the United States Environmental Protection Agency (US EPA) and the European Commission (EC 2010a). However, it is still up to the practitioner to choose the LCIA method. In the ideal case, an experienced LCA practitioner will be familiar with the main properties and characteristics of several of these methods, that is, be familiar with most parts of their documentation (Rosenbaum 2016). This experience allows the practitioner to run two impact models in parallel and compare the results. Significant differences in the results can lead to deeper investigation into the inventory and how the data were modeled in the impact assessment, hence, further aiding interpretation. Impact Indicators Are Not Intended to Reflect Actual Impact  If an LCA determines that a product system results in a GWP impact indictor of 3.4 CO2-­equivalents, this does not mean that exactly 3.4 kg of CO2 emissions will occur as a result the product. The total emission which is aggregated across the entire system is a “virtual” emission calculated based on the chosen functional unit for the study. Thus, this total does not represent an actual emission from a single source. Furthermore, geographical, technological, and temporal variations in making the same kg of material may have a different GWP.  Because they cannot be measured directly, impact assessment results are based on accepted environmental models based on potentials. These indicators are intended for use in supporting decision-­making (ACLCA 2014). Not All Inventory Data Can Be Characterized in an Impact Model  It is easy to get caught up in excitement of seeing the results of using impact methods, such as TRACI or ReCiPe. But it is also important to keep in mind that not all inventory data can be modeled for their related impact. That is, the user should also take into account inventory data that cannot be modeled via an impact model, such as noise. In these cases, the inventory data should also be part of the interpretation. This is also the case when impact data are not available, such as the release of nanoparticles into the environment whose effects on human and eco health are not yet fully understood. The early 1991 SETAC definition acknowledges the need to retain qualitative as well as quantified data (see box above Life Cycle Improvement Analysis). Therefore, it is recommended that uncharacterized elementary flows are reported before any interpretation or conclusions are drawn. The evaluation of uncharacterized elementary flows is of primary importance to assess the completeness of the LCIA coverage of the identified life cycle inventory flows. Indeed, a high number of elementary flows inventoried might be noncharacterized by the chosen models, even if their relevance in the assessed impact categories could be of primary importance (EC 2016).

14

M. A. Curran

2.6 Interpretation to Goal and Scope Definition, Inventory Analysis, and Impact Assessment Life Cycle Interpretation is a systematic technique to identify, quantify, check, and evaluate information from the results of the life cycle inventory and, if conducted, the life cycle impact assessment. The results from the inventory analysis and impact assessment are summarized during the interpretation phase. The relationship between interpretation and the other phases are explored in more detail in the following sections.

2.6.1 Goal and Scope The relationship between interpretation and goal and scope is elaborated in the ISO standard: The interpretation phase should deliver results that are consistent with the defined goal and scope and which reach conclusions, explain limitations and provide recommendations. (ISO 14040: 2006)

Goal and scope explains the intended purpose of the study, identifies the target audience and application, and specifies the methods and databases that are expected to be used in the study. The interpretation then needs to assess the findings of the LCA in relation to the goal and scope. Any deviations from the defined study goal and scope that may have occurred in creating the life cycle inventory or modeling the potential impact indicators need to be addressed in the interpretation. It may also happen that a revision to the goal and scope is in order. Information from phase 1 (goal and scope) is necessary for the adaptation to the objective of the study. This reflects the insight that every LCA is unique. Each depends on the objective(s) of the goal and scope definition phase. Therefore, the evaluation can only take place within the coverage of the goal and scope under study. However, if during the study and data collection phase it is found the original premise cannot be followed through or it is anticipated the original goal cannot be met, the goal and scope can be adapted and revised owing to the iterative approach of the method. Example of Revisiting the Study Goal  In 2009, researchers at the US Environmental Protection Agency set out to investigate the potential impacts of cotton socks embedded with nanoscale silver, considered beneficial for their potential antimicrobial properties (Meyer et al. 2011). The original goal was stated as follows: The screening level LCA will examine the impacts associated with the production and use of a pair of cotton socks containing nanoscale silver when compared to socks manufactured without the silver. Assumptions:

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

• • • • • •

15

Average pair of knit socks made from cotton fibers 60.8 g of fiber and 10.2 mg of nanoscale silver 30 nm silver nanoparticles 1-year useful life (50 wash cycles) All materials produced and used within the continental United States All silver is lost during washing

The results of the screening-level LCA were very useful in identifying the life cycle hotspots and the contribution of the silver to the overall impact. In addition, a key finding was that the impacts from washing the socks over their assumed lifetime dominated the results. However, it was quickly realized that a direct comparison between silver socks and plain cotton socks was not possible since the antimicrobial properties of the silver needed to be taken into account. That is, in order to do such a comparison, an additional product such as foot powder or spray that offered anti-­ microbial benefits had to be included in the study of the plain socks in order to lead to a fair comparison. Example of Interpreting Results Within the Study Scope  Some system boundaries of an LCA cover activities from the “cradle” (i.e., the extraction of raw materials, agricultural activities and forestry) up to the factory or farm gate (i.e., the point where the final product is ready to be transported for sale). Typically cradle-­to-­gate studies are useful in supporting B2B (business-to-business) sales. As such, they are useful in identifying hotspots within the supply chain. But the interpretation of the results must be confined to the limited boundaries and not misrepresent findings as if they represent the full life cycle. For example, where claims of a product being “more sustainable” than another are not supported in a cradle-to-gate study, it could be appropriate to say that the production of the product was found to carry less environmental impact than the alternative product that was studied. Example of Interpreting Results Within the Confines of the Goal  The use of a single impact indicator is sometimes applied in studies using LCA methodology. The most popular of these types of studies is Carbon Footprint, which focuses on accounting for greenhouse gas releases and calculating global warming potential. Other single-issue studies include chemical toxicity, energy demand, water footprint, material flow, etc. Such “streamlining” may be done to either simplify the data collection effort (i.e., reduce the amount of data needed) or to focus on the impact category the client has decided is the most relevant to their cause. Since this type of limited study results in a less complete inventory, the practitioner must be careful to present the findings in support of the study and within the confines of the goal, without making claims of overall environmental superiority and the like, which would go beyond the goal and scope.

16

M. A. Curran

2.6.2 Inventory Analysis Interpretation of the inventory analysis depends on a variety of factors such as the purpose and context under which the inventory is developed, the extent and reliability of the data, and the intended use of the data. This suggests that care must be used in presenting and communicating the results of a life cycle inventory. Interpretation of Life Cycle Inventory aims to: • Improve the inventory model to meet the needs derived from the study goal • Perform a sensitivity analysis to check for limitations in the appropriateness of the life cycle inventory work • Aid in understanding the underlying assumptions when secondary datasets are used in the model • Combine the above points to refine the LCA model • Draw appropriate and robust conclusions (EC 2016) Examination and documentation regarding the quality and completeness of the data and applied calculation methods and assumptions to determine if the inventory is sufficient to support the findings. Inventories can be subject to the following problems in interpretation: –– Those unfamiliar with inventories may misinterpret them as characterizing the actual or potential environmental impact of the products, processes, or activities. –– Public use of inventory data that are incomplete or not presented in context can mislead consumers into believing they are being informed of the total impact or of the most important impacts associated with the product, or that one product is better for the environment than another. –– Readers might infer a higher degree of accuracy to inventories than the quantity or quality of data allows. –– The use of national or aggregated data can mask regional or site-specific variations in energy and material requirements and pollution or waste generated. Example of Interpreting the Inventory  It is not always possible to fully satisfy everything that was foreseen for the LCA. Modifying the goal and scope, in these cases, does not always make sense. For example, if only older data sets for a specific process are available, and these data do not meet the data quality requirements established in the goal and scope, it is often more convenient to address these shortcomings in a limitation section in the final report. There is, however, also the possibility to directly address limitations in the goal and scope in the interpretation phase. Explaining limitations of conclusions and recommendations is very important in decision support in order to avoid overstating and misinterpreting LCA results (Ciroth 2016).

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

17

2.6.3 Impact Assessment At times, impact assessment results can help identify errors or gaps in the inventory data, which were not obvious when examining the inventory data tables directly. The impact indicators when viewed together can flag where there may be a problem with the underlying data. Other elements subject to interpretation in the LCIA phase include: • • • •

The relevance of applying different characterization models The relevance of applying different normalization sets The relevance of applying different weighting sets (valuation) The identification of the most sensitive elements in determining the final results, conclusions, and recommendations • The coverage of characterization factors compared to the inventoried elementary flows Section 5.5 of the ISO 14040 standard states: The LCIA results are based on a relative approach; they indicate potential environmental effects, and they do not predict actual impacts on category endpoints; the exceeding thresholds or safety margins, or risks. This includes a warning of over-interpretation of results of the impact assessment. This caution extends throughout the entire 14,040 framework (Klöpffer and Grahl 2014). LCIA results do not predict impacts on category endpoints, exceeding thresholds, safety margins, or risks. (ISO 14040:2006)

2.7 Interpreting Procedural Versus Numerical Results In general, a distinction between procedural and numerical approaches can be drawn: • Procedural approaches include all types of analyses that deal with the data and results in relation to other sources of information, like expert judgment, reports on similar products, intuition, reputation of data suppliers, and so on. • Numerical approaches include those approaches that somehow deal with the data that is used during the calculations, without reference to those other sources of information, but as algorithms that use and process the data in different ways, so as to produce different types of “smart” data reduction that provide an indication of reliability, key issues, discernibility, robustness, and so on. This distinction helps us understand some important roles of interpretation. On one hand, it is about comparing the data and results with previous findings and putting the results in the context of decision-making and limitations. On the other hand, it is devoted to a systematic analysis with the help of statistical and other decision-analytic techniques. The latter type may be incorporated in software.

18

M. A. Curran

Indeed, an increasing number of software packages contain options for running Monte Carlo analysis, doing sensitivity analysis, carrying out statistical significance tests, etc. (Heijungs and Guinee 2012).

2.8 Importance and Implementation of the Iterative Approach Based on Interpretation As shown in Fig. 2.3, an LCA study is a highly iterative process. The LCA practitioner is required to revisit the goal and scope after preliminary inventory work is completed to ensure the study is moving forward in line with the original goal. The practitioner also revisits the impact assessment in relation to the inventory analysis. Ideally, the overall quality of the data improves in accuracy, precision and completeness with each iteration. As mentioned earlier, and supported by the ISO framework for LCA (Fig. 2.1, right-hand side), interpretation is not something that is done at the end after the data collection and impact modeling tasks have been completed. Rather, the goal and scope, inventory, and impact assessment results are revisited throughout the study. Owing to the iterative approach of LCAs, the evaluation requires some experience. This is taken into account in Annex B of ISO 14044 Examples of Life Cycle Interpretation (ISO 14044: 2006b). It is meant as a support for practitioners to help them understand how to interpret LCA studies. Annex B (page 36) states “It may also be possible to undertake this structuring procedure for individual impact

Overall data quality (accuracy,precision,completeness)

3rd Iteration

better data for key processes and flows(background and foreground)

2nd Iteration

revision of scope definition? better data for key processes (background and foreground) more specific data for foreground processes

1st Iteration

full product system specified data as available easily available secondary data

LCI

Goal and Scope

LCI

Goal and Scope

LCIA

LCIA

LCI

Goal and Scope

Evaluation

Evaluation

LCIA

Evaluation

Time and ef fort

Fig. 2.3  Life cycle assessment is a highly Iterative process. (EC 2010b)

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

19

categories for a more detailed examination.” Guidance for identifying “significant issues” consists of (1) structuring of information and (2) subsequent determination of any significant issues. Structuring of Information  The structuring of the available data and information is an iterative process undertaken in conjunction with the LCI and (if performed) LCIA phases, as well as with the goal and scope definition. This structuring of information may have been completed previously in either the LCI or LCIA and is intended to provide an overview of the results of these earlier phases. This facilitates determination of important and environmentally relevant issues, as well as the drawing of conclusions and recommendations. The use of different structuring approaches can offer additional insight. Possible ways to group the study results include the following approaches: (a) Grouping by individual life cycle stages; for example, raw material acquisition, production of materials, product manufacture, use, final disposal (b) Grouping by processes; for example, transportation, energy supply, fabrication (c) Grouping by processes under different degrees of management influence; specifically foreground and background processes1 (d) Grouping by individual unit processes offers the highest resolution possible On the basis of this structuring process, any subsequent determination is performed using analytical techniques. The standard includes detailed guidance on how to perform checks for: –– Completeness –– Sensitivity –– Consistency Determination of Any Significant Issues  The identification of significant issues contains two interrelated aspects: (1)There are the main contributors to the LCIA results, that is, the most relevant life cycle stages, processes and elementary flows, and the most relevant impact categories; and (2) there are the main choices that have the potential to influence the precision of the final results of the LCA. These can be methodological choices, assumptions, foreground, and background data used for deriving the process inventories, LCIA methods used for the impact assessment, as well as the optionally used normalization and weighting factors (EC 2016). Depending on the goal and scope of the study and the level of detail required, the following methods can be recommended for determining significance: (a) contribution analysis, (b) dominance analysis, (c) influence analysis, and (d) anomaly assessment. These are covered in detail in Chap. 3 of this book.

 While foreground processes are under the control of the decision maker for whom an LCA is carried out, background processes cannot be directly influenced by the decision maker. 1

20

M. A. Curran

2.9 Conclusions The key aim of LCA is to provide decision makers with comprehensive and understandable information; this task depends on proper interpretation of the results of an LCA study. Over the years, LCA methodology evolved from including a phase in which an improvement assessment was conducted to including the less demanding activity of interpretation. Life Cycle Interpretation is a systematic technique to identify, quantify, check, and evaluate information from the results of the life cycle inventory and, if conducted, the life cycle impact assessment. Not to be viewed as the “last step” in conducting an LCA, the interpretation phase entails the evaluation of the results of the inventory analysis as well as the results of the impact assessment, and keeping in line with the stated goal of the study, with a clear understanding of the uncertainty and the assumptions used to generate the results. The results from the inventory analysis and impact assessment are summarized during the interpretation phase. The ISO framework shows direct links between interpretation and the other three phases. Therefore, the iterative nature of interpretation is crucial in conducting a reliable, defensible LCA. While the ISO standard outlines the essential components to be included in the interpretation phase of an LCA, better, more detailed guidance on how to interpret LCA results is needed.

References Ciroth A (2016) Goal and scope connection to the interpretation phase. Chapter 5 “Goal and Scope Definition in Life Cycle Assessment” (Curran MA ed). In: LCA compendium – the complete world of life cycle assessment (Klöpffer W, Curran MA, series eds). Springer, Dordrecht, pp 161–168 Curran MA (2014) Strengths and limitations of life cycle assessment. Chapter 6 “Background and Future Prospects in Life Cycle Assessment” (Klöpffer W ed). In: LCA compendium – the complete world of life cycle assessment (Klöpffer W, Curran MA, series eds). Springer, Dordrecht, pp 189–206 EC (2010a) (European Commission  – Joint Research Centre  – Institute for Environment and Sustainability). International Reference Life Cycle Data System (ILCD) Handbook – framework and requirements for life cycle impact assessment models and indicators. First edition March 2010. EUR 24586 EN. Publications Office of the European Union, Luxembourg EC (2010b) (European Commission  – Joint Research Centre  – Institute for Environment and Sustainability). International Reference Life Cycle Data System (ILCD) handbook – general guide for life cycle assessment  – detailed guidance. First edition March 2010. EUR 24708 EN. Publications Office of the European Union, Luxembourg EC (2016) (European Commission  – Joint Research Centre  – Institute for Environment and Sustainability). JRC technical report: guide for interpreting life cycle assessment results. Zampori L, Saouter E, Schau E, Cristobal J, Castellani V, Sala S. EUR 28266 EN Egger AE, Carpi A (2008) Data analysis and interpretation in VisionLearning. Vol. POS-1 (1); accessed on-line October 22, 2017, https://www.visionlearning.com/en/library/ Process-­of-­Science/49/Data-­Analysis-­and-­Interpretation/154

2  Interpretation, Critical Review, and Reporting: Scientific Outline of Interpretation

21

Fava JA, Denison R, Jones B, Curran MA, Vigon B, Selke S, Barnum J (eds) (1991) A technical framework for life-cycle assessments. Workshop held 18–23 August 1990. Society of Environmental Toxicology and Chemistry (SETAC), Washington, DC Heijungs R, Guinée JB (2012) Chapter 2. An overview of the life cycle assessment method – past and future. In: Curran MA (ed) Life cycle assessment handbook: a guide for environmentally sustainable products. Wiley-Scrivener. ISBN: 978-1-118-09972-8. Beverly, Massachusetts, USA ISO 14040 (2006) Environmental management  – life cycle assessment  – principles and framework. International Organization for Standardization Geneva ISO 14044 (2006a) Environmental management – life cycle assessment – requirements and guidelines. International Organization for Standardization Geneva ISO 14044 (2006b) Environmental management  – life cycle assessment  – requirements and guidelines, Annex B examples of life cycle interpretation. International Organization for Standardization Geneva Klöpffer W, Grahl B (2014) Life cycle assessment: a guide to best practice. Wiley-VCH. Weinheim, Germany Meyer D, Curran MA, Gonzalez M (2011) An examination of silver nanoparticles in socks using screening level life cycle assessment. J Nanopart Sci 3(1):147–156 Prado V, Rogers K, Seager T (2012) Chapter 19. Integration of MCDA tools in valuation of comparative life cycle assessment. In: Curran MA (ed) Life cycle assessment handbook. Scrivener-Wiley. Beverly, Massachusetts, USA Rosenbaum R (2016) Selection of impact categories, category indicators and characterization models in goal and scope definition. Chapter 2 “Goal and Scope Definition in Life Cycle Assessment” (Curran MA ed). In: LCA compendium – the complete world of life cycle assessment (Klöpffer W, Curran MA, series eds). Springer, Dordrecht, pp 63–122 Schenck R, White P (2014) Environmental Life Cycle Assessment: Measuring the Environmental Performance of Products, American Center For Life Cycle Assessment: Vashon Island, Washington, USA. ISBN-978-0-9882145-5-2

Chapter 3

Data Quality Analysis as Part of Interpretation Daniela Kölsch and Sönke Giebeler

Abstract  This chapter focuses on Data Quality Analysis (DQA) as part of interpretation. As a starting point of their considerations, Daniela Kölsch and Sönke Giebeler clarify the relation between data quality and data uncertainty, as these terms are closely interwoven with each other. Whereas uncertainty refers to lack of knowledge, data quality should be considered as an umbrella term which includes criteria such as uncertainty and variability. Against this background, Kölsch und Giebeler define DQA as an approach to explore to which extent specific data quality requirements are fulfilled. The authors set forth the broad range of data which has to be managed in LCA and refer to several attempts to structure these data. Furthermore, the several types of data quality indicators and the operationalization are described in detail. Based on this theoretical framework, more insight could be provided by analyzing best practice examples, taking into account the type of data used, the quality indicators proposed, and the scoring system applied. Keywords  Data quality analysis (DQA) · Data quality indicators · Interpretation · Life cycle assessment (LCA) · Life cycle impact assessment (LCIA) · Life cycle inventory analysis (LCI) · Life cycle thinking · Pedigree matrix · Quality scoring system

D. Kölsch Bayer Technology Services GmbH, Leverkusen, Germany e-mail: [email protected] S. Giebeler (*) OEKO-TEX® Association, Zurich, Switzerland e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_3

23

24

D. Kölsch and S. Giebeler

3.1 Introduction 3.1.1 Incorporation of Data Quality Analysis into the Contextual Framework of Life Cycle Thinking Due to rising awareness of sustainability strategies in politics and private and publishing sectors, life cycle thinking is getting an upsurge. More and more studies are modeling resource consumption, environmental releases and potential environmental and human health impacts of product systems (Lloyd and Ries 2007). Hence, data issues received increased attention in the life cycle assessment (LCA) community. Case studies, reports, surveys, and other research efforts raise steadily the availability of datasets. Therefore, it is crucial to value the quality of information provided. It is of utmost importance that LCAs are performed accurately and profoundly. For this purpose, quality parameters, such as information on uncertainty or representativeness, are increasingly integrated as background data in LCA-software1 (Henriksson et al. 2014). On the one hand, extended databases and new methodologies could simplify and ease LCA procedures; on the other hand, the flood of new information could lead to imprudent data treatment, inasmuch as data is assimilated and reused with insufficient reconsideration. Thus, critical reflection has to be sharpened and data use needs to be treated with care. Assessing the life cycle of a product system means that a practitioner is confronted with various types of data. This includes both numerical values, such as consumption data, and characterization factors, as well as inherent characteristics like representativeness or fuzziness of the system. Therefore, it is inevitable that practitioners deal with the challenge of data quality. Early approaches focusing on data quality were devised, for instance, by the SETAC Working Group ‘Data Availability and Data Quality” (Bretz 1998; Huijbregts et  al. 2001) and by the United States Environmental Protection Agency (EPA) (1995) with the “Guidelines for Assessing the Quality of Life-Cycle Inventory Analysis’. More research work followed by LCA scientists, inter alia from van den Berg et al. (1999), Bauer et al. (2004), Frischknecht et  al. (2007), European Commission (EC 2010), United Nations Environment Programme (UNEP)/Society of Environmental Toxicology and Chemistry (UNEP/SETAC 2011), Weidema et al. (2013). Nonetheless, ‘clear definitions of parameters of how the input parameters should be defined and what they need to enclose’ are currently missing (Henriksson et al. 2014). In order to prevent any confusion, it is important to distinguish between data quality and data uncertainty. Van den Berg (1999) noted that ‘it may be necessary to make a distinction between the “quality” of databases and the uncertainty of a particular LCA result’. The most common definition of uncertainty2 refers to a lack  Software providers, such as SimaPro, Umberto, and GaBi, implemented Monte-Carlo-Simulation. In addition, ecoInvent introduced quantitative uncertainty values to their database. 2  The term ‘uncertainty’ could be defined by distinguishing between epistemic and aleatory uncertainty. Epistemic uncertainty describes a lack of knowledge; aleatory uncertainty (often used under 1

3  Data Quality Analysis as Part of Interpretation

25

of knowledge, due to missing information or systemic fuzziness (Heijungs and Huijbregts 2004). Data quality should moreover be considered as an umbrella term incorporating data criteria such as uncertainty and variability. Both terms, data quality and uncertainty, are closely interwoven with each other, which jeopardizes their notional cutoff. However, ‘data quality and uncertainty should be analyzed separately, noting that data quality is not a part of data uncertainty but data uncertainty is a part of data quality’ (Cooper and Kahn 2012). Data quality analysis (DQA), as proposed here, is an approach to check to what extent specific data quality requirements are fulfilled. These requirements could be expectations that are stated, for example, in documented information, generally implied as a common practice or that are obligatory due to other general regulations, for example, the International Standard Organization (ISO 9000:2005). It has to be noted that these requirements are subject to the study-specific framework. Data quality requirements should already be defined within the goal and scope of an LCA (ISO 14044:2006). In the stricter sense, data quality focuses on LCI and Life cycle impact assessment (LCIA) (EC 2010), where data are described by its inherent attributes. During the LCI phase, data quality indicators (DQIs)3 could be used to characterize relevant information by means of several quality criteria, such as representativeness, completeness, spread or inherent uncertainty (Van den Berg 1999). In the LCIA phase, DQA is as an optional element that portrays the reliability of the LCIA results. In case of an LCA study that strives to be communicated to any third party, data quality assessment is a must within the LCI analysis as well as in the life cycle interpretation. DQA is understood as a holistic and iterative approach that covers all LCA phases. A quality assessment should achieve a ‘better understanding concerning the reliability of the collection of indicator results’ (ISO 14044:2006). When assessing data quality, each dataset is scored, whether qualitatively or quantitatively. This is why DQA is mostly classified as a semi-quantitative analysis tool. The quality criteria and rating scales are typically outlined in a pedigree matrix (Ciroth et al. 2013). Chapter 2 provides different types of data, quality indicators, and scoring systems embedded in such a matrix. Single quality scores, resulting from the quality judgment, may be grouped and aggregated to an overall data quality rank. Some options for such an aggregation are outlined in Chaps. 3 and 4. Information on data characteristics should be interlinked with the results of the preceding LCA phases (Goal and Scope, LCI and LCIA) and subsequently discussed in the interpretation phase (ISO 14044:2006). This means to prove requirements, assumptions and findings on reliability; to check their degree of validity; and to identify significant issues. Completeness, sensitivity and consistency tests4 could the term ‘variability’) represents an inherent randomness of a system being analyzed or its environment (Thunnissen 2003; Iaccarino 2008). 3  Van den Berg (1999) distinguishes between data quality assessment via qualitative indicators and via probability distributions. 4  Besides these approaches, provided by ISO 14044:2006, some other analyses were developed, for instance by EPA (2006), who distinguishes between contribution analysis, dominance analysis and anomaly assessment.

26

D. Kölsch and S. Giebeler

supplementary enhance confidence and reliability of LCA results. ‘Thus, data quality in a broader sense could support conclusions and recommendations of studies and indicate to which degree the data basis of the study meets requirements regarding reporting, transparency, review, reproducibility, etc.’ (EC 2010).

3.1.2 Key Questions in this Article: Goals of Data Quality Analysis in Interpretation Disregarding methodological or terminological inconsistencies (Heijungs and Huijbregts 2004), a practitioner has to draw conclusions by means of transparent and comprehensible interpretation. LCA is a dynamic, flexible and volatile research field. Even studies, compliant with a similar methodological basis, may vary in their findings due to the practitioners’ experience, the availability of datasets or the model premises. ‘The sourcing of representative unit process data is moreover influenced by value judgments, epistemological perspectives and ethics, which may further influence results’ (Lazarevic et al. 2012). Nonetheless, evaluation methods, specified by ISO or other guidelines,5 have to fulfill certain requirements. This applies particularly to comparative LCA studies. However, sufficient space for own decisions and actions has to be preserved, because uncertainties, assumptions and proxies are inherent in LCA modeling. Leaving such a broad discretion presupposes that a practitioner acts conscientiously and with sufficient critical acclaim. Qualitative data analysis, inasmuch as it is understood as an iterative process within the entire LCA, focuses on the interrelation between all four LCA phases. During this article the following questions should be discussed. • Is DQA a proper method to identify significant issues, such as data gaps and deficiencies, and how do they affect the interpretation of study results? • Is DQA applicable on all stages of LCA or is it merely an issue of life cycle inventory? • To what extend could DQA be standardized − is a fixed framework essential for an effective quality analysis or rather a hindrance for a broad-minded interpretation? • In conclusion, does DQA generate added value or cause disproportionate additional effort? This chapter outlines both opportunities and constraints, performing data quality analyses. Therefore, several efforts are shown how to handle and to characterize different types of data according to their inherent properties. A literature survey (Ch. 3) delineates how DQA is implemented in LCA studies. Further, a case study

 Guidelines from EPA (2006), EC (2010, 2012), UNEP/SETAC (2011), as well as other scientific discussions provide a range of systematic approaches that may facilitate practitioners handling with data issues. 5

3  Data Quality Analysis as Part of Interpretation

27

(Ch. 4) should illustrate the application of a DQA. Conducting such a quality assessment should unveil important drivers and key points and should indicate what kind of additional provisions need to be conducted. Without anticipating the closing discussion (Ch. 5), DQA might encourage a practitioner to conduct further investigations, such as sensitivity analyses, completeness or consistency checks, which could enhance the quality of datasets and increase the robustness of the entire study.

3.2 Methodology In order to clarify the terminological framework applied in this chapter, the compound term “data quality” should be dissected first. Data, which are essential for the performance of an LCA study, are understood as any ‘re-interpretable representation of information6’ that could be used for communication, interpretation, or processing (International Standard Organization (ISO)/International Electrotechnical (IEC) 2382–1:1993). Quality specifies the ‘degree to which a set of inherent characteristics fulfills [certain] requirements’ (ISO 9000:2005). Inherent characteristics could be any information about data, such as uncertainty, reliability completeness, but also age, geographical representativeness or technological level (Weidema and Wesnaes 1996). According to ISO 14044: 2006, data quality is defined as follows: Data quality Characteristics of data that relate to their ability to satisfy stated requirements (ISO 14040:2006). How data could be classified and characterized by means of its inherent attributes is illustrated in the following chapters.

3.2.1 Types of Data An LCA practitioner is cluttered with a vast amount of data and is confronted with various types of data as well as data quality at different phases of the LCA. Within the first phase of an LCA (goal and scope), the determination of a functional unit (FU) and the choice of an appropriate reference flow is essential (Klöpffer and Grahl 2014). This decision should consider the availability of information and prescribed system boundaries with high priority. With reference to the Product Environmental Footprint (PEF) Guideline (EC 2012), data collection (Life Cycle Inventory) is divided into foreground processes and background processes. Foreground processes include all relevant information of the core process (e.g. producer’s site and other processes operated by the

 Information is any knowledge concerning objects, such as facts, events, things, processes or ideas, including concepts that within a certain context has a particular meaning (ISO/IEC 2382–1:1993). 6

28

D. Kölsch and S. Giebeler

producer or its contractor), whereas background processes give access to information on relevant up- and downstream life cycle processes (e.g. infrastructure, buildings or electricity generation) (EC 2012). Usually, foreground data are fed with appropriate information, known as primary data. Primary data are usually collected from the origin itself. This includes gathering activity data from the producers of goods, operators of processes and services, as well as their associations (EC 2010). It is also known as specific data (EC 2012). It could be collected, measured or calculated. However, if no primary data are available, is of insufficient quality, or does not fit the goal and scope, the use of secondary data could be more appropriate. Secondary data sources give access to primary and generic data (EC 2010). According to PEF (EC 2012), secondary data are termed as generic data. It includes information from public sources like statistics (e.g. country-specific databases, third-party LCI databases), from peer-reviewed literature, other studies, interviews or from expert judgments. Although secondary data do not provide one with study-­ specific information, generic data do not automatically signify a lower quality than specific data. Especially for background processes, generic data are a meaningful alternative (Klöpffer and Grahl 2014). All relevant elementary flows are collected and assembled in an inventory. Thus, the inventory includes activity data from the product system (flow of raw material, energy, emissions, etc.) and generic data, like country-specific electricity generation or other consumption data (Weidema and Wesnaes 1996). Each dataset in the inventory is burdened with specific environmental impacts during the third LCIA phase. All in- and outputs are subsequently transformed with a corresponding characterization factor (CF). The characterization factor, ‘derived from a characterization model, […] is applied to convert an assigned life cycle inventory analysis result to the common unit of the category indicator’ (ISO 14044:2006). Both fore- and background data should be evaluated with high attention to the study framework. With reference to Coulon et al. (1997), one could assess data quality either by use of stochastic parameters, such as standard deviation or spread, or via data quality indicators (DQIs). Data quality indicators could be used to sharpen the environmental profile of an LCA study. An overview of common DQIs, which are used to measure the performance of data, is shown in the following chapter.

3.2.2 Data Quality Indicators All relevant data that is collected and summarized in the life cycle inventory could be parameterized by means of various quality requirements. Data quality indicators should indicate the level of representativeness and aggregation of the gathered datasets (UNEP/SETAC 2011) providing objective information about the underlying data (Wang et al. 1993). It is here differentiated between geographical, temporal, technological representativeness, completeness, precision/uncertainty and methodological appropriateness and consistency. Whereas completeness and p­ recision/ uncertainty can be quantified, the other components are more of qualitative nature

3  Data Quality Analysis as Part of Interpretation

29

Table 3.1  Data quality indicators (DQIs) based on ISO 14044:2006and ILCD Handbook (EC 2010) Data quality indicator 1. Representativeness

 (a) Time-related

 (b) Geographical

 (c) Technological

2. Completeness

3. Precision/ uncertainty 4. Methodological appropriateness and consistency

Description Draw a precise picture of the inventory data that represents flow characteristics and environmental impacts of a system as close to reality as possible. Consider divergence of cause (LCI) and effect (LCIA), at least in interpretation of results. Gives information about the age of data and a certain time frame at which the considered data has been collected. This time context should be in line with the intended application. Under consideration of other issues, as, e.g., the technological representativeness, the validity of the datasets should be evaluated. There exists a close link to technological representativeness due to scientific and methodological changes. A process or system is clearly characterized by the origin of its elements. The descriptive information reveals the geographical area (continent, country, region, site, market) where it is operated, produced or consumed. It should be noted that markets often have a different delimitation than typical geographical locations. Technical and technological characteristics of the input data represented in the inventory dataset, with focus on the link between process (activity) and product (results of activity, expressed in FU). Depending on the system boundaries, all relevant elementary flows (inputs, outputs, byproducts, sub-processes, etc.) should be included and quantified in the life cycle inventory. The term uncertainty is used to express the lack of precision, whereas precision incorporates the variability of the data values for each data expressed (e.g. variance). Consistency describes the degree of uniformity of the applied methodology during the life cycle analysis.

(EC 2010). These data quality indicators, expressed in semi-quantitative numbers, may be used to judge the consistency of individual data quality on overall data quality goals (Weidema and Wesnaes 1996). An overview of data quality indicators complying with ISO 14044:2006 and the ILCD Handbook (EC 2010) is provided in Table 3.1. The categorization of data following the proposed quality indicators aims to exhibit data gaps and to reveal inconsistencies (EPA 2006). It helps getting an improved understanding of problems with data quality within LCAs (Weidema and Wesnaes 1996).

3.2.3 Operationalization of Data Quality Analysis In order to gain reliable LCA results, a systematic DQA should be established. By use of various types of instruments, one could evaluate data by means of its inherent attributes. This chapter − making no claim to be exhaustive − outlines different

30

D. Kölsch and S. Giebeler

options on how quality requirements could be operationalized. ISO 14044:2006 suggests performing completeness, sensitivity and consistency checks. These tools could be used to assess the robustness of the model (EC 2012). Within these checks, one will inevitably be confronted with quality issues like data gaps, temporal, geographical or technological inaccuracies or methodological inconsistencies. 1. Completeness Check A completeness check monitors the availability of data. All relevant information of the fore- and background system across all life cycle stages need to be gathered (ISO 14040:2006; ISO 14044:2006; EC 2012). The scope of the study, quality goals, limitations and boundaries need to be kept in mind (EC 2012; EPA 2006). Whenever data are missing, it is inevitable to estimate the potential impact due to the lack of information. At least, it has to be documented why a data gap exists and how it is dealt with (ISO 14044:2006; EPA 2006). Finally, an accurate statement constitutes to what extent the dataset is complete. The degree of completeness determines whether additional efforts are essential or even the study-specific set-up need to be revised (EPA 2006; EC 2010). A checklist could help to capture all relevant processes as well as to identify data gaps. A possible structure of such a completeness check is proposed in ISO 14044:2006 (Table B.9). 2. Consistency Check A consistency check strives to determine whether the application of methodologies is consistent throughout the whole life cycle approach or not (ISO 14044:2006; EC 2010, 2012). Therefore, a consistency check investigates if the data gathered and assumptions made are in line with the goal and scope of study (ISO 14044:2006). One could distinguish between the consistency along the life cycle itself and between different product systems (ISO 14044:2006; EC 2010). Performing such a check could reveal discrepancies in regional and temporal issues, but also check the robustness of allocation rules and system boundaries. All phases − goal and scope, life cycle inventory, LCIA and interpretation − are affected (ISO 14044:2006; EC 2010). If inconsistencies are detected, their contribution to the overall result has to be evaluated. A consistency check could end up in a comparative summary of data (ISO 14044:2006, Table B.13). 3. Sensitivity Check The sensitivity analysis should be understood as an iterative process along the whole LCA that strives to enhance the reliability of final LCA results (ISO 14044:2006). It is useful to perform sensitivity checks at each life cycle phase. Additional effort for sensitivity analyses comply with the size and complexity of the inventory (ISO 14044:2006), but also with the systemic conditions presupposed. With reference to ISO 14044:2006, one can distinguish between sensitivity checks on allocation rules, data uncertainty or characterization data. EPA (2006) distinguishes between three techniques – contribution, uncertainty and sensitivity analysis  – for performing a sensitivity check. Depending on the type of study, theappropriate method should be chosen (EC 2010). How results of such sensitivity checks could look like is shown in ISO 14044:2006 (Table B.10–12). A sensitivity

3  Data Quality Analysis as Part of Interpretation

31

check which reveals no significant differences does not necessarily imply that the considered study is free from uncertainties (ISO 14044:2006). However, it helps to determine whether uncertainty ‘affect[s] the decision-maker’s ability to confidently draw comparative conclusion’ (2006). To increase practitioner’s efficiency in performing LCAs, one could implement these checks into a systemic approach. For instance, Henriksson et al. (2013) developed a scheme where data are characterized by specific properties. Starting with an input flow, one has to distinguish between primary and secondary data. Following the string of the proposed decision tree, the number of data points, inherent uncertainty and spread, unrepresentativeness and distribution need to be considered. This is one possibility which could ease the procedure of gathering and characterizing data. Once such a structured scheme is established, it could work as a supportive instrument, which tries to portray input data by means of its inherent quality. 4. Quality Scoring Systems According to the data quality indicators, shown in Sect. 3.2.2, scoring systems are used for translating quality issues into numerical values. Data quality is therefore evaluated by means of data quality parameter values on a qualitative or subjective dimension (Wang et  al. 1995). There is no broad consensus on a universal scoring system. The range of such scales and the interpretation of scoring values differ from practitioner to practitioner: Weidema and Wesnaes (1996) proposed a scale from 1 (best quality) to 5 (lowest quality), whereas van der Sluijs et al. (2003) performed systems with scores from 0 to 4, respectively, from 0 to 2. LCA Digital Commons presents a two-tiered scale at flow level using the indicators A and B (Cooper and Kahn 2012). Most of the publications (e.g. EC 2010; van den Berg et al. 1999; Weidema 1998) adopted to scoring systems with a scale from 1 to 5, where 1 is the best and 5 is the lowest score. A score of 0 indicates that the data are not applicable. According to the ILCD Handbook (EC 2010), flow data are characterized in terms of six basic quality indicators: technological, geographical and temporal representativeness; completeness; precision/uncertainty; and methodological appropriateness and consistency. Detailed quality requirements for the respective score are listed in the ILCD Handbook (EC 2010, p.330/331). 5. Pedigree Matrix The pedigree matrix is a notation that could be used to evaluate LCA data by means of its inherent quality characteristics. It is a semi-quantitative approach that links descriptive values with a numerical ranking. The pedigree matrix was introduced to uncertainty analyses by Funtowicz and Ravetz (1990), while Weidema and Wesnaes (1996) transferred the pedigree matrix into LCA.  Their main purpose was that the matrix should be ‘applicable for all types of processes in all sectors of society’ (Weidema and Wesnaes 1996). Also known as ‘Numeral Unit Spread Assessment Pedigree’ (NUSAP), the matrix includes a set of pedigree criteria to code qualitative expert judgments into discrete numeral values (van der Sluijs et al. 2005). Each cell of the matrix includes a linguistic description for each value. Generally, there are no further formal requirements on the

32

D. Kölsch and S. Giebeler

structure of the matrix (Ciroth et al. 2013). Matching numerical values with quality indicators makes the pedigree matrix a semi-quantitative instrument. Most matrices, applied in LCA studies, refer to the pedigree criteria defined by Weidema and Wesnaes (1996). Depending on goal and scope, the quality matrix is modified: indicator categories are adjusted, new scoring systems implemented (Rousseaux et al. 2001; Lewandowska et al. 2004; Schuurman 2004), or even economic and social dimensions are integrated (Junior et  al. 2014). Moreover, matrices get extended with additional stochastic techniques, such as standard deviations, distribution functions or uncertainty factors (Frischknecht et al. 2007). The reader is referred to Ciroth et al. (2013), who developed empirically based uncertainty factors for flow data in EcoInvent, starting from a qualitative assessment by a pedigree matrix (Table 3.2). The publication from van den Berg et al. (1999) provides a survey on quality assessment giving an overview of quality goals, quality indicators and quality scores in LCA literature. As shown in Table 3.2, Weidema and Wesnaes (1996) introduced indicators on data level only. They declare that these indicators could not be aggregated to system level because the scores do not represent an amount of quality. However, other scientists proposed methods for aggregation. These cover, for example, the aggregation of quality scores to system level using equal weights for each dataset (Wrisberg et al 1997) or the use of target quality goals (Rousseaux et al. 2001). These two methods are discussed within the case study of May and Brennan (2003) in this chapter. Wrisberg et al (1997) and Lindeijer et  al (1997) introduced a set of indicators on flow, process and system level. Within their framework, data quality parameters could be aggregated stepwise to system level, resulting in indicators in accordance to reliability, completeness and representativeness. Either way, it is important to prove case wise which quality indicators are applied, if they are adopted on data, process or system level and whether they are aggregated or not. Within the case study shown in this chapter, the data quality requirements from Weidema and Wesnaes (1996) were applied. They recommend not to sum up quality indicators; nonetheless, the approach from the ILCD Handbook (EC 2010) is checked on its applicability here.7 It consolidates data quality indicators and brings them together to an overall quality rating for each dataset. Therefore, the DQIs in Table  3.3 are slightly modified. In contrast to the PEF guideline (EC 2012) the ILCD-formula includes a weighting parameter Xw which attaches importance to low quality scores (EC 2010). The PEF guideline (EC 2012) excludes a weighting factor as it is easier to achieve a ‘good’ quality rating without such a parameter. In order to maintain a critical distance, the approach from ILCD is applied here.

 Please see also the publication of Maurice et al. (2000). The authors use an aggregated indicator as an intermediate indicator for the identification of elementary flows of unit processes. 7

3  Data Quality Analysis as Part of Interpretation

33

Table 3.2  Data quality pedigree matrix (Weidema and Wesnaes 1996) Indicator score Reliability

1 Verified data based on measurements

2 Verified data partly based on assumptions or non-verified data based on measurements Completeness Representative Representative data from a data from a sufficient smaller sample of sites number of over an sites but for adequate adequate period to even periods out normal fluctuations

3 Non-verified data partly based on assumption

4 Qualified estimate (e.g. by industrial expert)

5 Non-qualified estimate

Representative data from adequate number of sites but from shorter periods

Representative unknown or incomplete data from a smaller number of sites and/or from shorter periods

Temporal correlation

Less than 3 years difference to year of study Geographical Data from correlation area under study

Less than 6 years difference to year of study Average data from area in which the area under study is included

Less than 10 years difference to year of study Data from area with similar production conditions

Technological Data from correlation enterprises, processes, and materials under study

Data for processes and materials under study but from different enterprises

Data for processes and materials under study but from different technology

Representative data but from a smaller number of sites and shorter periods or incomplete data from an adequate number of sites and periods Less than 15 years difference to year of study Data from area with slightly similar production conditions Data on related processes or materials but same technology

Age of data unknown or more than 15 years Data from unknown area or area with very different production conditions Data on related processes or materials but different technology

3.2.4 Inclusion of DQA Results in Interpretation According to ISO 14044:2006, “a study [that] is used to support a comparative assertion” should address data quality requirements. Irrespective of this formal requirement, data analyses are beneficial for the identification of quality issues within LCA. Once a DQA is established, it could help a practitioner reveal data gaps to evaluate interrelations and to judge data issues in accordance with study-specific quality goals. The selection of relevant data, whether of good or bad quality, should be made with regard to the goal and scope of the study and should take the indented

34

D. Kölsch and S. Giebeler

Table 3.3  Data quality rating adapted from ILCD Handbook (EC 2010) DQR 

DQR TeR GR TiR C R Xw i

TeR  GR  TiR  C  R  X  4 w i4

Overall data quality Overall data quality level rating (DQR) ≤1.6 ‘High quality’ >1.6 to ≤3 ‘Basic quality’ >3 to ≤4 ‘Data estimate’ Data quality rating of the LCI dataset Technological representativeness Geographical representativeness Time representativeness Completeness Reliability Weakest quality level obtained (i.e. highest numeric value) among the data quality indicators Number of applicable (i.e. not equal “0”) data quality indicators

study applications into account. With reference to data requirements from the European Commission (EC 2015)8, one has to distinguish between relevant and non-relevant processes. All processes contributing more than 80% to an impact category are classified as most relevant processes. A ‘Data Needs Matrix’ indicates options for assessing the relevance of processes considering the level of influence a company has on them. For instance, a relevant process run by the company has to achieve a DQR less than 1.6. Such requirements are certainly beneficial in order to identify the main drivers of the environmental profile of a product. However, due to the uniqueness of a study, it seems to be inappropriate to prescribe a default procedure on how to apply DQA for interpretation. Data quality requirements, in terms of a fixed set of DQIs or quality thresholds for the emission profile, could weaken users’ critical appreciation. Nevertheless, integrative data analyses are indispensable, because ‘if an LCIA stops at the characterization stage, the LCIA interpretation is less clear-cut’ (EPA 2006). As well as the LCA itself, DQA is an iterative process, which covers issues in every LCA phase. Therefore, DQA findings should be considered with high attention. This does not mean that quality scores should be seen as absolute numbers delivering the truth. These values rather serve as indicators highlighting hot spots for further quality improvements. The interpretation phase leaves space for such evaluations and reinforces feedback loops. One has to bear in mind that data quality analysis, especially semi-quantitative scoring via pedigree, ineluctably involves subjective value judgments (Weidema 1998). For this purpose, to decrease practitioners’ influence, numerical methods were introduced by several scientists and increasingly applied in LCA. However, data analysis, as a supportive instrument for the interpretation of the LCA procedure  A first draft of ‘Data requirements in Product Environmental Footprint Category Rules (PEFCRs)’ prepared by the European Commission provides instructions concerning the choice of datasets to be used when calculating an Environmental Footprint. 8

3  Data Quality Analysis as Part of Interpretation

35

and its results, is often not sufficiently recognized, although it could reveal critical instances of lower data quality and avoid misinterpretations (Cooper and Kahn 2012; Klöpffer and Grahl 2014). Within interpretation the practitioner has to decide whether data inconsistencies or inaccuracies are of substantial importance for the study results. Thus, an improved data management aids not only the comprehension of uncertainties and limitations but also benefits future LCA studies. In addition, DQA findings could be used in the final report and critical review for further recommendations.

3.3 Data Quality Analysis in Practice The following chapter gives a rough overview on DQA in practice. Therefore, best practice examples are analyzed. It is focused on the type of data used, quality indicators proposed and scoring systems applied in the respective study. Moreover, it is shown what kind of limitations the study authors had to deal with, how quality challenges are met and which recommendations are made. Study: Life Cycle Inventory Data Quality Issues for Bioplastics Feedstocks Grabowski A, Selke SEM, Auras R, Patel MK, Narayan R (2015) This study reviews currently available life cycle inventory data for biopolymer feed stocks and assesses the data quality for selected feed stocks of corn, sugarcane and soy. Grabowski et al. (2015) recognize that the ‘[a]vailability of appropriate, high-quality data is a problem in LCA (LCA) of biopolymers and other bio-based materials that limits the accuracy and usefulness of study results’. In order to collect all relevant data, diverse data providers (ecoinvent, GaBi, US LCI, LCA Food DK, LCA Commons) were used. About 287 datasets were gathered for a total of 22 different feed stocks. ‘The majority of these datasets are from Europe and the USA, with most of Asia, the Middle East, and Africa having very limited data available’. Subsequently, to a completeness check, a pedigree matrix scoring system was established. Grabowski et al. (2015) refer to quality indicators and data quality rankings from ILCD (EC 2010) and rating level definitions proposed by van den Berg et al. (1999). They note that ‘reliability’ from van den Berg et al. (1999) and ‘precision/uncertainty’ from ILCD (EC 2010) are used equivalent. Based on the matrix scores, an overall quality score for each dataset was calculated. According to the ILCD Handbook (EC 2010), a dataset with DQR less than or equal to 1.6 is considered high quality, while a dataset with a DQR between 1.6 and 3 is considered to be of basic quality. Any dataset with a DQR between 3 and 4 is considered to be an estimate. A score of 5 means that the criterion was not evaluated or is unknown. Grabowski et al. (2015) introduced half scores when no clear distinction could be made. The overall results of the data quality evaluation for the considered feed stocks of corn, sugarcane and soy are provided in Table 3.4.

36 Table 3.4  Overall data quality rating for the datasets of corn, sugarcane and soy

D. Kölsch and S. Giebeler Feedstock Corn Sugarcane Soy

Data quality rating 1.4–2.6 1.8–3.7 1.6–4.5

Within the evaluation categories9, technological correlation scored best for all three kinds of datasets. Therefore Grabowski et al. (2015) conclude that there is ‘no urgent need to investigate data gaps due to different technologies’. This means ‘that data from different process than the one under study rarely had to be used as a substitute to fill a data gap’. They noticed that geographical variation could be complex and difficult to this model. A large variation within single regions using uniform technology makes it hard to predict correlation between regions. Hence data within a region could not be substituted easily. The corn datasets scored fairly well, whereas soy and sugarcane showed significant variation, for exampl,e due to sensitivity of growing conditions. Grabowski et al. (2015) pointed out that ‘[g]eographical and technological differences are often strongly related, with quality issues bridging both categories’. Considering temporal representativeness, the collection updated data are of major importance, because LCA studies coping with crop issues are inevitably sensitive to changes (e.g. precipitation, climatic conditions) (Grabowski et al. 2015). The quality indicator ‘Completeness’ was dominated by methodological inconsistencies. Grabowski et al.’s (2015) main concern relates to gaps due to missing information on land use changes. Some datasets contained land use data, but it appeared to be impossible to distinguish between direct and indirect land use changes. In this study, the issue of land use changes ‘is the largest problem in the category of completeness’; nevertheless, the significance of this methodological challenge varies depending on the system under study (Grabowski et  al. 2015). Uncertainty could only be evaluated for ecoinvent data, since no uncertainty information for other input data were available. Therefore, no overall uncertainty scores were assessed. The study authors deduce that more datasets with regional variations in crop cultivation, as well as more data on land use change, are needed. Therefore, more regionally explicit data that accurately models the technology and conditions are essential. ‘Newer data for cultivation of feed stocks […] would also be beneficial’ (Grabowski et al. 2015). In order to evaluate and accentuate environmental impacts of bioplastic production data gaps need to be resolved. Study: Application of Data Quality Assessment Methods to an LCA of Electricity Generation May JR and Brennan DJ (2003)

 Reliability, Completeness, Temporal, Geographical and Technological correlation (adapted from van den Berg et al. 1999). 9

3  Data Quality Analysis as Part of Interpretation

37

The study reviews several assessment methods for estimating uncertainties, distinguishing between numerical and qualitative uncertainty. Quantitative uncertainty analyses performed by Monte-Carlo technique are not discussed here. A case study on Australia’s electricity generation serves as an example for the data analysis.About 90 of Australia’s electricity is produced by use of fossil fuels. But there are significant differences between the generation of black and brown coal, for example in mining systems, electrical generation efficiency and transport distances. Therefore, environmental impacts of black and brown coal were assessed. Most of the relevant data were obtained both from confidential and public sources and represent parts of Australia’s black coal and brown coal electricity generation industry. The impact categories, climate change, acidification, and resource depletion, are considered and evaluated by use of CML impact factors. A qualitative uncertainty analysis was obtained using a single set of indicators proposed by Weidema and Wesnaes (1996). Data were evaluated applying parameters with scores from 1 to 5. Two methods (Wrisberg 1997; Rousseaux et al. 2001) are applied to translate data level indicators into aggregated data quality indicators. Applying the Wrisberg (1997) method, data level indicators were aggregated stepwise to system level using equal weights for all environmental flows. In doing so, qualitative uncertainty scores for each processing step (mining, transport, generation and transmission) were calculated (May and Brennan 2003). Due to unverified data and lack of data for some facilities, the aggregated quality scores for reliability (2.86) and completeness (3.16) were highest. Temporal, geographical and technological representativeness differed only slightly, ranging from 1.47 to 1.40. May and Brennan (2003) address the dualism of DQA and LCIA by comparing quality scores with the relative contribution to the total indicator. Unfortunately, it is not shown precisely what forms the basis of the relative contribution. The study authors observed that the transmission phase contributes lowest to all quality categories. Therefore a decrease in quality would not influence the overall quality significantly. The generation stage dominated the impact results, but had a similar number of streams contributing to the scores as mining and transport. May and Brennan (2003) conclude that ‘[t]his method fails to establish whether quality is good in data that counts towards the impacts’. Thus, the study authors applied to another method proposed by Rousseaux et al. (2001), where each data point is compared to a defined target quality goal (e.g. 2 out of 5). It showed ‘more promise in locating and identifying the cause of deficiencies in data quality’. Nevertheless, it was insufficient in providing details concerning quality of impact results either (May and Brennan 2003). The study authors note that the establishment of target quality scores did not influence the results of aggregation significantly. They argue that both methods estimating qualitative uncertainties (Wrisberg 1997; Rousseaux et al. 2001) fall far short of providing deeper insights into the data’s contribution to environmental impacts. In addition, May and Brennan (2003) combined the qualitative uncertainty analysis with numerical uncertainty measures referring to Kennedy et  al. (1997) and Meier (1997). It aims to convert qualitative uncertainty into a quantitative uncertainty profile. The combined uncertainty analysis could not reveal any additional

38

D. Kölsch and S. Giebeler

value neither for their findings of the quality analyses nor for their LCIA results. Hence, May and Brennan (2003) infer that no further judgment could be delivered. They conclude that the numerical and qualitative uncertainty analysis should be treated separately. According to them, LCA results should include a ‘quantitative assessment, using a skewed probability distribution […], supported by the qualitative assessment of Wrisberg’. Literature Survey on Data Quality Analysis Methods in Practice Table 3.5 outlines the results of a literature survey which was carried out by the authors of this chapter. All studies selected for this survey are published in scientific journals and are publicly available. A detailed presentation of the content of the studies is omitted. The survey rather serves as an overview of best practice examples dealing with data quality. It is a screening that should indicate which quality issues are addressed in the studies shown below. Particular attention was paid to the choice of quality indicators and to the application of scoring systems in the respective study. Studies reviewed do not necessarily follow the terminological understanding, as it is described in Chap. 2. The given footnotes provide remarks on the use and understanding of quality indicators. The screening of the LCA studies disclosed that ‘representativeness’ seems to be implemented consistently by the majority. Only minor adjustments are made, for example, to time scale (Rousseaux et al. 2001). A completeness check also appears to be applicable without major effort. In most cases, the pedigree matrix of Weidema and Wesnaes (1996) forms the basis of the data quality scheme. Sometimes the scoring system is slightly modified, but it is mostly kept in its original design. It is remarkable that DQIs are frequently modified or newly added. Thus, using different terminology and categories leads to limited comparability. Besides this, DQA almost entirely focuses on input data (primary and secondary). Other data, such as data from characterization models, are often treated with less attention. However, one can conclude that − irrespective of methodological choices − quality analysis could provide a better understanding of underlying data. Such supportive instruments, whether data scores are aggregated or not, could indicate the data quality level along the entire LCA. Thus, data quality analyses embedded in interpretation phase could increase transparency and confidence.

3.4 Case Study: Analysis of Data Quality in the Environmental Assessment of the Extraction and Processing of a Raw Material for Pharmaceutical Products This chapter outlines how to conduct a DQA and to interpret its results in accordance to the findings of a life cycle impact assessment. A case study is provided which deals with the assessment of the environmental impacts of a mining process

✓ ✓ ✓ ✓ ✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓

✓e

✓

✓

✓f

✓f

✓

✓d

✓

DQIa Representativeness Precision/ Time Geo Tech Completeness uncertainty × ✓ ✓ ✓ ✓

b

a

Data quality indicators adapted from EC (2010) Aggregated data quality indicator (ADQI) c An ‘×’ indicates that the DQI is not explicitly applied in the respective study d Additionally ‘Justness of data’ and ‘Repeatability of system’ are measured e Wang and Shen (2013) refer to availability of data f The study authors refer to ‚Reliability’ proposed by Weidema and Wesnaes (1996) g Junior et al. (2014) refer to a study developed by Zhang and Haapala (2012)

Study topic Bioplastic feedstocks (Grabowski et al. 2015) Polyethylene bottles (Rousseaux et al. 2001) Electricity generation (May and Brennan 2003) Whole-building embodied energy analysis (Wang and Shen 2013) Industrial pumps (Lewandowska and Foltynowicz 2004) Machining work cell (Junior et al. 2014)g

Table 3.5  Data quality analysis in practice

×

×

✓

×

✓d

Methodological appropriateness and consistency ×c

✓

✓

✓

✓

✓

1–5

1–0.2

5–1

1–5

1–5

From best to ADQIb lowest 1–5 ✓

Scoring system

3  Data Quality Analysis as Part of Interpretation 39

40

D. Kölsch and S. Giebeler

and the following processing steps. The execution of a qualitative data analysis should therefore check to what extent data quality influences the LCIA results and their interpretation.

3.4.1 Presentation of Study The objective of the study was to evaluate potential environmental impacts during the extraction and processing of a raw material for pharmaceutical products. Due to the fact that the study is part of a research project that includes confidential information, some details are omitted deliberately. However, this has no effect on the evaluation of the data quality analysis. The aim of this study was to compare two production sites (in the following named as Site A and Site B), located within different regions and using different technologies. For both production sites, a detailed DQA was conducted. The results are shown in order to provide a basic understanding of the procedure of DQA. The present study is a “cradle-to-gate” approach that considers the following processing steps: • Step 1: Extraction (Site A & B). • Step 2: Pre-processing (only at Site A). • Step 3: Processing (Site A & B). A screening LCA10 was conducted to better understand the impact of these processing steps. Five impact categories were assessed within the present study: climate change, resource depletion (fossil and mineral), acidification, eutrophication and water consumption. The data analysis applied here focuses on the interdependence of DQA and LCIA results according to climate change and resource depletion. The functional unit was determined as a certain amount of processed raw material. In the following, 1 kg of processed raw material is defined as reference. No commercial by-products are manufactured. Thus, the environmental load was fully allocated to this one product.

3.4.2 Including Data Quality Analysis into Procedure of LCA A DQA was performed to disclose data gaps and to identify relevant quality issues. After determination of goal and scope, all relevant in- and outputs of the two alternatives (A & B) were compiled and quantified. For both sites, fore- and background data were collected in accordance with site-specific requirements, such as  A screening LCA is only a preliminary and quick LCA study, which is for internal use only. It includes limited system boundaries, limited impact assessment categories, normally low data quality and no critical review requirements. The aim of such a screening LCA is to quickly identify the most important processes that contribute most significantly to a product’s life cycle. 10

3  Data Quality Analysis as Part of Interpretation

41

technological or geophysical conditions. Activity data were collected by questionnaires, providing site-specific information, for example, electricity demand or material consumption. In the case of missing information or insufficient documentation, databases or other public sources (e.g. energy statistics databases, product information sheets or other relevant studies) were used to fill data gaps with adequate assumptions. Characterization factors were primarily taken from CML11 included in LCA software. It should be noted that this study is a screening LCA that follows the basic rules of ISO 14044:2006 and ILCD (EC 2010). The DQA was conducted using the pedigree matrix proposed by Weidema and Wesnaes (1996). In order to simplify the quality judgment, the rating level definitions shown in Table 3.2 were applied. Both activity data from foreground systems (e.g. power consumption) and background data (e.g. dataset for electricity generation) were ranked by means of a scoring system from 1 to 5, which indicates the quality of the respective dataset − the lower the score, the better the quality. A score of 0 indicates data that could not be assessed, for example due to missing information on temporal or geographical acquisition. If so, it is documented why an evaluation was inapplicable. For production site A and B, numerous datasets were evaluated. Table 3.6 outlines an excerpt of results from the quality judgment distinguishing between foreground and background data. The shown scores represent the final DQA results after additional quality checks. The provision of a sensitivity analysis is described below. All foreground datasets that are ranked best with a score of 1 were provided directly from the manufacturer. It is assumed that these information are up-to-date and do represent the technological specifications and geographical conditions at site appropriately. The occurrence of the raw material at production site A was mainly dependent on the geophysical realities. Therefore, it was difficult to gain reliable information on the geographical representativeness of the raw material. This led to an assumption in terms of a quality score of 3 in geographical representativeness. Table 3.6  Data quality indicator values and overall data quality ratings for selected datasets Production site A Electricity Steam Raw material B Electricity Steam Raw material

Foreground data TeR GR TiR 1 1 1 1 1 1 1 3 2 1 3 2 1 1 1 1 1 1

C 1 1 2 2 1 1

R 1 1 2 2 1 1

DQRF 1.00 1.00 2.44 2.44 1.00 1.00

Background data TeR GR TiR 1 2 2 1 4 4 0 4 0 1 1 2 1 3 2 0 4 0

C 2 2 4 2 2 4

R 3 2 3 2 2 3

DQRB 2.44 3.22 3.86 1.78 2.44 3.86

For a detailed description of the data quality indicators and scores, as well as of the applied aggregation method please, see Sects. 3.2.2 and 3.2.3 and the ILCD Handbook (EC 2010) DQRF Data quality rating – Foreground data, DQRB Data quality rating – Background data  The CML 2001 method includes characterization factors that are updated when new knowledge on substance level is available. It was developed by the Institute of Environmental Science at Leiden University. 11

42

D. Kölsch and S. Giebeler

In contrast, the occurrence of raw material at production site B could be determined more precisely by use of measurement, which in turn resulted in a score of 1. The electricity consumption at production site B was approximated based on its production capacity. That is why, the uncertainties of the foreground data for electricity are higher compared to production site A.  One has to bear in mind that this quality assessment is based on subjective value judgments. Using the quality requirements from the pedigree matrix, the representativeness indicators were mostly easy to apply. For the indicators’ completeness and reliability, it was more challenging to rank foreground as well as background data. Background data was mainly taken from databases (GaBi database by think step). The appropriateness of the considered datasets was judged with regard to on-­ site conditions. Some quality indicators such as geographical representativeness suffer due to missing or insufficient background information. The background data for the raw material was evaluated worst. Both technological and temporal representativeness were categorized as not applicable; the other quality scores do represent assumptions, mostly based on estimations. Missing or insufficient background information is not only an issue in the present study but also a common challenge LCA practitioner has to cope with. Subsequent to the value judgment described before, an overall data quality rating of each dataset was calculated referring to the ILCD Handbook (EC 2010). Here, it is assumed that the quality rating definitions from ILCD may lead to similar scores or at least to comparable results as those from Weidema and Wesnaes (1996). However, in a full LCA, one should strive for methodological consistency in order to prevent any disharmony. In accordance with the ILCD equation (Table  3.3), aggregated scores DQRF and DQRB for each dataset were calculated. Zero values were accounted as not applicable and were excluded in the calculation. These aggregated DQRs could be beneficial to compare quality judgments between datasets. Nonetheless, a single DQR value could not reveal information on the significance for the study. Hence, in order to get a holistic picture, the results of the DQA should be interpreted with regard to the findings of the LCIA. The Handbook of ILCD (EC 2010) misses the opportunity to address possibilities how to treat this issue. It is also not specified how to handle the dualism of foreground and background data. As issues concerning foreground data were already discussed, the following discussion focuses on background data assuming that it does more likely represent the appropriateness of the dataset. In this case, the DQRB scores furthermore indicate a worst-­ case scenario, due to the fact that they are almost entirely higher than the DQRF scores. For purposes of completeness, Table 3.7 provides both overall quality scores from foreground and background data vis-à-vis the corresponding LCIA results. The latter are expressed by the percentage contribution to each impact category. The single DQRB-values were not added up to an overall DQR as it diminishes the relative share of the respective flow. Either one has to discuss the results on a detailed level or implement a weighting scheme that addresses the correlation between data quality and environmental impact. It should be noted, as Table 3.7 does not claim to

3  Data Quality Analysis as Part of Interpretation

43

Table 3.7  Overall data quality ratings calculated using the ILCD equation compared to the environmental impact Production site Dataset A Electricitya Steam Raw material B Electricity Steam Raw material

DQRF 1.00 1.00 2.44

DQRB 2.44 3.22 3.86

Contribution to GWP (%) 74 12 0

Contribution to ADP fossil (%) 65 18 0

Contribution to ADP elements (%) 0 0 96

2.44 1.00 1.00

1.78 2.44 3.86

72 26 0

59 39 0

0 0 99

The electricity datasets for both processing steps at production site A were aggregated together due to the fact that they are based on information from the same source and thus evaluated equally with the same quality scores a

be exhaustive, the respective contribution does not add up to 100%. Nonetheless, the main drivers within one impact category are identified and shown below. As well as described above, Table  3.7 outlines the final DQA results after an iterative process of quality improvements. After a review of the first DQA findings, it could then be determined whether to intensify the data research or to desist from further quality checks. This interpretation work is a recurring challenge a practitioner has to tackle within his analytical procedure. As shown in Table 3.7, both electricity datasets, for site A and B, contribute more than half to the global warming potential (GWP) and the resource depletion (ADP) of fossil fuels. Compared to the other background data, the overall quality of the electricity datasets were valued best. However, due to the high contribution to the final result (GWP: 74% and 72%; ADP fossil: 65% and 59%), some additional sensitivity checks were performed in order to check data quality. For both production sites, background information on power generation from LCA databases was compared with information from public sources. The result of this quality check was that the background data for B remained untouched, whereas the background system for electricity at site A was specified: An adoption of the energy mix to the country-specific conditions revealed a better scoring in all three representativeness categories. This led to the final overall quality score of 2.44 for the background electricity dataset at production site A. As the “raw material-flow” does not affect GWP and ADP fossil at all, it is even more important to the abiotic resource depletion potential of elements. For this purpose, a sensitivity analysis was carried out to check the influence of the amount of raw material used during the production. Even halving the input amount did not lead to a substantial reduction in resource depletion. Thus, the DQRF could not be improved. The underlying background data of the raw material resulted in a DQRB score of 3.86.

44

D. Kölsch and S. Giebeler

3.4.3 Discussion The findings of the quality analysis emphasize data weaknesses, which are brought together in a well-structured scheme. This alone makes it easier to discuss and reconsider decisions taken within the study scope; not only for the practitioner themself but also for other interested parties who may interpret study results. Moreover, one could detect quality gaps very precisely, so that additional efforts to gather data could be performed purpose- and goal-oriented. Aggregated quality scores could therefore give a first rough idea where potential data issues may occur. Beyond that, a quality analysis could indicate the need for additional checks, such as completeness or consistency checks. These additional provisions could not only raise quality scores but rather strengthen practitioners’ faith in its interpretation process. The sensitivity analysis, performed to reassess energy consumption data, reveals the conflict between subjective expert judgment and potential quality improvements. Nevertheless, it is striking that the demarcation of quality scores within one category is not as evident as it may be presumed. The quality requirements of the three representativeness indicators are specified quite precisely, which facilitates the DQA procedure. In contrast, for the indicators’ reliability and completeness, the borders are sometimes blurred, so that it is not clear which score one has to apply. Further, a statement on the overall quality of a study or a single process step is hard to predict, inasmuch as the DQA excludes the linkage to the impact assessment. The results shown above reveal that a dataset with high contribution to one impact category could simultaneously be meaningless to another. Even within an impact category, a ‘bad’ quality score does not necessarily imply that the results are useless. Once a flow or process has only little contribution to the overall environmental impact, its quality might be less critical. Though, a dataset could be a suitable option for one study, while being of insufficient quality for the use of another study.

3.5 Conclusions 3.5.1 The Practitioner as Expert and Decision Maker Due to the fact that LCA is a multidisciplinary approach being used not only in product development and improvement and strategic planning but also in public policy planning and marketing (Klöpffer and Grahl 2014), it is influenced by manifold interests. That is why quality issues should not only be discussed internally between LCA practitioners but should also incorporate, for instance, stakeholders’ values and knowledge. Usually, decision makers rank as the best performing discipline (Lloyd and Ries 2007). It is of high importance to encourage critical debates involving all relevant participants. In this course, It is the practitioner who should take responsibility as an LCA expert by disclosing limitations and risks, but also

3  Data Quality Analysis as Part of Interpretation

45

giving recommendations. Finally, it should be the practitioner to deduce as to whether data weaknesses contribute significantly to the final statement.

3.5.2 Data Quality Analysis as an Immanent Part of Interpretation DQA as an Inherent Part of Interpretation Crosses All LCA Phases  It is apparent that the mere evaluation of data quality usually focuses on life cycle inventory. However, a DQA− defining quality goals, establishing quality requirements, and calculating quality scores − needs the findings to be reflected and critically scrutinized. This is why the interpretation phase is of particular importance as the final conclusion is drawn on the interaction of decisions and actions across all LCA phases. The interpretation of quality issues coupled with insights into datas contribution to environmental impacts could foster critical debates; not only on quality aspects but also on the entire study scope. Primary and Secondary Data Sources  Performing an LCA study means that a practitioner has to gather information from primary or from secondary data sources. Usually, a practitioner is aware of the strengths and qualitative lacks of primary data. It is often gathered as site-specific data via personal communication. Thus, it could be discussed and refined easily. The ILCD Handbook (EC 2010) recommends, even as a general rule, to use specific data for foreground processes concluding that ‘generic or average secondary background data may [only] be used to identify the need for more representative or specific data’. Usually, priority is therefore given to primary data, whereas secondary data should be accessed if no primary data is available (Henriksson et al. 2014). If specific data are gathered responsibly and checked for applicability in an adequate manner, it is surely a good option, even though primary data could incorporate high uncertainties. In contrast to primary data, tracing and scrutinizing secondary data are often more complicated and could be time-consuming. Nevertheless, generic data could be an appropriate substitute for primary data of low quality. Klöpffer and Grahl (2014) even note that ‘generic data are indispensable for a conduct of complete inventories and LCAs’. Particularly if generic data are estimated with a higher quality level than primary data, generic data should be preferred and could also be used for the foreground system (EC 2010). Hence, it is essential that the data supplier provides sufficient background information on the quality of datasets. A comprehensive and transparent supply of data quality information is insofar eligible, as the LCA practitioner gets an extensive knowledge about the circumstances of the respective product system. However, it is the practitioner who should ensure that the chosen dataset represents the considered system in an appropriate manner. Beyond that, it has to be noted that a good appropriateness of a dataset in a certain study does not necessarily imply a fair applicability in another study. It seems to be even

46

D. Kölsch and S. Giebeler

more challenging to deal with background data as ‘predefined’ parameters. Somehow, a practitioner needs to rely on established LCA methodologies and practices. Nevertheless it could be thorny taking background data or data from characterization models for granted. If discrepancies arise from such data, it is essential that there is a feedback loop between the practitioner and scientists. Conducting DQA could not only be used to identify main drivers for fuzziness and to detect low quality spots, but also to get a deeper understanding of the model itself. In combination with stochastic modeling, which is an effective method for assessing the probability of LCA results (Lloyd and Ries 2007), the interpretation of data issues could be eased. In doing so, concerns of LCA actors ‘whether high levels of uncertainty and the resulting lack of significance between outcomes may deem LCA outcomes meaningless’ (Huijbregts et  al. 2004) may be diminished. Indeed, high uncertainties may influence the outcome of a study; but DQA should rather serve as an instrument that improves decision-making ‘by identifying the likelihood that an alternative will have a lower environmental impact than others or the likelihood of exceeding inventory or impact thresholds’ (Lloyd and Ries 2007). DQA should moreover reveal recommendations, such as alignment of data collection strategies, scenario analyses, or sensitivity checks. This could highlight issues where improved understanding is needed and subsequently provide the basis for further evaluations, whether numerical or not. Data Quality Matrix  However, as described in Sect. 3.2.3, there is no coherent framework, for example, on how to design a data quality matrix. The matrix’ intentions is ‘to be generally applicable for all types of processes in all sectors of society’ (Weidema 1998). On the one hand, it is useful for quantifying uncertainties at unit process level, on the other hand, inherent uncertainties and spread are often neglected (Henriksson et al. 2013). As mentioned before, an LCA practitioner needs to consider study specific conditions. This could be one reason, why authors modified and reinterpreted pedigree matrices for their purpose. It seems to be a thin line between general rules on how to treat data and restrictions that jeopardize practitioners’ flexibility. A sharpened terminological framework (Lloyd and Ries 2007) and a standardized DQA procedure could be of considerable benefit for diminishing inconsistencies in data use and data assessment. Performing DQA under stringent conditions does certainly strengthen the procedure of life cycle approaches. It could increase the reliability, consistency and interpretability of LCA results and improve the robustness of the whole study model. At least quality lacks and weaknesses could be detected. Beyond that, the quality of an LCA study is determined by subjective value judgments from the practitioner, especially by her or his experience and critical mind. The degree of meticulousness and accuracy are therefore crucial, not only for the study outcome but also for the DQA results. ‘In some instances, it may not be clear which product or process is better because of the underlying uncertainties and limitations’ (EPA 2006). Even though, if data gaps or quality weaknesses are uncovered properly, LCA results could still be valuable. Once a DQA is applied, it could serve as a decision-making tool that identifies both the source and type of deficiencies in data quality (May and Brennan 2003).

3  Data Quality Analysis as Part of Interpretation

47

Data Quality Scores  Semi-quantitative approaches could be used to combine these subjective quality judgments with numerical value propositions. It has been debated at length in the LCA community whether or to what extent these qualitative statements should be quantified and summed up. A pending issue is whether quality scores should be aggregated at system level or be left as single scores. The interpretation of quality scores does not only depend on the purpose but also on the process of the respective study; different processes, weighted with the same score, may not necessarily represent the same importance.12 That is why Weidema and Wesnaes (1996) do not recommend the usage of aggregating data quality scores. In contrast, Wrisberg et al. (1997) and Lindeijer and van den Berg (1997) ‘suggest the aggregation of indicators at system level and add of two indicators on system completeness and reliability’ (Weidema 1998). According to them, aggregated data quality ranks might be useful for comparisons between quality indicators. By consolidating single scores, it should be indicated which step contributes most to the overall data quality. High amounts of datasets could be notably analyzed more easily by use of aggregated scores. Nevertheless, it could be more time-consuming and error-prone. Aggregation of indicators disregards that all indicators are weighted equally and their independency is diminished. Further, the practitioner has to be aware that the higher the degree of aggregation, the higher the risks of misinterpretations. Due to the reason that low quality data points could determine the quality of an entire flow (Weidema and Wesnaes 1996), the identification of low quality does not necessarily reveal information how critical the quality issue really is. In order to benefit from data quality analyses, Weidema et  al. (2013) propose to use quality information for supporting uncertainty analyses, for example by use of Monte-­ Carlo-­Simulation for assessing reliability at system level. Collaboration Among Software and Database Providers  Irrespective of any numerical quality assessment method, instead of striving for a single and absolute score, data quality rather indicates a level of appropriateness (Bauer et al. 2004). Thus, any kind of analysis is only as good as it is performed and interpreted by the practitioner. That is why Klöpffer and Grahl (2013) account non-numerical methods as fixed part of interpretation ‘because mathematical methods cannot solve problems that results from value choices’. As insinuated, there is no coherent framework how to link DQA and LCIA results or how to treat the interaction of foreground data and background data. Hence, the establishment of an applicable approach that imbeds both dimensions − quality and quantity − is essential. This requires not only collaboration between scientists, experts, and practitioners but also a close linkage to software- and database providers. DQA is an immanent part of interpretation, which includes all LCA phases. It is not only beneficial for the management of data issues within a single LCA step but  ‘A score 2 for one indicator is not necessarily of the same importance as a score 2 on another indicator, nor is a score 4 necessarily twice as problematic as a score 2 on the same indicator’ (Weidema 1998). 12

48

D. Kölsch and S. Giebeler

also for the quality assurance of the whole study. Having data quality analysis as kind of ‘fixed part’ in LCA, it could certainly benefit LCA practices and strengthen the study outcome. However, it is even more important that an LCA practitioner acts sober-minded and self-reflective, never neglecting the uniqueness of every life cycle approach. A well-balanced conglomeration of scientific diligence, critical acclaim and pragmatism permits an LCA practitioner to evaluate data in terms of quality consciously and prudential.

References Bauer C, Schebek L, Warsen J, Wursthorn S (eds) (2004) Proceedings of the International Workshop on Quality of Life Cycle Inventory (LCI) Data: Forschungszentrum Karlsruhe, October, 20th to 21st, 2003, Wissenschaftliche Berichte/Forschungszentrum Karlsruhe Technik und Umwelt, vol 6941. Forschungszentrum, Karlsruhe Bretz R (1998) Society of Environmental Toxicology and Chemistry (SETAC) LCA Workgroup: Data availability and data quality. Int J Life Cycle Assess 3(3):121–123. https://doi.org/10.1007/ BF02979390 Ciroth A, Muller S, Weidema B, Lesage P (2013) Empirically based uncertainty factors for the pedigree matrix in ecoinvent. Int J Life Cycle Assess. https://doi.org/10.1007/s11367-­013-­0670-­5 Cooper JS, Kahn E (2012) Commentary on issues in data quality analysis in LCA. Int J Life Cycle Assess 17(4):499–503. https://doi.org/10.1007/s11367-­011-­0371-­x Coulon R, Camobreco V, Teulon H, Besnainou J (1997) Data quality and uncertainty in LCI. Int J Life Cycle Assess 2(3):178–182. https://doi.org/10.1007/BF02978816 European Commission (EC) (2015) Data requirements in Product Environmental Footprint Category Rules (PEFCRs): Version 1.0, European Commission  – Joint Research Centre  – Institute for Environment and Sustainability H08 Sustainability Assessment Unit European Commission (EC) (2012) Product Environmental Footprint (PEF) Guide, European Commission  – Joint Research Centre  – Institute for Environment and Sustainability H08 Sustainability Assessment Unit EC (2010) International Reference Life Cycle Data System (ILCD) handbook – general guide for LCA – detailed guidance. EUR, 24708 EN. Publications Office, Luxembourg Frischknecht R, Jungbluth N, Althaus H-J, Doka G, Dones R, Heck T, Hellweg S, Hischier R, Nemecek T, Rebitzer G, Spielmann M, Wernet G (2007) Overview and methodology. Ecoinvent Report No. 1, Dübendorf Funtowicz SO, Ravetz JR (1990) Uncertainty and quality in science for policy, Theory and Decision Library, Series A: Philosophy and Methodology of the Social Sciences, vol 15. Springer, Dordrecht Grabowski A, Selke SEM, Auras R, Patel MK, Narayan R (2015) Life cycle inventory data quality issues for bioplastics feedstocks. Int J Life Cycle Assess. https://doi.org/10.1007/ s11367-­015-­0853-­3 Heijungs R, Huijbregts MA (2004) A review of approaches to treat uncertainty in LCA. In: Pahl-­ Wostl C, Schmidt S, Jakeman T (eds) iEMSs 2004 International Congress: “Complexity and Integrated Resources Management”. International Environmental Modelling and Software Societey, Osnabrueck Henriksson PJG, Guinée JB, Heijungs R, de Koning A, Green DM (2014) A protocol for horizontal averaging of unit process data –including estimates for uncertainty. Int J Life Cycle Assess 19(2):429–436. https://doi.org/10.1007/s11367-­013-­0647-­4 Huijbregts M, Heijungs R, Hellweg S (2004) Complexity and integrated resource management: uncertainty in LCA. Int J Life Cycle Assess 9(5):341–342. https://doi.org/10.1007/BF02979428

3  Data Quality Analysis as Part of Interpretation

49

Huijbregts MAJ, Norris G, Bretz R, Ciroth A, Maurice B, von Bahr B, Weidema B, de Beaufort, A S. H. (2001) Framework for modelling data uncertainty in life cycle inventories. Int J Life Cycle Assess 6(3):127–132. doi: https://doi.org/10.1007/BF02978728 Iaccarino G (2008) Quantification of uncertainty in flow simulations using probalilistic methods. http://web.stanford.edu/group/uq/docs/iaccarino_vki_notes.pdf. Accessed 29 Apr 2015 ISO (9000:2005) Quality management systems – Fundamentals and vocabulary International Standard Organization (ISO) (14040:2006) Environmental management  – LCA  – Principles and framework (ISO 14040:2006) International Standard Organization (ISO) (14044:2006) Environmental management  – LCA  – Requirements and guidelines (ISO 14044:2006) International Standard Organization (ISO)/International Electrotechnical (IEC) (2382–1:1993) Information technology – Vocabulary – Part 1: Fundamental terms Junior DJ, Gomes J, Bork CA (2014) Reliability of the sustainability assessment. Proc CIRP 15:361–366. https://doi.org/10.1016/j.procir.2014.06.034 Kennedy DJ, Montgomery DC, Rollier DA, Keats JB (1997) Data quality. Int J Life Cycle Assess 2(4):229–239. https://doi.org/10.1007/BF02978420 Klöpffer W, Grahl B (2014) LCA (LCA): a guide to best practice. Wiley-VCH, Weinheim Lazarevic D, Buclet N, Brandt N (2012) The application of life cycle thinking in the context of European waste policy. J Clean Prod 29–30:199–207. https://doi.org/10.1016/j.jclepro. 2012.01.030 Lewandowska A, Foltynowicz Z (2004) Comparative LCA of industrial objects part 2: case study for chosen industrial pumps. Int J Life Cycle Assess 9(3):180–186. https://doi.org/10.1007/ BF02994192 Lewandowska A, Foltynowicz Z, Podlesny A (2004) Comparative lca of industrial objects part 1: LCA data quality assurance – sensitivity analysis and pedigree matrix. Int J Life Cycle Assess 9(2):86–89. https://doi.org/10.1007/BF02978567 Lindeijer EW, van den Berg N (1997) Kwaliteitsbeoordelingsprocedure LCA data. Stichting RIONED. in: Weidema BP (1998) Multi-user test of the data quality matrix for product life cycle inventory data. Int J Life Cycle Assess 3(5):259–265. https://doi.org/10.1007/BF02979832 Lloyd SM, Ries R (2007) Characterizing, propagating, and analyzing uncertainty in life-cycle assessment: a survey of quantitative approaches. J Ind Ecol 11(1):161–179. https://doi. org/10.1162/jiec.2007.1136 Maurice B, Frischknecht R, Coelho-Schwirtz V, Hungerbühler K (2000) Uncertainty analysis in life cycle inventory. Application to the production of electricity with French coal power plants. J Clean Prod 8(2):95–108. https://doi.org/10.1016/S0959-­6526(99)00324-­8 May JR, Brennan DJ (2003) Application of data quality assessment methods to an LCA of electricity generation. Int J Life Cycle Assess 8(4):215–225. https://doi.org/10.1007/bf02978474 Meier MA (1997) Eco-efficiency evaluation of waste gas purification systems in the chemical industry, LCA documents, vol 2. Eco-Informa Press, Bayreuth Rousseaux P, Labouze E, Suh Y, Blanch I, Gaveglia V, Navarro A (2001) An overall assessment of life cycle inventory quality: application to the production of polyethylene bottles. Int J Life Cycle Assess 6(5):299–306. https://doi.org/10.1007/BF02978792 Schuurman A (2004) Data quality assessment method for LCI-data of the Dutch building industry. In: Bauer C, Schebek L, Warsen J, Wursthorn S (eds) Proceedings of the international workshop on quality of life cycle inventory (LCI) data: Forschungszentrum Karlsruhe, October, 20th to 21st, 2003, vol 6941. Forschungszentrum, Karlsruhe, pp 81–86 Thunnissen D (2003) Uncertainty classification for the design and development of complex systems. California Institute of Technology, Division of Engineering & Applied Science UNEP/SETAC (2011) Sonnemann G, Vigon BW (eds) Global guidance principles for LCA databases: A basis for greener processes and products: ‘Shonan guidance principles’. United Nations Environment Programme, Nairobi

50

D. Kölsch and S. Giebeler

United States Environmental Protection Agency (EPA) (1995) Guidelines for assessing the quality of life-cycle inventory data. United States Environmental Protection Agency, Solid Waste and Emergency Response, Washington, DC United States Environmental Protection Agency (EPA) (2006) Life cycle assessment: principles and practice. United States Environmental Protection Agency, Office of Research and Development, Cincinnati, Ohio van den Berg, NW, Huppes G, Lindeijer EW, van der Ven BL, Wrisberg MN (1999) Quality assessment for LCA. http://www.leidenuniv.nl/interfac/cml/ssp/publications/quality.pdf. Accessed 27 Feb 2015 van der Sluijs J, Kloprogge P, Risbey J, Ravetz J (2003) Towards a synthesis of qualitative and quantitative uncertainty assessment: applications of the Numeral, Unit, Spread, Assessment, Pedigree (NUSAP): system communication to the international workshop on uncertainty, sensitivity, and parameter estimation for multimedia environmental modeling van der Sluijs J, Craye M, Funtowicz S, Kloprogge P, Ravetz J, Risbey J (2005) Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: the NUSAP system. Risk Anal 25(2):481–492. https://doi.org/10.1111/j.1539-­6924.2005.00604.x Wang E, Shen Z (2013) A hybrid data quality indicator and statistical method for improving uncertainty analysis in LCA of complex system – application to the whole-building embodied energy analysis. J Clean Prod 43:166–173 Wang RY, Reddy MP, Kon HB (1995) Toward quality data: an attribute-based approach. Decis Support Syst 13(3–4):349–372. https://doi.org/10.1016/0167-­9236(93)E0050-­N Weidema BP (1998) Multi-user test of the data quality matrix for product life cycle inventory data. Int J Life Cycle Assess 3(5):259–265. https://doi.org/10.1007/BF02979832 Weidema BP, Bauer C., Hischier R., Mutel C, Nemecek T, Reinhard J, Vadenbo CO, Wernet G (2013) Overview and methodology: Data quality guideline for the ecoinvent database version 3. http://www.ecoinvent.org/fileadmin/documents/en/Data_Quality_Guidelines/01_ DataQualityGuideline_v3_Final.pdf. Accessed 27 Feb 2015 Weidema BP, Wesnæs MS (1996) Data quality management for life cycle inventories—an example of using data quality indicators. J Clean Prod 4(3–4):167–174. https://doi.org/10.1016/ S0959-­6526(96)00043-­1 Wrisberg MN (1997) A semi-quantitative approach for assessing data quality in LCA.  In: Proceedings seventh annual meeting of SETAC-Europe, Amsterdam Zhang H, Haapala KR (2012) Integrating sustainability assessment into manufacturing decision making. In: Dornfeld DA, Linke BS (eds) Leveraging technology for a sustainable world. Springer, Berlin/Heidelberg, pp 551–556

Chapter 4

Quality Assurance by International Standards: The ‘Critical Review’ Walter Klöpffer and Matthias Finkbeiner

Abstract  This chapter explains the development of the most important step of the ‘Critical Review’ from the beginning in the late 1960s, over SETAC, here, especially, the ‘Code of Practice’, to ISO 14040-43 (1997–2000) and ISO 14040 (2006) as well as ISO 14044 (2006). While ISO 14040-43 (1997–2000) can be regarded as the first series or the first edition, the update in 2006 as ISO 14040 (2006) and ISO 14044 (2006) has elaborated the commonly accepted rules for LCA (Life Cycle Assessment). They are the ‘core standards’: • ISO 14040 (2006): Environmental management − Life cycle assessment − Principles and framework. • ISO 14044 (2006): Environmental management − Life cycle assessment − Requirements and guidelines. Moreover, the chapter describes the importance of the Technical Specification ISO 14071 for the performance and quality of the critical review process. Keywords  Code of Practice (1993) · Critical review · Interpretation · ISO (International Standard Organisation) · ISO 14040-14043 (1997–2000) · ISO 14040 (2006) · ISO 14044 (2006) · ISO/TC207/SC5 · LCA · LCI · Life cycle assessment · Life cycle inventory analysis · Peer review · SETAC · Technical specification ISO 14071 (2014)

This chapter was co-written, reviewed and accepted by Walter Klöpffer before, sadly, passing away on January 29, 2023. May he rest in peace. W. Klöpffer LCA Consult & Review, Frankfurt am Main, Germany M. Finkbeiner (*) Chair of Sustainable Engineering, Institute of Environmental Technology, Technische Universität Berlin, Berlin, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_4

51

52

W. Klöpffer and M. Finkbeiner

4.1 Introduction This chapter explains the development of the most important step of the critical review process from the beginning in the late 1960s, over SETAC (Code of Practice 1993), to ISO 14040-43 (1997–2000) and ISO 14040 (2006), as well as ISO 14044 (2006).  The roots of LCA (Life Cycle Assessment) reach back to the late 1960s when a relatively simple but convincing method was presented under the name ‘Resource and Energy Profile Analysis’ (REPA) at the Midwest Research Institute in the USA. Similar methods, called ‘proto-LCAs’, were developed in the following years, leading to a moderate growth of the young environmental assessment method in North America and Europe. The Society for Environmental Toxicology and Chemistry (SETAC) took the leadership in establishing a framework for the new method, now called ‘Life Cycle Assessment’. The first document dealing with the ‘peer review’ of LCA studies goes back to the ‘Code of Practice’ (1993). The term ‘critical review’ was introduced later by the International Standard Organisation (ISO). It has been coined for the special peer review of LCA studies. International standards for LCA were developed in the 1990s by ISO Technical Committee (TC) 207 (Environmental Management) as part of the ISO 14000 family of environmental management standards. The committee within ISO/TC207 dealing with LCA is Subcommittee 5 (SC5). So the complete name of the LCA unit is ISO/TC207/SC5. The ISO-LCA standards 14040-43 (1997–2000) are a first series or a first edition. • ISO 14040 (1997): Environmental management  – Life cycle assessment  – Principles and framework. • ISO 14041 (1998): Environmental management – Life cycle assessment: Goal and scope definition and inventory analysis. • ISO 14042 (2000): Environmental management  – Life cycle assessment: Life cycle impact assessment. • ISO 14043 (2000): Environmental management  – Life cycle assessment: Interpretation. The division of LCA methodology into successive phases  – principles and framework; goal and scope definition, inventory; impact assessment; interpretation (formerly improvement)  – was directly inspired by SETAC ‘Code of Practice’ (1993), which was the most authoritative publication to be referred to. In 2006, these standards have been updated according to a revised ISO 14040 standard and a new standard 14044 containing all requirements: • ISO 14040 (2006): Environmental management − Life cycle assessment− Principles and framework. • ISO 14044 (2006): Environmental management − Life cycle assessment − Requirements and guidelines.

4  Quality Assurance by International Standards: The ‘Critical Review’

53

The objective of the revision of the standards ISO 14040, 14041, 14042 and 14043 was to improve readability, while leaving the requirements and technical content unaffected, except for errors and inconsistencies. It was the intention. 1. To gather all requirements (‘shalls’) in one new standard, keeping the structure of ‘goal and scope’, ‘inventory’, ‘impact assessment’ and ‘interpretation’ as separate chapters. 2. To maintain ISO 14040 as a framework document, but transferring all requirements (‘shalls’) to the new standard, adding to ISO 14040 a requirement (‘shall’) of compliance with the requirements (‘shalls’) of the new standard. ISO 14040 and ISO 14044 (2006) have become the commonly accepted rules for LCA. They are the ‘core standards’: A decisive distinction is made between LCA studies to be used for ‘comparative assertions’ disclosed to the public and others which are used for internal purposes. A ‘comparative assertion’ is defined in ISO 14040 (1997). Very important for the further development is the statement in ISO 14040ff that LCA is about environmental impacts of product systems, excluding economic and social ones. Moreover, the chapter describes the importance of the Technical Specification ISO 14071 for the performance and quality of the critical review process. Due to the increasing relevance of critical review in the application of LCA as well as carbon and water footprint studies, the Technical Specification ISO 14071 was developed (2014). This TS is not a stand-alone document, but adds additional requirements and guidelines to ISO 14044. This Technical Specification provides the necessary information for carrying out a critical review for any type of LCA study and the necessary competencies of the reviewers. ‘Critical review process and tasks’ is the centrepiece of the standard 14071.

4.2 The Critical Review: Why and How It Came About A first doctoral thesis was written at the Technical University of Berlin (Franke 1983). The quality of this thesis was checked in the usual academic procedure for PhD candidates. No formal quality assurance was used; however, in the early reports issued by consulting and industrial firms, a final reading by a superior was made.1 During the second half of the 1980s, it turned out that the absence of any obligatory and formal quality control in the private sector was a weak point of the life cycle

 At Battelle Institute.V. in Frankfurt/M, a daughter company of Battelle Memorial Institute in Columbus, Ohio, there was an obligatory language check in addition (also for reports written in German). 1

54

W. Klöpffer and M. Finkbeiner

methods, especially in comparative studies. This has to do with the greatest strength of the LCA method(s): the combination of cradle-to-grave analysis with the unique feature of LCA, the functional unit. This combination allows comparisons of product systems2 under environmental aspects, even between goods and services as long as they provide the same or a very similar function. Of course, this combination constitutes the great charm of LCA and made the method(s) popular, but also vulnerable. In the same period (the late 1980s), the number of studies increased strongly, still proto-LCAs produced without a harmonized – much less a standardized – methodology. Misinterpretations of comparative life cycle studies became known, some due to the absence of a common methodology, some evidently by misuse for marketing purposes. In this situation, the Society for Environmental Toxicology and Chemistry (SETAC) took the leadership in establishing a framework for the new method, now called ‘Life Cycle Assessment’3 (Fava et al. 2014). SETAC had the necessary authority for such a harmonization activity since it is a reputable international scientific society and has a ‘tripartite’ structure in membership and management involving academic, industrial and governmental members in all decision levels. The procedure used by SETAC for the harmonization of LCA was the organization of well-documented (‘Pellston’) workshops attended by invited speakers and writers (Fava et al. 2014) both in the USA and in Europe. The first of these workshops was held in Smuggler’s Notch (Vermont) in August 1990. The documentation of this event (Fava et al. 1991) may be considered as the birth of modern LCA. It provided definitions and a first structure (the famous ‘SETAC triangle’) and gave advice on how to conduct an LCA study properly. Many questions remained open, however, leading to further Pellston workshops in the following years. In the context of quality assurance, only the workshop in Wintergreen, Virginia, October 1992 (Fava et al. 1994), is mentioned here. This workshop dealt with a typical quality issue: data quality and how to measure and document it. This was (and still is) a major problem in Life Cycle Inventory analysis (LCI) and plays a major role in reviewing LCAs. At the time of the Wintergreen workshop, only few LCI data were available, and traditional methods of data treatment failed due to the small amount and (often poor) quality of data. Frequently no original data were at hand and had to be estimated with the help of technical handbooks. An early example is the book by Boustead and Hancock (1979). The main result of the Wintergreen workshop was a clear definition of the problem and suggestions how to proceed further. Most important for further development of the LCA review process is a strong statement about data quality assessment as an integral part of LCA, especially with regard to input (LCI) data. The most obvious data quality issues named in the report

 Products are defined in ISO 14040ff as goods and services (tangible and non-tangible products).  There is an old dispute about the correct spelling: with hyphen (Life-Cycle Assessment) or without? The editors of this compendium adopted the version without hyphen in accordance with ISO 14040 and the International Journal of Life Cycle Assessment. 2 3

4  Quality Assurance by International Standards: The ‘Critical Review’

55

concern the representativeness, age of the technology analyzed, completeness of coverage of the data categories and reproducibility and overall uncertainty of the study. There is an interesting definition of data quality on top of the data quality concepts elaborated by SETAC: Data quality was defined as the degree of confidence in individual input data and in the data set as a whole and ultimately in decisions made by using the data.

This definition means that in LCA, there is no absolute data quality, but the whole data set should be suitable for a specific study. This point is extremely important for the review process, since such a decision finally needs the experience of reviewers and cannot be done with mathematical means alone. Furthermore, the answer is strongly context dependent; hence, the great importance of ‘goal definition and scoping’ (compare the nearly identical wording for phase 1 according to ISO 14040). A three-stage systematic data quality assessment (DQA) process is suggested as a first step of any LCA. It is based on data quality goals (DQG). If no DQA is possible in stage III, the result may be either: place limitations or restrictions on study results

or abandon the study

We will see later that the ‘double arrow’ scheme in modern LCAs allows correcting steps in the study also during the work, provided that the correction is done in written form. This gives more flexibility since the availability of certain key data often becomes evident during the LCI work (e.g. a supplier promised to provide foreground data from his facility but finally did not). The stepwise procedure proposed is sound, however, and reduces the risk of performing a study without a convincing result. Please note that in comparative LCAs there are at least two data sets which have to be of similar quality in order to allow a fair comparison of the product systems on the basis of the functional unit chosen. A final judgement about data quality will always be a main point in the last phase of any proper LCA, which is Interpretation, as given in ISO 14044. Data quality indicators (DQI) play a major role in the discussions of the Wintergreen report, but also the development of new data categories specific for use in LCAs has been discussed. Since isolated data are of limited use in a method needing many consistent data with a defined (or at least known) quality, the next step was the establishment of databases. These should use uniform data formats, the oldest and best known in LCA being the SPOLD4 format, originally developed for data transfer. The continued work in that direction recently culminated in international rules known as the

 Society for the promotion of life cycle development (SPOLD).

4

56

W. Klöpffer and M. Finkbeiner

‘Shonan Guiding Principles’ (Sonnemann and Vigon 2011; Sonnemann et al. 2011). This report also contains a chapter about the review process needed for the (LCI) data quality assurance.

4.3 The LCA Peer Review According to the SETAC Guideline 1993 The first document dealing with the ‘peer review’5 of LCA studies explicitly goes back to the SETAC workshop report ‘Guidelines for Life-Cycle Assessment: A Code of Practice’ (SETAC 1993).6 These guidelines were derived from presentations and discussions of ca. 50 invited participants from the USA and Europe. The workshop took place in Sesimbra, Portugal, immediately after the first SETAC world congress in Lisbon, March/April 1993. The impatiently awaited publication was available only 5  months after the workshop. The ‘Code of Practice’ can be considered as a blueprint or seed paper for the first series of ISO standards 14040-14043 (next section). The ‘peer review’ is treated in chapter 7 of the code in more detail than the ‘critical review’ in the ISO standards (Sect. 4 and 5). Only the recent Technical Specification ISO TS 14071 (2014) (Sect. 6 ff) goes beyond and provides the details for performing a critical review correctly. The high importance attributed to the review is already visible in the first sentence of chapter 7 in (SETAC 1993): The development and use of a peer review process will be a key feature in the advancement of Life-Cycle Assessments. The rationale for the review is given as follows: 1. The peer review process enhances the scientific and technical quality of LCAs. 2. The process helps to focus study goals, data collection, and provides a critical screening of study conclusions, thereby enhancing study credibility. Already in the introduction it is postulated that a peer review of LCA studies used for public policy and advertising applications ‘should be more extensive than that traditionally used for the publication of research in professional journals’. This clear distinction was later also formally recognized by ISO in creating the new term ‘critical review’ to be used in LCAs for comparative assertions. In the code, three reasons are given for the greater effort needed for the review of comparative LCAs: regulatory and public-policy implications, proprietary information and the protection of it and complexity of the data. In a nearly visionary insight, it is stated

 The term ‘critical review’ was introduced later by the International Standard Organisation (ISO).  Editors: Frank Consoli, David Allen, Ian Boustead, James Fava, William Franklin, Allan A. Jensen, Nick de Oude, Rod Perriman, Dennis Postlethwaite, Beth Quay, Jacinthe Séguin and Bruce Vigon. 5 6

4  Quality Assurance by International Standards: The ‘Critical Review’

57

that a ‘more multidisciplinary peer review process than is required in most scientific studies’ will be needed. The same is true for the proposal of an ‘interactive peer review’, ideally conducted in three phases: • At the beginning (goal, scope, boundaries, planning of data collection). • By checking the progress after initial data collection and modelling and. • In the reporting phase ‘to review the adequacy of the study and the credibility of the conclusions’. The repeated use of the word ‘credibility’ hints to the main problem of the still young and sometimes misused method. Although the interactive review is considered as ‘desirable’, a review of the final study report and supporting data is admissible. This type of review has later been called ‘a posteriori’ (Klöpffer 2005). It is also recognized that many details are study-dependent, for example, the number of meetings of the peer review panel and the level of documentation. The main difference identified is the one between studies released to the public and others intended for internal use only, for example for product improvement. The general selection criteria for LCA reviewers ‘include experience with the technical framework, design, and conduct of life-cycle studies; some experience or expertise with the subject matter of the study; a willingness to interact positively as part of a peer review panel; and a lack of any conflicts of interest that would potentially jeopardize the independence of the peer review’. The composition of the panel should depend on the audience, which is different for external LCAs (e.g. regulators, consumers and environmental activists as well as the scientific community) and internal LCAs. In the latter case, the sponsor is the main audience since the main aim is mostly product optimization. Such studies are often done ‘in house’ by an LCA expert or a small group. LCA practitioners, process experts, customers and suppliers are suggested as reviewers in that case. It is also stated that in LCAs, for internal use, in general, the sponsor will compose the panel, but also the practitioner can do that. For external studies, the procedure discussed in the ISO section of the workshop is suggested: The sponsor invites the chair of the panel who will be responsible for the composition of the panel. Sponsor and practitioner may help in this process, but ‘The final choice … is the responsibility of the chair of the peer review panel’. Sponsor and practitioner are not allowed to act as panel members of an LCA intended for public release. Nothing is said about the size of such panels which will in practice – at least in most cases –be limited by financial considerations. In the revision of the ISO standards in 2006, a minimum number of three panel members was introduced (Finkbeiner et al. 2006). The peer review process is described in detail on the basis of the ‘ideal’ case of the three phases presented above. This part made the code useful even after appearance of the ISO standards and makes a good reading still today. It is suggested that the first part, essentially Goal and Scope, data availability, boundaries, functional unit (including alternatives), etc. is done in an early stage of the LCA,

58

W. Klöpffer and M. Finkbeiner

since ‘early review also reduces the chance that a panel would disagree with specific study conclusions’. Further peer review items described in some detail concern ‘peer review of data’, ‘review of the final report’ and ‘documentation’. While the data review overlaps with the contemporary Wintergreen report (Fava et al. 1994), and the documentation is well-treated in the first series of ISO standards (Klöpffer 2005), the guidelines for review provide a first structure for the review of the whole report. The following questions give guidance: • Was the methodology adequately documented? Was the documentation of sufficient detail for someone to repeat the study? • How were the data aggregated, summarized and presented? • Are sensitivity analyses adequate? • Were the conclusions appropriate based on the data and analysis? The comments by the reviewers should be communicated in written form with the practitioner and the sponsor; they should be discussed and also revised if the draft report has been modified. A detail not to be neglected: LCA reports should not be altered between the final review and public release. And: The project report…should contain the conclusions and recommendations of the peer review panel. This ‘should’ will later be replaced by a ‘shall’ during the ISO process. Since the LCA peer review process was relatively new when the code appeared, there were not many qualified reviewers available, which was an obstacle for full-­ scale introduction. Also the costs were noted as hindering the process. While the number of suitable reviewers has strongly increased since 1993, the number of reviewers per review stayed rather constant in order to keep the costs relatively limited (Klöpffer 2005, 2012).

4.3.1 Application of the ‘Code of Practice’ in Real LCI and LCA Studies Between the publication of the code and the first ISO standard (ISO 1997) there was a relatively short time during which the Code of Practice was the only international guidance for performing an LCA study, including the ‘peer review’.7 One example is the ‘peer review’ of the Life Cycle Inventory study of the European Surfactant study (Klöpffer et  al. 1995, 1996). The practitioner of the study was Franklin Associates Ltd.8 (Janzen 1995), the sponsor of a group of surfactant and detergent producers (ECOSOL) under the umbrella of CEFIC9 (Brussels). The study was

 For another major harmonization effort by the Scandinavian countries, see Lindfors et al. 1995.  FAL, now Franklin Associates, a Division of ERG (Eastern Research Group). 9  The European Chemical Industry Council or CEFIC (former French name: Conseil Européen des Fédérations de l’Industrie Chimique). 7 8

4  Quality Assurance by International Standards: The ‘Critical Review’

59

accompanied by a large group of ecological industrial experts (Stalmans et  al. 1995). The voluminous reports were studied by the whole review panel, and the data and model check was done by Gustav Sundström (Sweden), the doyen of the group, in the USA. An unexpected problem surfaced during this review with regard to data quality control: An evident error of electricity generation in Italy (primary energy mix) could not be fully explored since the data provider refused to lay open the original data for scrutiny. After an extra sensitivity analysis (CEFIC-ECOSOL) and an emergency session of the peer review panel and sponsor, the review could be successfully concluded; the error introduced by the wrong data set did not invalidate the main conclusions of the report (Klöpffer et al. 1995). Including the emergency session, there were four review panel meetings, and the one dealing with the final review report lasted only 2 days. The publication of the data and of the background information was done in a series of papers in ‘Tenside, Surfactants, Detergents’ (1995) (see Stalmans et al. 1995).

4.3.2 PVC in Sweden (1996) An interesting critical review10 performed before the publication of ISO 14040 (ISO 1997) concerned the total life cycle impacts of PVC in Sweden (Ayres et al. 1996; Tukker et  al. 1996). The LCA-type study was prepared under time pressure (the Swedish parliament had planned a complete ban of PVC!) and several simplifications were due to this condition. The structure of the study was a substance flow analysis using the border of Sweden as geographic system boundary. In addition to the substance flow (a truncated LCI), an impact assessment according to the ‘CML method’ (Heijungs et al. 1992/1993) was added. The whole endeavour was labelled as a ‘level I’ (simplified LCA) study by the critical review panel, who met four times ‘face-to-face’ with the practitioner and the commissioner at Schiphol Airport between April and October 1996. The finalization of the study was delayed by the finding of the critical review panel that different PVC types had strongly diverging toxicity ratings. The error was difficult to spot, but, finally, the source was found: for calculating the toxicity, a programme developed for chemical risk assessment was used. In the version used, substances with unknown toxicity (no data) got an unreasonably high (worst case) rating as a substitute for the unknown ‘real’ value. It turned out that the presumably highly toxic PVC used a new, untested plasticizer with longer chains (higher molecular weight, less volatile and soluble) than the mostly used and critically discussed Di(ethylhexyl) phthalate (DEHP). Finally, the report was discussed with some severe reservations by the critical review team, but all in all, it was accepted (Ayres et al. 1996). The delay of several months was, in retrospect, no disaster since the Swedish parliament finally decided not to ban PVC.  It is usual praxis to use draft standards (especially the draft international standard – DIS) before the official publication; hence the use of the term ‘critical review’ in this study. 10

60

W. Klöpffer and M. Finkbeiner

Many LCA studies of PVC revealed weak points of this material, most of which can be avoided by using better techniques over the whole life cycle and by a proper use of the long-lived polymer. The use of PVC in short-lived bottles, etc. disappeared soon in Europe. In the long run, PVC, similar to aluminium, profited both from the criticism by environmental groups and from improved and carefully reviewed LCAs. The ‘PVC in Sweden’ LCA was much discussed in the late 1990s and flowed in a very detailed PhD thesis by the project leader (Tukker 1998).

4.4 The Critical Review in the First Series of ISO Standards (1997–2000) The first international LCA standard appeared 4  years after SETAC’s ‘Code of Practice’ (ISO 1997; Marsmann 1997) as ISO 14040. It contains the basic features, definitions and rules of any correct Life Cycle Assessment study: Principles and Framework. The importance of the ISO standards for Life Cycle Assessment has been discussed by Finkbeiner (2014). The term The Constitution of LCA coined by this author for the ISO LCA standards describes very well the role and rank of these standards. ISO 14040 was followed by three other LCA standards dealing with Goal and Scope Definition and Life cycle Inventory Analysis (ISO 1998), Life Cycle Impact Assessment (ISO 2000a), and Interpretation (ISO 2000b). The term ‘critical review’ has been coined for the special peer review of LCA studies. Very important for the further development is the statement that LCA is concerned with environmental impacts of product systems, products being defined as goods and services ‘from cradle-to-grave’. It is acknowledged in ISO 14040 that other aspects  – especially economic and social ones  – contribute to the overall impact of a product system, but the ISO standards14040ff deal with the environmental aspect only. LCA (alone) is therefore not a sustainability assessment. It is important to stress this point, since the term ‘sustainability’ has been used inadequately recently (Klöpffer and Grahl 2014). The critical review in the first series of ISO LCA standards has been treated before (Klöpffer 2005; Lichtenvort 2005), so that a short summary of the main findings in the standards may be appropriate here. Most information about the critical review can be found in ISO 14040. There, the following definition is given: Critical review is a technique to verify whether an LCA study has met the requirements of this international standard for methodology, data and reporting. Whether and how to ­conduct a critical review, as well as who conducts the review, shall11 be defined in the scope of the study.

11

 ‘shall’ (bold letters) is very strong in ISO language, meaning ‘must without exception’.

4  Quality Assurance by International Standards: The ‘Critical Review’

61

The main emphasis is on meeting the requirements given by the standards. This is further confirmed by the following list of requirements, which is very useful for structuring critical review reports: The critical review process shall ensure that:

• the methods used to carry out the LCA are consistent with this international Standard; • the methods used to carry out the LCA are scientifically and technically valid; • the data used are appropriate and reasonable in relation to the goal of the study; • the interpretations reflect the limitations identified and the goal of the study; • the study report is transparent and consistent. Please note that hardly any absolute requirements are formulated apart from consistency with the standard and the goal of the study. A decisive distinction is made between LCA studies to be used for ‘comparative assertions’ disclosed to the public and others, for example for internal purposes. A comparative assertion is defined as (ISO 1997) environmental claim regarding the superiority or equivalence of one product versus a competing product which performs the same function.

Since such claims can clearly be misused, a strict regulation with regard to the critical review is necessary in order to minimize such misbehaviour. For such studies, the critical review according to ISO 14040, section 7.3.3, has to be used. This review is now mostly called the ‘panel method’, since the original name (review by interested parties) is partly misleading. The ‘review by interested parties’ differs from other forms of critical review by a chairperson (invited by the commissioner) who is responsible for the team, has the main contact with commissioner and practitioner and writes the critical review report. The co-reviewer(s) can be selected according to special knowledge required in a specific study or in order to represent groups of people concerned with the results, hence the name ‘interested parties’. The standard ISO 14040 (ISO 1997) defines an interested party as: Individual or group concerned with or affected by the environmental performance of a product system, or by the results of the life cycle assessment.

Examples are competitors, users (e.g. consumer protection groups), government environmental protection agencies and lobbying groups. Industry associations, which, in general, represent several competing companies, are especially suited as interested parties, but frequently also act as commissioner. Some large LCA studies have advisory boards to which are invited to attend project meetings by the commissioner and discuss the results with the project team or the critical review team or at least with the chair. In such cases, the advisory board acts as ‘interested party’ without being formally part of the critical review team. There is also an ambiguity in the wording of section 7.3.3, ‘Review by interested parties’, insofar as the text reads: ‘This panel may include other interested parties that will be affected by conclusions drawn from the LCA study….’. As the weak

62

W. Klöpffer and M. Finkbeiner

formulation (‘may’ instead of ‘shall’) shows, the inclusion of interested parties is optional, not mandatory. Clearly, the results of LCA studies can influence the environmental legislation, especially by ordinances which define in detail how a law has to be executed. Standards are in general, however, scientific/technical conventions, not laws. Their frequent use in legislation makes the distinction often ambiguous. For LCA studies without comparative assertions, critical reviews are recommended by ISO 14040 (ISO 1997), but not obligatory. A review by one internal (7.3.1) or external (7.3.2) independent expert was sufficient according to the first version of the standard (ISO 1997). This was changed later (ISO 2006a), see section 5. Despite some inconsistencies, the critical review according to ISO 14040-43 was a great success and contributed to the establishment of LCA as a serious scientific endeavour. In the same series of standards, the last phase (formerly called improvement assessment or valuation) was transformed into ‘Interpretation’ (Lecouls 1999; Marsmann 2000) in order to avoid subjective interpretations as much as possible. In the same direction points the ban on ‘weighting’ (as part of LCIA) for studies containing comparative assertions. Also in the preparation time of the first series of LCA standards (1995) ‘The International Journal of Life Cycle Assessment’ was founded (Heinrich 2014), offering a platform for applied as well as theoretical LCA research and development. The quality assurance in this journal is done by traditional peer review. Some manuscripts have been ‘critically reviewed’ before submission, but most have not. This does not constitute an offence against ISO 14040, since scientific papers serve other purposes (e.g. method development or improvement) than studies performed to prove environmental superiority of one product over another. A critical review by one independent expert is recommended for all LCA studies (Lichtenvort 2005). This voluntary application of critical reviews by individual experts is quite common and successful.

4.5 The Critical Review According to ISO 14040 and 14044 (2006) The first series of ISO LCA standards (Sec. 4) has been updated only once (Finkbeiner et al. 2006; Finkbeiner 2013, 2014), during which process the number of base standards has been reduced from four to two (ISO 2006a, b). Despite this drastic change in form, not much was changed in content with one notable exception: The critical review has been tightened (Klöpffer 2012). The newer standards now consist of a general standard, again called 14040 (ISO 2006a), containing principles and framework and a specific standard (ISO 2006b) containing all requirements and guidelines. It was the intention of the ISO update team to gather all requirements (‘shalls’) in one new standard (ISO 14044) and maintain ISO 14040

4  Quality Assurance by International Standards: The ‘Critical Review’

63

as a framework document, but transferring all requirements (‘shalls’) to the new standard, adding to 14040 a requirement (‘shall’) of compliance with the requirements (‘shalls’) of the new standard (Finkbeiner et al. 2006). The two standards are coupled together by one ingenious ‘shall’ in the new ISO 14040 (section 5.1) requesting full conformity of the two standards: When performing an LCA, the requirements of ISO 14044 shall apply.

Finkbeiner et al. (2006) point out that such a request for conformity did not exist between the former 14040 (ISO 1997) and 14041-43 (ISO 1998, 2000a, b). This requirement is highly relevant for the critical review process, since without this request for conformity with the strict ISO-requirements in performing an LCA, the following could happen: If a ‘new’ LCA method based on the relative open standard 14040 is combined with non-ISO elements (instead as 14044), a new pseudo-­standard ‘based on ISO 14040’12 could result. It is one duty of the critical reviewers (independent expert or panel) to find out whether such deviations have been committed in a specific study. A similar, although somewhat weaker, request for conformity can be found in ISO 14044 (section 4.1, General requirements): See ISO 14040 for the principles and framework to be used to conduct an LCA.

The two statements together clearly show that the two new LCA standards cannot be used separately, if a specific LCA is to be qualified as to be ISO-conform. In addition, both standards contain the other one as a normative reference. If standards are normatively referenced, they are according to the ISO Directives indispensable for the application. The critical review according to the new standards can be performed in one of two forms (formerly three): 1. Critical review by internal or external expert (14040 section 7.3.2; 14044 section 6.2). 2. Critical review by a panel of interested parties (14040 section 7.3.3; 14044 section 6.3). With regard to (1) an expert independent of the current LCA should perform and document the review; the expert ‘should be familiar with the requirements of LCA and should have the appropriate scientific and technical expertise.’ Since this last point is not trivial, this type of critical review is sometimes performed by a small team of reviewers. In contrast to (2), such a team has no chairperson and is not entitled to review LCAs containing or supporting comparative assertions.

 Life cycle methods deviating from the ISO standards (often called ‘footprint’) can be used for internal use, e.g. as part of Life Cycle Management (LCM). The sum of such life-cycle based methods is often called ‘tool box’. 12

64

W. Klöpffer and M. Finkbeiner

The review by a panel is best described in ISO 14044 section 6.3 (ISO 2006b): A critical review may be carried out as a review by interested parties. In such a case, an external independent expert should be selected by the original study commissioner to act as chairperson of a review panel of atleast three members. Based on the goal and scope of the study, the chairperson should select other independent qualified reviewers. This panel may include other interested parties affected by the conclusions drawn from the LCA, such as government agencies, non-governmental groups, competitors and affected industries. For LCIA, the expertise of reviewers in the scientific disciplines relevant to the important impact categories of the study, in addition to other expertise and interest, shall  be considered. The review statement and review panel report, as well as comments of the expert and any responses to recommendations made by the reviewer or by the panel, shall be included in the LCA report.

It is interesting that the two ‘shalls’ of these sections concern the expertise of the critical reviewers with regard to life cycle impact assessment and the inclusion of the review report into the LCA study report. The actual performance of the review ‘by interested parties’ is considered by weak ‘may’ and ‘should’ orders only. For good reasons, since the inclusion of ‘interested parties’ (especially ‘competitors’) is not always possible for reasons of confidentiality and competition. Environmental protection groups are often invited and, as mentioned in Sect. 4, industry associations and members of environmental protection agencies. Frequently, however, only experts are invited, who, in such cases, are expected to consider the rights of the competitors and other non-invited interested parties adequately, in addition to all other duties requested by critical reviewers. A very important question concerns the condition(s) under which a critical review according to the (necessarily more expensive) panel method has to be applied. Here, ISO 14044 presents a clumsy but unambiguous formulation (Finkbeiner et al. 2006; ISO 2006b section 6.1): In order to decrease the likelihood of misunderstandings or negative effects on external interested parties, a panel of interested parties shall conduct critical reviews on LCA studies where the results are intended to be used to support a comparative assertion intended to be disclosed to the public.

The important change compared to the previous standards (Sec. 4) is contained in the second part, saying that the mere intention to use the results of an LCA for comparative assertions ‘intended to be disclosed to the public’ requires a critical review according to the panel method. This very restrictive formulation shows that the prevention of fraud or misuse of LCA studies in marketing did not get less attention since the times of Sesimbra (Sec. 3). The same is true for the potential political misuse of comparative LCA studies. The much better quality of (generic) data and methodological progress (software) compared to the early times may be even seen as a chance to manipulate LCA studies without too much work. Another hint to the serious nature of this problem consists in the observation that the panel method is nearly exclusively performed in the case of comparative assertions intended to be disclosed to the public (Koffler 2013). It would be best if all LCA studies be critically reviewed, at least by one independent expert. If the main results are submitted

4  Quality Assurance by International Standards: The ‘Critical Review’

65

for publication to a serious journal, the anonymous journal review by at least two ‘peers’ is the rule. LCA results reported in the popular non-reviewed literature (without a source where the full study can be obtained) should be regarded with caution. There persists a contradiction with regard to the necessity to include ‘interested parties’ in the new standards (ISO 2006a, b). The ‘shall’ highlighted in the above citation ‘In order to decrease the likelihood of misunderstandings…shall conduct critical reviews….’ is clearly at variance with weak formulations, such as ‘This panel may include other interested parties affected by the conclusions…’ (Klöpffer 2012, 2013). This formulation is taken from section 6.3: ‘Critical review by panel of interested parties’ (14044, ISO 2006b). A very similar formulation can be found in section 7.3.3 (14040, ISO 2006a). The evident contradiction cannot be solved within the existing main standards. Needless to say that the fuzzy wording makes it easier to establish a panel since it will always be correct  – with or without interested parties. The other major deviation in the two new standards compared to the older series of four standards is, with regard to the critical review, that there is no distinction between internal and external independent experts in the weaker form of critical review (one or more experts without a chair). The question is whether or not an internal expert – even if not connected to the current LCA – can be really independent. In a large company with a quality or sustainability accounting department, this may be indeed possible, but in a small firm, it is doubtful. In practice, and especially for internal use, full independence of the reviewers may not be so crucial, since no comparative assertions are allowed for this type of critical review. Exaggerated positive judgements about the environmental behaviour of a product – based on internal LCAs  – may nevertheless leak out into non-peer reviewed journals, company announcements, web sites or newspapers. Also data collections/LCI studies for materials or energy are often reviewed according to ISO 14040 section 7.3.2 and 14044 section 6.2. If the data are to be used by public databanks, independent external experts are preferred over internal ones (Klöpffer 2009).

4.6 The Technical Specification ISO 14071: How It Came About In the previous section, the rather concise core requirements and guidelines for critical review according to ISO 14040 and ISO 14044 (2006) are described. Due to the increasing relevance of critical review in the application of LCA as well as carbon and water footprint studies, the Technical Specification ISO 14071 (2014) was developed. Background and motivation for this standard document are presented in Sec. 6.1, followed by a brief history of the standardization process in Sec. 6.2. The actual technical content of ISO/TS 14071 is summarized in Sec. 7 and 8.

66

W. Klöpffer and M. Finkbeiner

4.6.1 Background and Motivation The LCA-related ISO standards have been in permanent development (Finkbeiner et al. 2006; Finkbeiner 2013, 2014; Klüppel 2002; Lecouls 1999; Marsmann 1997, 2000) since new applications of LCA emerge and even companies and other large economic/ecologic systems are considered as subjects of life cycle–based assessments. Such an extension of the product-level (micro) LCA to company-level (meso) LCA to national/global (macro) LCA has been proposed as one result of the CALCAS project (Zamagni et al. 2009). Another extension of LCA aims at Life Cycle Sustainability Assessment (LCSA) (Finkbeiner et al. 2010; Klöpffer 2003, 2008; UNEP/SETAC 2011). These developments increase the pressure for more and better data, methods, etc. Quality assurance becomes even more important, and the critical review is in the centre of interest. As shown in Sec. 4 and 5, the critical review has been part of the ISO standards 14040ff from the beginning of LCA standardization, albeit in very concise form. Therefore, there is hardly any guidance in the standards on how the review should be done (Curran and Young 2014). This gap has now been closed by the development of the ‘Technical specification (TS)’ (ISO 2014). This TS is not a stand-alone document, but adds additional requirements and guidelines to ISO 14044 (ISO 2006b). Despite the concise content in the core standards of LCA, the critical review element was quite successful in the LCA community and a common practice emerged in the market place that satisfied all stakeholders. One of the key features is that the critical review system is not built around an accreditation scheme, but quality assurance by the credibility of the individual reviewer. The reviewer is personally accountable for the work. In contrast, accreditation schemes try to ensure quality by bureaucracy (Finkbeiner 2013). While this critical review scheme is generally accepted in the LCA world, it was a challenge to defend it against the bloated verification approaches of other schemes which sometimes have even three separate standards: one for the verification process, one for verification bodies and one for the competence of verifiers. As an example, several different conformity assessment options were discussed for the carbon footprint standards or other upcoming labelling initiatives. Critical review according to ISO 14040and ISO 14044 was in a sense competing with verification according to ISO 14025, but also the bureaucratic accountant approach of greenhouse gas verification (Finkbeiner 2014). To fill the gap compared to alternative schemes, the basic idea of ISO/TS 14071 was to document the well-established critical review process and practice in a more formal way. The intention was to provide additional requirements and guidelines for conducting a critical review and the competencies required. The idea to develop the Technical Specification ISO 14071 ‘Environmental Management -Life Cycle Assessment – Requirements and Guidelines for Critical Review Processes and Reviewer Competencies’ was introduced by Finkbeiner in his role as chair of the ISO committee of LCA(ISO TC207/SC5) during the 2010

4  Quality Assurance by International Standards: The ‘Critical Review’

67

TC207 plenary meeting in Leon, Mexico. The committee unanimously agreed to set up a task group to work on the scope and outline of such a document. After a survey for participation and leadership, Philippe Osset (France) was selected as convenor, Chen Liang (China) asco-convenor and the secretariat was provided by Mélanie Raimbault (AFNOR – France). The task group completed its work and presented its report of the 2011 meeting of ISO/TC 207/SC5 held in Oslo. The committee adopted the convenor’s report, which concluded with the recommendation to submit a new work item proposal (NWIP) based on the outline developed by the task group. After advice from the Portfolio Task Group of TC207, the NWIP was distributed to the national member bodies for voting.

4.6.2 Standard Development Process The actual standardization process started with the positive vote on the new project. The new work item was accepted in February 2012 with an almost unanimous vote (only the Netherlands and USA disapproved). A new working group (WG9) was established. The leadership team of the preparatory task group was selected to continue by leading WG9. The secretariat was still provided by AFNOR (Mélanie Raimbault was replaced by Sandrine Delalieux during the process). The first WG meeting was held in Bangkok in June 2012 and resulted in the first working draft (WD1). In November 2012, the second WG meeting was held in Tokyo and produced WD2. In March 2013, the third WG meeting in Paris reached a relatively mature WD3, which represented already a good share of the final document. After some further changes during the fourth WG meeting in Gaborone in June 2013, it was decided that the document is ready to be submitted for voting. For Technical Specifications, the ISO Directives foresee only one round of voting, if the document meets the acceptance criteria. For ISO/TS 14071, this was no problem, because the vote was almost unanimous (only Netherlands disapproved). A final WG meeting was held in Berlin in December 2013. The purpose of this meeting was to discuss the final remaining comments received during the vote and to do some editorial polishing. Finally, ISO/TS 14071 was officially published by ISO Central Secretariat in May 2014. In conclusion, the process of ISO/TS 14071 was an overall positive example of international standardization. The working group had a relatively small, but consistent membership with global representation and the relevant competencies on board. The active contribution from SETAC through a liaison mirror group represented in the WG by Chris Koffler was also appreciated. The WG leadership ensured a proper process, which resulted in the completion of the standard ahead of the usual deadline for standards.

68

W. Klöpffer and M. Finkbeiner

4.7 Overview of ISO TS 14071 (2014) 4.7.1 General ISO TS 14071 is a slim 11-page document (ISO 2014). It consists of a foreword, introduction, a main part with five chapters and two (informative) annexes. As said in the introduction, the TS addresses especially the critical reviews of LCAs containing comparative assertions for which a critical review is mandatory. Recent practice showed that also many LCA studies without this imperative request are critically reviewed according to the ISO standards, mostly in the less demanding form by independent internal or external experts. For these voluntary reviews, a more detailed description of the review process is useful as well. An important statement in the introduction points to a peculiarity of the critical review of LCAs relating to the status of reviewers: It is one of the key features of critical review that it does not relate to an accreditation scheme, but ensures quality by making the individual reviewer personally responsible for the work and by giving priority to the content rather than the form.

According to the introduction, other LCA-related standards are likely to benefit from this TS: This Technical Specification might be applicable to other standards that require independent review of LCA-based procedures and information (e.g. ISO 14045 (ISO 2012a), ISO 14025 (ISO 2006c), ISOTS 14067 (ISO 2012b)), but might need to be adapted to the specific fields of application. Other reference standards can be included in the critical review process.

It would have been useful to elaborate this last point in the main part of the TS, since the terminology is not used consistently in the various life cycle standards. In ISO 14025 ‘verification’ is often used in the sense of critical review (Grahl and Schmincke 2011; Schmincke and Grahl 2007). ISO 14045 uses the term ‘value’ for life cycle costing but also for other, unspecified types of value, but value is prohibited in ISO 14040+44 (at least for comparative assertions). Other forms of truncated LCAs (mostly called ‘footprints’) use only one impact assessment category. This would be no tragedy if these studies were used only for the purpose they have been created for; what we observe, however, is that especially the ‘carbon footprint’13 is misused as single impact pseudo-LCA for product comparisons. It is one duty of the critical review to hinder such undesirable trends in an early stage.

 Clearly a misnomer, since some potent greenhouse gases (GHG) do not have a carbon atom in the molecule, e.g. N2O, SF6, NF3. 13

4  Quality Assurance by International Standards: The ‘Critical Review’

69

4.7.2 Scope, Normative References and Terms and Definitions In the scope part, it is said that this Technical Specification provides the necessary information for carrying out a critical review for any type of LCA study and the necessary competencies of the reviewers. In particular, the following items are provided: • Details of a critical review process, including clarification with regard to ISO 14044:2006. • Guidelines to deliver the required critical review process, linked to the goal of the life cycle assessment (LCA) and its intended use. • Content and deliverables of the critical review process. • Guidelines to improve the consistency, transparency, efficiency and credibility of the critical review process. • The required competencies for the reviewer(s) (internal, external and panel member). • The required competencies to be represented by the panel as a whole. There are two normative references which are indispensable for the correct use of the TS: ISO 14040:2006 (ISO 2006a). ISO 14044:2006 (ISO 2006b). The TS includes 10 terms and definitions from (section 3.1) ‘independent internal expert’ (competent person, employed in a full-time or part-time role (by) the commissioner of the LCA study or by the practitioner of the LCA study, but not involved in defining the scope or conducting the LCA study) to (3.10) ‘reviewer’ (independent internal expert or independent external expert performing a critical review, or interested party taking part in a critical review panel). There are self-explaining definitions, such as ‘panel member’ (3.3): reviewer taking part in a critical review panel and ‘commissioner of the LCA study’ (3.4): organization (or group of organizations) that finances the LCA study according to ISO 14040 and ISO 14044. Not trivial and somewhat unexpected is definition (3.5) ‘commissioner of the critical review’: organization (or group of organizations) that finances the critical review of the LCA study according to ISO 14040 and ISO 14044. This definition tries to address the common role of commissioners of a critical review considering the different constellations possible. Typically, the commissioner of the study also commissions the critical review. However, the contracts for the reviewers are not always directly made with the study commissioner, because the reviewers are sometimes contracted by the practitioner (e.g. consultant) of the study. As in such a case, the review fees are included in the practitioner’s contract and the term financing was chosen in the definition. A third and relatively new constellation is also covered by this definition. Recently, for example, as part of governmental and eco-labelling initiatives, the commissioner of the study might be a

70

W. Klöpffer and M. Finkbeiner

private organization while the critical review is commissioned by a public body. It is near consensus that the interactive review, first proposed by SETAC (SETAC 1993), is advantageous and should be chosen whenever possible (Klöpffer 2005, 2012). The reporting of the results is split into the critical review report (3.7) and the critical review statement (3.8).The first is a documentation ‘of the critical review process and findings, including detailed comments from the reviewer(s) or the critical review panel, as well as corresponding responses from the practitioner of the LCA study.’ The critical review statement (3.8), on the other hand, is a ‘conclusive document aggregating the conclusions from the reviewer(s) regarding the LCA study, and stating unambiguously whether the LCA study is in conformance with ISO 14040 and ISO 14044’. Finally, there is a definition of an ‘interested party’ (3.9), as an ‘individual or group concerned with or affected by the environmental performance of a product system, or by the results of the life cycle assessment.’

4.7.3 Defining the Scope of the Critical Review Section 4 of the TS ‘Critical review process and tasks’ is the centrepiece of the standard 14071 and contains most of the relevant information. It is therefore treated here and in Sect. 4.8 in due detail. The title of this section corresponds to that of 4.1 in the technical specification. Reading the following sections should increase the understanding of the TS, but not replace a careful reading of the original text. Standards are written in a style which is suitable for unambiguous statements, and therefore often not an easy reading. Section 4.1 of the TS starts with definitions out of ISO 14044 which are known to any person who has critically reviewed LCA studies. Please note that ‘LCA report’ and ‘study report’ are used synonymously and may contain confidential material not or not completely to be used in a third party report according to ISO 14044 (5.2). ISO 14044:2006 offers only two ways of conducting a critical review: It can be based on expert review, ISO 14044 (6.2) or panel review, ISO 14044 (6.3). Of course, the members of a panel should also be experts, but not all have to be. For both types of review, the TS offers in addition three features which can be used to structure and set up the review process. The review may be performed concurrently or at the end of the study. The first choice has already been proposed by SETAC (1993) as ‘interactive review’ and by Klöpffer (2005, 2012) as ‘accompanying review’. The advantages of the first approach are clearly explained in the TS. They consist in the possibility of an early warning by the reviewers, thus reducing the length of the final phase of critical reviews considerably. The only advantage of the alternative (‘a posteriori’) is that it can be applied to a finished LCA study.

4  Quality Assurance by International Standards: The ‘Critical Review’

71

A moderate form of a review at the end is starting with the draft final report of the LCA study. 1. The review includes or excludes an assessment of the life cycle inventory (LCI) model. Since LCI is an integral part of each LCA according to ISO 14040 and 14044 (ISO 2006a, b), it is difficult to understand how this basis can be omitted from scrutiny. The intention is to clearly document, whether the review just assessed the LCI documentation as given in the report or whether the review had access to and performed plausibility checks on the LCI model itself. Another plausible reason could be that the LCI had been reviewed before and the results are accessible to the review-team. 2. The review includes or excludes an assessment of individual data sets. For reasons of confidentiality, data are often provided by industry only if there are at least three independent sources so that an anonymous average can be calculated. Such data are very useful as generic data, but also for specific LCA studies. This item (3) gives more flexibility to the process of critical review. There are some statements known from base standards about the independency of the reviewers and the impossibility to verify or validate the goals of an LCA chosen by the commissioner. How is this to understand? Value statements are not scientific but may be ethically relevant. A reviewer who cannot agree with the goal of a study should explain his/her reservation and refuse to take part in the review.

4.7.4 Selecting, Contracting and Replacing External Reviewers The content of this section is described in ISO 14040 and 14044 only marginally and justifies a clear presentation in the TS. With regard to the selection of an internal or external independent expert, both the commissioner or the practitioner may invite a suitable person and additional experts may be included. This has been established practice,14 but now it is acknowledged in the TS. In contrast to the panel method, there is no chairperson. With regard to the panel method, the commissioner of the LCA should select an external independent expert to act as chairperson of the review panel. This person should select other independent qualified reviewers and may include interested parties. The commissioner and or the practitioner may propose suitable persons to the chairperson. This practice, too, has been established and is now codified in the TS. Since the panel members get a lot of confidential information during the review process, it is clear that an atmosphere of mutual trust must be created. This does not exclude controversial discussions about the methods used (or not used) by the practitioner, but the basis has to be constructive (as clearly requested by this TS).

 Some commissioners do not care about the critical review, but order it together with the study report. 14

72

W. Klöpffer and M. Finkbeiner

A novelty in this TS concerns a ‘self-declaration’ which shall be submitted by the panel members before the contracting (an example is given in TS, Annex B). By this measure, misuse of the review should be prevented. Contracting is a must for external experts but facultative for independent internal experts and interested parties. Contracting often includes signing a confidentiality agreement. Rules for the replacement of reviewers or the chairperson are given in section 4.2.3 of the TS; the leading role plays the commissioner of the LCA study. Any such events have to be documented in the critical review report and statement.

4.8 The Review Procedure According to ISO TS 14071 4.8.1 Type of Critical Review The general introduction given in section 4.3.1 of the TS recommends a close cooperation between the review team (panel members/chair), the commissioner and the practitioner15 (see also the ‘critical review triangle’, Klöpffer 2012). Phone/web conferences and physical meetings are encouraged as well as the exchange of draft reports (including the draft critical review statement). Clearly, neither the commissioner nor the practitioner should rewrite the critical review (CR) statement, but get a fair chance for suggesting minor improvements. If major objections with regard to the CR statement persist, a reply can be written by the commissioner and/or the practitioner, which will be part of the study report as the CR statement (ISO 2006b). The last sentence of section 4.3.1 reads: All parties involved should strive to establish conformance with ISO 14040, ISO 14044 or this Technical Specification by working together constructively and cooperatively.

The procedure of CR is described for the two typical ways of producing a critical review statement: ‘at the end of the LCA study’ or ‘concurrent’ with the study – the interactive critical review requested by SETAC (1993) and Klöpffer (2005, 2012). In the first case, the draft final LCA report is the ideal starting point, in the second it is the draft Goal and Scope chapter. A review of the finished and approved LCA study report16 (truly ‘a posteriori’) is possible, too, but has the disadvantage that changes in response to any criticism would require a re-opening of the life cycle assessment process. The final critical review comments will always relate to the last version of the report submitted by the practitioner to the reviewer(s). This version is in general approved by the commissioner so that the comments by the reviewers are the last

 There are cases where commissioner and practitioner are identical.  Such an LCA report cannot be published if it contains comparative assertions, but it may exist as an internal document in a company or authority. 15 16

4  Quality Assurance by International Standards: The ‘Critical Review’

73

ones to be taken into account by the practitioner (and the commissioner, who might be interested in the final formulation of the critical review report). The following milestones for comments in a concurrent critical review are suggested in the TS (section 4.3.4): • • • • • •

The goal and scope definition. Inventory analysis, including data collection and modelling. Impact assessment. Life cycle interpretation. Draft LCA report. The critical review statement shall be issued for the final version of the LCA report.

This procedure ideally fits the ISO structure of LCA, but would mean four intermediate and one final critical review report (+ the CR statement at the end). With a minimum of three reviewers in a panel, this may surpass the budget in many cases, especially if phone conferences and/or face-to-face meetings are planned in order to discuss the results. In theory, however, this is the perfect way to proceed, and some big LCA studies may adhere to this schedule in real life.

4.8.2 Critical Review Report and Statement In order to prevent the misuse of the LCA study and of the CR report and statement, it has to be unambiguously defined to which version of the report (report number, date, etc.) the report and statement belong. In contrast to old-fashioned printed, signed and numbered documents, modern communication makes it easy to fake reports.17 As a principle, a CR statement and review report belongs only to one LCA study report which has to be clearly identified. In the case of an updated LCA study, the reviewers (either new ones or the team of the original study) may refer to the previous CR and concentrate to the added items; the conformity of the updated report as a whole with the standards ISO 14040+44 has to be checked, though. Executive summaries, press releases and other truncated summaries should be submitted to the practitioner and to the reviewers ‘for feedback to ensure consistency with the LCA study’. A problem which is not addressed concerns the submission of a publication on the LCA study and the critical review to a not-peer-reviewed, often highly sector-specific journal.18 Such a paper may be more extensive than the ‘truncated summary’ mentioned in the TS. The problem of dissemination of such papers is recognized, though.

17 18

 Actually, such fakes may even be produced by error and mistake.  As the joke says: ‘Soap & Towel’.

74

W. Klöpffer and M. Finkbeiner

The critical review report shall contain all reviewer comments and recommendations and the responses by the practitioner of the LCA study (section 4.5). The critical review statement and the responses by practitioner and/or commissioner shall be included in the LCA study report as stipulated by ISO 14044. Due to the requirement of a detailed critical review report, the CR statement can be relatively short. Also the CR report can be shortened, if the team agrees, by omitting language corrections and other editorial comments or ‘inadvertent mistakes’. In practice, this will be done by templates used by the chairperson and the other members of the CR panel. The practitioner answers either by pointing out where in the report modifications have been done or by a response to the reviewers. With regard to the content of the critical review statement, the TS gives the following advice: The critical review statement may highlight any particular strengths, limitations and remaining improvement potentials of the LCA study or the critical review process.

With regard to signing the statement, it is said that the chairperson shall sign and the other reviewers should sign it. In the review by independent experts, these shall sign as individual(s), not representing any organization(s). This is in perfect agreement with the special feature of the critical review (as opposed to certification and verification schemes), the personal responsibility of the reviewers stated in the introduction. The TS continues in section 4.5 with a detailed list of items that have to be reported in a critical review statement in order to exactly identify the study to which it belongs and the exact nature of the CR, for example performance in parallel or at the end of the LCA study, inclusion or exclusion of checking the LCI model and/or individual data sets and, finally, whether the study was found to be in conformance with ISO 14040 and ISO 14044 or not. Most of these requests have been good CR practice in recent years, but here they are presented in an authoritative text which should improve the praxis of CR. How to deal with dissenting reviewers in a panel if no compromise was found19? Also good practice, but never written down: he or she may write a minority statement and the chairperson shall include it into the critical review statement. A statement of non-conformance shall be based exclusively on the failure to meet one or more of the requirements of ISO 14044:2006, [ISO 2006b] section 6.1. This section (‘General’) reads as follows: The critical review process shall ensure that

• the methods used to carry out the LCA are consistent with this International Standard, • the methods used to carry out the LCA are scientifically and technically valid, • the data used are appropriate and reasonable in relation to the goal of the study, • the interpretations reflect the limitations identified and the goal of the study, and, • the study report is transparent and consistent. 19

 It is the duty of the chairperson to suggest compromise formulations.

4  Quality Assurance by International Standards: The ‘Critical Review’

75

The scope and type of critical review desired shall be defined in the scope phase of an LCA, and the decision on the type of critical review shall be recorded. In order to decrease the likelihood of misunderstandings or negative effects on external interested parties, a panel of interested parties shall conduct critical reviews on LCA studies where the results are intended to be used to support a comparative assertion intended to be disclosed to the public.

These sentences, especially the bullets, belong to the most cited ones in critical review reports and statements. They nevertheless contain a flaw in the form of a vague formulation with regard to the ‘interested parties’, taken from the first series of ISO LCA standards (Klöpffer 2005, 2012). Their inclusion is not obligatory, as the formulation ‘a panel of interested parties’ suggested. In the TS, this formulation is not used anymore, but simply the ‘panel review’. The ‘interested parties’ are clearly an optional variant of the panel method. In the review statement ‘it may be stated which interested parties were involved in the critical review process’. The inclusion of interested parties is certainly easier if the commissioner of the LCA study and/or of the critical review is an industrial association or an environmental agency rather than a single company.

4.8.3 Critical Review Tasks 4.8.3.1 The Chairperson The main duties of the chairperson consist of (for details see ISO 2014, 4.7.1) • Setting up a competent and independent critical review panel (minimum 2 persons in addition to the chair). • Distributing the tasks and coordinating the whole review process; ensuring that all panel members have a common understanding of the tasks… including the fact that the comments should be based on ISO 14040 and ISO 14044. • Communication within the panel and with the practitioner, resolving conflicting statements (alternative: minority statement by deviating member). Creating a good relationship between the members and with practitioner and commissioner (‘critical review triangle’, Klöpffer 2012). • Ensuring that the critical review report and the critical review statement are generated and approved by the panel. The procedure of the critical review is highly dependent on the progress made by the practitioner whose work depends on the financing by the commissioner. It happens from time to time that a concurrent critical review is interrupted or even finished before the task has been achieved. The chairperson should be able to deal with difficult situations, although no general rule can be given for such events.

76

W. Klöpffer and M. Finkbeiner

4.8.3.2 The Reviewer It should be remembered that also the chair is a reviewer with a certain range of knowledge (mostly the LCA method in great detail, but it may also be the system(s) to be studied or their environmental relevance). It is only the management duty described in section 7.3.1, which distinguishes his/her position in the panel. The reviewer (see section 4.7.2 in ISO 14071) • Shall provide comments, including justifications regarding the whole content of the report. • May use the template (see Annex A) when providing the comments. • Shall contribute to the critical review report. • Shall express agreement or disagreement on the critical review statement and provide the reasons for disagreement: justifications shall be based exclusively on the requirements of the relevant ISO standards, especially 14044. A real critical review may be much more detailed than expressed in these minimum requirements, especially since there are several critical review steps in the case of a concurrent review. The interaction between the chairperson and the other reviewers, but also of the whole panel with practitioner and commissioner (in case of meetings) may be intense and contribute to the success of the review and  – finally – of the LCA study as a whole. The interaction should not become too close, however, so that the critical distance between the review team on the one hand and the practitioner is not lost.

4.9 Competencies of the Reviewers The first sentence in chapter 5 (ISO 2014) reads: The reviewer(s) shall be familiar with the requirements of LCA according to ISO 14040 and ISO 14044, or shall have the appropriate scientific and technical expertise.

The ‘or’ indicates that there may be two types of reviewers in the critical review panel:20 LCA experts and product/system/data experts. This view is confirmed by an important statement below the list of required competencies in the same chapter 5: The set of competencies can be provided by different panel members or a group of experts, apart from proficiency in the language used for the study.

The list of required competencies (shall have knowledge of, and proficiency in) contains on first place ISO 14040 and ISO 14044. As shown in 7.2, the main task of the review panel is to confirm that the LCA study was performed according to the relevant standards.

20

 The same is true for critical reviews performed by independent experts.

4  Quality Assurance by International Standards: The ‘Critical Review’

77

The next item is the LCA methodology and current practice, followed by critical review practice. There is a lot of environmental science contained in the next requirement, taken from ISO 14044: the scientific disciplines relevant to the important impact categories of the study.

Actually, the ISO standards do not provide a list of impact categories, indicators and characterisation factors. This means that at least in theory such a list – tailor-made for the problem at hand – has to be provided by the practitioner. There are books and websites recommending the important impact categories and the most common methods for quantification (e.g. Guinée et al. 2002), but the final responsibility rests with the author(s) of the study. The reviewer(s) (at least the LCIA specialist) has to be able to follow the arguments put forward by the practitioner. Technical knowledge is claimed for the next requirement: environmental, technical and other relevant performance aspects of the product system(s) assessed.

Finally, the reviewer has to master the language used for the study. Additionally, further competencies may be required, depending on the goal and scope of the study. The reviewer shall demonstrate his/her qualification by means of a curriculum vitae and a list of relevant publications. In addition, a self-declaration shall be requested by the organization that contracts the reviewer.21 An example of a self-­ declaration is provided in Annex B of TS 14071.

4.10 Conclusion In the early days of LCA, the credibility of the method was damaged by wrong claims about the environmental superiority of products. LCA was partly misused as a kind of ‘greenwashing machine’. The establishment of the international standards of Life Cycle Assessment-LCA (ISO 14040 series) helped to overcome some of this misuse and led to the acceptance of life cycle assessment all around the world and by all stakeholders. The conformity assessment built into ISO 14040 and ISO 14044 (2006), the critical review, has ensured the quality and credibility of LCA studies, especially for the most contentious application, namely the studies for comparative assertions intended to be disclosed to the public.

 In general, this is the commissioner, but the contracting may be delegated, for example, to the practitioner. 21

78

W. Klöpffer and M. Finkbeiner

So far, the critical review has been a success story. The process was fine-tuned and better documented by the recent ISO/TS 14071. The future of critical review looks promising as there is an increasing demand for critical reviews. However, this depends on the quality of the critical reviews which has to be ensured.

4.11 Outlook In order to continue the quality control of Life Cycle Assessment studies we need an open discussion on the international standards described above and their improvement where necessary. As one step in this direction, a new section was created in the Int J Life Cycle Assess (Klöpffer 2017): Critical Review and LCA Standards.

Whereas the development of new LCA methods, especially with focus on Life Cycle Inventory Analysis (LCI) and Life Cycle Impact Assessment (LCIA), is well documented, quality assurance including the critical review is presented much less frequently. We hope that the new section attracts submissions, hopefully also about the standard ISO TS 14071 (2014). Open and critical discussion is the fundament to any scientific progress and the basis for improvements. As of 2023, ISO TS 14071 is  revised into a full standard, while the main content of TS is expected to be confirmed.

References Ayres RU, Klöpffer W, Lindfors L-G (1996) Critical review report of the study ‘A PVC substance flow analysis for Sweden’. Stockholm Boustead I, Hancock GF (1979) Handbook of industrial energy analysis. Ellis Horwood Ltd, Chichester Curran MA, Young SB (2014) Critical review: a summary of the current state-of-practice. Int J Life Cycle Assess. https://doi.org/10.1007/s11367-­014-­0778-­218 Fava JA, Denison R, Jones B, Curran MA, Vigon B, Selke S, Barnum J (eds) (1991) SETAC workshop report: a technical framework for life cycle assessments, August 18–23 1990, Smugglers Notch, Vermont. SETAC, Washington, DC Fava J, Weston RF, Jensen AA, Lindfors L, Pomper S, De Smet B, Warren J, Vigon B (eds) (1994) Life-cycle assessment data quality: a conceptual framework. Workshop report. SETAC and SETAC Foundation for Environmental Education. Wintergreen, Virginia, October 1992 Fava JA, Smerek A, Heinrich AB, Morrison L (2014) Chapter 2: the role of the Society of Environmental Toxicology and Chemistry (SETAC) in life cycle assessment (LCA), development and application. In: Klöpffer W (ed) LCA compendium  – the complete world of life cycle assessment, Vol 1: background and future prospects in life cycle assessment. Springer, Dortrecht, pp 39–84 Finkbeiner M (2013) From the 40s to the 70s  – the future of LCA in the ISO 14000 family. Editorial Int J Life Cycle Assess 18(1):1–4

4  Quality Assurance by International Standards: The ‘Critical Review’

79

Finkbeiner M (2014) Chapter 3: the international standards as constitution of life cycle assessment: the ISO 14040 series and its offspring. In: Klöpffer W (ed) LCA compendium  – the complete world of life cycle assessment: background and future prospects in life cycle assessment. Springer, Dordrecht, pp 85–106 Finkbeiner M, Inaba A, Tan RBH, Christiansen K, Klüppel H-J (2006) The new international standards for life cycle assessment: ISO 14040 and ISO 14044. Commentary Int J Life Cycle Assess 11(2):80–85 Finkbeiner M, Schau EM, Lehmann A, Traverso M (2010) Towards life cycle sustainability assessment. Sustain For 2:3309–3322.; Open access. https://doi.org/10.3390/su2103309 Franke M (1983) Umweltauswirkungen durch Getränkeverpackungen: Systematik zur Ermittlung der Umweltauswirkungen von Prozessen am Beispiel von Einweg- und Mehrweg-­ Getränkebehältern. Dissertation, Technical University Berlin Grahl B, Schmincke E (2011) ‘Critical review’ and ‘Verification’ cannot be used synonymously. A plea for a differentiated and precise use of the terms. LCM conference Berlin 2011. Download from http://www.lcm2011.org/papers.html. Session: Critical Review and Verification of LCA Guinée JB (final editor), Gorée M, Heijungs R, Huppes G, Kleijn R, Koning Ade, Oers L van, Wegener Sleeswijk A, Suh S, Udo de Haes HA, Bruijn H de, Duin R van, Huijbregts MAJ (2002) Handbook on life cycle assessment – operational guide to the ISO standards. Kluwer Academic Publication, Dordrecht. ISBN 1-4020-0228-9 Heijungs R, Guinée JB, Huppes G, Lamkreijer RM, Udo de Haes HA, Wegener Sleeswijk A, Ansems AMM, Eggels PG, van Duin R, de Goede HP (1992) Environmental life cycle assessment of products. Guide (Part 1) and backgrounds (Part 2) October 1992, prepared by CML, TNO and B&G. Leiden 1992. English Version 1993 Heinrich AB (2014) Chapter 5: life cycle assessment as reflected by the International Journal of Life Cycle Assessment. In: Klöpffer W (ed) LCA compendium – the complete world of life cycle assessment, Vol 1: background and future prospects in life cycle assessment. Springer, Dordrecht, 145–188 Hunt R, Franklin WE (1996) LCA – how it came about. Personal reflections on the origin and the development of LCA in the USA. Int J Life Cycle Assess 1(1):4–7 ISO (1997) International Standard Organization: environmental management – life cycle assessment – principles and framework. ISO 14040 ISO (1998) International Standard Organization: environmental management – life cycle assessment: goal and scope definition and inventory analysis. ISO 14041 ISO (2000a) International Standard Organization: environmental management – life cycle assessment: life cycle impact assessment. ISO 14042 ISO (2000b) International Standard Organization: environmental management – life cycle assessment: interpretation. ISO 14043 ISO (2006a) International Standard Organization TC 207/SC 5: environmental management – life cycle assessment – principles and framework. ISO 14040:2006 ISO (2006b) International Standard Organization TC 207/SC 5: environmental management – life cycle assessment – requirements and guidelines. ISO 14044:2006 ISO (2006c) International Standard Organisation ISO/TC 207/SC3: environmental management – environmental labels and declarations – Type III environmental declarations – principles and procedures ISO 14025:2006 ISO (2012a) International Standard Organisation ISO/TC 207/SC5/WG7: environmental management – eco-efficiency assessment of product systems – principles, requirements and guidelines ISO 14045:2012 ISO (2012b) International Standard Organisation ISO/TC 207/SC7: carbon footprint of products  – requirements and guidelines for quantification and communication ISO/DIS 14067: 2012 (Draft) ISO (2014) International Standard Organization TC 207/SC 5: environmental management – life cycle assessment – critical review processes and reviewer competencies – additional requirements and guidelines to ISO14044:2006, ISO/TS 14071:2014

80

W. Klöpffer and M. Finkbeiner

Janzen DC (1995) Methodology of the European surfactant life-cycle inventory for detergent surfactants production. Tenside Surfactant Deterg 32:110–121 Klöpffer W (2003) Life-cycle based methods for sustainable product development. Int J Life Cycle Assess 8(3):157–159 Klöpffer W (2005) The critical review process according to ISO 14040-43: an analysis of the standards and experiences gained in their application. Int J Life Cycle Assess 10(2):98–102 Klöpffer W (2008) Life-cycle based sustainability assessment of products. Int J Life Cycle Assess 13(2):89–94 Klöpffer W (2009) Experiences with the critical review process of aluminium LCI data. Int J Life Cycle Assess 14(Suppl 1):45–51 Klöpffer W (2012) The critical review of life cycle assessment studies according to ISO 14040 and 14044: origin, purpose and practical performance. Int J Life Cycle Assess 17(9):1087–1093 Klöpffer W (2014) Chapter 1: introducing life cycle assessment and its presentation in ‘LCA compendium’. In: Klöpffer W, Curran MA (eds) LCA compendium – the complete world of life cycle assessment, Vol ‘Background and future prospects in life cycle assessment’ (Klöpffer W, ed). Springer, Dortrecht, pp 1–38 Klöpffer W (2017) Introducing a new section in the international journal of life cycle assessment: critical review and life cycle assessment standards. Int J Life Cycle Assess 22(7):1015–1016 Klöpffer W, Grahl B (2014) Life cycle assessment (LCA) – a guide to best practice. Wiley-VCH, Weinheim Klöpffer W, Grießhammer R, Sundström G (1995) Overview of the scientific peer review of the European life cycle inventory for surfactant production. Tenside Surfactant Deterg 32:378–383 Klöpffer W, Sundström G, Grießhammer R (1996) The peer reviewing process  – a case study: European life cycle inventory for surfactant production. Int J Life Cycle Assess 1(2):113–115 Klüppel H-J (2002) The ISO standardization process: Quo Vadis? Int J Life Cycle Assess 7(1):1 Koffler C (2013) Regarding your article ‘The critical review of life cycle assessment studies according to ISO 14040 and 14044 – origin, purpose and practical performance’. Int J Life Cyle Assess (2012) 17:1087–1093. Comment. Int J Life Cycle Assess 18(2):300–301 Lecouls H (1999) ISO 14043: environmental management  – life cycle assessment  – life cycle interpretation. Editorial. Int J Life Cycle Assess 4(5):245 Lichtenvort K (2005) Comment on ‘the critical review process according to ISO 14040-43: an analysis of the standards and experiences gained in their application. Int J LCA 10(2):98–102 (2005)’. Int J Life Cycle Assess 10(4):232 Lindfors L-G, Christiansen K, Hoffmann L, Virtanen Y, Juntilla V, Hanssen O-J, Rønning A, Ekvall T, Finnveden G (1995) Nordic guidelines on life-cycle assessment. Nordic Council of Ministers. Nord 1995:20. Copenhagen 1995 Marsmann M (1997) ISO 14040 – the first project. With a foreword by Merkel A. Int J Life Cycle Assess 2(2):121–123 Marsmann M (2000) The ISO 14040 family. Int J Life Cycle Assess 5(6):317–318 Schmincke E, Grahl B (2007) The part of LCA in ISO type III environmental declarations. Supplement issue 1. Int J Life Cycle Assess 12:38–45 SETAC (1993) Society of environmental toxicology and chemistry: guidelines for life-cycle assessment: a ‘code of practice’. From the SETAC Workshop held at Sesimbra, Portugal, 31 March – 3 April 1993, 1st edn, 69 pp, Brussels and Pensacola (Florida), August Sonnemann G, Vigon B (eds) (2011) Global guidance principles for life cycle assessment databases – ‘Shonan guiding principles’. UNEP/SETAC Life Cycle Initiative, Paris, 156 pp. ISBN: 978-92-807-3174-3. Download from http://lcinitiative.unep.fr Sonnemann G, Vigon B, Broadbent C, Curran MA, Finkbeiner M, Frischknecht R, Inaba A, Schanssema A, Stevenson M, Ugaya C, Wang H, Wolf MA, Valdivia S (2011) Process on ‘global guidance for LCA databases’. Int J Life Cycle Assess 16:95–97 Stalmans M, Berenbold H, Berna JL, Cavalli L, Dillarstone A, Franke M, Hirsinger F, Janzen D, Kosswig K, Postlethwaite D, Rappert T, Renta C, Scharer D, Schlick K-P, Schul W, Thomas H, Van Sloten R (1995) European life-cycle inventory for detergent surfactants production. Tenside Surfactant Deterg 32:84–109

4  Quality Assurance by International Standards: The ‘Critical Review’

81

Tukker A (1998) Frames in the toxicity controversy. Based on the Dutch chlorine debate and the Swedish PVC debate. Proefschrift (PhD thesis). Universal Press, Veenendaal, The Netherlands, September Tukker A, Kleijn R, Oers L van, Smeets ERW (1996) A PVC substance flow analysis for Sweden. Report for Norsk Hydro by TNO Centre for Technology and Policy Studies and Centre of Environmental Science (CML) Leiden, Apeldoorn, November UNEP/SETAC Life Cycle Initiative (2011) Valdivia S ‘Towards a life cycle sustainability assessment. Making informed choices on products’. UNEP/SETAC Life Cycle Initiative, Paris. ISBN: 978-92-807-3175-0. Download from http://lcinitiative.unep.fr Zamagni A, Buttol P, Buonamici R, Masoni P, Guinée JB, Huppes G, Heijungs R, van der Voet E, Ekvall T, Rydberg T (2009) Co-ordination action for innovation in life-cycle analysis for sustainability. D20 blue paper on life cycle sustainability analysis. Deliverable 20 of work package 7 of the CALCAS project. Revision 1 after the open consultation, August. http://www.estis.net

Chapter 5

The Terms “Critical Review” and “Verification” Addressing Quality Assurance in Life Cycle Assessment and ISO Type III Declarations Birgit Grahl and Eva Schmincke Abstract  Quality assurance in Life Cycle Assessment (LCA) is a conformity assessment against defined criteria that are suitable to reflect robustness and significance of the results. In LCA, “critical review” and “verification” are used for quality assurance. The chapter interprets the quality assurance via critical review, the quality assurance via verification, and finally verification and PCR (product category rules) review. “Critical review” was introduced in ISO 14040: 1997, adopted in ISO 14040:2006, ISO 14044:2006, that is, by the revision of ISO 14040: 1997, and specification is given in (ISO/TS 14071:2014). In the context of quality assurance in LCA and related tools like ISO type III declarations according to ISO 14025:2006, occasionally the term “verification” is used. This term was significantly coined in Quality Management Systems and defined in ISO 9000:2015. The definition of verification from ISO 9000:2000, identical with the definition in ISO 9000:2015, was transferred to ISO 14025:2006. The verification approach and the peer review approach are different in methodology. The verification process leaves an extensive range of possibilities to define objective evidence. Every document, however, using the term “verification” must precisely define the requirements correlated with the respective objective evidences. The basic idea of Environmental Product Declarations (EPD) according to ISO 14025 is to provide liable and relevant information on environmental issues for companies in the chain management (Type III environmental declarations).

B. Grahl (*) Industrielle Ökologie, Heidekamp, Germany e-mail: [email protected] E. Schmincke Tübingen, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_5

83

84

B. Grahl and E. Schmincke

Keywords  Chain management · Conformity criteria · Critical review · Environmental product declarations · EPD · ISO 14025:2006 · ISO 14040: 1997 · ISO 14040:2006 ISO 14044:2006 · ISO 9000:2000 · ISO 9000:2015 · Objective evidence · PCR · PCR · Peer review · Product category rule · Product category rules · Quality management · Type III environmental declarations

5.1 Introduction Quality assurance of Life Cycle Assessment (LCA) studies increases reliability and credibility of the results. In LCA studies according to (ISO 14040:2006) and (ISO 14044:2006), all four phases (goal and scope definition, inventory analysis, impact assessment, interpretation) need to be addressed and must also be reflected in quality assurance. Methods chosen for quality assurance must therefore be suitable to consider all methodological requirements, which are methods and assumptions, calculation procedures and numerical results, the interpretation, and, last but not least, a consistent elaboration of all issues. Just like the iterative approach of the elaboration process of an LCA, the quality assurance also needs an iterative perspective. In general, quality assurance has the character of a conformity assessment against defined criteria (ISO 9000:2015). Applying this principle in the context of LCA, the reference documents against which conformity is stated shall consider the entire system of an LCA and define reference criteria that are suitable to reflect robustness and significance of the results. In the LCA context, two terms are used for quality assurance: critical review and verification. The aim of this chapter is to scrutinize the usage of the terms concerning quality assurance in LCA with the objective of increasing transparency regarding conformity criteria.

5.2 Quality Assurance via Critical Review in Life Cycle Assessment “Critical review” was introduced in (ISO 14040: 1997), adopted in (ISO 14040:2006, ISO 14044:2006), that is, by the revision of (ISO 14040: 1997), and specification is given in (ISO/TS 14071:2014) (see Chap. 4 of this book by W.  Klöpffer and M. Finkbeiner). (ISO 14044:2006) defines, in paragraph 6.1, the following five questions to be answered by the critical review when stating conformity or nonconformity to ISO 14040 and ISO 14044. The critical review process shall ensure that: 1. the methods used to carry out the LCA are consistent with this International Standard, 2. the methods used to carry out the LCA are scientifically and technically valid,

5  The Terms “Critical Review” and “Verification” Addressing Quality Assurance…

85

3. the data used are appropriate and reasonable in relation to the goal of the study, 4. the interpretations reflect the limitations identified and the goal of the study and, 5. the study report is transparent and consistent.

These key questions have the character of conformity criteria to analyze a specific LCA study in the sense of a peer review, that is, the examination of a scientific work by independent experts in the same area of expertise. The necessity for the general character of those five criteria is that in ISO 14040 and 14044 (2006), methodological frame settings are given with a high degree of freedom. Adaption and specification of methodological setting are necessary in specific LCA studies. The high degree of freedom in the standards ensures their applicability for any LCA regardless of the product system investigated: In the LCA phase “goal and scope definition,” numerous specifications, for example, regarding system boundaries, allocation rules, considered impact categories, characterization models, or optional elements of impact assessment, have to be defined for a specific study. The review must investigate whether all specifications are in line with the requirements in ISO 14040 and 14044 (2006); all specifications must be comprehensively described and, possible consequences must be reflected in the interpretation stage of an LCA; and if necessary, sensitivity analyses are to be included. For the reviewer, all results must be plausible considering the specifications applied. It is usual practice for critical reviews that the correctness of all items of primary and other data cannot be checked because of the amount of data to be considered, but the data used and the results have to be reviewed for appropriateness, consistency, and plausibility. As a consequence, the critical review has three tracks of analysis: 1. Are all mandatory methodological requirements to conduct an LCA according to IS0 14040 and ISO 14044 (2006) implemented? 2. Are all specifications of a specific LCA Study in line with ISO 14040 and 14044 transparently and consistently handled in the specific study? This also includes value choices on which specifications may be based. 3. Are the results plausible under consideration of the specifications in the LCA study? These three tracks of analyses are relevant throughout all five questions mentioned above. In order to answer the questions the reviewer has to investigate the LCA study carefully, including data quality and the modeling of the product system. In a peer review, the conformity criteria for quality assurance can only be answered competently when the reviewer not only knows the standards but also has a good overview over the scientific and technical topic under investigation. Therefore, (ISO 14071:2014; chapter 5) defines: The reviewer(s) shall be familiar with the requirements of LCA according to ISO 14040 and ISO 14044, or shall have the appropriate scientific and technical expertise. The reviewer(s) shall have knowledge of, and proficiency in, the following: –– ISO 14040 and ISO 14044; –– LCA methodology and current practice, particularly in the context of LCI, (including data set generation and data set review, if applicable);

86

B. Grahl and E. Schmincke –– critical review practice; –– the scientific disciplines relevant to the important impact categories of the study; –– environmental, technical and other relevant performance aspects of the product system(s) assessed; –– the language used for the study …the set of competencies can be provided by different panel members or a group of experts, apart from proficiency in the language used for the study.

5.3 Quality Assurance via Verification In the context of quality assurance in LCA and related tools like ISO type III declarations according to (ISO 14025:2006) (see para 3), occasionally, the term “verification” is used. As discussed above, a clear definition of conformity criteria is needed for a specific application. The term “verification” was significantly coined in quality management systems and defined in ISO (9000:2015, 3.8.12): Verification confirmation, through the provision of objective evidence, that specified requirements have been fulfilled Note 1 to entry: The objective evidence needed for a verification can be the result of an inspection or of other forms of determination such as performing alternative calculations or reviewing documents. Note 2 to entry: The activities carried out for verification are sometimes called a qualification process. Note 3 to entry: The word “verified” is used to designate the corresponding status.

The definition and notes use quite unspecified terms, and hence, there is a large scope for interpretation. The key message is that “specific requirements” have to be fulfilled. Thus, the core task of documents using the term “verification” in the context of quality assurance is to define these requirements precisely. In order to decide if a requirement is fulfilled or not, the definition requires that “objective evidence” shall be used. Thus, both “specific requirements” and “objective evidence” must be precisely defined and correlated in reference documents against which a verification shall be performed. Without this correlated specifications, the verification process remains undefined. If the requirements and the respectively correlated objective evidence are defined, a checklist may be used as a tool in conformity assessment. Hence, for any application, the thorough elaboration of conformity criteria (requirements) and correlated proofs of conformity (objective evidence) is the central requirement if a document refers to “verification” as quality assurance tool. The above definition leaves an extensive range of possibilities to define objective evidence; nevertheless, in a verification process, a verifier needs a clear reference of conformity criteria and respective proofs. Objective evidence is defined in (ISO 9000:2015, 3.8.3):

5  The Terms “Critical Review” and “Verification” Addressing Quality Assurance…

87

objective evidence data (3.8.1) supporting the existence or verity of something Note 1 to entry: Objective evidence can be obtained through observation, measurement (3.11.4), test (3.11.8), or by other means. Note 2 to entry: Objective evidence for the purpose of audit (3.13.1) generally consists of records (3.8.10), statements of fact or other information (3.8.2) which are relevant to the audit criteria (3.13.7) and verifiable.

Thus, the definition including notes specifies the possibilities, but the large range of interpretation still remains. Nevertheless, it is important to point out that the definition refers to existence or verity. As a consequence, the wording of the requirements must consider that an objective evidence can be correlated (Fig. 5.1). However, “verity”  – which means something that is true  – is a widely discussed term in philosophy and hardly ever to be proofed. In the pragmatic context of “objective evidence,” the requirement must imply the precise definition of the issue that shall be declared as true, the precise definition which evidence shall be used to demonstrate verity and a precise description of the procedure to check. Figure 5.1 shows some examples. The advantage of such a general definition of objective evidence is that users of the quality management system are relatively free to define what is reasonable for their practice. The risk is that it may not be possible to define the objective evidence precisely enough for every requirement listed. For the examination (auditing) of quality management and environmental management systems, correlated lists of requirements and proofs (sometimes called auditing lists) are quite common. Definition of reqiurements

Wording must consider that an objective evidence can be correllated (Examples for environmental management system one site)

Objective evidence

Must demonstate existence or verity of fulfillment of requirement.

A clearly defined fraction of hazardous waste (waste type; share of moisture) shall be reduced by 5 w/w% compared to the previous year

 Reliable records of the specified waste fraction for this and the previous year must be available.  Metric (calculation of percentage)

 The COD* in effluent water shall be reduced by 5% compared to previous year

 Reliable measure results of COD for this and the previous year.  Metric (calculation of percentage)

 environmental policy is formulated

 analysis of existing documents (other means)  Yes or No

 In every hall of production at least one first aid kit is placed next to the machines, and be clearly visible.

 Observation  Yes or No

*Chemical Oxygen Demand

Fig. 5.1  Examples of correlated requirements and objective evidence in a verification system

88

B. Grahl and E. Schmincke

The verification approach and the peer review approach are different in methodology. For quality assurance of scientific works, the verification approach cannot cover the full complexity of such studies. The following simple example, referring to the first key question in the critical review process (see above Sect. 5.2), shows the difference: the methods used to carry out the LCA are consistent with this International Standard.

It is of course possible to check if “goal and scope” is defined. The answer will be “yes” or “no,” but this answer is not really useful to show the quality of a study. Going deeper into the methodological requirements of an LCA study, the functional unit must be defined. Again the question may be “yes” or “no,” and again this is trivial. It must be checked if the functional unit is defined according to the definition and explanations in the standard and additionally if the chosen functional unit is reasonable in the context of the goal and the intended application of the study. To answer this question, a clear objective evidence cannot be defined, but the peer reviewer (as defined above) needs profound methodological and technical knowledge of the subject. Every document, however, using the term “verification” must precisely define the requirements correlated with the respective objective evidences.

5.4 Verification and Product Category Rule Review in ISO 14025 The definition of verification from (ISO 9000:2000), identical with the definition in (ISO 9000:2015), but without the notes, was transferred to (ISO 14025:2006). The basic idea of Environmental Product Declarations (EPD), according to (ISO 14025), is to provide liable and relevant information on environmental issues for companies in the chain management (Type III environmental declarations). Because the declarations are published, also the business-to-consumer communication must be considered. The information in an EPD is structured as follows: For one part, an LCA study is conducted and the numerical results (impact category indicators and selected inventory data) are listed in a table. For the other part, according to ISO 14025, an EPD may include additional environmental information which is not covered by the listed LCA-results (e.g., emission of volatile substances in the use phase of a product according to a defined analytical method). This possibility is provided in paragraph 7.2.3 of ISO 14025. Generation and publication of EPD is organized in programs which state the required information declared in an EPD, the format as to how it is declared, and also the procedure for quality assurance according to the demands of ISO 14025 (Del Borghi 2013; Hunsager et al. 2014). Each program states program rules and methodological specifications for conducting an LCA according to (ISO 14040:2006) and (ISO 14044:2006) for specified product categories called Product Category Rules (PCR). The demands for additional information are also stated in

5  The Terms “Critical Review” and “Verification” Addressing Quality Assurance…

89

PCRs. In declaration programs, the underlying LCA is performed based on the provisions agreed upon in the PCR. The PCR document specifies goal and scope as well as the main assumptions governing the calculation rules, for example, the declared unit, system boundaries, allocation rules, impact categories including characterization models. The PCR may also give guidance for calculating averages if data of different companies are considered. The advantage of PCR is that results of different specific products within one product category (e.g., linoleum, PVC, cork or wood-floor covering with specified technical properties) are generated based on the same methodological specifications (e.g., PCR for floor coverings (IBU 2017). The results (LCA and additional information) of products under consideration are declared in one EPD for each product or average, thus a comparative assertion is explicitly not provided. Nevertheless, the declarations of different products based on the same PCR exist in parallel and a comparison is implicitly intended. If EPD are used for comparison, certain restrictions are to be respected.. The intention, when including PCRs into the procedure according to ISO 14025, is to simplify specific LCAs by predefining the issues that need to be specified in the first phase of an LCA “goal and scope definition.” ISO 14044 specifies in paragraph 4.2.3 the items that have to be considered, unambiguously stated, and clearly described to characterize the goal and scope: Goal: –– the intended application; –– the reasons for carrying out the study; –– the intended audience, i.e. to whom the results of the study are intended to be communicated; –– whether the results are intended to be used in comparative assertions intended to be disclosed to the public. Scope: –– the product system to be studied; –– the functions of the product system or, in the case of comparative studies, the systems; –– the functional unit; –– the system boundary; –– allocation procedures; –– LCIA methodology and types of impacts; –– interpretation to be used; –– data requirements; –– assumptions; –– value choices and optional elements; –– limitations; –– data quality requirements; –– type of critical review, if any; –– type and format of the report required for the study.

90

B. Grahl and E. Schmincke

This comprehensive list shows that all methodological aspects have to be addressed already in the first LCA phase. The level of detail that can be defined in a PCR depends on the definition of the product category. If the definition is narrow (e.g., linoleum floor covering for classrooms), the amount of specific products covered by the PCR is small, but the settings for the abovementioned methodological issues can be very precise. The degree of freedom setting system parameter is low, and quality control is simplified significantly (Fig. 5.2). If the definition of a product category is too narrow, a larger number of PCR is necessary to cover the handling of all relevant specific products under a program (e.g., PCR for all materials and applications that may be relevant for floor coverings). If, on the other hand, the definition of a product category is wide, the degree of freedom defining methodological settings in goal and scope definition must be higher and the simplification of quality control reduced. Thus, the inspection effort is higher. Concerning the additional information, the PCR can specify additional technical, health-related, or environmentally relevant information about the product, not represented sufficiently by LCA results. The quality control according to ISO 14025 is a two-step process: PCR review and EPD verification. Degree of freedom for setting system parameter and modelling

Inspection effort

Variability of products covered by one PCR

Fig. 5.2  Inspection effort of an EPD depends on the degree of freedom in the PCR (schematic diagram)

5  The Terms “Critical Review” and “Verification” Addressing Quality Assurance…

91

PCR Review According to ISO 14025, the PCR review shall be conducted by a review panel of at least three persons, and the interests of interested parties shall be considered. The result of the PCR review shall be included in the PCR. Conformity to (ISO 14040:2006) is a substantial aspect to be examined. Insofar the PCR review has the character of a critical review (see Sect. 5.2). EPD Verification Regarding Environmental Product declarations which are based on a PCR the standard (ISO 14025) uses the term “verification” by independent verifiers, whereby it is understood that “verification” refers to the conformity of an EPD to the PCR. To define the actual meaning of “verification,” a closer look to the procedure is helpful. The process of quality assurance can be structured into quality control of the LCA part (project report) and quality control of the EPD. Quality Control of the LCA Part For the LCA part, the quality control has to scrutinize if all methodological settings that are defined in the PCR are sufficiently implemented in the specific study and all requirements from ISO 14040, 14044 are consistently considered. To do that, a transparent study report must be provided by the LCA practitioner to demonstrate how the provisions in the PCR are handled. The investigation of this study report has the character of a critical review, for example, consistent methodological implementation according to the settings in the PCR, correct modeling of the product system, and reliable data  – especially if data in reliable databases are missing – have to be checked. If the PCR is valid for a wide range of specific products, this examination is more complex as if only a small range is covered because the degree of freedom in the provisions of the PCR must be higher. If the LCA study report passes the examination successfully, the LCA results can be used in the EPD. As in any critical review of an LCA, the verification of data, in the sense of investigating defined objective evidence for all data used, is not included. Quality Control of the EPD Quality control of the EPD refers to existing documents: The PCR, which should contain clear provisions according the format of the EPD; the LCA data to be declared, which must be transparently documented in the project report; and the necessary additional information. Thus, requirements for declared EPD data are clearly defined in existing documents (PCR and project report), and examining the content of the EPD compared to these documents is a conformity check. The answer concerning conformity may be “yes” or “no” (see also Fig. 5.1). In detail, the verifier has to investigate the following requirements: –– Existing document: LCA studyreport: –– Are the LCA results and other LCA-specific information in the EPD consistent with the numbers and methodological settings in the LCA study report?

92

B. Grahl and E. Schmincke

Quality Control Type: Verification

Requirements of PCR (format, additional information) correctly transferred to EPD? LCA results correctly transferred to required format?

Quality Control Type: Critical Review PCR define all aspects of goal and scope for LCA study, format of EPD and type of additional information

LCA study of a specific product in this product category

PCR-Review: Third party review by review panel

Critical Review: Study conducted according to methodological settings in PCR? Plausibility and consistency check

EPD

Fig. 5.3  Structure of quality control in the ISO 14025 system

–– Existing document: PCR (occasionally supplemented by additional documents of a program): –– Are all Information included in the EPD in the correct format? –– Existing document: PCR: –– Is all additional specific information according to the PCR provisions given, credible, and, if required, proven by quotable sources? The verification of the EPD is a conformity check related to defined existing documents. ISO 14025does not differentiate between the examination of the LCA study report and the conformity check of the EPD. The whole process is called verification, leading occasionally to confusion. Figure  5.3 summarizes the different steps of quality control in the ISO 14025 system. Based on ISO 14025, two standards specify the ISO type III declaration in EPD for the construction sector: (EN 15804:2012) and (ISO 21930:2017) working as frame PCR. Both documents use the term “verification” like ISO 14025, and thus, the reflections above are true for this documents as well.

5.5 Conclusion Critical review and verification are different methodological approaches to ensure quality control, and thus, the synonymic use of the terms may lead to confusion.

5  The Terms “Critical Review” and “Verification” Addressing Quality Assurance…

93

Concerning the critical review as peer review approach, the requirements are defined in ISO 14040 and ISO 14044 (2006), and ISO TS 14071 (2014) provides further guidance. Using the verification approach requires the clear definition of requirements and correlated objective evidences.

References Del Borghi A (2013) LCA and communication: environmental product declaration. Int J Life Cycle Assess 18:293–295 EN 15804 (2012) Sustainability of construction works—environmental product declarations— core rules for the product category of construction products Hunsager EA, Bach M, Breuer L (2014) An institutional analysis of EPD programs and a global PCR registry. Int J Life Cycle Assess 19:786–795 IBU (Institute Construction and Environment e.V., Berlin) (2017) PCR part B: requirements on the EPD for floor coverings. www.ibu-­epd.com ISO 14025 (2006) Environmental labels and declarations – type III environmental declarations – principles and procedures. Technical committee ISO/TC 207/SC 3 environmental labelling ISO 14040 (1997) Environmental management  – life cycle assessment  – principles and framework. Technical committee ISO/TC 207/SC 5 life cycle assessment ISO 14040 (2006) Environmental management  – life cycle assessment  – principles and framework. This standard was last reviewed and confirmed in 2016. Therefore this version remains current. Technical committee ISO/TC 207/SC 5 life cycle assessment ISO 14044 (2006) Environmental management – life cycle assessment – requirements and guidelines. Technical committee ISO/TC 207/SC 5 life cycle assessment ISO 21930 (2017) Sustainability in buildings and civil engineering works—core rules for environmental product declarations of construction products and services. Technical committee ISO/ TC 59/SC 17 sustainability in buildings and civil engineering works ISO 9000 (2015) Quality management systems – fundamentals and vocabulary. Technical committee ISO/TC 176/SC 1 concepts and terminology ISO TS 14071 (2014) Life cycle assessment – critical review processes and reviewer competencies. Additional requirements and guidelines to ISO 14044. Technical committee ISO/TC 207/ SC 5 life cycle assessment

Chapter 6

Benefits from Critical Review and Communication Hans-Jürgen Garvens

Abstract  The critical peer review (CPR) of Life Cycle Assessment (LCA) studies is a task of value. The majority of benefits lie in the projects and are not realized from the outside. The effect to the outside concerns communication. A peer review can be conducted on various levels. As a measure of quality assurance and to assist the interpretation of results, it is already meaningful to ask some colleagues for their opinion. For credibility and reliability, external reviews are most suitable. The best option is to include interested parties, which will also support communication. Communication of LCA results need to be suitable for the respective target audience. Often results are over-interpreted or simplified too much. The publication of an assertion “A is better than B” is meaningless without some background information and some limitations of that statement. The background information and limitations are meaningless, if the target audience cannot understand them. Keywords  Category rules (PCR) · Communication · Critical peer review (CPR) · Environmental product declarations (EPD) · Interested parties · Interpretation · Life Cycle Assessment (LCA) · LCA results

6.1 Introduction: Variability Is Credibility’s Enemy Humans tend to expect a certain stability in the answer whether A or B is environmentally better.

H.-J. Garvens (*) LCA Consultant and Review, Wandlitz, Germany Umweltbundesamt, German Environment Agency, Berlin, Germany School of Engineering – Technology and Life, HTW Berlin University of Technology and Economics, Berlin, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_6

95

96

H.-J. Garvens

From system boundaries, data quality, robustness of the models, assumptions to overcome data gaps and methodological choices on all LCA stages, as well as other issues, the results of two or more LCAs on the same issue may vary significantly. There is a variety of possibilities to narrow the variability. ISO 14040 and 14044 are often criticized for not giving clear and detailed recommendations on how to act, that is, on multifunctional processes or how to regard credits from products from the end of life stage and many other issues. The room left by ISO standards on LCA is often filled with guidelines on how to model the inventory, how to assess potential impacts on specific issues and how to evaluate the results in view of data quality, data gaps, and overall robustness. Finally, some such guidelines have also methods for interpretation included (i.e., JRC 2018). ISO itself has a built-in quality assurance concept demanding critical peer reviews (CPR) if scenario comparisons shall be published to the general public. Already in 1993 SETAC proposed “A Code of Practice” (Klöpffer 2012), where a peer review process was described. Originally, it was aimed at interested parties to take part in the process when an LCA is worked on or at least after its finalization, but prior to publication. Alongside the information on the subject of the project, the participants gained a lot of knowledge about the LCA methodology and its development. It depended and still depends today on the willingness of NGOs, companies, authorities, or others to participate. Many LCAs based on ISO 14040 and 14044 (until 2006 also ISO 14041 to 14043) were, and a few still are, conducted on such bases. However, the majority of projects today are conducted with an expert’s panel instead of an interested party panel review or without review at all. A few other projects have a kind of advisory board accompanying the process to participate on key decisions and to overlook the data used. Based on software solutions and readily available databases, many commissioners and practitioners tend to avoid the cost of any accompanying process – accepting that if published the project cannot claim accordance with ISO standards – or the project results are kept inside the companies. Some special (software) solutions for companies are reviewed once and getting applied without reviewing the single results. The inclusion of parties from outside the LCA project has valuable communicational effect. All participants are able to influence modelling, evaluation, and interpretation prior to publication, and all are on the same experienced level on LCA methodology. Stakeholders tend to bring divergent interests into the project, pushing the interpretation of results to be as comfortable to them as possible. The part of aligning the interpretation and the recommendations from the results in such situations is a demanding step. Having aligned interpretation and recommendations on the other hand has great communicational benefits. The discussions on results were held already and not brought to public. CPR can be seen as a discussion process in the project (partly) replacing discussion in the public by intended target groups. The variability in the results and interpretation narrows down. From the participation of externals, credibility and

6  Benefits from Critical Review and Communication

97

reliability gained significantly and the communication already started prior to the first press release. As any industrial product, also scientific “products” may come without any kind of quality control and quality assurance measures. And as in industrial production, those elaborates, which are quality controlled, are more expensive to some extent. And like industrial products, scientific studies also do not need to prove conformity with national or international standards unless contracted. It is always a matter of request. The customers – in most LCAs addressed as target audience – demand the quality standard. Often these are fellow scientists (i.e., in need of LCI data), large commercial chains, NGOs, or authorities (Curran 2011; UBA 2015; Frischknecht et al. 2015).

6.2 Critical Peer Review with Interested Parties Historical overview: At the beginning (1970–1990), there were many (methodological) discussions in each project; it took long to form a common basic framework. It was not very common that aside from the commissioners and authors, interested parties from science took part in the projects and formed LCA methodology. In the years prior to and in the first years of standardization (1990–2000), the amount of projects was rising and the awareness of the sensitivity of results rose accordingly. Conflicting assertions were published and the need of credibility forced all participants to include other stakeholders and the interested public (NGO) already in the process prior to publication. Under the guidance of ISO standards (from 1996), the critical peer review was included in most projects since the method LCA by itself needed more credibility. Later, the large panels were more and more relegated by review panels of external experts. Fewer methodological issues were discussed. The focus moved to the availability of data and the result interpretation. Since about 2010, LCA is getting to be more and more a business case, where cost, timeline, and effectiveness are acting as ruling measure. ISO TS 14071 (2014) condenses these developments and further formalizes the critical review process. At the same time, more and more related methods, like sustainability assessment, carbon and water foot printing, and product category rules, demand further reviews. If other parties are to be included in the LCA project it developed to a standard, then at least three independent experts are invited. Many commissioners consider well the need of such experts’ panel because of the cost. The value for the process and the communication from the review is often underestimated. There are developments to aid the review process by fully reviewed software products (reviewed LCI databases), lowering the labor cost of the review but raising the material cost (Weidema 2013). The few projects from the early stages attracted a lot of attention in science as well as in public. The numerous to-day LCAs cannot attract as much attention as before; however, it is still and will be mandatory to maintain reliability, credibility,

98

H.-J. Garvens

and quality of LCA and other assessments and to provide as much communicative value as possible. LCAs are made for communication. In a business case LCA, it is practically not possible to include stakeholders or interested parties into the review panel. It is only possible to fulfil the demands by contracting independent experts. In larger projects covering industry sectors or for setting regulations, still NGO and stakeholders play a steering role.

6.3 In-Projects Benefits It is often not possible to distinguish between benefits from an LCA and the CPR within an LCA project. Some effects from an LCA will only be intensified by a CPR – like creditability, and some are unaffected. In-project effects of CPR are easier to identify and often even lower the effort by the practitioner. The ISO standards 14044 and 14071 allow the review performed concurrently (accompanying, interactive) or at the end of the study. Only the concurrent mode has effects on the study itself. Like Klöpffer (2012) and many others, the accompanying review is to be preferred. The review at the end of a study (“a posteriori”) may only decide if ISO standards are met or not. From the complex and throughout review process, the independent experts often have many comments and improvement suggestions, which only can be regarded if the study and the LCA report are not completed at that stage. The effort for a review is not exploited effectively if the review outcome is limited to a conformity statement. In critical review panels according to ISO 14044 6.3, it is the task of the review chair to invite participants to the panel; however, all project partners may suggest possible members to the chair. Reviewers should not only be selected by knowledge but also if they represent academia, industry, or authorities and preferably different geographical origins. If the panel of experts does not involve interested parties or stakeholders, the reviewers need to anticipate their positions. Anticipating stakeholder interests is never fully possible. However, an appropriate selection of experts might make this task easier. The meetings often provide a societal discussion in a nutshell. Editorial issues are seldom discussed directly in meetings due to the amount of time used. The review process itself requires extensive communication. Aside from written comments, the project partners often discuss issues directly. Physical meetings are preferable if the partners have never met before; however, financial considerations often lead to phone and/or web meetings only. The meetings will best be held after submission of comments. Issues in the CPR are technical (i.e., data availability, data appropriateness, data quality, methodological issues) and focused on the interpretation of the results. Regarding the technical issues as a result from the review process, they may span from the assessment of LCI data with regard to their quality and appropriateness to the inclusion of further processes and data provision. From the evaluation of

6  Benefits from Critical Review and Communication

99

data quality and appropriateness, a clearer view on data limitations is provided – a prerequisite for a thoughtful interpretation of the results. The extent of LCI data review can also vary largely. On completely reviewed LCI databases, the review can be limited to appropriateness and compilation in the model. However, only few LCA projects do not come with newly collected, specific foreground process data, which need further attention. Methodological choices like process and system allocation is a further issue which is often broadly discussed in review panels. Benefits from the review of the impact assessment are directly linked to the interpretation in view of goal and scope. Since there is no allegation as to which impact categories are to be used, their selection is an important discussion step between reviewers and practitioners in the early stages after the definition of goal and scope. Aside from expected impact categories by the target group (i.e., authorities), all impacts of the systems in view shall be included with regard to effects on humans and the environment (legally protected areas). On the one hand, the selection of impact categories has to be sufficiently complete and data have to be sufficiently available. On the other hand, each LCA project shall stay in time and cost limitations. A contribution analysis is a good basis for the discussion of the selection of impact categories. Although the review of LCI data, the compilation models, and the LCIA needs to follow traditional horizontal and vertical test schemes, a review cannot provide a verification of the results (Grahl and Schmincke 2011). Optimum result is the validation of the data and models used as far as the latest state-of-the-art. The final step of a study’s review is to carefully evaluate the interpretation of the results. Any interpretation has to be avoided which is not substantially based on the results – also in view of uncertainties and data limitations. Of course, the goal of the study must be considered by the interpretation. Many practitioners tend to over-­ interpret small result differences. Reviewers have to arbitrate, if the comparative assertions are sufficiently backed by the study. Most review comments, and thus improvements of the LCA, regard data and interpretation issues. The largest benefits could be derived here. The final task of the reviewers is, according to ISO TS 14071, the CPR statement with inclusion of the main discussion results (also inclusion of minority votes) and the CPR report. The statement is to be included in the published study report and is only valid for that single report. The statement might be commented by the commissioner and the practitioner of the LCA.  The complete CPR report on the critical review, including all comments and detailed responses, is typically not published. Results of a critical review process are highly dependent on the personal attitude of reviewers. The attitude of interested parties (NGOs) and stakeholders is not debated. Participating in a concurrent, interactive review of an LCA will force all participants to work for the success of the project to the utmost possible degree. Including interested parties and stakeholders will broaden the bases of public acceptance. The review process will force the authors to a more careful interpretation of the results in view of the limitations.

100

H.-J. Garvens

Nearly all reviews are to some extent concurrently performed since the outcome has much positive impact on the study itself.

6.4 Outside the LCA Project The main reason to conduct an LCA is the communication of its results. The main reason to conduct a critical review is the perception of the results by the intended target audience. As explained before, there are many in-project benefits from CPR. However, only a few of those are included in the short CPR report attached to the LCA report. The CPR report does not contain much of the work done, when finally everything is in line with ISO or other standards. The holistic assessment of a product or process system and its impacts on humans and the environment is always a time-consuming exercise. There is an obvious need to limit the efforts. Once published the calculation models and estimations as well as the data might be reconstructed by others and could be used for alternative interpretations  – depending on the interest of those doing the recalculation. A critical peer review process and its subsequent review report and statement lead to an objectification of a subsequent public discussion. Those results and assertions backed by the CPR are less attackable and thus more valuable for the commissioner. The best option is to include such stakeholders or interested parties into the peer review panel in order to give them sufficient insight and to avoid reconstructions and the uncertainties that come along with it. Since participation in a CPR is voluntary, they shall at least be invited to participate. The critical review process is one brick in the arch of credibility, reliability and proof of quality of an LCA. There are many more bricks in that arch – to be solid and sustainable is the need of every one of them.

6.4.1 Business to Authorities Many LCAs have been done to support political decisions. Various authorities have issued regulations or recommendations on LCAs used to tender results to authorities and gain acceptance. Europe: In Europe, the Joint Research Center in Ispra issued a Handbook (JRC 2010) where 12 individual requirement schemes are included. Aside from the LCI data review, many of these schemes come with mandatory inclusion of interested parties and stakeholder panels. The schemes also cover product category rules (PCR), environmental product declarations (EPD), carbon footprint, and other assessments based on life cycle thinking and require reviews for them. The requirements often include involvement of interested parties, especially if results are to be disclosed to public or support political decisions. There is no advice

6  Benefits from Critical Review and Communication

101

given on how to act if interested parties are not interested to take part in a review process. One might come to the conclusion that it is not possible to publish a valid LCA (where required) when interested parties did not take their part in a critical review panel. Such exclusion criteria need interpretation in the future towards the possibility to conduct a valid LCA where interested parties were invited to participate, but declined. The European requirements lack clarity on the differentiation of micro level (LCI data or assessments) and meso/macro level. These terms are used for different requirements; however, no clear definition is given for distinction. The JRC claims are extremely ambitious. The intended quality standard of valid assessments is very high; however, the practical implementation lacks application. As such, the ILCD handbook is an example of how high ambitions cannot be realized to the extent intended, if too far from practice. In the United States, the EPA issued a code of conduct for LCAs (US-EPA 2006), where critical review is also dealt with. EPA included recommendations for a review of three or more parties for quality assurance reasons if the results are to be used in public forum. In addition, a need for data (LCI) review is expressed. Basically  – being a recommendation  – they are not necessarily to be met. However, the recommendation leaves much more room for voluntary measures – left to the responsibility of commissioners. That might be better understandable if one takes into account that under US legislation, wrong statements can be legally opposed more easily and with more effect than in other countries. As such, the need of quality assurance is claimed powerfully, and it is expressed that an accepted way of quality assurance is the critical peer review. Until 2014, there was no formal requirement in Germany – however, full accordance with ISO standards is one of the internal check items with the authorities. As long as ISO demands a critical review, it is prerequisite for a positive valuation by the authorities in Germany. From 2015, at least beverage LCA guidelines (UBA 2015) include minimum requirements for quality assurance from a critical peer review according to the standard. That peer review according to UBA 2015 also needs to formally check all other requirements of these guidelines, not only conformity to ISO 14040 and 14044. Even if there was no formal requirement until 2014, most LCAs in Germany came with a critical review if the submission to authorities was intended.

6.4.2 Business to Public/Stakeholders Business to public needs to fulfil the same requirements by ISO than business to authorities. The role of enforcement is taken over by NGOs or other stakeholders. The main role of the CPR is to give rise to credibility. Public awareness on CPR is low. It is more the role of NGOs and other interested parties to enforce quality standards as well as to use the results of LCAs for their purposes. Their interest on the quality of the results is very demanding. Mostly they even try to recalculate results.

102

H.-J. Garvens

Quality and transparency of a study need to be extraordinary high, as perquisite for acceptance by NGO. Stakeholders are participating in the commercial chain (especially retailers and material suppliers). And the retailers may ask for an independent review of results. The general public deals mostly with the interpretation of LCAs by others, like authorities or NGOs. The public trusts their assessment of quality of the underlying LCA. At all deliveries to public or to stakeholders, the knowledge about LCA methodology at the recipients might be low. All communication needs to consider this. If the target audience could already take part as an interested party in the project, that part of the communication is already much easier. A communication like “A is environmentally better than B” to the public might be tricky when the considerations and limitations of data, models and the method itself are hidden and unknown.

6.5 Critical Review – From Task to Communicational Benefit – A Value-Based Approach If nobody knows about the value of LCA results – nobody requests LCA. If nobody derives value from critical review – nobody requests CPR. Box 6.1: Critical Review Peer review is a review by colleagues Subjective comments based on experience Typical for scientific journals Only for communicational benefits Credibility ISO 14044 defines tasks to reviews, not benefits. –– The methods used to carry out the LCA shall be consistent with ISO 14044 and scientifically and technically valid. –– The data used shall be appropriate and reasonable in relation to the goal of the study. –– The interpretations shall reflect the limitations identified and the goal of the study. –– The study report shall be transparent and consistent. In order to improve communicational value, additional tasks like the anticipation of stakeholders’ interests are included. The efficiency of LCA is not higher, the less effort is necessary to get results. The value is higher, the more value the target audience(s) can drag out of these results. That value means communication and decision support. Since decisions on strategic or tactical level, where Life Cycle Thinking can be used with a lot of value, are rare, communication is the predominant value from LCA.

6  Benefits from Critical Review and Communication

103

Value is high on debated products like single use packaging as a proof the results are as reliable as possible. That value is lower, if the commissioner by itself is a scientific organization, NGO, or an authority. Even large national and supranational practitioners are trusted for providing reliable results and interpretations without further measures. Value is also high on new developments. About a decade ago, the first bioplastics were assessed using LCA.  Today carbon dioxide as feedstock is more and more discussed and thus assessed by LCA.  All new developments urgently need a stakeholder process by interested parties. Following initial LCA projects, a panel of experts will be commissioned to review the next few LCA projects. In the twentieth LCA report, only internal review might be necessary to check, if everything is in line with what others calculated. The more experience on an issue exists, the less effort will be taken for an additional review.

6.6 The Role of Guidelines Making it easy – from model to result in a few dozen hours. Making it complex – hundreds of pages of regulations to be met. Over the last decade, the number of guidelines on LCA has risen extraordinarily. As ISO 14040:2006 and 14044:2006 try to close the gap, they leave a lot of room for assessing issues. However, they mostly need a few hundred pages to address all relevant topics. Many of these guidelines are additionally accompanied with case studies with even more details on how to do things. There is a tendency to develop software and databases meeting the requirements of such guidelines. They flag everything on issue (data quality, data gaps, assumptions, etc.) to make everything computational. Having such guidance and software, it makes the task easy to model a scenario and to press a key to get result charts. Everything could be provided - from the result bars, evaluation of the data quality to the interpretation with implemented (hidden) weighting and normalization, the results, what are the 80% most relevant impact categories, which LCA stages are most relevant and to the most relevant processes. But - do practitioners still know what they got? The data modelling – foreground and background processes decide what kind of answer could be given. What does it mean that in some background process (i.e., plastics) economical allocation was applied? What does it mean if human or eco-toxicity show up in the most relevant impacts? It needs years of experience and in-depth knowledge to take into account the various aspects from the basic data, data quality and missing data, the data and distribution model of USEtox, knowledge about the low robustness factor applied in weighting the potential toxicity impact assessment numbers, etc. From all that, it could as well be very alarming to see toxicity having a high score in the most relevant impacts it could be also quite negligible depending on why the tox- category came up.

104

H.-J. Garvens

Shortcut Evaluation and Interpretation  To know what you really got is a prerequisite to telling others what you got. Recommendations and regulations from guidelines could not exchange for knowledge and experience of the practitioners writing the LCA report. From the numerous single results from scenarios and sensitivities, you get a lot of pages of evaluation and interpretation. It aids a lot if others accompany these steps to finally refine, fraction, and condense the final outcomes. Is There a Need for Rollback?  Guidelines and software makes it easy to produce numbers. An easy access to numbers and results makes the effort smaller. As long as the effectivity is not affected, less effort is good. All too frequent, not only is the effort smaller but also the reliability. Of course, nobody wants to miss modelling software and LCI databases. No rollback. But be careful of what you really got. Discuss that with colleagues; discuss that with external experts; it is best to discuss that with interested parties prior to publication. Then you will better know what you got.

6.7 Communication of LCA: LCAs Are Made for Communication Communication to Those Who Are Interested and Capable to Understand the Communication “A seems better than B. I made some calculation from our database and evaluated GWP.” Other LCA practitioners will read: “Okay, there is some probability, that A has advantages in greenhouse gases. No individual foreground data. Does not mean much.” The general public does not understand the second sentence and concludes “A is better than B.” Only when the results are sufficiently accompanied with a rise of understanding of the results, a commissioner of an LCA will also get a rise of value. For all practitioners, rise of demand might then follow. Decision makers, for example, still tend to simplify, zooming in on one environmental problem to find a quick fix that may have broader implications over time. (Strothmann et al. 2015)

Results Need Additionally to Come with Some Amount of Information on Their Meaning From the general public there are only very few people who can derive the importance of separate waste collecting from a waste management LCA. To overcome the “I do not need all these bins” mindset, communication needs to transport the idea behind GWP, resource use, and other impacts and the basic principles. This information needs to be substantial enough to encourage the willingness to pay, take more personal efforts, etc. It has never been difficult to transport information on

6  Benefits from Critical Review and Communication

105

more sustainable measures, which in parallel lead to less effort, whether these are daily decisions by the general public or on a more general level of industrial or governmental decision makers. The appropriateness of the information matters and how it is heard (and understood). The communication has to be suitable for the audience. The European waste directive (article 22 of the directive 2008/98/EC – separate collection of bio-waste) is an example of how LCA results were requested by European policy makers in order to gain sustainability but without sufficient communication. The national regulations interpreted the European directive in their own way, often setting certain amounts of material to be collected separately, often only for the sake of the ease monitoring amounts. The communication as well from the LCA as from the regulation lacks suitability for those, who shall act accordingly. Broken links in information delivery aided that development resulting in inefficient practices with several adverse effects. I assume overall advantages for the environment from the regulation. But the communication cannot answer the question of the normal citizen: “Do I need all these bins?” So, how to encourage people? In the rural country I am living in, communication was not encouraging but enforcing. If not enough amount will be collected, private composting has to be denied. Well, that is not the intended outcome of an LCA. Besides the national authorities responsible for the transfer of the regulations into national law, where is the information for the regional authorities who need to implement separate collection? Of course, that information needs to be much different than the information given to the national authorities or the public. Where is the information for the citizens on why they should accept yet another bin on their yard, encouraging them to fill it? It should be a task for the LCA practitioner to author the basic communication to all target audiences separately, suitable for the respective group in terms of understanding the statements and their background.

6.8 Conclusion Refine and condense results with the help of discussions with others as far as possible. Others may be fellow colleagues for internal quality assurance, externals for credibility and reliability or interested parties for preliminary discussion and communication. Communicate LCA results with all necessary additional information suitable for the respective target audience. Disclaimer  This publication does not necessarily reflect the opinion or the policies of the German Environment Agency or those of the German Emission Trading Authority (DEHSt) at the German Environment Agency.

106

H.-J. Garvens

References Curran MA (2011) Maintaining quality: critical peer review (CPR) as the demand for life cycle assessments increases: LCM 2011. Germany, Berlin Frischknecht R, Wyss F, Büsser Knöpfel S, Lützkendorf T, Balouktsi M (2015) Cumulative energy demand in LCA: the energy harvested approach. Int J Life Cycle Assess 20:957–969. (see also UBA 2015) Grahl B, Schmincke E (2011) “Critical Review” and “Verification” cannot be used synonymously. A plea for a differentiated and precise use of the terms. LCM 2011, Berlin JRC (2010) European Commission  – Joint Research Centre  – Institute for Environment and Sustainability: International Reference Life Cycle Data System (ILCD) handbook  – review schemes for life cycle assessment. First edition March 2010. EUR 24710 EN. Luxembourg. Publications Office of the European Union JRC (2018) Nessi S, Bulgheroni C, Konti A, Sinkko T, Tonini D, Pant R, Environmental sustainability assessment comparing through the means of life cycle assessment the potential environmental impacts of the use of alternative feedstock (biomass, recycled plastics, CO2) for plastic articles in comparison to using current feedstock (oil and gas). Draft report for stakeholder consultation. Ispra Klöpffer W (2012) The critical review of life cycle assessment studies according to ISO 14040 and 14044. Int J LCA 17(9):1087–1093 Strothmann P, Bricout J, Sonnemann G, Fava J (2015) Communication and collaboration as essential elements for mainstreaming. Life cycle management. In: Sonnemann G, Margni M (eds) Life cycle management. LCA compendium – the complete world of Life Cycle Assessment. Springer, Dordrecht, p 280. https://doi.org/10.1007/978-­94-­017-­7221-­1_12 UBA (2015) Guidelines on beverage packaging LCAs. Environment Protection Agency, Germany’s research project. See, e.g. Frischknecht et al., Int J Life Cycle Assess 2015 US-EPA (2006) Life Cycle Assessment: principles and practice. https://cfpub.epa.gov Weidema B (2013) Radically reducing the costs of panel critical reviews according to ISO 14040. LCM 2013, Gothenburg

Chapter 7

Cost-Benefit Analysis of Critical Reviews: Learning from Practice Christian Bauer and Andreas Detzel

Abstract This chapter deepens the issue of “cost-benefit analysis of critical reviews” on the basis of practical experiences. It differentiates between direct and indirect advantages and also gives magnitudes of the financial costs of a review with three external experts in relation to the total cost of a life cycle assessment. Interestingly, regardless of the objective and scope of individual studies, there are recurring issues that can be conflicting in the review process. Against this background, the chapter provides practical advice on how to ensure a favourable relationship between expenditure and benefits of the review. Keywords  Benefits · Comparative assertions · Cost analysis · Critical reviews · ISO · Interpretation · Life Cycle Assessment (LCA) · Practical advice · SIG Combibloc

Acronyms CR ELCD EPD GHG ILCD PCR PEF

Critical review European life cycle data centre Environmental product declarations Greenhouse gas International reference life cycle data Product category rules Product environmental footprint

C. Bauer (*) Group Environment, Health & Safety, SIG International Services GmbH, Linnich, Germany e-mail: [email protected] A. Detzel IFEU GmbH, Heidelberg, Germany e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_7

107

108

C. Bauer and A. Detzel

7.1 Introduction The critical review as an element of an ISO compliant Life Cycle Assessment (LCA) study takes a prominent portion of the resources which are needed if a comparative assertion is intended to be disclosed to the public. Despite the additional efforts to accommodate the critical review process, the advantage is the additional expertise involved, which increases the value of the different outcomes. SIG Combibloc1 has commissioned a variety of LCAs carried out by Institut für Energie- und Umweltforschung (ifeu Institute2), Heidelberg, Germany, to understand the environmental hotspots of its products and also their performance compared to competing product systems in all relevant market segments. Based on these experiences, the following will reveal costs and benefits involved with the current critical review (CR) practice. It will furthermore point out how time and costs can be optimized by addressing potential issues right from the start.

7.2 SIG’s Commitment to ISO-Conforming Critical Review For SIG Combibloc, examining all the key environmental categories throughout the entire product lifecycle is a fundamental basis for assessing the environmental profile of their packaging. Accordingly, SIG has conducted lifecycle assessments for Europe in all of its market segments. These assessments compare the standard paperboard carton against the relevant alternative packaging materials in each segment. The environmental impact profile of beverage cartons has been investigated by means of lifecycle assessments. In 2000, based on a detailed lifecycle assessment carried out by ifeu Institute on behalf of Germany’s Federal Environment Agency (UBA), the beverage carton was rated on par with the ‘refillable glass bottle system’ and considered as ‘environmentally favorable packaging’ in the context of the German Packaging Ordinance. As a consequence, the carton, in contrast to disposable plastic and glass bottles and beverage cans, is exempted from Germany’s ‘one-way deposit’ system for non-­refillable containers. In parallel, SIG is actively involved via ‘The Alliance for Beverage Cartons and the Environment’ (ACE) in preparing representative publicly available LCI (Life Cycle Inventory) datasets in order to support LCA practitioners and database owners, such as the ‘European Life Cycle Data Centre’ (ELCD) and ecoinvent.

 SIG Combibloc (www.sig.biz) is one of the world’s leading system suppliers of carton packaging and filling machines for beverages and food. The company supplies complete systems, including both the packaging materials and the corresponding filling machines. 2  Ifeu Institute (www.ifeu.de) was founded in 1978 and is an independent non-profit environmental research and consulting organisation with a global outreach. 1

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

109

In addition, ACE is active in preparing Product Category Rules (PCRs) in the EPD (Environmental Product Declarations) programme ‘International EPD System’, hosted by environedec. All LCAs commissioned by SIG have been critically reviewed, usually by a panel of three experts. Throughout the different studies, the composition of the panels has been changed to avoid fatigue and to gather additional insights. All reviews have been conducted concurrently with the LCA study, which has led to a specific experience-based format for contracting and arranging reviews. Table 7.1 reveals a generic timeline and milestone layout, which is used to set up an LCA study. The current procedure for setting up a panel is that, at first, the panel chair is contracted and consulted in contracting further panel members. The reviewers are asked to estimate their costs based on a rough briefing document provided by ifeu Institute which contains major elements of the goal and scope phase and a work plan. After having formally agreed on confidentiality, the review covers the four phases of the LCA as milestones. In particular, following the goal and scope phase and the Life Cycle Impact Assessment (LCIA) phase, the need to be in discussion with the panel, the practitioners and the commissioners is indispensable and most helpful to avoid unnecessary iterations. Depending on the ambiguity of the LCIA results, the interpretation phase is accomplished by a third discussion round, which is followed by the finalization of the critical review (CR) (Curran and Young 2014; Klöpffer 2005) report, which is then attached to the study. This work plan ensures that the reviewer’s tasks, as laid down in ISO 14044, paragraph 6.1 (ISO 14044:2006), are also accomplished concerning the correctness of the method, data appropriateness and consistency along the study, while limitations and transparency are assessed once a draft of the final report is available. During the recent years, this approach has been optimized by improving the cost/benefit relation for all stakeholders. Table 7.1  Generic LCA project setup including critical review milestones LCA phase Project scoping Goal and scope and LCI LCIAa

Interpretation

Work item Commissioning of ifeu Institute, preparation of project outline and contracting review panel Selection of representative product and data collection for drafting Goal and Scope First meeting with CR panel to discuss Goal and Scope Preparation of draft report containing first results Second meeting with CR panel to discuss first LCI and LCIA results Preparation of final draft Third meeting with CR panel to discuss final draft (optional) Preparation of final report and review report

Life Cycle Impact Assessment

a

Timing (tentative) Before the study starts 1st month 2nd month 3rd month 4th month 5th month 6th month 6th month

110

C. Bauer and A. Detzel

7.3 Benefits of Critical Reviews The most important benefit of a critical review is that it ensures fulfilment of an essential ISO 14040 (ISO 14040:2006) and ISO 14044 (2006) requirement. Without having conducted a critical review, a comparative assertion out of an LCA study cannot be claimed to be based on LCA and in consequence is not safeguarded by the consensus laid down in ISO 14040 (ISO 14040:2006). Apart from this rather trivial but, nevertheless, fundamental value, reviews deliver a variety of benefits for the commissioner which may be best grouped into direct and indirect ones. Direct benefits increase quality, correctness and appropriateness of the LCA study itself. Indirect benefits are considered as those relating to the application and communication of the LCA results.

7.3.1 Direct Benefits Increasing the quality of an LCA study via the critical review starts with the language. In the case of SIG, LCA is applied and results are communicated at a European level, which means that the entire life cycle approach is developed and communicated in English prior to being translated into German or other supported company languages. Apart from the commissioner and the practitioner, the CR panel plays an important role as the first and immediate audience to ensure that the language is clear and unambiguous. While the main part of an LCA report may be written in technical language without restrictions in terms of length and detail, it is particularly important that condensed conclusions and summaries should not leave any space for ambiguous interpretations. Developing a concise verbal representation of the study and findings, also in view of later communication, is therefore more challenging than just assessing the correctness of text and figures. The development of a concise language must start with proper specifications during the goal and scope phase, and in particular in the definition of the functional unit as reference point for all claims and observations. As the functional unit is the direct reference to the assertion, its definition needs to be carefully balanced to capture the objective of the LCA adequately. A short and tangible definition which has a high value for communication bears the risk that study results are extended far beyond the scope of the analysis. A sufficiently comprehensive definition including specific assumptions and alike increases the relevance and representativeness of the analysis considerably. Another area in which a CR panel adds value to an LCA is that reviewers bring in additional expertise. The current LCA practice faces numerous challenges, starting with proliferation of additional guidance at the product and sector level, competing databases and datasets and different impact assessment methods and models. In these areas, the panel helps to discuss appropriateness and maturity of data, guidance and models in view of the defined goal and scope.

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

111

The CR panel furthermore has the crucial role to help settle which scenarios and modelling variants are justifiable and necessary to deliver robust results without diluting the value of the analysis. Thereby, the panel can also act as a moderator between commissioner and practitioner to determine and negotiate necessary and sufficient update of data, models and methods, which can be agreed as state of the art for the defined study.

7.3.2 Indirect Benefits An indirect benefit of carrying out an LCA study with a concurrent CR panel for the commissioner is to receive immediate feedback on a given assertion by the review panel. A CR panel does not consist of experts alone, but may also be seen as a critical audience in the perspective of a critical consumer or stakeholder. In the case of comparative assertions, the reviewer even has the task to represent the perspective of competing products in the context of the study. Developing a communication including an environmental assertion in mutual discussion with a CR panel may help to anticipate feedback and reactions that can be expected once the study results are published. The same holds true if the intended application is rather a hotspot analysis to detect environmental improvement potentials. In this case, the CR panel can help to sharpen the analysis towards research and development by articulating this oftentimes-­important outside view. In addition, the critical evaluation of recent methodological LCA developments can help to inform the corporate LCA strategy for future LCA studies as it helps to understand the level of scientific maturity and robustness of LCI and LCIA developments.

7.4 Costs of Critical Reviews Apart from the benefits, the CR causes costs in a considerable dimension compared to the costs of the study itself. A three-expert review panel usually causes costs in the order of 30–40% of the costs of the LCA study work. On top of this, additional costs may be caused by requirements brought up by the CR panel, which end up in an extension of the originally planned scope for the LCA work in the course of the study. Therefore, while the direct reimbursement can be calculated based on the project outline, the full project costs remain open given that additional work, data and considerations are requested from the CR panel. Having highlighted the advantages of critical review and pointing out that it is a relevant factor regarding timing and costs of an LCA, it is important to ensure that a dedicated study must not be converted into a playing field of LCA debate as this bears huge risks of delaying timing and thus increasing costs. A principal attitude of

112

C. Bauer and A. Detzel

all parties involved, that is commissioner, practitioner and reviewers, should be oriented towards practicality. The reviewer has to find a balance between adequate diligence and practicality. It often helps if reviewers have their own LCA practitioner track record. Fundamental to the costs of an LCA study is the application of the core principles of LCA as laid down in ISO 14040 (ISO 14040:2006). In particular, a strict and rigid interpretation of principle 5 (management of iterations) and principle 8 (priority of the scientific approach) may impose additional analyses and investigations, which may exceed the capacities of the commissioner in terms of time and resources.

7.5 Managing the Cost-Benefit Relation The critical review requirements in ISO 14044 (ISO 14044:2006) set out a clear framework on which elements should be taken into specific consideration by a critical review. This gives a first good guideline, for example, to avoid contentious discussions about the goal and scope of the study. On the other hand, each requirement relies on expert judgement and practical experience of the reviewer. There are several points where reviewer’s opinion and practitioner’s opinion may not always coincide. This bears the risk for the commissioner that additional costs might be triggered by unforeseen discussions and iterations. Based on experiences gathered in LCAs commissioned by SIG and carried out by ifeu institute, potential areas of conflict and options to prevent cost impacts are addressed hereafter. Table 7.2, for this purpose, lists the binding requirements and the corresponding interpretation which have proven to be essential for the cost-­ benefit relation. More in detail are recurrent topics which have been areas of debates within almost all LCA studies commissioned by SIG:

7.6 Definition of the Functional Unit The definition of the functional unit is one of the core elements of any LCA. Based on this, a number of assumptions for modelling the product systems under examination have to be made. The function of packaging reveals a variety of aspects from product protection to transport behaviour and convenience during the use phase. From this end, each packaging system has unique features which complicate the definition of a single function as a common denominator. In SIG studies, the smallest denominator for packaging systems is used: the provision of a product unit which is available on shelves for the consumer: 1000 L of food/juice/milk packed and shelf-ready in the European market. Consensus among the commissioner, practitioner and reviewers about the definition of the functional unit is a key factor.

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

113

Table 7.2  One set of review requirements – two interpretations “Shall” requirements for the critical review from ISO 14044 The methods used to carry out the LCA are consistent with this International Standard

The methods used to carry out the LCA are scientifically and technically valid

Drivers for benefits Clear interpretation of goal and scope in view of principle modelling approach Sufficiency in terms of additional guidelines, standards and protocols Practical and realistic determination of scientific ambitions Simple – but adequate – specification of functional unit Practical decision and evaluation process, probably backed up by sensitivity analyses

Drivers for costs Demand for consequential and attributional modelling Demand for consistency with related standards (e.g. carbon footprint, ISO/ TS 14067:2013) or even PCR from EPD schemes following ISO 14025 (ISO 14025:2006) Define challenging requirements to conduct LCA-related research, for example on environmental impacts Extremely specific definition of functional units

Reviewer demands use of data from reviewed LCI databases, while datasets offered do not provide the quality required for the purpose of a study The interpretations Focus on clear documentation Request all limitations to be reflect the limitations of the interpretation as embedded in the assertion identified and the goal of support for unambiguous Request sensitivity analysis to the study assertions explore aspects without relevance for Request a meaningful set of the intended application sensitivity analyses supporting the intended application The study report is Transparency related to the Transparency related to embedded transparent and technical documentation of business relations, trading consistent the study regarding essential information and other confidential assumptions and data choices data Request a consistent report Request a report as true record of revealing all elements each project step required in ISO 14044 The data used are appropriate and reasonable in relation to the goal of the study

1. Assumptions Regarding Modelling of the Product System(s) The functional unit acts like a compass to determine the reference flows of a product system. In the case of packaging, this starts with the weight and material composition of the different types of packaging and packaging components assessed. It also includes aspects like the primary and secondary raw materials and end-of-life recovery and disposal rates and routes. Weights and material composition of packs on a dedicated consumer market usually are within a certain range; available end-of-life data often are not as packaging- or material-­ specific as would be required. It is therefore good practice to select figures which can be substantiated by empirical data or experience but which  – in cases of

114

C. Bauer and A. Detzel

doubt/uncertainty – are conservative3 regarding SIG’s packaging systems and on the favourable side for the competing packages examined. This also makes life easier for the reviewers, who have to ensure that the study is unbiased and therefore helps reduce controversial discussions and data gathering loops. In SIG’s LCA studies, market data are used to identify representative competing packaging systems for which the bill of materials is determined. End-of-life parameters like recycling rates are taken from official European or, if applicable, national statistics. 2. Type of LCA Approach (Consequential and Attributional Approach) The decision on which modelling approach (consequential or attributional) to select is directly linked to the objectives of an LCA study. As the LCAs commissioned by SIG intend to compare the environmental profiles of products as such (and not the macro-economic consequences of bringing these products on the market), SIG LCAs usually follow the attributional approach. This has implications on how to deal with methodological choices, such as allocation. However, currently, the philosophies underlying each approach are still debated among LCA experts, some of who go as far as to accept only one as the single correct approach. The commissioner, practitioner and critical reviewers should agree on the adequate approach from the start, including the methodological and data choices involved. Otherwise, this topic will be a constant source of conflict throughout the study. 3. Interpretation of ISO 14044 Requirements The provisions in ISO 14044 define framework requirements which at some point during the implementation in a distinct LCA study require interpretation by the practitioner. This leads to choices which imply a certain degree of value judgement, and as a consequence, different views about an adequate choice may exist between practitioners and critical reviewers. Here are three examples: A. Allocation procedure in the ISO standard The debate is whether ISO provides a hierarchy or a stepwise approach. In section 4.3.4.2, ‘Allocation procedure’, the ISO 14044 (ISO 14044:2006) standard clearly says two things:

1. The making of choices about handling of co-products follows a ‘stepwise procedure’, which means that allocation should follow rather a decision tree approach than a given hierarchy of individual allocation choices. 2. Furthermore, if the choice is that of carrying out a system expansion, it should be borne in mind that ISO 14044 (ISO 14044:2006) says that this is done by including ‘the additional functions related to the co-products’. There is a highly controversial discussion in the LCA world about what system expansion means (see section B below).

 Conservative in this context means that the least favorable assumption for SIG’s packaging systems is used as baseline. 3

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice





115

Practitioners and reviewers should discuss the adequate approach for handling co-products in a given LCA context during the goal and scope phase of an LCA and seek a common view and understanding. B. System expansion (ISO 14044, § 4.3.4.2) The discussion here usually is whether substitution credits comply with the ISO 14044 (ISO 14044:2006) term of ‘system expansion’. In ISO 14044, the term ‘system expansion’ is addressed exclusively in the context of allocation with reference to including additional functions to the system examined (see above). The result of such an expansion is a so-called basket-of-benefits. What are substitution credits? According to Brander and Wylie (2011), this is an ‘approach for avoiding allocation…., which involves identifying the products that are substituted by the co-product(s) of the product that is studied and quantifying the environmental burdens associated with those products. The avoidance of these burdens is then credited to the product that is studied’. It is obvious that in this case, the environmental attributes of a beverage carton would not only be those directly caused by the beverage carton system itself but would be influenced by the environmental attributes of any product that is assumed to be substituted by co-products of the beverage carton system. So far in SIG’s LCAs, an allocation approach, as addressed above under A, including sensitivity analyses, has been applied as the baseline. However, as the perception among LCA experts differs about if, when and how to apply system expansion within product LCAs, this is again a topic which needs to be clarified right from the start between practitioners and reviewers. C. Significance of the differences (ISO 14044, § 5.3.1) According to ISO 14044 (ISO 14044:2006) for LCA studies supporting comparative assertions intended to be disclosed to the public, the significance of the differences has to be evaluated. Such an evaluation typically is done by means such as uncertainty analysis and sensitivity analysis. Sensitivity analysis requires that assumptions, parameters and data with relevant influence on the LCA results are first identified and then varied in order to find out if and at which point these variations will change the outcome of the comparative assertion. Uncertainty analysis in current practice usually is done by applying mathematic-­statistical procedures based on data of probability distribution of each inventory data point. However, existing primary data inventories do not provide such numbers derived on real measurements. It is therefore debatable whether an uncertainty analysis with currently available inventories can provide robust results. On the other hand, in databases such as ecoinvent, datasets have been created including estimates about possible standard deviations. Although these data do not reflect real measurements, a practitioner may be asked by a reviewer to carry out an uncertainty assessment (‘this is better than nothing’ argument). This may not

116

C. Bauer and A. Detzel

only cause ­additional work time but also disputes about how to interpret the outcome of such analysis. Practitioners and reviewers should therefore discuss the handling of uncertainty issues already during the goal and scope phase. In SIG’s LCA studies, differences of indicator scores in the LCIA profile smaller than 10% usually are considered as insignificant when comparing indicator results per impact category. On top of that, to prevent an overinterpretation of uncertain LCIA scores, normalization is applied, which helps to determine the scale of calculated potential impacts and their differences in terms of relevance. Furthermore, sensitivity analysis is used in SIG’s LCA studies to address uncertainty regarding the scope of validity of the base scenarios.

7.7 LCIA Methods A variety of LCIA methods are developed and available for the LCA practitioner. The maturity and acceptance of methods oftentimes differs amongst practitioners who might be in favour, for example for more endpoint related approaches or rather conservative midpoint methods. The same holds true for the level of geographic specificity. A balance between pragmatism and academic aspiration has to be met. For SIG, reference is made to the LCIA methods that have been developed by the German Environmental Protection Agency, whose method guideline is currently being updated and amongst others incorporate elements of methods proposed by CML 2002 (Guinée 2001), ReCiPe4 and ILCD.5 It is important to recognize that any LCIA method informs on the interpretation phase, but this is not the only and decisive element to derive sound conclusions. In particular in areas where • No sufficiently robust indicators and characterization factors are established, as, for example, for impacts from land occupation and transformation on ecosystems • Availability of good quality inventory data is lacking, as, for example, is often the case for toxicity related categories or impacts from water related use The interpretation phase is important to address such aspects if considered relevant… The selection of an LCIA method including the underlying models needs to be fixed, as early as possible, back to back with decisions on data collection. In view of the iterative approach and the analysis of relevance and significance, it is important  ReCiPe includes a package of LCIA methods to provide assessments on both midpoint and endpoint levels. 5  The International Reference Life Cycle DataSystem (ILCD) Handbook provides governments and businesses with a basis for assuring quality and consistency of life cycle data, methods and assessments. 4

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

117

to revisit the models behind the individual indicators. Reviewers and practitioners should stick to the method initially chosen, and changing the method at a later stage of the study should be avoided for the sake of keeping timing and costs within scope. In SIG LCAs, the currently updated UBA method (UBA 1999) is used as core reference to identify appropriate LCI and LCIA indicators and as guideline to interpret flows and impacts. Necessary adaptations of the used UBA LCIA catalogue have been pursued in parallel to scientific progress in LCIA and LCI data availability.

7.8 LCIA Results As requested in ISO 14044 (ISO 14044:2006), LCIA results need to be discussed by category indicator. At this phase, it helps a lot to discuss the results already together with an indicative conclusion to decide on the further steps of the analysis. In SIG LCA studies, a proposal for potential conclusions is always presented from the practitioner, together with the environmental impact profiles of the packaging systems compared for discussion with the CR panel.

7.8.1 Knowledge of Inventory Data and Databases Primary data are ideal but not always available. For secondary data, several databases are available. Any selection on database and dataset level influences the results. Thus, even inventory data from databases have to be scrutinized. However, sometimes, opinions of practitioners and reviewers regarding validity of datasets from individual databases diverge. Adequateness and quality of data are key points for the robustness of the outcome of an LCA study. It is therefore worthwhile to strive for a common understanding between reviewers and practitioners about inventory data suitable for the purpose of the study as early as possible in the course of an LCA study, possibly before scenario modelling starts. In SIG LCA studies, the appropriateness of data in view of the goal and scope is discussed with the CR panel based on the process flow diagram on dataset level. As lined out in the previous paragraphs, a major potential for improving the cost-­ benefit relation is to agree on issues which are mainly caused by different possible interpretations of ISO 14044 (ISO 14044:2006) requirements, as early as possible during an LCA study. Nevertheless, each LCA study is unique and delivers new or unexpected insights about the product systems and their environmental performance. The trade-off between a mechanistic analysis and an approach which is driven from finding to finding needs to be carefully managed within each study. A checklist may help to ask the right questions at the right moment to ensure that goal and scope stay in focus throughout the entire project. The suggested guidance

118

C. Bauer and A. Detzel

in ISO/TS 14071 (ISO/TS 14071:2014) might serve as useful reference in future LCA studies. A clear understanding on the need to fix consensus is essential to moderate the discussions between the critical review panel, commissioner and practitioner. In this context, the chair of the CR panel has the responsibility to fix consensus for certain aspects. For each request from the CR panel regarding data gathering and modelling variants, a clear justification and rationale in relation to the goals of the study should be fixed – best in writing. It is a big difference if scenarios are defined to ‘have a look’ on what happens, or if the scenarios serve to verify a distinct conclusion. This is also true for experiments with different LCIA methods – which can be achieved by pushing a button. In this case, the reviewer, practitioner and commissioner should determine necessary work items and not possible calculations. In SIG’s LCA studies, additional requests from the CR panel regarding modelling choices have been implemented on the basis of a clear reasoning in relation to the goal and scope of the study and have been documented in the critical review report.

7.9 Recent Developments During the recent years, progress has been made to reduce the overall costs for LCAs – including the CR. Table 7.3 reveals recent developments and their impact on the cost-benefit relationship. As the roles and tasks of the critical reviewers are also subject of interpretation, this helps of course to achieve more clarification on procedural and practical aspects. The development of ISO/TS 14071 (ISO/TS 14071:2014) to address the critical review process is highly appreciated to agree on further minimal requirements giving directions for commissioners, practitioners and reviewers. Apart from guidance that specifies good LCA practice in coherence to the established ISO 14044 standards, a variety of organizations have published their own life cycle framework delivering additional guidance which is not in line with ISO 14044 requirements. Their value for an LCA study is limited and may lead to confusion. During recently carried out LCA studies, SIG and ifeu Institute were more and more asked to consider guidance documents in addition to ISO 14040/44 that are relevant in very specific cases (e.g. French BPX 350 (Ingwersen and Subramanian 2014), PEF and GHG6 protocol (2012). These guidance documents seek simplification of LCA studies by giving prescriptions, for example, for a certain modelling choice. These approaches may be valid if applied in the respective context. However, mixing the requirements set by those methods with ISO 14044 conformity requirements is a complex challenge, and results obtained are potentially as contentious as the interpretation of the LCA requirements themselves.

 Greenhouse Gas Protocol for corporate accounting and reporting https://ghgprotocol.org/

6

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

119

Table 7.3  Recent developments Work streams with the potential to reduce costs Beneficial aspects ISO/TS 14071a More clarity to moderate responsibilities and procedures of a panel review Sector guidelines (food, Source of documented packaging) and other best practice in order to proprietary justify data and methodologies (e.g., modeling choices PEFb) High-quality assured Potential to avoid deep LCI data (e.g., as dataset review during established for ecoinvent LCA study or ELCD)

LCIA model rankings (e.g. as established within ILCD)

Potential to ease reduction of indicators sets and interpretation

Product category rules in Potential to serve as EPD schemes reference for system descriptions

Costly aspects Additional set of requirements that need interpretation from commissioner and practitioner

Additional set of requirements that may, for example, exceed the necessary data collection efforts that originate from goal and scope and create confusion by using different terminologies Additional need to assess appropriateness of quality assurance schemes in view of data quality requirements in the specific study context. This might even imply extensive check of underlying documentation Additional argumentation and evidence needed if recommended LCIA models are not appropriate. Overstating the relevance of LCIA scores for interpretation Additional effort to understand the modelling choices in the corresponding EPD scheme Additional complexity to match product-­ related requirements with the concept of the functional unit EPDs often use a generic approach not necessarily suitable for addressing the specific goals of a distinct LCA study

ISO/TS 14071 (ISO/TS 14071:2014) PEF Product Environmental Footprint (Finkbeiner 2014)

a

b

Product category rules are established within EPD schemes which deliver LCA based environmental information on product level to support informed choices B2B and B2C. The underlying PCRs contain findings out of a variety of LCA studies and are a good reference to understand relevant aspects of a specific product. It needs to be decided, case by case, if the provisions of a PCR can support modelling aspects in LCA. With the carbon footprint (Inaba et al. 2016) and the water footprint (Berger et al. 2016), also LCA-based approaches are standardized supporting the quantification of a single environmental aspect in the form of an indicator. While the carbon footprint relates directly to climate change, the water footprint relates to a variety of data categories and impact models relating to resource use aspects but also to emissions. In view of an LCA study, it should be decided in the beginning of the project how far footprinting standards need to be considered.

120

C. Bauer and A. Detzel

7.10 Conclusion Methods which are established outside ISO may contain modelling approaches and also may reveal stakeholders’ consensus which can deliver important arguments to support modelling choices and assess, for example scientific consensus. Examples for such proprietary guidance documents are the PEF communication of the EU Commission and the ILCD handbook. A clarification to which extent these proprietary methods are considered for the LCA study itself is essential in the beginning of a study. This holds true also for data which are created and provided following these methods. The given examples show a variety of activities being under way, which ease data, model and tool access and provide additional ready-made system information. This has a high potential to abbreviate contentious discussions during the LCA study by referencing mature third party consensus on essential aspects. However, in the short term, the proliferation of so-called streamlined and practical solutions will increase the efforts for commissioners, practitioners and the review panel as more additional requirements are available that first need to be discussed before being applied.

References Berger M, Pfister S, Motoshita M (2016) Water footprinting in Life Cycle Assessment – how to count the drops and assess the impacts? (chapter 3) (Ed Finkbeiner M). In: Klöpffer W, Curran MA (eds) Special types of Life Cycle Assessment. LCA Compendium – the complete world of Life Cycle Assessment. Springer Brander M, Wylie C (2011) The use of substitution in attributional life cycle assessment. GHG Measure Manage 1(3–4):161–166. https://doi.org/10.1080/20430779.2011.637670 Curran MA, Young SB (2014) Critical review: a summary of the current state-of-practice. Int J Life Cycle Assess 19:1667–1673 Finkbeiner M (2014) Product environmental footprint—breakthrough or breakdown for policy implementation of life cycle assessment? Int J Life Cycle Assess 19:266–271 GHG Protocol (2012) Corporate value chain (scope 3) accounting and reporting standard, 2011. http://www.ghgprotocol.org/ Guinée JB (2001) Life Cycle Assessment: an operational guide to the ISO standards. Centre of Environmental Science (CML), Leiden University. http://www.leidenuniv.nl/cml/lca2/ index.html Inaba A, Chevassus S, Cumberlege T, Hong E, Kataoka A, Lohsomboon P, Mercadie C, Mungcharoen T, Radunsky K (2016) Carbon footprint of products (chapter 2) (Ed Finkbeiner M). In: Klöpffer W, Curran MA (eds) Special types of Life Cycle Assessment. LCA Compendium – the complete world of Life Cycle Asessment. Springer Ingwersen WW, Subramanian V (2014) Guidance for product category rule development: process, outcome, and next steps. Int J Life Cycle Assess 19:532–537 ISO 14025 (2006) Environmental labels and declarations, type III—environmental declarations— principles and procedures. Geneva ISO 14040 (2006) Environmental management – Life Cycle Assessment – principles and framework. Geneva

7  Cost-Benefit Analysis of Critical Reviews: Learning from Practice

121

ISO 14044 (2006) Environmental management  – Life Cycle Assessment − requirements and guidelines. Geneva ISO/TS 14067 (2013) Carbon footprint of products requirements and guidelines for quantification and communication. Geneva ISO/TS 14071 (2014) (ISO/TS 14071—environmental management – Life Cycle Assessment – requirements and guidelines for critical review processes and reviewer competencies) Klöpffer W (2005) The critical review process according to ISO 14040-43. An analysis of the standards and experiences gained in their application. Int J Life Cycle Assess 10(2):98–102 UBA (1999) Methode des Umweltbundesamtes zur Normierung von Wirkungsindikatoren, Ordnung (Rangbildung) von Wirkungskategorien und zur Auswertungnach ISO 14042 und 14043 (Version ‘99) Texte 92. Berlin, UBA 99

Chapter 8

Reporting and Communication Pere Fullana i Palmer

Abstract  Reporting an LCA study is described in the ISO standards 14040 and 14044. There are specificities depending on whether the aim is to perform a third party report or a comparative assertion. The standards and a number of other guiding documents talk about the content to be reported but not about how this content may be communicated and the philosophy behind communication. This chapter reflects on how, on the one hand, LCAs should be reported and, on the other hand, how they should be shaped depending on the object to be reported and the audience to receive the report. The chapter addresses the meaning of reporting. As it has many meanings, a discussion on how LCA reporting can adapt the meanings is presented. Some implicit aspects of LCA nature are considered, such as being iterative, comparative or weighted. Keywords  Communication · Interpretation · ISO 14040 · ISO 14044 · LCA · Life Cycle Assessment · Reporting · Stakeholders

8.1 Introduction The title of this volume is based, according to the standard ISO 14040:2006, section 3.4, on the fourth phase of a Life Cycle Assessment (LCA), the interpretation phase, and is extended by the important topics ‘review’ and ‘reporting’. The term life cycle interpretation is defined in ISO 14040 as the phase of life cycle assessment in which the findings of either the inventory analysis or the impact assessment, or both, are consistent with the defined goal and scope in order to reach conclusions and recommendations. P. Fullana i Palmer (*) UNESCO Chair in Life Cycle and Climate Change ESCI-UPF, Passeig Pujades 1, Barcelona, Spain e-mail: [email protected] © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5_8

123

124

P. Fullana i Palmer

Interpretation replaced older concepts such as ‘improvement assessment’ (ISO 14044:2006, section 2.2) and/or valuation (much discussed in Germany, see Klöpffer and Grahl 2009, 2014). The reason for declining the latter by ISO was that ‘subjective’ elements should be avoided in the otherwise scientific  – as far as possible ‘objective’ – LCA study (ISO 14044, section 6). Although ‘subjective’ is not (necessarily) ‘arbitrary’ (Klöpffer 1998), the last stage of an LCA study had to be profoundly changed (ISO 14043:2000; Lecouls 1999). Interpretation is now the counterpart of the phase goal and scope and essentially has to secure that all the phases of an LCA are well tuned in and consistent with each other. In addition, the plausibility and accuracy of the results have to be checked with suitable methods, such as sensitivity analyses and/or error calculations. The structure of the LCA phase ‘Interpretation’ is described as follows (ISO 14044, section 4.5): • Identification of significant issues • Evaluation with the elements Completeness check, Sensitivity check and Consistency check • Conclusions, limitations and recommendations The term review means to conduct a critical review as a way to deal with quality assurance under a confidentiality agreement. If parts of the underlying data of an LCA study are confidential, but there is a need for credibility and transparency (always for comparative LCA studies intended to be disclosed to the public), a reviewer who has seen the full report, including the confidential data, may certify its validity. ISO also poses high standards on reporting, especially in the case of third party reports, i.e. for the external use of LCA studies. A critical review (also named ‘peer review’) by independent experts should be performed for each LCA (optional) (ISO 14044, section 6.2) but is mandatory if the study is intended to be used in comparative assertions intended to be disclosed to the public. In that case, the review has to be performed in the strongest form according to the panel method (ISO 14044, section 6.3). The standard ISO 14040:2006, chapter 6, states: A reporting strategy is an integral part of an LCA study. An effective report should address the different phases of the study. It should describe results and conclusions of the LCA in an adequate form, address the data, methods and assumptions applied in the study and the limitations thereof. Reporting (ISO 14044:2006, section 5) and Critical Review Report (ISO 14044, section 6) are separate items outside of interpretation, but evidently belong to the phase ‘Interpretation’ of any LCA study. The performance of the critical review should preferably be done in the interactive or accompanying mode, as proposed by SETAC (1993). According to ISO 14040+44, it can also be performed a posteriori, if the draft final report is available (Klöpffer 2012). Report Format The scope definition phase in the ISO standards 14040+44 calls for describing the type and format of the report required for the study to be an exclusive chapter on

8  Reporting and Communication

125

reporting outside the goal and scope chapter (see above), but does neither regulate nor propose types or formats of reports. The ILCD Handbook (ILCD 2010) rather calls it form and level of reporting. Those depend primarily on three factors: the type of deliverables, the purpose and intended applications and the intended target audience. The form and level of reporting should be defined in goal and scope as well. Forms of reporting are: –– ‘Classical’ detailed project report, i.e. an often comprehensive text document, typically with graphics and tables, that provides all relevant details –– A more condensed and formalized, electronically exchangeable report –– A very condensed executive summary report that condenses the detailed project report to its essence in non-technical language Further, the ILCD Handbook distinguishes in goal and scope between three levels of final reporting: –– Reports or data sets for internal use –– Reports or data sets for external use (i.e. to be made available to a limited, well-­ defined list of recipients with at least one organization that has not participated in the Life Cycle Inventory (LCI)/LCA study) –– Comparative assertion reports that are foreseen to be made available to the (nontechnical) public It is important to decide on the kind of desired report very early, may be even before the goal and scope definition. Electronically exchangeable reports and automated routines based on standards to produce reports out of LCA software and database models are performed today in daily practice. Surely this requires a clear understanding of the related standards and the conversion of the requirement of the standards into electroniclogic and processes. Different templates can be generated, which represent typical goal and scope settings for the user. Ideally, these templates should be chosen before the analysis starts.

8.2 The Meaning of Reporting Let us start with the basics. Reporting may have different meanings,1 some of which may be relevant to life cycle studies. ‘To make or present an official or formal account of…’ is a somehow good definition, as LCA deals with environmental accounting, although seldom official, often formal and rarely on a regular basis. For instance, an environmental product declaration (EPD) is a way of presenting LCA results in a standardized format (a formal report), mainly on a

  The Free Dictionary by Farlex, http://www.thefreedictionary.com/reporting, visited on 2019-04-06. 1

126

P. Fullana i Palmer

business-to-­business basis and often with a cradle to gate boundary. Because of changing technologies and other boundary conditions, EPD programmes define for how long an EPD is valid for a certain product category.2 After this time, which may vary from 3 to 5 years, a new EPD must be developed. So one can say an EPD is a condensed LCA report made and presented on a regular basis. As a basis for the condensed information in an EPD, a transparent study report is mandatory (see Grahl and Schmincke, this book, Chap. 5.) If per “official” one thinks of ‘recognized and authorized’, then EPDs fit in as well, for they are certified by an accredited or qualified programme operator. Other ‘official’ LCAs may be those used within green public procurement tenders, in which a certain weight is given to an offer depending on the performed LCA.3 In addition, LCA is a methodology which often appears in legislation for environmental decisions, and public officers use their results to decide among options. Many examples of this may be found for packaging design and packaging waste management alternatives (Flanigan et  al. 2013). To a less official extent, LCA is very often a mandatory part of EU research ‘calls for tender’ and must be one of the project deliverables. The current activity in the EU concerning Product Environmental Footprint (PEF) is also an example where ‘official’ LCAs play a role. ‘To relate or tell about…’ is a good definition as well, for, although often forgotten, LCA is as deep as the goal and scope defines and, therefore, might be just a semi-quantitative life-cycle point of view to help an environmental decision. There is an interesting ‘fight’ between practicability and reliability within the LCA community (Baitz et al. 2013). To re-examine some often forgotten fundamentals of LCA described in ISO 14040 (2006) is quite clarifying. • ‘4.1 Principles of LCA 4.1.5 Iterative approach LCA is an iterative technique. The individual phases of an LCA use results of the other phases. The iterative approach within and between the phases contributes to the comprehensiveness and consistency of the study and the reported results. • 4.1.8 Priority of scientific approach Decisions within an LCA are preferably based on natural science. If this is not possible, other scientific approaches (e.g. from social and economic sciences) may be used or international conventions may be referred to. If neither a scientific basis exists nor a justification based on other scientific approaches or international conventions is possible, then, as appropriate, decisions may be based on value choices’. • ‘4.3 Key features of an LCA a) LCA assesses, in a systematic way, the environmental aspects and impacts of product systems, from raw material acquisition to final disposal, in accordance with the stated goal and scope; c) the depth of detail and time frame of an LCA may vary to a large extent, depending on the goal and scope definition;’

 For instance, the EPD programme operated by AENOR, Global EPD, has PCR for ceramic coverings (RCP (2013)  – Reglas de Categoría de Productos para la preparación de la Declaración Ambiental de la categoría ‘Recubrimientos cerámicos’), which state a validity of 5 years for EPDs following these rules. 3  In early 1998, the Barcelona municipality was already asking for an LCA to be performed for any city furniture to be contracted, being the LCA results and improvement measures a 10% of the whole scoring procedure. 2

8  Reporting and Communication

127

Being an iterative technique means that the first iterations do not need to be a fully fledged LCA.  Even the final iteration, if ‘in accordance with the stated goal and scope’, might be quite a simplification, as the ‘depth of detail and time frame may vary to a large extend’. Although LCA is, in principle, a robust scientific methodology, ‘as appropriate, may be based on value choices’. The big problem about these clauses is that they might be applied without transparency and poor results may be reported without highlighting their large uncertainty. The definition ‘To write or provide an account or summation of for publication or broadcast…’4 helps us understanding that a report is not necessarily a written one, as many times the intended audience only needs an oral presentation of quick results to go on towards their decision. The word ‘publication’ is essential as well, for many LCA practitioners come from the academia and they think that a fully formatted and filled-in scientific report is needed by the commissioner or intended audiences, while what they may need is an adapted trustful summary for sending to a newspaper for marketing purposes. ‘To submit or relate the results of considerations concerning…’ is a somehow more generalist definition and one that more clearly fits within the broadest view of what LCA is. The two essential words are ‘submit’ (to someone!) and ‘results’ (of a study). We scientists tend to listen to and write for ourselves, while including more assumptions than results in our reports. The rest of the world may be expecting something else, which, unfortunately, may be covered by somebody else quicker and more attractive than us. Of course, one must not confuse a primary LCA scientific report with a shorter secondary report, which may summarize or further explain relevant aspects for a particular recipient. In addition, those summary reports should include enough information for them not to be taken out of context. However, which one is the most important? On the one hand, scientists would strongly vote for the first one, as it is the basis for all others and to which one has to go back for clarification, methodologies, uncertainties and context, all in all, reproducibility. On the other hand, most of decision makers will never read such a report and will rely on whatever summary arrives to their hands. Therefore, if scientists want to make the world go round in an environmentally friendly direction, they should put more effort in knowing how to write this second type of report and, accordingly, spend (and budget) the sufficient amount of time of senior expertise to write or to review them.

8.3 The Aim of Reporting: Object or Subject Oriented Reporting An LCA is not an objective per se but a means to obtaining information for somebody. Practitioners may concentrate their efforts, for example, to perfectly follow ISO 14044 clauses and to deepen into data gathering and data quality assessment,   The Free Dictionary by Farlex, http://www.thefreedictionary.com/reporting, visited on 2019-04-06. 4

128

P. Fullana i Palmer

which is fine. However, although not part of the standard, it is better to take as a lead the intended application (eco-design, marketing, eco-labeling, etc.) and have it always in mind, shaping up the study to reach this object. Even more, if LCA has to make a change and be useful, it is even better to have, as their motivation, to fulfill the different stakeholders’ needs. These needs never come alone. If, during the goal definition, enough senior time is given to studying each stakeholder involved and discussing their possible needs, LCA study shaping will be more effective (results will match needs) and efficient (multiple solutions will be achieved with less effort). Therefore, practitioners do not have to choose between the intended application (object) and the stakeholder needs (subject) as a guide to perform and report an LCA but keep both in mind and make them fully compatible. It gives much satisfaction to the practitioner to see how his or her LCA study has solved a high number of topics to a high number of stakeholders than to see how she or he considered a high number of options with a high number of data. These solutions must come with clear reporting, or they will not be understood, accepted and applied. The words above should never be taken as an excuse for corrupting an LCA study, shaping it to give results that match someone’s wishes. LCA has been much criticized for being complex enough (with so many variables) that any result could be obtained which may suit the commissioner’s desires. The orientation of the LCA study can suit the process of getting the best results for a decision without a bias to what the commissioner would want as a result. The practitioner must ensure this, and she or he must be sure that the results are reported in a way which is not giving an undeserved benefit to the commissioner. A ‘short LCA report’ might be taken as an oxymoron by most LCA practitioners in academia. Working with marketing people to create summary reports to send short and appealing messages out of an LCA study without having the sensation that much of the context is left out is not easy for a scientist. However, it is much better to work with them to find innovative solutions for third party communication than to sign the fully scientific and detailed LCA study and avoiding personal involvement into the secondary but broadly sent out short reports or messages. If we do not do this job, someone else will do it, and we know how. Most scientists have stories to tell about how a journalist misinterpreted what they said in an interview and were made fools in front of the big audience. However, recent studies (Sumner et  al. 2014) show that only 20% of the news is miscommunicated if there is no exaggeration or misinterpretation of the scientific claim; however, if we scientists do not communicate properly, then the majority of the news will be wrong. The biggest problem found by these authors is that scientists ‘just don’t care’. Ours is the main blame.

8  Reporting and Communication

129

8.4 Reporting in Science We may ask ourselves if LCA is science and if an LCA report is a scientific report. ISO 14040, 4.1.8 clause (‘Priority of scientific approach’), already presented above, is very clear on this. In principle, LCA follows natural science. If not possible, it follows social and economic sciences. Only if those are not feasible, LCA might be based on value choices. The most important factor is transparency, stating very predominantly the LCA nature. Generally, the structure of an LCA follows the scientific method. The hypotheses are stated in the goal and scope definition. The methods for experimentation and testing the hypotheses, together with the raw results, are described in the inventory analysis and impact assessment. The discussions considering whether the results fit the hypotheses are presented in the interpretation phase, including the implications of the different findings and stating the potential limitations of the experimentation. Therefore, we may say that LCA methodology fits fairly well within a scientific framework. Whether LCA reporting fits scientific reporting is another story. And if there is no external review or verification, it is up to each practitioner to follow general scientific rules. For instance, scientific reports tend to have a very strong introduction, with a clear hypothesis and the reasons why you believe the hypothesis is valid and what science is expected to gain out of the experiment. This sounds very much like goal and scope definition, but these are generally weak when compared with the inventory phase. The hypothesis on an LCA study may be that introducing a packaging waste collection system based on refunding returned packaging to the shop is not better than maintaining the existing one based on collecting different fractions on the street. However, the underlying goal may be from marketing against the new system to improving open-loop recycling allocation methodology. These scientific steps in reporting are generally confused by LCA practitioners. How previous research is used in an LCA report is something which will show its scientific nature. A good LCA inventory includes all citations from which data have been collected. If alternative data come from different sources, then scientific arguments must be reported when choosing one of them or a sensitivity analysis should be reported together with the baseline results. However, this long list of citations is not the most important part of the background literature to include. Any good scientific report must include all (conflicting) previous reports on the same issue and discuss the differences. It might happen that the commissioner is more interested in knowing his/her case, but the practitioner should be sure that the job is properly performed. Benchmarking it against others is a reasonable way to test. All in all, it is a question of transparency as well. Any materials and methods section of a scientific report is essential to ensure reproducibility. But how is this ensured for an LCA study? A discussion within the editorial board of the International Journal of Life Cycle Assessment (Curran 2014)

130

P. Fullana i Palmer

has dealt with the requirements of LCA case studies publication. Arguments in favour of reproducibility and transparency to follow rigorous scientific principles were put against more application-oriented reports provided by industry, following the two worlds of LCA (Rebitzer et al. 2001) and the balance between theory and practice (Baitz et  al. 2013). For instance, for the background system, in which datasets from databases are used (GaBi, ecoinvent, ELCD, etc.), the only requirement would be properly stating the names of the datasets. However, for the foreground system, documentation on specific flows, amounts and traceability should be required. Between how the results were obtained and how the results should be interpreted lays the results section itself. There, tables and graphs are presented. This quantitative information might be taken out of the heavy context of the sections of methods and interpretation. Therefore, although we put much effort into describing how we got there and the meaning of it elsewhere, we should avoid misleading tables and graphs if taken out of context, as they very easily may be taken that way in secondary reports. Self-explanatory figures should be preferred, even if some information is repeated in the text. Scientific journals do not like authors to repeat information in their papers; however, when writing an LCA report, we must be sure that figures or tables will not be used out of context in a secondary report.

8.5 Implicit Comparative Nature of LCA An LCA study may be designed as a comparative assertion. In this case, the report must contain any information of compared systems and a justification of the frame conditions under which the comparison is reasonable. In addition, ISO14044 asks for specific items to include in the report and for critical reviewing, which are somehow often forgotten by practitioners. In many other occasions, an LCA is performed for one single product or service for internal use. This does not mean that no comparison has taken place, as very often the commissioner wants to know which are the weak points and how would the product look if those are solved, or how the service is evolving along time, etc. If no benchmarking is performed, no improvement can be derived from, and the LCA might be useless to motivate change. However, LCA practitioners should anticipate if this kind of “not comparative” LCA study could be used for external purposes, deriving in secondary reports which may take information of their report together within information from an LCA report of a competing product. These two LCA studies may have been performed with a rather different goal and scope definition, thus resulting in non-­ comparable results. To avoid misinterpretation by an audience and to avoid the commissioner or the commissioner’s stakeholders to misuse the report, a clear statement on this issue of not-for-comparison must be included into the report.

8  Reporting and Communication

131

8.6 Using Relative or Absolute Information Data processing of absolute numbers into relative ones is a common procedure in scientific data analysis when any type of comparison is at stake. It is essential to understand the contribution of parts within a system, to prioritize or to choose between options, as we better understand comparative distances than absolute ones. However, absolute numbers must be kept at hand, not only to have a reproducible report but also, again, to make them relative to certain baselines. We may find that system A has double the impact of system B in a certain impact category, but that both systems have a negligible impact in respect to a known and accepted reference. If we want to produce effective improvement, the actions must be referred to both, really big, hot spots and those which are more susceptible to be improved. Both absolute and relative results must be used in LCA communication; however, for the large majority of audiences, relative figures will be more easily understood (they may not be certain of what DALY means or if 23 kg of CO2 is a lot or negligible). The role of the practitioner is to avoid that the results used for communication are meaningless to the environment or that they are used without any indication of the uncertainty of the shown differences between the systems under confrontation. Whether the results are absolute or relative, the content-related implications usually need a detailed analysis and careful interpretation, and the practitioner must stress this in any part of the report which may be taken as communication material in a secondary report.

8.7 Implicit Weighting A kind of relative information which is very often asked by the commissioner or any target audience is the relative importance of the different impact categories being reflected in the report. Although it is clearly recognized as non-scientific based on ISO 14044 (clauses 4.1 and 4.4.3.4.2) and never to be used for comparative assertions (clause 4.4.5), there is always weighting in LCA reporting, whether it is made explicit or not. When some impact categories are not chosen, they are weighted zero. If a number of impact categories are shown in the results without any explicit weighting, they are implicitly and equally weighted with the weigthing factor “one”. The report should reflect the implications of this choice, and a qualitative weighting assessment should be performed by the practitioner or others with less environmental knowledge will. This assessment may be, at least, a clear warning that in different locations or times, some categories may be more important than others, and if action must be taken, it should follow the most relevant category results. In addition, if in absolute numbers, the system has small impacts for a certain impact category, this should be interpreted by the practitioner to guide the commissioner on not basing the decision in this one.

132

P. Fullana i Palmer

8.8 Reporting According to ISO 14044 The main LCA standard ISO 14044 basically mandates: 5.1.1 The type and format of the report shall be defined in the scope phase of the study. The results and conclusions of the LCA shall be completely and accurately reported without bias to the intended audience. The results, data, methods, assumptions and limitations shall be transparent and presented in sufficient detail to allow the reader to comprehend the complexities and trade-offs inherent in the LCA.  The report shall also allow the results and interpretation to be used in a manner consistent with the goals of the study.

The first part of the clause strongly states scientific needs of a proper LCA report for the reader mainly to understand all its boundaries and limitations. In the last sentence in this clause, the word ‘goals’ is very easily read as ‘aims’ or ‘objectives’, and probably this was the intention of the editors. However, the ‘goal’ (in singular) of the study (clause 4.2.2 of ISO 14044) includes the ‘intended audience’. Unluckily, due to the ever-growing LCA applications and intended audiences, none of them could be part of the standard. Nevertheless, they are the most important objects to consider before writing a report. The commissioner wants mainly to magnify the benefit out of the report for the foreseen application, and if this is too difficult to get from the full scientific report, LCA will always be the impossibly understandable and never-ending data gathering tool. Having said this, it is important to add that LCA is being more and more introduced into the decision-making processes, thanks to its intrinsic scientific value. This value must not decrease but be complemented by communication skills and emotional intelligence (Goleman 2006). It is very surprising to find that not many scientific papers have been published to develop LCA reporting beyond the poor guidance of ISO 14044. When looking for contributions listed in the Web of Science, the results are very clear. Only one (Vares and Hakkinen 2009) result appears if both words ‘LCA’ and ‘reporting’ are demanded in the title and another one (Harding 2013) when the search is done with the words ‘Life Cycle Assessment’ and ‘reporting’. When using the word ‘report’ instead, six documents are listed, but none of them relevant for the methodological aspect. Although the guidance in ISO 14044 is not very extensive, numerous requirements are specified that are very often not sufficiently implemented in LCA reports. ISO differentiates between ‘General requirements’ (5.1), ‘Additional requirements and guidance for third-party reports’ (5.2) and ‘Further reporting requirements for comparative assertion intended to be disclosed to the public’ (5.3). The titles of these sub-clauses indicate that, in each case, all requirements of the previous sections must be considered. Reporting the facts of all listed criteria requires a great deal of time, effort and ability of understandable writing. That is not always in the mindset of commissioners and practitioners when calculating the fee for an LCA project, and thus, reporting is very often only poorly processed, while it is absolutely essential if the practitioner wants to influence to make a positive change in the environment.

8  Reporting and Communication

133

8.9 Planning the Process of Reporting Before executing a task, it is always reasonable to plan how to perform it. LCA standards and guides, such as the ILCD Handbook (2010), give a main structure and some specifications for third party reports and for reporting on comparative studies to be disclosed to the public. Clause 5.2.1.1 of ISO 14040 (2006) states, ‘The scope should be sufficiently well defined to ensure that the breadth, depth and detail of the study are compatible and sufficient to address the stated goal’. Within the scope (clause 4.2.3.1. of ISO 14044), the ‘type and format of the report required for the study’ shall be considered and clearly described. It is very wise to define the different reports, their aims and the foreseen target audiences before starting to frame the study and collect data. Some principles of life cycle management (Fullana-i-Palmer et  al. 2011) may help to guide the process. It is essential to note that in order to fulfill ISO requirements, there is a component which has to be deeply considered: the amount of resources that the practitioner has at his/her disposition to perform the task of reporting. As for an LCA study development as a whole, following the ‘good enough is best’ principle is good guidance until the next iteration, and for the reporting part, more resources should be delivered if the practitioner wants the study to be used for change. If we agree that an effective report is needed to give value to the LCA study, enough senior human resources should be delivered for the content, especially to the interpretation and application chapters, while more junior-like human resources could be used for delivering an attractive format. As mentioned previously, an LCA study has usually more than one report: the “full” study report, with all scientific features included, and a number of internal and external reports in a diversity of contents and formats, written and oral, physical and virtual. All of them must be anticipated when the scope is defined, and human and financial resources must be accounted for or the practitioner will be confronted with a strong dilemma at the end of the study: whether to painfully ask for more unforeseen payments, to deliver a poor reporting result (according to the commissioner), or to work out the additional reporting activities for free. Finally, if the LCA study may have policy or investment implications and the LCA practitioner wants to influence the decisions to be made, the ‘trust beats certainty’ principle should be kept in mind. Certainty is based on reason, while trust is based on emotions. If the practitioner’s brand does not communicate enough trust, she or he may try to find allies with better branding to help writing and communicating the report. This issue needs to be discussed during the goal and scope with the commissioner, as it may have financial implications, and the possible allies may demand to be involved in the study right from the start in order to agree with the results.

134

P. Fullana i Palmer

8.10 Conclusions Many LCA practitioners tend to believe that an LCA study is a thorough scientific string of data and analyses to deliver the truth, and, therefore, they deliver their efforts to have a big and sound report complete with plenty of data and formulae. Others see LCA as a means to change the world for good, and, therefore, they try hard to sharpen their tools to convince those making decisions. Those two perspectives are compatible; however, the resources to do the job are limited, and the practitioner must prioritize where to use them. Whichever perspective is applied, the communication of the results is made by people with certain tools and certain skills. For the LCA to be useful and make a change, choosing the right people with the right tools and skills is, at least, as important as having good results.

References Baitz M, Albrecht S, Brauner E, Broadbent C, Castellan G, Conrath P, Fava J, Finkbeiner M, Fischer M, Fullana-i-Palmer P, Krinke S, Leroy C, Loebel O, McKeown P, Mersiowsky I, Möginger B, Pfaadt M, Rebitzer G, Rother E, Ruhland K, Schanssema A, Tikana L (2013) LCA’s theory and practice: like ebony and ivory living in perfect harmony? Int J Life Cycle Assess 18(1):5–13 Curran MA (2014) How many case studies should we publish, if any? Int J Life Cycle Assess 19(1). https://doi.org/10.1007/s11367-­013-­0667-­0 European Commission – Joint Research Centre – Institute for Environment and Sustainability: International Reference Life Cycle Data System (ILCD) Handbook (2010) General guide for Life Cycle Assessment – detailed guidance. First edition March 2010. EUR 24708 EN. Publications Office of the European Union, Luxembourg Flanigan L, Montalvo T, Frischknecht R (2013) An analysis of Life Cycle Assessment in packaging for food & beverage applications UNEP/SETAC life cycle initiative Fullana-i-Palmer P, Puig R, Bala A, Baquero G, Riba J, Raugei M (2011) From Life Cycle Assessment to life cycle management: a case study on industrial waste management policy making. J Ind Ecol 15(3):458–475 Goleman D (2006) Emotional intelligence. Bantam Books, New York Harding KG (2013) A technique for reporting Life Cycle Impact Assessment (LCIA) results. Ecol Indic 34:1–6 ISO 14043 (2000) International Standard Organisation: environmental management – Life Cycle Assessment: interpretation. Geneva ISO 14044 (2006) International Standard Organisation: environmental management – Life Cycle Assessment: requirements and guidelines. Geneva ISO-14040 (2006) Environmental management – Life Cycle Assessment – principles and framework. Geneva Klöpffer W (1998) Subjective is not arbitrary. Editorial. Int J Life Cycle Assess 3(2):61–62 Klöpffer W (2012) The critical review of Life Cycle Assessment studies according to ISO 14040 and 14044: origin, purpose and practical performance. Int J Life Cycle Assess 17(9):1087–1093 Klöpffer W, Grahl B (2009) Ökobilanz (LCA) – Ein Leitfaden für Ausbildung und Beruf. Wiley-­ VCH, Weinheim Klöpffer W, Grahl B (2014) Life Cycle Assessment (LCA) – a guide to best practice. Wiley-VCH, Weinheim

8  Reporting and Communication

135

Lecouls H (1999) ISO 14043: environmental management – Life Cycle Assessment – life cycle interpretation. Int J Life Cycle Assess 4(5):245 Rebitzer G, Fullana P, Jolliet O, Kloepffer W (2001) Advances in LCA and LCM: an update on the liaison of the two LCA-planets. Int J Life Cycle Assess 6(4):187–191 SETAC (Society of Environmental Toxicology and Chemistry) (1993) Guidelines for Life-Cycle Assessment: a ‘Code of Practice’. From the SETAC workshop held at Sesimbra, Portugal 31 March–3 April 1993. Brussels/Pensacola Sumner P, Vivian-Griffiths S et al (2014) The association between exaggeration in health related science news and academic press releases: retrospective observational study BMJ 2014; 349:g 7015. In White M (2014) How scientists contribute to bad science reporting. Pacific Standard, books and culture, visited 2015-06-01. http://www.psmag.com/books-­and-­culture/ scientists-­contribute-­bad-­science-­reporting-­96801 Vares S, Hakkinen T (2009) LCA tool for cement manufacturers for system optimization and for environmental reporting. In: Koukkari H, Nors M (2009) Life Cycle Assessment of Products and Technologies. Book series: VTT symposia 262, pp 122–133. VTT, Espoo, Finland

Index

B Benefits, 5, 15, 46–48, 68, 95–105, 108–111, 113, 128, 132

G Goal and scope definition, 2, 9, 14–17, 19, 52, 60, 73, 84, 85, 89, 90, 126, 129, 130

C Chain management, 88 Code of Practice (1993), 52, 56, 58–59, 96 Communication, 5, 27, 45, 73, 75, 88, 95–105, 110, 111, 120, 123–134 Comparative assertions, 4, 33, 53, 56, 61–65, 68, 72, 75, 77, 89, 99, 108, 110, 111, 115, 124, 130–132 Conformity criteria, 84–86 Cost analysis, 108–120 Critical peer review (CPR), 96–102 Critical review (CR), 1–5, 7–20, 35, 40, 52–78, 84–93, 95–105, 108–120, 124

I Interested parties, 2, 44, 61–65, 69–72, 75, 91, 96–105 International Standard Organisation (ISO), 2, 9, 25, 52, 84, 96, 108, 123 Interpretation, 1–4, 8–20, 25–27, 29–31, 33–35, 38, 40, 43–48, 52, 53, 55, 60–62, 73, 74, 84–87, 89, 96–104, 109, 110, 112–119, 123, 124, 129–133 ISO 14025:2006, 4, 68, 86, 88 ISO 14040, 17, 52–55, 59–66, 69–72, 74–77, 84, 85, 91, 93, 96, 101, 123, 124, 129 ISO 14040:1997, 52, 53, 61, 62, 84 ISO 14040:2006, 2, 9, 14, 17, 27, 30, 52, 69, 77, 84, 85, 88, 91, 103, 110, 112, 123, 124, 126, 133 ISO 14040-14043 (1997–2000), 52 ISO 14044, 11, 18, 55, 62–64, 66, 69–72, 74–77, 84, 85, 89, 91, 96, 98, 101, 102, 109, 112–115, 117, 118, 124, 127, 130–133 ISO 14044:2006, 3, 10, 25, 27–31, 33, 41, 52, 53, 62–65, 69, 70, 74, 77, 84, 85, 88, 93, 103, 109, 110, 112, 114, 115, 117, 124 ISO 9000:2000, 88

D Data quality, 4, 12, 16, 24–48, 54–56, 59, 85, 89, 96, 98, 99, 103, 119, 127 Data quality analysis (DQA), 4, 25, 55 Data quality indicators (DQIs), 4, 25, 28, 29, 55 E Environmental product declarations (EPD), 88–92, 100, 109, 113, 119, 125, 126

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 M. A. Curran (ed.), Interpretation, Critical Review and Reporting in Life Cycle Assessment, LCA Compendium – The Complete World of Life Cycle Assessment, https://doi.org/10.1007/978-3-031-35727-5

137

138 ISO 9000:2015, 84, 86, 88 ISO/NP TS 14074, 2–4 ISO/TC207/SC5, 52, 66 L LCA framework, 9–11 LCA results, 4, 5, 9, 10, 16, 20, 24, 65, 88, 91, 102, 105, 110, 115, 125, 126 Life cycle assessment (LCA), 8, 24, 52, 84, 96, 108, 123 Life cycle impact assessment (LCIA), 1, 3, 12–14, 17, 19, 20, 25, 52, 60, 62, 64, 77, 89, 99, 109, 111, 116–119 Life cycle inventory analysis (LCI), 9, 11, 12, 19, 54–56, 58–60, 65, 71, 74, 85, 97–101, 104, 108, 109, 111, 113, 117, 119, 125 Life cycle thinking, 24–26, 100, 102 M Multi-criteria decision analysis (MCDA), 10

Index Peer review, 8, 52, 56–60, 62, 85, 88, 93, 96, 100–102, 124 Practical advice, 5 Product category rules (PCR), 4, 88–92, 97, 100, 109, 113, 119 Q Quality assurance, 4, 48, 52–78, 84–93, 96, 97, 101, 105, 119, 124 Quality management, 87 Quality scoring system, 31 R Reporting, 1–5, 7–20, 26, 57, 60, 70, 123–134 S SIG Combibloc, 108 Society for Environmental Toxicology and Chemistry (SETAC), 13, 52, 54–60, 66, 67, 70, 72, 96, 124 Stakeholders, 5, 66, 77, 96–103, 109, 111, 120, 128, 130

O Objective evidence, 86–88, 91, 93 P Pedigree matrix, 25, 31–33, 35, 38, 41, 42

T Technical specification ISO 14071 (2014), 53, 56, 65–76 Type III environmental declarations, 88