Transactions on High-Performance Embedded Architectures and Compilers II [1 ed.] 3642009034, 9783642009037

Citation preview

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Alfred Kobsa University of California, Irvine, CA, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos University of California, Los Angeles, CA, USA Doug Tygar University of California, Berkeley, CA, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

5470

Per Stenström (Ed.)

Transactions on High-Performance Embedded Architectures and Compilers II

13

Volume Editor Per Stenström Chalmers University of Technology Department of Computer Science and Engineering 412 96 Gothenburg, Sweden E-mail: [email protected]

Library of Congress Control Number: Applied for CR Subject Classification (1998): B.2, C.1, D.3.4, B.5, C.2, D.4

ISSN ISSN ISBN-10 ISBN-13

0302-9743 (Lecture Notes in Computer Science) 1861-306X (Transactions on HiPEAC) 3-642-00903-4 Springer Berlin Heidelberg New York 978-3-642-00903-7 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. springer.com © Springer-Verlag Berlin Heidelberg 2009 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 12632398 06/3180 543210

Editor-in-Chief’s Message

It is my pleasure to introduce the second volume of Transactions on HighPerformance Embedded Architectures and Compilers. This journal was created as an archive for scientific articles in the converging fields of high-performance and embedded computer architectures and compiler systems. Design considerations in both general-purpose and embedded systems are increasingly being based on similar scientific insights. For example, a state-of-the-art game console today consists of a powerful parallel computer whose building blocks are the same as those found in computational clusters for high-performance computing. Moreover, keeping power/energy consumption at a low level for high-performance general-purpose systems as well as in, for example, mobile embedded systems is equally important in order to keep heat dissipation at a manageable level or to maintain a long operating time despite the limited battery capacity. It is clear that similar scientific issues have to be solved to build competitive systems in both segments. Additionally, for high-performance systems to be realized – be it embedded or general-purpose – a holistic design approach has to be taken by factoring in the impact of applications as well as the underlying technology when making design trade-offs. The main topics of this journal reflect this development as follows: – Processor architecture, e.g., network and security architectures, application specific processors and accelerators, and reconfigurable architectures – Memory system design – Power, temperature, performance, and reliability constrained designs – Evaluation methodologies, program characterization, and analysis techniques – Compiler techniques for embedded systems, e.g, feedback-directed optimization, dynamic compilation, adaptive execution, continuous profiling/ optimization, back-end code generation, and binary translation/optimization – Code size/memory footprint optimizations For the second volume of Transactions on HiPEAC we received 31 submissions and accepted 15 papers. Ten of these are regular submissions. We also set up a submission server, which has made the submission handling much more convenient for the authors, the reviewers, and the editors. I would like to thank Thomas van Parys and Sylvie Detournay of Ghent University for being instrumental in setting up this precious infrastructure. This volume contains 15 papers divided into two sections. The first section is a special part containing the top five papers from the Second International Conference on High-Performance Embedded Architectures and Compilers held in Ghent January 28-30, 2007. The second section is a set of ten regular papers. The topics of these general papers are a good pick of the ones covered by the journal, such as microarchitecture, memory systems, code generation, and performance modeling.

VI

Editor-in-Chief’s Message

The editorial board has worked diligently to handle the papers for the journal. I would like to thank all the contributing authors, editors, and reviewers for their excellent work. However, one of the editors, a dear colleague and friend, is no longer with us. In April 2007, Professor Stamatis Vassiliadis passed away after a long time of illness. He was a passionate visionary leader for our field, with too many contributions to mention, and was a key person behind the establishment of the HiPEAC Network of Excellence. We miss him immensely as a dear colleague and friend. I would like to welcome a new member of the editorial board. Dr. Georgi Gaydadjiev is an expert on reconfigurable computing, a field that he as well as Stamatis have contributed to heavily. We welcome Dr. Gaydadjiev aboard! Per Stenstr¨ om Chalmers University of Technology Editor-in-Chief Transactions on HiPEAC

Editorial Board

Per Stenstr¨ om is a professor of computer engineering at Chalmers University of Technology. His research interests are devoted to design principles for high-performance and embedded computer systems. He is an author of two textbooks and more than a hundred research publications. He regularly serves program committees of major conferences in the computer architecture field as well as actively contributing to editorial boards. He has been an editor of IEEE Transactions on Computers and is an editor of the Journal of Parallel and Distributed Computing and the IEEE Computer Architecture Letters. Further, he served as the General as well as the Program Chair of the ACM/IEEE Int. Symp. on Computer Architecture, the IEEE Int. Symp. on High-Performance Computer Architecture, the ACM Int. Conf. on Languages, Compilers, and Tools for Embedded Systems, the Int. Conf. on High-Performance Embedded Architectures and Compilers, and the IEEE Int. Parallel and Distributed Processing Symp. He is a member of the ACM and the SIGARCH, a Fellow of the IEEE, and a founding member of the Network of Excellence on High-Performance and Embedded Architecture and Compilation funded by the European Commission under FP6 and FP7.

Koen De Bosschere obtained his PhD from Ghent University in 1992. He is a professor in the ELIS Department at the Universiteit Gent where he teaches courses on computer architecture and operating systems. His current research interests include: computer architecture, system software, code optimization. He has co-authored 150 contributions in the domain of optimization, performance modeling, microarchitecture, and debugging. He is the coordinator of the ACES research network and of the European HiPEAC2 network. He can be contacted at [email protected].

VIII

Editorial Board

Jose Duato is professor in the Department of Computer Engineering (DISCA) at UPV, Spain. His research interests include interconnection networks and multiprocessor architectures, and he has published over 340 papers. His research results have been used in the design of the Alpha 21364 microprocessor, and the Cray T3E, IBM BlueGene/L, and Cray Black Widow supercomputers. Dr. Duato is the first author of the book Interconnection Networks: An Engineering Approach. He served as associate editor of IEEE TPDS and IEEE TC. He was General Co-chair of ICPP 2001, Program Chair of HPCA-10, and Program Co-chair of ICPP 2005. He has also served as Co-chair, Steering Committee member, Vice-Chair, or Program Committee member in more than 55 conferences, including HPCA, ISCA, IPPS/SPDP, IPDPS, ICPP, ICDCS, Europar, and HiPC.

Georgi Gaydadjiev is a professor in the computer engineering laboratory of the Technical University of Delft, The Netherlands. His research interests focus on many aspects of embedded systems design with an emphasis on reconfigurable computing. He has published about 50 papers on this topics in international refereed journals and conferences. He has acted as Program Committee member of many conferences and is subject area editor for the Journal of Systems Architecture.

Editorial Board

IX

Manolis Katevenis received his PhD degree from U.C. Berkeley in 1983 and the ACM Doctoral Dissertation Award in 1984 for his thesis on “Reduced Instruction Set Computer Architectures for VLSI.” After a brief term on the faculty of Computer Science at Stanford University, he has been in Greece, with the University of Crete and with FORTH, since 1986. After RISC, his research has been on interconnection networks and interprocessor communication. In packet switch architectures, his contributions since 1987 have been mostly in per-flow queueing, credit-based flow control, congestion management, weighted round-robin scheduling, buffered crossbars, and non-blocking switching fabrics. In multiprocessing and clustering, his contributions since 1993 have been on remote-writebased, protected, user-level communication. His URL is http://archvlsi.ics.forth.gr/∼kateveni.

Michael O’Boyle is a professor in the School of Informatics at the University of Edinburgh and an EPSRC Advanced Research Fellow. He received his PhD in Computer Science from the University of Manchester in 1992. He was formerly a SERC Postdoctoral Research Fellow, a Visiting Research Scientist at IRISA/INRIA Rennes, a Visiting Research Fellow at the University of Vienna, and a Visiting Scholar at Stanford University. More recently he was a Visiting Professor at UPC, Barcelona. Dr. O’Boyle’s main research interests are in adaptive compilation, formal program transformation representations, the compiler impact on embedded systems, compiler directed low-power optimization, and automatic compilation for parallel single-address space architectures. He has published over 50 papers in international journals and conferences in this area and manages the Compiler and Architecture Design group consisting of 18 members.

X

Editorial Board

Cosimo Antonio Prete is full professor of Computer Systems at the University of Pisa, Italy, and faculty member of the PhD School in computer Science and Engineering (IMT), Italy. He is coordinator of the graduate degree program in Computer Engineering and rector’s adviser for Innovative Training Technologies at the University of Pisa. His research interests are focused on multiprocessor architectures, cache memory, performance evaluation, and embedded systems. He is an author of more than 100 papers published in international journals and conference proceedings. He has been project manager for several research projects, including: the SPP project, OMI, Esprit IV; the CCO project, supported by VLSI Technology, Sophia Antipolis; the ChArm project, supported by VLSI Technology, San Jose; and the Esprit III Tracs project.

Andr´e Seznec is “directeur de recherches” at IRISA/INRIA. Since 1994, he has been the head of the CAPS (Compiler Architecture for Superscalar and Special-purpose Processors) research team. He has been conducting research on computer architecture for more than 20 years. His research topics have included memory hierarchy, pipeline organization, simultaneous multithreading, and branch prediction. In 1999-2000, he spent his sabbatical with the Alpha Group at Compaq.

Editorial Board

XI

Olivier Temam obtained a PhD in computer science from the University of Rennes in 1993. He was assistant professor at the University of Versailles from 1994 to 1999, and then professor at the University of Paris Sud until 2004. Since then, he has been a senior researcher at INRIA Futurs in Paris, where he heads the Alchemy group. His research interests include program optimization, processor architecture, and emerging technologies, with a general emphasis on long-term research.

Theo Ungerer is Chair of Systems and Networking at the University of Augsburg, Germany, and Scientific Director of the Computing Center of the University of Augsburg. He received a Diploma in Mathematics at the Technical University of Berlin in 1981, a Doctoral Degree at the University of Augsburg in 1986, and a second Doctoral Degree (Habilitation) at the University of Augsburg in 1992. Before his current position he was scientific assistant at the University of Augsburg (1982-89 and 1990-92), visiting assistant professor at the University of California, Irvine (1989-90), professor of computer architecture at the University of Jena (1992-1993) and the Technical University of Karlsruhe (1993-2001). He is a Steering Committee member of HiPEAC and of the German Science Foundation’s priority programme on “Organic Computing.” His current research interests are in the areas of embedded processor architectures, embedded real-time systems, organic, bionic, and ubiquitous systems.

XII

Editorial Board

Mateo Valero obtained his PhD at UPC in 1980. He is a professor in the Computer Architecture Department at UPC. His research interests focus on high-performance architectures. He has published approximately 400 papers on these topics. He is the director of the Barcelona Supercomputing Center, the National Center of Supercomputing in Spain. Dr. Valero has been honored with several awards, including the King Jaime I by the Generalitat Valenciana, and the Spanish national award “Julio Rey Pastor” for his research on IT technologies. In 2001, he was appointed Fellow of the IEEE, in 2002 Intel Distinguished Research Fellow, and since 2003 he has been a Fellow of the ACM. Since 1994, he has been a foundational member of the Royal Spanish Academy of Engineering. In 2005 he was elected Correspondent Academic of the Spanish Royal Academy of Sciences, and his native town of Alfam´en named their public college after him.

Table of Contents

Part I: Special Section on High-Performance Embedded Architectures and Compilers Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Per Stenstr¨ om and David Whalley Recruiting Decay for Dynamic Power Reduction in Set-Associative Caches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Georgios Keramidas, Polychronis Xekalakis, and Stefanos Kaxiras Compiler-Assisted Memory Encryption for Embedded Processors . . . . . . . Vijay Nagarajan, Rajiv Gupta, and Arvind Krishnaswamy

3

4 23

Branch Predictor Warmup for Sampled Simulation through Branch History Matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simon Kluyskens and Lieven Eeckhout

45

Data Cache Techniques to Save Power and Deliver High Performance in Embedded Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Major Bhadauria, Sally A. McKee, Karan Singh, and Gary S. Tyson

65

Combining Edge Vector and Event Counter for Time-Dependent Power Behavior Characterization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chunling Hu, Daniel A. Jim´enez, and Ulrich Kremer

85

Part II: Regular Papers Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Woojin Choi, Seok-Jun Park, and Michel Dubois

107

Fetch Gating Control through Speculative Instruction Window Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hans Vandierendonck and Andr´e Seznec

128

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Minwook Ahn and Yunheung Paek

149

Linux Kernel Compaction through Cold Code Swapping . . . . . . . . . . . . . . Dominique Chanet, Javier Cabezas, Enric Morancho, Nacho Navarro, and Koen De Bosschere

173

XIV

Table of Contents

Complexity Effective Bypass Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Aneesh Aggarwal A Context-Parameterized Model for Static Analysis of Execution Times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christine Rochange and Pascal Sainrat Reexecution and Selective Reuse in Checkpoint Processors . . . . . . . . . . . . Amit Golander and Shlomo Weiss

201

222 242

Compiler Support for Code Size Reduction Using a Queue-Based Processor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Arquimedes Canedo, Ben Abderazek, and Masahiro Sowa

269

Power-Aware Bus Coscheduling for Periodic Realtime Applications Running on Multiprocessor SoC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Khaled Z. Ibrahim and Smail Niar

286

Performance Characterization for the Implementation of Content Addressable Memories Based on Parallel Hashing Memories . . . . . . . . . . . Patrick Mahoney, Yvon Savaria, Guy Bois, and Patrice Plante

307

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

327

Introduction Per Stenstr¨ om1 and David Whalley2 1

Chalmers University of Technology, Sweden 2 Florida State University, U.S.A.

In January 2007, the second edition in the series of International Conferences on High-Performance Embedded Architectures and Compilers (HiPEAC’2007) was held in Ghent, Belgium. We were fortunate to attract around 70 submissions of which only 19 were selected for presentation. Among these, we asked the authors of the five most highly rated contributions to make extended versions of them. They all accepted to do that and their articles appear in this section of the second volume. The first article by Keramidas, Xekalakis, and Kaxiras focuses on the increased power consumption in set-associative caches. They present a novel approach to reduce dynamic power that leverages on the previously proposed cache decay approach that has been shown to reduce static (or leakage) power. In the second article by Magarajan, Gupta, and Krishnaswamy the focus is on techniques to encrypt data in memory to preserve data integrity. The problem with previous techniques is that the decryption latency ends up on the critical memory access path. Especially in embedded processors, caches are small and it is difficult to hide the decryption latency. The authors propose a compiler-based strategy that manages to reduce the impact of the decryption time significantly. The third article by Kluyskens and Eeckhout focuses on detailed architectural simulation techniques. It is well-known that they are inefficient and a remedy to the problem is to use sampling. When using sampling, one has to warm up memory structures such as caches and branch predictors. This paper introduces a novel technique called Branch History Matching for efficient warmup of branch predictors. The fourth article by Bhadauria, McKee, Singh, and Tyson focuses on static power consumption in large caches. They introduce a reuse-distance drowsy cache mechanism that is simple as well as effective in reducing the static power in caches. Finally, in the fifth paper by Hu, Jimenez, and Kremer, the focus is on a methodology to make accurate and fast estimations of the power consumption in a program. The authors note that while detailed power simulation is slow and inaccurate, real power measurements result in huge amounts of data. To this end, they present an infrastructure that can identify the program phases that best characterize the power consumption profile in the program. We do hope that you learn a lot and get inspiration by this excellent excerpt from the second HiPEAC conference.

P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, p. 3, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Recruiting Decay for Dynamic Power Reduction in Set-Associative Caches Georgios Keramidas1 , Polychronis Xekalakis2,⋆ , and Stefanos Kaxiras1 1

Department of Electrical and Computer Engineering, University of Patras, Greece 2 Department of Informatics, University of Edinburgh, United Kingdom {keramidas,kaxiras}@ee.upatras.gr, [email protected]

Abstract. In this paper, we propose a novel approach to reduce dynamic power in set-associative caches that leverages on a leakage-saving proposal, namely Cache Decay. We thus open the possibility to unify dynamic and leakage management in the same framework. The main intuition is that in a decaying cache, dead lines in a set need not be searched. Thus, rather than trying to predict which cache way holds a specific line, we predict, for each way, whether the line could be live in it. We access all the ways that possibly contain the live line and we call this way selection. In contrast to way-prediction, way-selection cannot be wrong: the line is either in the selected ways or not in the cache. The important implication is that we have a fixed hit time — indispensable for both performance and ease-of-implementation reasons. One would expect way-selection to be inferior to sophisticated way-prediction in terms of the total ways accessed, but in fact it can even do better. To achieve this level of accuracy we use Decaying Bloom filters to track only the live lines in ways — dead lines are automatically purged. We offer efficient implementations of such autonomously Decaying Bloom filters, using novel quasi-static cells. Our prediction approach affords us high-accuracy in narrowing the choice of ways for hits as well as the ability to predict misses — a known weakness of way-prediction — thus outperforming sophisticated way-prediction. Furthermore, our approach scales significantly better than way-prediction to higher associativity. We show that decay is a necessary component in this approach — way-selection and Bloom filters alone cannot compete with sophisticated way-prediction. We compare our approach to Multi-MRU and we show that without even considering leakage savings — we surpass it terms of relative power savings and in relative energy-delay in 4-way (9%) and more so in 8-way (20%) and 16-way caches (31%).

1

Introduction

Power consumption is a prime concern for the design of caches and numerous techniques have been proposed to reduce either leakage [11,16,18,23,25] or ⋆

This work has been conducted while Polychronis Xekalakis was studying at the University of Patras, Greece.

P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 4–22, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Recruiting Decay for Dynamic Power Reduction

5

dynamic power [13,14,30,31] but without a clear understanding of how to integrate them in a single implementation. We propose a new approach for reducing dynamic power in set-associative caches by exploiting a previous leakage-saving proposal. Specifically, we use Cache Decay [18] to identify dead cachelines, which we then exclude from the associative search. Our proposal is different from wayprediction proposals that precede it. We do not try to predict a single way for an access but instead we base our approach on information about what is live and what is decayed in the cache. In effect, instead of predicting a way number, we predict whether an access refers to a live line. We make this prediction separately for each cache way and we access only the ways that can possibly hold the line in a live state —we call this way-selection. Way-selection cannot be wrong: the accessed line is either among the selected lines or it is a miss. As a result, we have a fixed hit time as opposed to way-prediction. This is a significant advantage in terms of implementation complexity (with respect to cache pipelining) but more importantly, in terms of performance [10]. Variable hit latency in L1 not only slows down the cache, but creates many difficulties in efficiently scheduling depended instructions in the core. To achieve a level of accuracy to surpass previous proposals, we use a special form of membership-testing hash tables called Bloom filters [3] to track liveliness information for individual lines, per cache way. A normal Bloom filter (per way) would tell us whether a line possibly appeared before in some specific way, but without accounting for the possibility of the line being already dead. But, by decaying Bloom filter entries in concert with corresponding cache lines, a Decaying Bloom filter reflects the presence of the live lines only. A significant benefit of decaying Bloom filter entries is the early prediction of cache misses. This is due to the Bloom filters reflecting the presence of live data in the cache and responding negatively to accesses that are going to miss. Early miss prediction is a boost to both performance and power savings since the cache can be bypassed completely on misses [10]. In fact, the performance benefit of miss prediction more than offsets the performance penalty due to decay. For simple and efficient implementations we propose and study various forms of Decaying Bloom filters, including Self-Decaying Bloom filters which decay their entries autonomously and require no decay feedback from the cache. To support inexpensive and power-efficient implementations of self-decaying Bloom filters we propose novel quasi-static cells designed to offer the necessary decay functionality for Bloom filters. We compare our proposal to a sophisticated and highly-accurate way-prediction scheme called Multi-MRU [31](of similar predictive power to other complex schemes [2,13]) and our results show: – Self-Decaying Bloom filters of equivalent size to Multi-MRU, outpace it in terms of power and energy-delay by performing competitively on hits but doing much better on misses. – With increasing associativity, Self-Decaying Bloom filters, increase their distance from Multi-MRU in all metrics. – Decay is necessary for the Bloom filters to perform well. Structure of This Paper. Section 2 motivates the need for a new approach and presents our proposal. Section 3 discusses details for efficient implementations.

6

G. Keramidas, P. Xekalakis, and S. Kaxiras

Fig. 1. Way-Prediction implemented as an array of way-number predictions (or as way-presence vectors with at most one bit set per vector) and indexed by a hash of the address

Section 4 presents our methodology and Section 5 our results. Section 6 reviews previous work and Section 7 concludes with a summary.

2 2.1

Way-Selection and Decaying Bloom Filters What Is Wrong with Way-Prediction?

Way-prediction was initially proposed to address latency in set-associative caches [2,5,6]. These techniques also lead to power savings since they circumvent the associative search [12,13,17,30,31]. Way-prediction techniques aim to predict a single way where a reference might hit. Fig. 1 shows the general view of way-prediction for a 4-way set-associative cache. For an n-way cache, the waypredictor can be viewed either as an array of log2(n)-bit binary numbers or a n-bit presence vector. A correct prediction results in a fast hit and yields power benefits roughly proportional to the associativity (a single way out of n is accessed). On the other hand, a way misprediction, results in no power savings, and a slow hit, not only because the rest of the ways need to be searched with a subsequent cache access but also because the instructions that depend on the accessed value need to be flushed and re-issued in the core [10,26]. A variable hit latency also complicates cache pipelining adding to its overall complexity with special cases such as a slow-hit (a replay of a mispredicted hit). For fast scheduling of dependent instructions it is very important not only to have a fixed hit time (as we propose) but also to be able to predict misses early. Way-prediction techniques cannot predict misses — predictions are produced even on misses. Additional mechanisms to handle misses are required, e.g., predicting misses with an instruction-based predictor [31], or using Bloom filters to detect some of the misses [10]. Our approach eliminates the need to handle misses separately, encompassing the work of [10]. Some of the most sophisticated way-prediction techniques are those based on a combination of selective direct-mapping (DM) and way-prediction [13,31]. In this paper we compare against one such scheme called MMRU [31]. We will describe this scheme further in Section 2.

Recruiting Decay for Dynamic Power Reduction

7

Fig. 2. Decay and Way-Selection using Decay bits

2.2

Way-Selection

Cache decay was initially proposed to reduce leakage power in caches [18]. It is based on the generational behavior of cachelines, whereupon a cacheline initially goes through a “live time”, where it is accessed frequently, followed by a “dead time”, where it simply awaits eviction. Cache decay shuts-off unneeded cachelines after some period of inactivity — called decay interval — assuming the lines have entered their dead time. The decay interval is measured with coarse 2-bit counters, local to each line, that are advanced by a global cycle counter (local counters are reset with accesses to the lines). A decay (valid) bit in the tag shows whether the line can be accessed or is dead (Fig. 2). The simplest scheme to take advantage of cache decay for dynamic power reduction is to consult the decay status bits of the lines to enable accesses only to the live ways, and obtain dynamic power savings. We call this technique way-selection to distinguish it from way-prediction. The difference is that way-selection selects zero or more ways (up to all the ways) for each access whereas way-prediction strictly chooses a single way to access at all times. Because of this difference way-prediction can mispredict and require a second access, while way-selection cannot. In the case where all the lines in a set are dead, our technique predicts the miss simply by looking at the decay bits. Decay bits, however, do not offer very high accuracy: many times more than one line in a set is live and it is rare to find a completely dead set (in order to predict a miss). In other words, decay bits do not disambiguate beyond the cache index. Ideally, we want to disambiguate addresses at a chosen depth. Thus, we can predict with high probability whether a specific line is live rather than obtain a blanket prediction of which ways are live in the line’s set. The benefits are two-fold: high probability of identifying a single way for a line for possible hits and high probability of identifying misses early. Furthermore, decay bits are typically embedded in the tag array, while we would like to have a separate array that we can access prior to the cache, similarly to other way-prediction structures [5,13,31]. We accomplish our goals using appropriately sized Bloom filters [3] instead of decay bits as our predictors.

8

G. Keramidas, P. Xekalakis, and S. Kaxiras

Fig. 3. Bloom Filters (one per way) sized to disambiguate 2 tag bits (Bloom filters not drawn to scale)

Fig. 4. Behavior equivalence for 3 different implementations of a single Decaying Bloom filter entry which accommodates 3 distinct cachelines overlapping in time

2.3

Decaying Bloom Filters

Bloom filters are simply hash tables that implement a non-membership function. Each Bloom filter entry is a single bit: 1 denotes presence of an object hashed on this entry; 0 denotes absence of any object that can hash on this entry. A Bloom filter tells us with certainty when something is not present, but it cannot tell us what exactly is present because of possible conflicts in its entries. One can arbitrarily lower the probability of conflicts by using multiple hash functions for each inserted object [3]. Bloom filters are widely employed in software (i.e., in web caches, or in dictionary implementations) but recently several proposals incorporated Bloom filters in hardware [8,15,10,21]. Way-selection with Bloom filters is shown in Fig. 3 where the role of the decay bits now is performed by Bloom filters sized to disambiguate two additional tag bits. Bloom Filter Update and Decay. The main weakness of Bloom filters is that it is difficult to update them as there is no way to tell what is in them. In order to solve this problem, Peir et al. use helper structures called Collision Detectors [10] while Sethumadhavan et al. propose using small n-bit up-down saturating counters as Bloom filter entries [21]. In this case, a Bloom filter entry

Recruiting Decay for Dynamic Power Reduction

9

stores the number of objects that hash on it. As long as we attempt to delete objects previously inserted and their total number does not exceed the maximum count, the Bloom filter works as expected. In contrast, we use decay to update our Bloom filters: deletion of a dead line from the Bloom filters is equivalent to the decay of the corresponding Bloom entry. Note that, a Bloom entry stays live until the last of the lines that hash on it decays. Self Decaying Bloom Filters. Decaying Bloom filters track only live data and thus are more accurate. Their entries can decay using feedback from a decaying cache, or autonomously. The latter variety, which we call Self-Decaying Bloom filters (SDBF) is very appealing in terms of implementation simplicity since it does not require a feedback connection to the cache. While feedback or externally decayed Bloom filters are implemented with 2-bit saturating counters as in [21], SDBFs can be digital or analog implementations. Digital implementations use small (2-bit) decay counters corresponding to the local decay counters of the cache, so they can be quite expensive. Thus, for SDBFs, the analog design offers the least expensive implementation. We propose using decaying 4-Transistor quasi-static RAM cells [27] as Bloom filter entries. These cells were proposed for decaying branch predictors and transient non-architectural data [27]. Once written, they lose their charge because of leakage after some preset decay interval unless they are accessed and, consequently, recharged. Functionally, all the decaying Bloom filter implementations are equivalent. Fig. 4 shows the behavior of a single Bloom entry on which three distinct cachelines are hashed and decay after a some interval. The top diagram depicts the behavior of a 2-bit saturating counter which is externally decayed. As live lines are hashed on the entry its counter is increased to 3; it is decremented upon decay of the lines. The middle diagram shows the behavior of a digitally implemented (2-bit decay counter) SDBF for the same scenario. The decay counter is reset to its maximum count whenever a line is accessed and progressively decreases. Only when all lines decay the entry’s counter can fall to zero. Finally, the bottom diagram shows an analog SDBF where the behavior of the decay counter is replaced by the discharge behavior of a quasi-static memory cell.

3

Efficient Analog Implementations

Analog implementations of SDBFs are appealing because they are inexpensive and power-efficient. In terms of power, they require only very little dynamic power to recharge cells, but the digital implementations consume dynamic power to set, reset, increment, and decrement the counters. Although both represent a small overhead compared to the power of accessing the cache, analog implementations have a significant advantage both in dynamic and leakage power. We need, however, to carefully design them for the required functionality. This section discusses analog implementations issues and our solutions. A Decayed 4T Cell Yields a Random Value. Regardless of the value written in the 4T cell, decay means that its high node drops to the level of its low node

10

G. Keramidas, P. Xekalakis, and S. Kaxiras

Fig. 5. 5-T Asymmetrically Decaying Bit

(which floats slightly above ground at the decayed state). Thus, a decayed 4T cell is not capable in producing a significant voltage differential on its bit lines, causing the sense-amps to assume a random state at the end of their sense time. Furthermore, the cell itself might flip to a random state when connected to the bit lines. This is not a concern if the 4T stores non-architectural state, as for example branch predictions [27]. But in our case, we wish to have a specific decay behavior: ones decay to zeros to signify absence of the corresponding cacheline but zeros should remain intact since the benefit in Bloom filters comes from the certainty they provide. Fig. 5 shows our solution. We simply re-instate one of the pMOS transistors connected to Vdd of the 6T design in the 4T design. By doing so we create an asymmetric cell, that stores logical zeroes in a static fashion and flips decayed logical ones to zeroes. In Fig. 5 we can see a SPICE simulation of a typical flipping procedure. While the cell is in the flipping phase, reading its value is unpredictable. Fortunately, the window that this can happen chance of a reading error is very small (Table 1). We handle such 5T transient states as a soft error. Even though these reading errors might not be as infrequent as soft errors, they are still quite rare and this is why they are not a significant source of performance degradation. For our 5T cells it is important to control the retention and flip times. These times greatly depend on technology: as we move to nano-scale technologies, retention times and flip times become smaller. Frequencies scale too, but in a more conservative way. In our designs we need retention times in the range of a few thousand cycles but a 5T cell cannot achieve such long retention times by itself. To adjust its retention time we introduce an extra nMOS in series to ground. Forward-biasing such ground nMOS allows us to precisely control leakage currents, practically placing a resistor in the 5T cell’s path to ground. Table 1 shows a typical set of retention and flip times for 4T, 5T, and 6T cells, for 70nm technology, and a 5GHz clock, with and without ground transistors. Our aim here is not to give absolute numbers but to show that we have the capability to design retention times that suit our architectural needs. A Live Cell is Recharged When Read. This is an inherent property of the quasi-static cells and the basis of keeping accessed entries alive. However, the problem is that we inadvertently keep other Bloom-filter entries alive (which should have decayed) just by reading them. The correct refresh behavior is to refresh only the entry corresponding to a hit and nothing else. Our solution is to

Recruiting Decay for Dynamic Power Reduction

11

decouple an entry’s access from its refresh. Each entry is composed of two bits: an access bit (just for reading) and a refresh bit (for updating). The refresh bit is charged only in the case of a hit in the corresponding way while the access bit is redundantly refreshed with every access. The novelty of our design is that these two bits are coupled in such a way that when the refresh bit decays, it forces the access bit to promptly follow. This is accomplished simply by connecting the high node of the refresh bit to the leakage-control ground transistor of the access bit via a low-leak inverter. When the refresh bit decays, leakage in the access bit is increased by two orders of magnitude. The detailed design is shown in Fig. 6. Temperature and Process Variations. Our design also solves another problem with 4T decay: its dependence to temperature (leakage is exponential to temperature). Using an inexpensive decaying 4T Temperature sensor [19] we monitor temperature and bias the ground transistor appropriately to yield approximately stable decay times across a wide range of temperatures. 4T decay times converge to a specific value at high temperatures [16] so we only have to adjust decay times at low temperatures. Furthermore, as it is shown in [19] for 4T cells, the decay period of 5T cells is not affected by process variation, except by negligible amounts at low temperatures. The variability manifested in Ldrawn, and Tox, can drastically affect leakage current (the effect of the variation in channel dose has little impact). However, there is a physical self-calibration: as Ldrawn and Tox change, affecting the leakage current, the cell’s capacity also changes keeping a balance in the period needed for the cell to decay. Our SPICE simulations show that this counterbalancing effect is especially effective at temperatures above 40o C, where process variation of up to 20% results in negligible variation of the decay time (less than 2% impact) [19]. Finally we should note that the 4T design and it’s dependence to temperature can be exploited to save more leakage as shown in [20]. This gives our design another perspective, which however we choose not to address here since we are focusing on dynamic power consumption rather in static. Soft Errors and Short Decay Intervals. Soft-errors [28], 5T transient states, or thermal variations, can lead to Bloom decay errors which appear as artificially shortened decay intervals. We should note at this point, that due to the increased node capacity we expect our design to be quite resilient to ’1’s flipping to ’0’s (these flips affect the correctness of the our mechanism) [24]. In any case, with

Fig. 6. Decoupled access-refresh decay cells

12

G. Keramidas, P. Xekalakis, and S. Kaxiras Table 1. Retention and Flip Times for Various RAM Cells Type of Cell Retention Time Flip Time (cycles) 6T (cycle=0.2ns) - (cycles) 5T (cycle=0.2ns) 1500 (cycles) 75 5T+1 (cycle=0.2ns) 1500000 (cycles) 590000 4T (cycle=0.2ns) 700 (cycles) 4T+1 (cycle=0.2ns) 4500 (cycles) -

a shortened decay interval, a Bloom filter entry decays before its corresponding cacheline. The Bloom filter reports misleadingly that the line is not in the cache. Since the benefit comes from trusting the Bloom filter and not to search the cache, in case of an error we experience a Bloom-Induced Miss — the line is still in the cache. To handle Bloom-induced misses we could burden miss handling by always checking the cache to make sure that a miss is not Bloom-induced but we need not go that far. The solution is provided in cache hierarchies that support coherency — such as those in the majority of high-performance processors today. Coherent cache hierarchies typically adhere to the inclusion property [1], e.g., L1’s contents are strictly a subset of L2’s. To support inclusion, snooping caches implement an inclusion bit alongside their other coherence bits (MESI). The inclusion bit is set if the line exists at a higher level and cleared when the line ceases to exist in higher levels (by updating the lower-level tags upon replacement [1]). Under this scenario the L2 does not expect a miss from the L1 for a line whose inclusion bit is set — a sign of a Bloom-induced miss. Thus, the L2 acts as a safety net for L1 Bloom-misses. Bloom-induced misses are detected in L2 and L1 is advised of its mistake. The penalty for a Bloom-induced miss, besides a possible unsuccessful partial access of the L1, is an access of the L2’s tags. There is no other associated overhead, such as data transfer. Finally, the L1 must be re-checked after a Bloom-induced miss to locate the desired cacheline. It is evident that this safety-net protection is not offered for the L2 (the level before main memory) where the cost of a Bloom-induced miss — a main memory access — is also very high. In this case we use digital SDBFs or digital externally-decayed Bloom filters.

4

Methodology

Here, we discuss our simulation setup, details of the MMRU scheme we compare against and latency and power issues affecting the comparisons. Simulation Setup. For our simulations we use Wattch [7] and Cacti [29]. We simulate a 4-issue processor with 32K, 32-byte block, 3-cycle L1 cache and a 1M, 64-byte block, 8-way, 15-cycle L2 cache. For the L1 we assume, optimistically, a 3cycle pipelined design [26]. Although we do not penalize competing approaches with mis-scheduling overhead we refer the reader to [10] for its performance

Recruiting Decay for Dynamic Power Reduction

13

Fig. 7. Multi-MRU employs N MRU predictors (N == assoc) to disambiguate among different tags

Fig. 8. 4 partitions of 4 BFs (one per way) sized to disambiguate 2 tag bits. Total size is equal to MMRU and differences (with Fig. 7) are shown for tags *00 and *10.

quantification. Our performance results do reflect, however, the negative impact of slow hit times (3+3 cycles) for way-prediction as well as the latency benefits of predicting misses (1 cycle). We run all SPEC2000 benchmarks with the reference inputs, skipping the first billion instructions and collecting statistics for the second half billion. In all graphs the benchmarks are sorted by the base case miss-rate (lowest to highest). 4.1

MMRU and Bloom Filter Sizing

Multi-MRU or MMRU [31] is a virtual “selective-DM” extension of the Most Recently Used (MRU ) way-prediction [5]. MRU returns the most recently accessed way of a set as its prediction but MMRU allows multiple MRU predictors to disambiguate among different tags (Fig. 7). All tags in a set ending with the same bits are tracked by the same MRU table. Bloom Filter Sizing.While we can use Bloom filters of any size — trading their accuracy — we size the Bloom filters to equal aggregate size to MMRU way-predictors. This configuration is shown in Fig. 8 for 4-way caches. The Bloom filters are shown partitioned according to the two lower tag bits for direct comparisons with the MMRU. Note the important differences from Fig. 7: the four Bloom filters per way collectively allow more than one bit in the presence vector for the *00 tag; the presence vector for a decayed tag (e.g., tag *10) is 0 — it is the decay of the Bloom entries that give us the ability to predict misses.

14

4.2

G. Keramidas, P. Xekalakis, and S. Kaxiras

Latency Issues

Early way-prediction papers focused on reducing the latency of L1 set-associative caches [2,5,12,6]. For such schemes to work well, way-prediction must be performed prior to cache access. Approximate-address indexing schemes [5] or instruction-PC indexing [2,6] were proposed for this reason. PC-based schemes suffer from low prediction accuracy and implementation complexity [13]. Consistently, using the effective address yields the highest prediction accuracy [2,5,6,13]. Our view is that it is not as critical any longer to use approximate indexing schemes, given that L1 caches are not single-cycle. L1 caches are by necessity pipelined [26]. In this environment, predictor access can take place in the first pipeline stage or even occupy its own stage. Predictor access proceeds in parallel with decode and halts further operations in subsequent cycles once the predictor outcome is known. Our Cacti simulations show that accessing a small predictor structure of few kilobits is comparable in latency to L1 decode, in accordance to previous work. Power expended in cache decode, in parallel with predictor access, is fully accounted in our methodology. 4.3

Power Issues

Our approach introduces two sources of additional power: i) Bloom filters arrays, and ii) decay-induced misses. Our Cacti estimates showed that energy for accessing the largest predictor (Bloom or MMRU) is less than 1% of a full L1 access (in accordance to prior work [13,31]). We consider the extra dynamic power of decay-induced misses (extra accesses to lower level caches due to incorrect decay of cachelines) to be part of the leakage savings [18]. Having a decaying cache to reduce leakage, one would have to pay for this dynamic-power penalty regardless of any additional mechanism for dynamic power reduction. Of course, we do not count leakage savings as part of our benefits. However, we do take into account the performance penalty due to decay. As we will show, we turn this performance penalty into a performance benefit with way-selection and SDBF. 4.4

Decay Issues

With respect to adaptive decay for L1 caches [9] and for L2 [4], it is straightforward to adapt the decay in feedback-decayed or externally-decayed Bloom filters as well as digital SDBF, since they all mirror the decay of the cache. Analog SDBF can be made adaptive using our technique for adjusting retention times at various temperatures: we bias the ground transistor appropriately to obtain desired decay time [19]. This gives us the ability to adjust the decay interval for both the cache and the (analog) SDBF. Adapting the decay interval individually per Bloom entry (as would be required by [18]) is difficult but Velusamy et al. have shown that it is unnecessary since adapting a single decay interval is better [22]. Since adaptive Bloom filters are feasible in any variety, the problem of finding the best decay interval is largely orthogonal to our work.

Recruiting Decay for Dynamic Power Reduction

15

Fig. 9. 4-way SDBF vs. MMRU: ways accessed on hits, on misses and in total

5

Results

We show simulation results for three cases: first we show how decaying Bloom filters compare against MMRU for 4-way caches. We then show the scalability of our approach to higher associativity (8-way and 16-way) and we close by showing that decay is a necessary component in our approach. 5.1

Decaying Bloom Filters vs. MMRU

The main result of this section is that way-selection with Decaying Bloom filters is more power-efficient than the highly accurate MMRU. SDBF use a decay interval of 8Kcycles. They are coupled to a decaying cache (32KB) with the same decay interval. SDBF and MMRU are equal in cost and disambiguate 2 tag bits beyond the cache index. The top graph in Fig. 9 shows the number of ways accessed on hits for SDBF and MMRU. The numbers are normalized to the number of ways accessed by the base case without any prediction. SDBF lags behind MMRU — this is the result of selecting more than one way to access in our predictions. In contrast, SDBF does very well in reducing the number of ways accessed on misses (Fig. 9, middle), predicting zero ways for most misses for an average of 10% of the total ways of the base case for misses. In fact, the greater the miss rate of the benchmark, the better SDBF does compared to MMRU which always accesses 100% of the ways of the base case on misses. On average SDBF accesses 29% of the total ways of the base case, while MMRU accesses 31% (Fig. 9, bottom).

16

G. Keramidas, P. Xekalakis, and S. Kaxiras

Fig. 10. 4-way SDBF vs. MMRU: execution time, relative EDP

In terms of power savings the SDBF usually performs better than the MMRU resulting in a slight relative advantage of 5% for the SDBF (Fig. 10, middle). While SDBF speeds up many benchmarks, MMRU slows down most benchmarks (by about 1% on average, but more than 3% for the high-miss-rate ammp and art as shown in Fig. 10, top graph). The very small slowdown of MMRU attests to the fact that we do not simulate in enough detail the negative effects of variable hit latency on the cache pipeline or on the scheduling of dependent instructions [10]. On the other hand, it should be noted that SDBF speeds-up a decaying cache which is inherently at a performance disadvantage (compared to a normal non-decaying cache) because of decay-induced misses. Under these conditions the relative energy-delay product (EDP) of the SDBF is 9% better than MMRU’s (Fig. 10, bottom graph). 5.2

Scaling with Associativity

In this section we discuss how associativity affects SDBF and MMRU. Fig. 11 and Fig. 12 show the results for an 8-way 32KB L1 cache. SDBF improves dramatically in 8-way caches for both hits and misses. Its improvement in hits (Fig. 11, top graph) allows it to easily approach MMRU while further increasing its distance for the misses (Fig. 11, middle graph). Note also the improvement in the two high-miss-rate benchmarks ammp and art because of the improved prediction of misses. With respect to execution time (Fig. 12) for SDBF, overhead due to decayinduced misses tends to be more pronounced since the base 8-way cache has an improved miss rate. However, this is balanced by the improved prediction of misses and the resulting fast misses. Overall, we have no significant change for SDBF speed-up. For MMRU, we have a small increase in its slow-down (across most benchmarks) as a combined effect of decreased accuracy and improved

Recruiting Decay for Dynamic Power Reduction

17

Fig. 11. 8-way SDBF vs. MMRU: ways accessed on hits, on misses and in total

Fig. 12. 8-way SDBF vs. MMRU: execution time and relative EDP

miss-rate of the base case. In 8 ways, energy-delay is improved for MMRU but not as much as for SDBF which doubles its distance from MMRU to 21%. Finally, Table 2 compares the averages (over all benchmarks) for the various metrics for up to 16-ways. Ways accessed are reported as a percentage of the base case without prediction. As we can see SDBF matches the performance of MMRU on hits in 16-ways while at the same time increases its distance significantly for the misses. The result is that SDBF halves its percentage of total ways accessed (29% for 4-ways, 15% for 8-ways, 7% for 16-ways) with each doubling in associativity while MMRU improves much slower (31%, 20%, 14% respectively).

18

G. Keramidas, P. Xekalakis, and S. Kaxiras Table 2. Comparison of SDBF and MMRU for various cache associativities 4-ways 8-ways 16-ways MMRU SDBF MMRU SDBF MMRU SDBF ways accessed on hits 26% 30% 14% 15% 7% 7% ways accessed on hits 100% 10% 100% 5% 100% 2% total ways accessed 31% 29% 20% 15% 14% 7% power savings 59% 61% 69% 74% 75% 81% relative EDP 91% 79% 69%

Fig. 13. 4-way Non-decaying Bloom Filters vs. SDBF: ways accessed on hits, misses, total, and relative power savings

Power savings follow the trend of the total ways accessed and relative EDP improves with each doubling of the associativity: SDBF is 9%, 21% and 31% better than MMRU in EDP for 4-, 8- and 16-way caches.

Recruiting Decay for Dynamic Power Reduction

5.3

19

The role of Decay in Bloom Filters

In this section, se show that Decay is necessary for Bloom filters to be competitive. Fig. 13 compares SDBF to BF (insert-delete Bloom Filters). Without decay, BF accesses an average of 35% more ways than SDBF for hits — more than doubles for mcf. The situation is much worse for misses: BF accesses on average 270% more ways than SDBF. For 17 benchmarks the number shoots up more than 100%. This increases the total ways accessed for the BF of 37%. BF’s power savings are 16% lower than SDBF’s.

6

Related Work

Techniques for dynamic-power reduction in caches can be divided into two categories: cache resizing and way-prediction. Cache resizing (e.g., DRI cache [23], CAM-tag resizing [14]) is a straightforward technique to reduce dynamic, static, and precharge power in caches. However, our approach can be contrasted more directly to way-prediction. As we mentioned previously the most sophisticated way-prediction techniques are a combination of selective direct-mapping (DM) and way-prediction [13,31]. The idea of selective direct-mapping is to assign each line to a specific way based on its lower log2(associativity) tag bits. Either the line truly occupies its assigned way or this assignment is virtual and used simply to obtain predictions for this line [31]. There are two separate proposals for the combined selective-DM/way-prediction technique: by Zhu and Zhang [6,31] and by Powell et al. [13] based on an earlier (latency-focused) paper by Batson and Vijaykumar [2]. We have already described the Zhu et al. proposal [31] (MMRU), a virtual ”selective-DM” extension of the MRU way-prediction [5]. Zhu et al. also examine schemes where the DM assignment is actual rather than virtual but such schemes require line swapping — which consumes power — and complex replacement policies — which diverge from LRU and could have unpredictable consequences. The selective DM approach of Powell et al. [13] is an actual DM assignment where the lines are ”placed” in their DM position unless they generate conflicts. In terms of prediction power this scheme aims to place as many lines as it can in their DM positions and handle the rest with a way-predictor. MMRU tracks all such lines, both those in their DM position and those in other positions, thus yielding approximately the same prediction accuracy — for both schemes, on average 92% of the predictions result in first hits for 4-way caches [13,31]. Because of this prediction equivalence we compare our approach to MMRU noting that our conclusions would be similar with respect to the Powell approach. Bloom Filters. Bloom filters in hardware were proposed by Moshovos et al. as snoop filters in [15]. Sethumadhavan et al. use Bloom filters to detect absence of dependencies in Load/Store Queues thus avoiding fully-associative searches [21]. Peir et al. use Bloom filters to detect misses — absence of lines — in the cache and schedule dependent instructions accordingly [10]. Dharmapurikar et al. use Bloom filters to implement the Longest Prefix Match algorithm [27].

20

G. Keramidas, P. Xekalakis, and S. Kaxiras

Finally, way-selection is similar to the way-halting cache proposed by Zhang et al. [30] where accesses to ways that cannot possibly contain the desired line are ”halted”. Way-halting is limited by the use of expensive CAMs for the lower 4 bits of the tags to determine tag mismatch per way and thus difficult to scale to a large number of sets. In contrast, we use Bloom filters instead of CAM-based halt tags, thus providing better scalability. Our decay techniques, however, can benefit way-halting too, clearing the halt tags from useless entries and increasing their effectiveness in predicting misses.

7

Conclusions

Considerable work on reducing static and dynamic power in caches has been performed in recent years. We propose a dynamic power application for a leakagepower mechanism, allowing the seamless integration of the two. To achieve dynamic power reduction with decay we propose way-selection (similar to wayhalting) and Decaying Bloom filters. Way-selection offers a significant advantage over way-prediction: that of a fixed hit time. We also propose several variations of Decaying Bloom filters, most notably the analog Self-Decaying Bloom filters, that decouple decay in our predictors from decay in the cache. This gives us the flexibility to use Decaying Bloom filters with non-decaying caches (or for instance state-preserving Drowsy caches). We have performed the experiments for this case and obtained very similar results as with using decay caches. Decay, however, is necessary for the Bloom filters to be efficient. Another advantage of our approach is that it integrates prediction of misses. Despite the fact that we do not simulate some of the negative consequences of the variable hit time of way-prediction, we have shown that our approaches can outperform some of the most sophisticated way-prediction proposals. And this, without taking into account leakage savings of the integrated dynamic/leakage power mechanism.

Acknowledgments Stefanos Kaxiras is partially supported by the EU FP6 Integrated Project Future and Emerging Technologies, Scalable computer ARCitecture (SARC) Contract No. 27648, and the EU FP6 HiPEAC Network of Excellence IST004408. Georgios Keramidas is fully supported by a Grant (PENED 2003) which is funded by the European Commission by 75%, from the Hellenic State - Greek Ministry of Development - General Secretariat of Research and Technology (GSRT) by 25% and by private industry through the Operational Programme “Competitiveness”, Measure 8.3, 2000-2006. The equipment used for this work is a donation by Intel Corporation under Intel Research Equipment Grant #15842.

Recruiting Decay for Dynamic Power Reduction

21

References 1. Baer, J.-L., Wang, W.: On the inclusion properties for multi-level cache hierarchies. In: Proc. of the Int. Symp. of Computer Architecture (1988) 2. Batson, B., Vijaykumar, T.N.: Reactive-associative caches. In: Proc. of PACT (2001) 3. Bloom, B.: Space/time trade-offs in hash coding with allowable errors. Commun. ACM 13(7) (1970) 4. Abella, J., et al.: Iatac: a smart predictor to turn-off l2 cache lines. In: ACM TACO (2005) 5. Calder, B., et al.: Predictive sequential associative cache. In: Proc. of the Symp. on High-Performance Computer Architecture (1996) 6. Zhang, C., et al.: Two fast and high-associativity cache schemes. IEEE Micro. 17(5) (1997) 7. Brooks, D., et al.: Wattch: a framework for architectural-level power analysis and optimizations. In: Proc. of the Int. Symp. of Computer Architecture (2000) 8. Dharmapurikar, S., et al.: Longest prefix matching using bloom filters (2003) 9. Zhou, H., et al.: Adaptive mode control: A static-power-efficient cache design. Trans. on Embedded Computing Sys. 2(3) (2003) 10. Peir, J.-K., et al.: Bloom filtering cache misses for accurate data speculation and prefetching. In: Proc. of the Int. Conference on Supercomputing (2002) 11. Flautner, K., et al.: Drowsy caches: Simple techniques for reducing leakage power. In: Proc. of the Int. Symp. of Computer Architecture (2002) 12. Inoue, K., et al.: Way-predicting set-associative cache for high performance and low energy consumption. In: Proc. of ISLPED (1999) 13. Powell, M., et al.: Reducing set-associative cache energy via way-prediction and selective direct-mapping. In: Proc. of the Int. Symp. on Microarchitecture (2001) 14. Zhang, M., et al.: Fine-grain cam-tag cache resizing using miss tags. In: Proc. of ISLPED (2002) 15. Moshovos, A., et al.: Jetty: Snoop filtering for reduced energy consumption in smp servers (2001) 16. Powell, M., et al.: Gated-vdd: A circuit technique to reduce leakage in deepsubmicron cache memories. In: Proc. of ISLPED (2000) 17. Min, R., et al.: Location cache: a low-power l2 cache system. In: Proc. of ISLPED (2004) 18. Kaxiras, S., et al.: Cache decay: Exploiting generational behavior to reduce cache leakage power. In: Proc. of the Int. Symp. of Computer Architecture (2001) 19. Kaxiras, S., et al.: 4t-decay sensors: a new class of small, fast, robust, and lowpower, temperature/leakage sensors. In: Proc. of ISLPED (2004) 20. Kaxiras, S., et al.: A simple mechanism to adapt leakage-control policies to temperature. In: Proc. of ISLPED (2005) 21. Sethumadhavan, S., et al.: Scalable hardware memory disambiguation for high-ilp processors. In: Proc. of Micro., vol. 24(6) (2004) 22. Velusamy, S., et al.: Adaptive cache decay using formal feedback control. In: Proc. of the Workshop on Memory Performance Issues (2002) 23. Yang, S., et al.: An integrated circuit/architecture approach to reducing leakage in deep-submicron high-performance i-caches. In: Proc. of the Int. Symp. on HighPerformance Computer Architecture (2001) 24. Degalahal, V., et al.: Analyzing soft errors in leakage optimized sram design. In: Proc. of the Int. Conference on VLSI Design (2003)

22

G. Keramidas, P. Xekalakis, and S. Kaxiras

25. Li, Y., et al.: State-preserving vs. non-state-preserving leakage control in caches. In: Proc. of the Conference on Design, Automation and Test in Europe (2004) 26. Chishti, Z., et al.: Wire delay is not a problem for smt (in the near future). In: Proc. of the Int. Symp. of Computer Architecture (2004) 27. Hu, Z., et al.: Managing leakage for transient data: Decay and quasi-static 4t memory cells. In: Proc. of ISLPED (2002) 28. Borkar, S.: Design challenges of technology scaling. In: Proc. of Micro. (1999) 29. Wilton, S., Jouppi, N.: Cacti: An enhanced cache access and cycle time model. IEEE Journal of Solid-State Circuits 31(5), 677–688 (1996) 30. Zhang, C., Asanovic, K.: A way-halting cache for low-energy high-performance systems. ACM Trans. Archit. Code Optim. 2(1) (2005) 31. Zhu, Z., Zhang, X.: Access-mode predictions for low-power cache design. IEEE Micro. 22(2) (2002)

Compiler-Assisted Memory Encryption for Embedded Processors Vijay Nagarajan, Rajiv Gupta, and Arvind Krishnaswamy University of Arizona, Dept. of Computer Science, Tucson, AZ 85721 {vijay,gupta,arvind}@cs.arizona.edu

Abstract. A critical component in the design of secure processors is memory encryption which provides protection for the privacy of code and data stored in off-chip memory. The overhead of the decryption operation that must precede a load requiring an off-chip memory access, decryption being on the critical path, can significantly degrade performance. Recently hardware counter-based one-time pad encryption techniques [13,16,11] have been proposed to reduce this overhead. For high-end processors the performance impact of decryption has been successfully limited due to: presence of fairly large on-chip L1 and L2 caches that reduce off-chip accesses; and additional hardware support proposed in [16,11] to reduce decryption latency. However, for low- to medium-end embedded processors the performance degradation is high because first they only support small (if any) on-chip L1 caches thus leading to significant off-chip accesses and second the hardware cost of decryption latency reduction solutions in [16,11] is too high making them unattractive for embedded processors. In this paper, we present a compiler-assisted strategy that uses minimal hardware support to reduce the overhead of memory encryption in low- to medium-end embedded processors. In addition to the global counter used in [13], our technique uses additional counters. These counters, which are compiler controlled, are maintained using a small number of dedicated on-chip registers. Our experiments show that the proposed technique reduces average execution time overhead of memory encryption for low-end (medium-end) embedded processor with 0 KB (32 KB) L1 cache from 60% (13.1%), with single counter, to 12.5% (2.1%) by additionally using only 8 hardware counter-registers.

1

Introduction

There is significant interest in the development of secure execution environments which guard against software piracy, violation of data privacy, code injection attacks etc. [9,16,13,6,17,18,14]. Memory encryption is a critical component of such secure systems. While the code and data residing on-chip are considered to be safe, code and data residing off-chip can be subject to attacks. Thus, for example, to provide defense against software piracy and to protect the privacy of data, memory encryption is employed. Data leaving the chip must be encrypted before being sent off-chip for storing and when this data is referenced again by the processor, and brought on-chip, it must be decrypted. P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 23–44, 2009. c Springer-Verlag Berlin Heidelberg 2009 

24

V. Nagarajan, R. Gupta, and A. Krishnaswamy

The overhead of decryption operations that precede loads requiring access to off-chip memory can be very high and hence can lead to significant performance degradation. Techniques based on one-time-pad encryption or counter mode block ciphers [16,13,11] have been proposed that encrypt/decrypt data using a one-time-pad derived from a key and a mutating counter value. This enables the computation of the one-time-pad to be decoupled from the reading of encrypted data from off-chip memory. For high-end processors the performance impact of decryption has been successfully limited due to two reasons. First, the presence of fairly large on-chip L1 and L2 caches reduce the frequency of offchip accesses. Second in [16,11] additional hardware support has been proposed to reduce decryption latency. The techniques in [16] and [11] enable the onetime-pad to be precomputed while data is being fetched from off-chip memory by caching counter values and predicting them respectively. While the combination of above techniques is quite effective in limiting performance degradation in high-end processors, they rely on significant on-chip hardware resources. An additional cache is used [16] for caching the counter values. It is well known that caches occupy most of the on-chip space (about 50%) in embedded processors [21]. For this reason, several embedded processors are built without a cache as shown in Table 1. To obviate a large counter cache, the prediction technique [11] uses multiple predictions to predict the value of the counter and speculatively perform the multiple decryptions. As it performs multiple decryptions, 5 decryptions are performed in parallel, this technique requires multiple decryption units. In the prototype implementation of the AEGIS single-chip secure embedded processor [12], each encryption unit causes significant increase in the gate count (3 AES units and an integrity verification unit caused a 6 fold increase in the logic count). Thus, the hardware cost of decryption latency reduction solutions in [16,11] is too high making them unattractive for low- to medium-end embedded processors. For such processors the performance degradation is high because first they do not support on-chip L2 data caches and they only support small (if any) on-chip L1 data caches thus leading to significant off-chip accesses. Table 1 presents a list of commercial embedded processors, Table 1. Data Cache Sizes of Embedded Processors D-Cache Embedded Processor – 20 MHz to 700 MHz 0 KB [21] ARM7EJ-S, ARM Cortex-M3, ARM966E-S [22] SecureCore SC100/110/200/210 [23] NEC V850ES-20, Motorola MRC555-40 [23] Renesas M32R/E-40 2 KB [23] STMicro ST20C2 50 4 KB [19] NEC V832-143, Infineon TC1130 [19] NEC VR4181, Intel 960IT 8 KB [23] NEC V850E-50, Infineon TC11B-96 [19] NEC VR4181A, Hitachi AH7750S [19] Motorola MPC823E, IBM Power PC 405CR [19] Xilinx Virtex IIPro 16 KB [19] Motorola MPC8240, Alchemy AU 1000 32 KB [23] MIPS 20Kc, AMD K6-2E, AMCC 440GX [23] AMCC PPC440EP, Intel XSscale Core

Compiler-Assisted Memory Encryption for Embedded Processors

CRYPTO UNIT

D−cache

Key

I−cache

CPU Global Counter

cache block

25

CODE Encrypted using Key DATA Encrypted using Key & Counter Values

Counter Values Used

Encrypted

Not Encrypted

Fig. 1. Secure Processor

all of which have no on-chip L2 caches. Some of them do not even have an on-chip L1 data cache, while others have L1 data caches varying from 2 KB to 32 KB. In this paper, we develop a compiler-assisted strategy where the expensive task of finding the counter value needed for decrypting data at a load is performed under compiler guidance using minimal hardware support. Thus, the need to either cache or predict counter values using substantial on-chip hardware resources is eliminated. In addition to the global counter, our technique uses multiple compiler allocated counters for each application. The additional counters are implemented as special registers which reside on chip. The compiler allocates these counters for store instructions. Instructions that compute the one-time-pad (using the allocated counter value) are introduced preceding the stores and loads by the compiler. For a store, the counter used is the one that is allocated to it. For a load, the counter used is the one belonging to its matching store – this is the store that frequently writes the values that are later read by the load. Through its allocation policy, the compiler tries to ensure that when a load is executed, the value of the counter allocated to its matching store has not changed since the execution of the store. Thus, the pad needed for decryption can usually be determined correctly preceding the load using the counter associated with the matching store. In other words, a prediction for the counter that is to be used for a load is being made at compile-time and hard coded into the generated code. Our experiments show that for vast majority of frequently executed loads and stores, matches can be found that produce highly accurate compile-time predictions. When a prediction fails, the value of the pad is computed using the information fetched from off-chip memory (the counter value used in encryption). The global counter is used to handle all loads and stores that cannot be effectively matched by the compiler. Our experiments show that this compiler-assisted strategy supports memory encryption at a reasonable cost: minimal on-chip resources; and acceptable code size and execution time increases. The remainder of this paper is organized as follows. In section 2 we review how encryption/decryption is performed and motivate this work by showing the high overhead of memory encryption for low- and medium-end embedded processors. In section 3 we present our compiler-assisted strategy in detail. Experimental results are presented in section 4. Additional related work is discussed in section 5 and we conclude in section 6.

26

2

V. Nagarajan, R. Gupta, and A. Krishnaswamy

Background and Motivation

Let us begin by reviewing the encryption/decryption scheme used in [13] – the organization of secure processor is shown in Fig. 1. When a plaintext data value p is to be written to off-chip memory, the corresponding ciphertext c that is actually written is computed as follows: c = p ⊕ E n c r y p t (K , A d d r + C o u n t e r ) where ’+’ is the concatenation operator, K is the key that is kept secure inside the processor chip, A d d r is the (block) address to which p is to be written, and C o u n t e r is a value that is used once to produce the one-time-pad and hence is incremented after being used. A mutating counter is needed because in its absence the same pad, i.e. E n c r y p t K (A d d r ), will be repeatedly used during sequence of writes to the same memory location. Thus, a skilled attacker will be able to easily crack the ciphertexts stored in memory [16]. It should be noted that counter is needed only in case of data but not for code since code is read-only and thus never written back to off-chip memory. Ciphertext read from the off-chip memory is decrypted by the following operation performed on-chip: p = c ⊕ E n c r y p t (K , A d d r + C o u n t e r ). The off-chip memory is augmented with additional memory locations where the actual counter value used during encryption of data is stored. Thus, the counter value can be fetched along with the data so that decryption can be performed. However, doing so causes load latency to increase because computation of onetime-pad cannot be performed until the Counter value has been fetched. In the presence of an on-chip data cache, blocks of data are transferred across the chip boundary. All data items in a given block share the same counter value. Therefore, as described in [13], when a cache block is fetched from off-chip memory, the counter value which is one word can be fetched much faster than the larger cache block and therefore the computation of one-time-pad can be partly performed while the cache block is being fetched. This optimization takes advantage of the difference in sizes of a cache block and the counter and hence is more effective for high performance processors which have bigger cache blocks. It should be noted that the counter values are not stored in encrypted form because from these values an attacker cannot decrypt the data values stored in memory [2]. We measured the execution time overhead of supporting memory encryption according to [13] as described above for two processor configurations: a lowend processor configuration is based upon parameters provided in [23,7]; and a medium-end processor configuration uses parameters consistent with Intel Xscale. The decryption latency was computed based upon two real implementations of encryption and decryption hardware units described in [24] – the high performance implementation is used for medium-end processor configuration and lower performance implementation is assumed for low-end processor configuration. For low-end configuration we considered data cache sizes of 0, 2, and 4 KB while for

Compiler-Assisted Memory Encryption for Embedded Processors

60%

8K 16K 32K

50% 40% 30% 20%

qsort

mean

jpeg

lame

rijndael

crc

dijkstra

fft

stringsearch

sha

0%

adpcm

10%

bitcnts

qsort

mean

jpeg

lame

rijndael

crc

dijkstra

fft

string_search

sha

Percentage Overhead of Memory encrypti

Medium end parameters Processor speed: 600 MHz Issue: inorder L2 cache: None Data cache: 8, 16, 32 KB, 32B line Instruction cache: 32 KB Memory latency (1st chunk): 31 cycles Memory bus : 600 MHz 8-B wide Load/store queue size : 8 Decryption latency: 31 cycles Source : [25] 0k 2K 4K

adpcm

80% 70% 60% 50% 40% 30% 20% 10% 0%

bitcnts

Percentage Overhead of Memory encrypti

Low-end parameters Processor speed : 100 MHz Issue: inorder L2 cache: none Data cache: 0, 2, 4 KB, 16B line Instruction cache: 8 KB Memory latency (1st chunk): 12 cycles Memory bus: 100 MHz 4-B wide Load/store queue size: 4 Decryption latency : 14 cycles Source: [7] [23]

27

Fig. 2. Memory Encryption Overhead

medium-end configuration we considered data cache sizes of 8, 16, and 32 KB. The processor configurations are shown in Fig. 2. For low-end configuration the average overhead is 60% in absence of on-chip L1 data cache while it is 12.3% and 9.9% for 2 KB amd 4 KB L1 data caches. For medium-end configuration the average overhead is 20.2%, 14.3%, and 13.1% for 8 KB, 16 KB, and 32 KB L1 data caches respectively.

3

Compiler-Assisted Encryption

In this section, we show that determining the appropriate counter value can be transformed into a software (compiler) intensive task with minimal hardware support beyond the single global counter used in [13]. This transformation requires that we provide a small number of additional on-chip counters. These additional counters are allocated by the compiler to stores to ensure that matching loads (loads that read values written by the stores) can at compile-time determine which counter is expected to provide the correct counter value. We illustrate the basic principle of the above approach using the example in Fig. 3. Let us assume that function f in Fig. 3 contains a store Storef and its matching load Loadf as they both reference the same address Af . Following Storef and preceding Loadf the call to function g results in execution of Storeg zero or more times. For the time being let us assume that Ag can never be the same as Af . Fig. 3b shows how the key K and counter C0 are used preceding the stores to generate the one-time-pad by the technique in [13]. However, when we reach Loadf the generation of the one-time-pad requires the counter value Cf i n d – this value is fetched from off-chip memory in [13], found using caching

28

V. Nagarajan, R. Gupta, and A. Krishnaswamy

function f() { .... S toref Af g(); L o a d f Af .... }

function f() { .... Encrypt(K, Af +++ C0 ) Storef Af g(); Encrypt(K, Af + Cf ind ) Loadf Af .... }

function f() { .... Encrypt(K, Af + Cid1 +++ C1 ) Storef Af g(); Encrypt(K, Af + Cid1 + C1 ) Loadf Af .... }

function g() { while (..) { .... Storeg Ag .... }}

function g() { function g() { while (..) { while (..) { .... .... ++ Encrypt(K, Ag + Encrypt(K, Ag + Cid0 +++ C0 ) C0 ) Storeg Ag Storeg Ag .... .... }} }} (a) Original Code. (b) Single Counter. (c) Multiple Counters.

Fig. 3. Making Counter Predictable by Compile-time Allocation of Counters

in [16] and using prediction in [11]. Instead let us assume we are provided with additional on-chip counter-register C1 . We can use C1 to generate one-time-pad for Storef and then regenerate this pad at Loadf again by using C1 as shown in Fig. 3c. C1 is guaranteed to contain the correct counter value for Loadf because Storeg does not modify C1 as it still relies on counter C0 . Thus, by using a separate counter-register for Storef and its matching load Loadf , we are able to avoid the interference created by Storeg . Under the assumption that Af and Ag are never the same, the one-time-pad produced in the above manner for Loadf is always correct. From the above discussion it is clear that we need to use a separate counterregister for Storef . However, as we can see, we have also introduced a prefix to the counters, counter-id (Cid). Next we explain the reason for adding the counter-ids. Recall that in the above discussion we assumed that Af and Ag are never the same. Now we remove this assumption and consider the situation in which Storef Af and Storeg Ag can write to the same address. Since these stores use different counter-registers, the values of these counter-registers can be the same. When this is the case, the pads they use to write to the same address will also be the same. Obviously this situation cannot be allowed as the purpose of using counters to begin with was to ensure that pads are used only once. Hence by using a unique prefix (counter-id) for each counter, we ensure that the onetime-pads used by Storef and Storeg are different even if their counter-register values match (C1 = C0 ) and they write to the same addresses. In other words, the counter, in our scheme is composed of the counter-id and the contents of the counter-register. Since the counter-registers are allocated at compile time, the counter-id of each allocated counter-register is known at compile time. Hence the counter-ids are hard coded into the instructions that are introduced (before stores and loads) to compute the pads. Finally, we would like to point out that in the above example when Af and Ag happen to be the same, it has another consequence. The pad computed at Loadf in Fig. 3c will be incorrect. One way to view our technique is that it relies

Compiler-Assisted Memory Encryption for Embedded Processors

29

on the compiler to specify the identity of the counter that is likely to contain the needed counter value at a load but not guaranteed to contain it. When the counter value is incorrect, first we must detect that this is the case. As is the case in [13], when off-chip memory is written to, in associated storage the counter value that was used to produce the value is also stored. In our case, since each counter is prefixed by a counter-id, in addition to the counter value, the counter-id is also written. When data is fetched from memory, the counter-id and counter value are also fetched and compared with the counter-id and and counter value used to precompute the pad. If the match fails, we must recompute the pad before performing decryption; otherwise decryption can proceed right away and latency of computing the pad is hidden. Thus, incorrect precomputation of a pad does not impact the correctness of decryption but rather it affects the performance as pad computation time cannot be hidden. The task of the compiler is to identify matching (store,load) pairs and allocate counter-registers (and counter-ids) accordingly. Finally, the counter register and counter-id assignments made to the stores and loads are communicated to the hardware through special instructions described in the following section which are introduced preceding the appropriate stores and loads. Stores and loads that cannot be effectively matched are simply handled using the global counter – in other words no instructions are introduced to handle them. Fig. 4 summarizes the modified secure processor model needed by our technique. The global counter (C0 ) is used in the encryption of all data written by stores that do not have any matching counterparts. Counter-registers (C1 , C2 , ...Cn ) are the counters that are allocated by the compiler to stores. Data memory contents are now encrypted and for each block of memory in it, a word of memory containing additional information is present which includes the counter-register value and the counter-id that was used to encrypt the data. Security of the scheme. Note that the use of multiple counters does not adversely affect the security of the encryption scheme. The concatenation of the (counter-id, counter value) pair in our scheme can be considered as the unique counter and hence our compiler-assisted encryption scheme is a direct implementation of the counter-mode encryption scheme, which is proved to be a secure symmetric encryption scheme [2]. Further it is shown that the the counters may be safely leaked without compromising the security [2]. Hence the counter values and counter-ids are not stored in encrypted form. CPU C0 Global Key K0 I−cache

Cn

CRYPTO UNIT

C1 C2 . . .

CODE

Global Counter

Counter Registers

cache block

Original + special instructions Encrypted DATA Encrypted

using compiler controlled counters: C 0 , C 1, C 2..... C n

D−cache

Encrypted

Fig. 4. Secure Processor

(counter id, counter value) pairs used Not Encrypted

30

V. Nagarajan, R. Gupta, and A. Krishnaswamy

In the absence of an on-chip L1 cache, each load and store involves movement of a single word across the chip boundary. Each time a value is stored in memory the corresponding (counter-id, counter value) pair is also written. In the presence of an on-chip L1 cache, a cache block is transferred across the chip boundary. In this case, when a store is performed, the (counter-id, counter value) pair may be written to the L1 cache. These pairs can then be written to the memory when the block is written back. In this case, the space overhead of holding (counter-id, counter value) pairs in cache is 12.5% for 32B cache block. To avoid the space overhead the (counter-id, counter value) pairs can be updated in memory in a write through fashion. We have explained the basic principles of our technique. Obviously, the number of counter-registers needed for a program to carry out the above task is an important issue, since counter-registers require additional on chip hardware. The size of the counter-id field is directly dependant on the number of counters allocated. Although counter-ids do not take up on-chip resource, it is still desirable to keep the size of this field minimal, since counter ids are to be stored along with each memory block. As our experiments later show, with careful allocation of counters, it is sufficient to support 8 counter-registers and generate up to 32 counter ids (5 bits of counter-id). In our implementation we use 32 bits to store the combined counter-id and counter value. When data value stored is encrypted using the Global Counter, we indicate the use of Global counter by having the most significant bit as 1 and remaining 31 bits represent the Global Counter Value. When data stored is encrypted using an allocated counter-register, the most significant bit is always 0, next 7 bits represent the counter-id (this restricts the maximum number of counter-ids to 128, which is more than sufficient), and the remaining 3 bytes are used to store the counter-register Value. Using 3 byte counter-register is sufficient in our design because multiple counter- registers are used and each such register is shared by a smaller number of stores. The Global Counter, on the other hand, is shared by a larger number of stores and therefore we maintain a 31 bit Global Counter. Our experiments presented later confirm that smaller size counters are adequate. 3.1

Instruction Set Support

The ISA is extended with instructions that manipulate the counter registers. We introduce two new instructions: the incctr instruction is used to increment a specified counter-register (Ck ); and the counter instruction that specifies the counter-register (Ck ) and counter-id (Cid), to use for encryption/decryption. These instructions are implemented using an unimplemented opcode in the ARM processor shown below. To specify the two instructions we have introduced an 1 bit opcode (as only two new instructions are needed). Additional bits are used to specify the Ck and Cid fields. Unimplemented ARM Instruction 1111 1111 xxxx [31..24] [23..0]

Compiler-Assisted Memory Encryption for Embedded Processors

31

L: ... for(i=..) { ... .... str , value1i A[i] = value1i bcond BR if () A[i] = value2i ; str , value2i value3i = A[i] BR: ... ... .... ldr value3i , } bcond L L: ... ... incctr ctrk counter Cidk ,ctrk str , value1i bcond BR str , value2i BR: ... ... counter Cidk ,ctrk ldr value3i , bcond L

Fig. 5. Illustrating Use of Instructions

We now explain with the help of the example shown in Fig. 5, how the various hardware components interact. The source level code shows an array location being updated to one of two values and the same array location being read later in the loop. This results in two store (str) instructions and one load (ldr) instruction as shown. We assume that the compiler matches the first str instruction with the ldr instruction. As a result, the compiler uses the same (counter-id, counter-register) values for this pair. The incctr instruction is used at the head of the loop, before the store, to increment the counter value associated with this load/store pair. The counter instruction is used to specify the appropriate counter-id and counter-register before both the load and the store for the precomputation of the pad. The first store uses the (counter-id, counter-register) pair specified in the counter instruction. The second store, which was not matched to any load by the compiler, uses the global counter. Since these unmatched stores are not preceded by counter instructions the hardware is able to detect such stores and subsequently the global counter value is incremented automatically after each such store instruction. Both stores write the (counter-id, counter value) pair to the memory along with the encrypted data. When the load is encountered, the (counter-id, counter value) pair specified in the program is used to compute the pad, where the the counter value refers to the contents of the counter-register specified in the counter instruction. As the data word to be loaded is being fetched from memory, the corresponding (counter-id, counter value) pair is also fetched. To enable this, we assume that the memory bus width is twice as large as the original. Now the counter-id and the value of the counter specified in the counter instruction are compared with the ones retrieved from memory. If there is a match, the precomputed pad is correct and can be XOR’ed with the fetched data value for decryption. If the comparison fails, the counter-id and the counter value retrieved from memory are used to generate the correct pad and decrypt the data. It is worth noting that wrong prediction effectively adds the decryption latency on to the critical path and stalls the pipeline additionally

32

V. Nagarajan, R. Gupta, and A. Krishnaswamy

by this amount. The failure in our example, corresponds to the case where the second store instruction is executed before the load instruction. 3.2

Compiler Support

Next we describe the compiler support employed to assign counter-ids and counter-registers to selected stores and their matching loads. Before we present this algorithm we will first discuss how opportunities for sharing counter-registers across multiple stores arise and can be exploited. At first glance it may seem that the counter allocation problem that we encounter here is similar to the classical register allocation problem. As we shall see in the following paragraph, it is the unique nature of the sharing opportunities that arise due to the semantics of the counter based encryption mode, that distinguishes our allocation from the classical register allocation problem. Sharing Counter-Registers. We assign different generated counter-ids to different static stores. Therefore, two stores being handled using two different counter-ids need not always use different counter values and hence different counter-registers. Two different counter-registers are necessary only if these two static stores interfere with each other as was illustrated by our example in Fig. 3. Next we show that the above observation can be exploited to enable counterregister sharing in certain cases. We describe two general categories of sharing opportunities: intraprocedural sharing and across procedure sharing. We describe these opportunities using examples given in Fig. 6. In the first example in Fig. 6, function f contains three stores which have all been assigned different generated counter-ids (Cid1 , Cid2 , and Cid3 ). Let us assume that we have a counter-register Cf which is incremented each time the function f() { Cf++ g ; function f() { .... ++ Cf ; Encrypt(K, Af + Cidf + Cf g ) .... Storef Af Encrypt(K, A1 + Cid1 + Cf ) .... no (in)direct calls to f/g() Store1 A1 Encrypt(K, Af + Cidf + Cf g ) while (..) { Loadf Af .... .... Encrypt(K, A2 + Cid2 + Cf ) } Store2 A2 .... function g() { } Cf++ g ; Encrypt(K, A3 + Cid3 + Cf ) .... Store3 A3 Encrypt(K, Ag + Cidg + Cf g ) .... Storeg Ag Encrypt(K, Al + Cidl + Cf ) .... no (in)direct calls to f/g() Load Al Encrypt(K, Ag + Cidg + Cf g ) .... Loadg Ag } .... } (a) Intraprocedural Sharing.

(b) Across Procedure Sharing.

Fig. 6. Sharing Counters Across Multiple Stores

Compiler-Assisted Memory Encryption for Embedded Processors

33

function f is entered. The counter Cf ’s value can be used without any restriction by Store1 and Store3 since they are executed at most once in the function. While Store2 is executed multiple times, if the compiler can determine that during each iteration of while loop address A2 changes, then Store2 can also use Cf ’s value that is computed upon entry to f for deriving its pad. In other words, during a single invocation of f , it is safe for all three stores to use the same value of Cf to derive their pads, as they use different counter-ids. The load at the end of the function now knows to refer to counter-register Cf irrespective of which of the three stores it is matched to and the counter-id Cidl is one of the three depending upon which store the load is matched to based upon the profile data. The above example showed how three different stores shared the same counter registers. But note that the three stores used unique counter-ids. The second example in Fig. 6 illustrates across procedure sharing of both counter-ids and counter-registers. In this example two functions are shown such that each of these functions contains a counter-register to handle its local pairs of matching stores and loads: (Storef , Loadf ) in function f and (Storeg , Loadg ) in function g. Moreover these functions are such that in between the matching pair in one function, no function call exists that can lead to the execution of either function. In other words, execution of matching store load pair in one function cannot be interleaved by execution of the matching pair in the other function. Thus, when we reach a store (Storef or Storeg ), counter Cf g is incremented and used and when we reach a load (Loadf or Loadg ) Cf g ’s value can be used as it has not changed since the execution of the corresponding store. Since the execution of these functions can never be interleaved, sharing the same counter-register by the two functions does not lead to any interference. It should be noted that due to sharing of the counter-register, the two store load pairs in this example are guaranteed never to use the same counter value and thus they can safely share the same counter-ids. Thus across procedural sharing can lead to reduction of counter-ids as well as counter-registers. In conclusion, first code can be analyzed intraprocedurally to allow sharing of counters among multiple stores in a function. Then, by analyzing the call graph and hence the lifetimes of pairs of functions, we can determine if two functions in a pair can use the same counter. Based upon the examples already presented, the conditions under which we allow sharing are as follows: – Intraprocedural sharing. Given a function, a subset of stores in the function share the same counter register, if each of the stores in the subset writes to a unique address during a single invocation of the function. To simplify counter-register allocation, we assign at most one counter to each function and this counter is used to cover a subset of stores that satisfy the above condition. The remaining stores make use of the default global counter. – Across-procedure sharing. Given two functions, they can be assigned the same counter (counter-id, counter-register) if there does not exist a pair of store live ranges, one from each function, such that the execution of these live ranges interfere (i.e., can be interleaved with each other). Here a store live range is the program region that starts from the store instruction and

34

V. Nagarajan, R. Gupta, and A. Krishnaswamy

extends up to and including all of its matching loads. If a function call is present within a store live range of one function that can lead to the execution of another function and hence its store live range, then the execution of two store ranges interfere with each other. To simplify the analysis for computing interferences, we apply this optimization only to those functions whose store live ranges are local to the function (i.e., the store and all its matching loads appear within the function). Counter-id and Counter-register Allocation Algorithm. Now we are ready to describe the steps of the complete compiler algorithm required by our technique. This algorithm operates under the constraint that we are given a certain maximum number of counter-ids (N K ) and counter-registers (N C ). The algorithm is formalized and presented in Fig. 7. 1. Match Stores and Loads. The first step of our algorithm is to carry out profiling and identify matching static store and load pairs. If a load matches multiple stores (as at different times during execution it receives values from different static stores) only the most beneficial matching pair is considered because in our technique a load is matched to a single store. Note that a store may match multiple loads during the above process, this is allowed by our technique. 2. Find Counter-register Sharing Opportunities. We look at function at a time and identify the subset of stores in the function that satisfy the intraprocedural counter sharing condition given earlier. Given a function fi , fi .Stores denotes the subset of stores identified in this step. During counter allocation in the next step, if a function fi is allocated a counter-register, then the stores in fi .Stores will make use of that counter. For every pair of functions fi and fj , we also determine whether these functions can share the same counter according the across procedure counter sharing condition presented earlier. 3. Construct Weighted Function Interference Graph We construct an interference graph, where the nodes of the graphs are the functions. There exists an edge between two functions, if Share(fi , fj ) is false. For each function fi we compute a priority which is simply the expected benefit resulting from allocating a counter-register to function fi . 4. Allocate Counter-registers. In this step we color the interference graph using a greedy heuristic and allocate counter registers to functions. 5. Assign Counter-ids. For each counter-register C, we examine the set of functions F that have been assigned this counter-register. Each function f ∈ F would have used as many counter-ids as the number of stores in the f.Stores set. However, the same counter-ids can be shared across the functions in F . Therefore the number of generated counter-ids that are needed to handle the functions in F is the maximum size of the f.Stores set among all f ∈ F . The above process is repeated for all counter-registers. It should be noted that in this process we may exceed the number of counter-ids available NK . However, this situation is extremely rare. Therefore we use a simple method to handle this situation. We prioritize the counter-ids based upon the benefits of

Compiler-Assisted Memory Encryption for Embedded Processors Step 1: Match Loads and Stores Let {l1 ,l2 . . . lm } and {s1 ,s2 . . . sn } be the set of static loads and stores Let {f1 ,f2 . . . fp } be the set of functions Let ReachCount(si , lj ) = k if static store si reaches lj k times Let M atchingLoad(si ) = lj if ReachCount(si , lj ) is maximum for i = 1 to n (i) Compute M atchingLoad(si ) end for Step 2: Find Sharing Opportunities for each function fi Compute the set fi .Stores that satisfies intraprocedural sharing criterion end for Let boolean function Share(fi , fj ) be true if fi , fj can share a counter for each function pair (fi , fj ) (i) Compute Share(fi , fj ) using across-procedural sharing criterion. end for Step 3: Construct Weighted Function Interference graph G = (V, E) where V = F ,the set of functions E(fi , fj ) = true iff Share(fi , fj ) = f alse Let DynamicCount(si ) be the dynamic execution count for the static store for each fi  (i) Compute P riority(fi ) = DynamicCount(sj ), sj ∈ fi .Stores end for Step 4: Assign counter registers to functions using a greedy allocation algorithm Let M = φ, where M (ci ) denotes set of functions assigned to counter ci Sort F = {fi } in decreasing order of priority for i = 1 to p (i) Try to assign fi to a previously assigned counter cj iff Share(fi , M (cj )) is true; Otherwise, allocate new counter (ii) Add the new counter-to-function mapping to M end for Step 5: Assign counter-ids Let M = φ, where M (cidi ) denotes set of stores assigned to counter id cidi  Let S = fi .Stores, ∀ fi that has been allocated a counter Prioritize each static store within S based on same heuristic used for functions. for each si ∈ S (i) Try to assign a previously assigned cid iff si and each of M (ci ) come from different functions and all of the above functions are allocated the same counter register; Otherwise assign a new counter-id; If we have exhausted counter-ids goto step 6 (ii) Add the new counter-id-to-store mapping to M end for Step 6: Generate Code for each si that has been allocated a counter-id (i) Insert counter instructions before the si and before M atchingLoad(si ) end for for each fi that has been allocated a counter register (i) Insert incctr instruction at the start. end for

Fig. 7. Compiler Allocation Algorithm

35

36

V. Nagarajan, R. Gupta, and A. Krishnaswamy

the stores with which the counter-ids are associated. The stores corresponding to the low priority counters are then handled using the global counter such that the number of generated counter-ids does not exceed NK . 6. Generate Code. All static stores that have been assigned a counter-id and allocated a counter register, and their matching loads, are now known. Thus we can generate the appropriate instructions for computing the pad preceding each of these instructions. All stores and loads that have not been assigned a generated counter-id and a counter register during the above process will simply make use of the global counter and thus no code is generated to handle them.

4

Experimental Evaluation

We conducted experiments with several goals in mind. First and foremost we study the effectiveness of our approach in reducing execution time overhead of memory encryption over the scenario where memory encryption is implemented using simply the global counter-register. This study is based on the low- and medium-end processor configurations with no L1 cache and varying L1 cache sizes as described in Fig. 2. We also evaluate the effectiveness of our compiler techniques in detecting opportunities for counter sharing. Since code size is an important concern in embedded systems, we measure the static code size increase due to our technique. Finally, we study the impact of using smaller sized counterregisters (31 bits for global counter and 24 bits for additional counter-registers). Our implementation was carried out using the following infrastructure. The Diablo post link time optimizer [20,5] was used to implement the compiler techniques described in the paper. The Simplescalar/ARM simulator [3] was used to collect profiles and simulate the execution of modified binaries. As we mentioned earlier, the processor configurations (low and medium) from Fig. 2 are used. We use the 128 bit AES algorithm to compute the one-time-pad using a crypto unit in hardware as mentioned before. Experiments were performed on the MiBench benchmark suite [7]. The small datasets from MiBench were used to collect profiles and the large datasets were used in the experimental evaluations. Not all the benchmarks could be used in the experiments because at this point some of them do not go through the Diablo infrastructure being used. 4.1

Evaluation of Sharing Optimizations

We studied the effectiveness of our counter sharing strategy by conducting the following experiment. For each program we examined the profile data and identified all of the matching store load pairs, i.e. pairs that can benefit by our technique. The number of statically distinct stores covered by these pairs represents the number of counter-ids, and hence the number of counters, that will be needed if no counter sharing is performed. Application of sharing techniques

Compiler-Assisted Memory Encryption for Embedded Processors

37

Table 2. Number of Counter-registers Used: APS+IS:IS:Unopt Benchmark

Threshold of Dynamic Loads Covered 100% 99% 98% bitcount 19:46:106 (20%) 3:8:40 (8%) 2:5:16 (1%) sha 24:55:161 (15%) 2:3:25 (8%) 1:2:24 (4%) adpcm 12:29:58 (21%) 7:19:46 (15%) 7:18:44 (16%) fft 23:59:191 (12%) 8:20:108 (7%) 8:19:102 (8%) stringsearch 16:34:69 (23%) 8:18:48 (17%) 8:17:45 (18%) crc 23:52:137 (17%) 15:35:88 (17%) 12:28:54 (22%) dijkstra 25:60:174 (14%) 6:14:76 (8%) 5:13:67 (7%) rijndael 21:47:226 (9%) 8:17:140 (6%) 7:15:130 (5%) jpeg 40:138:560 (7%) 10:24:217 (5%) 6:19:178 (3%) lame 49:144:1344 (4%) 9:30:811 (1%) 7:23:660 (1%) qsort 25:57:164 (15%) 7:15:52 (13%) 5:11:45 (11%)

reduces this number greatly. While the intraprocedural sharing reduces the number of counter-registers, the across-procedural sharing reduces both the number of counter-ids and counter-registers. Next we present results that show the reductions in number of counter-registers and counter-ids that is achieved by our optimizations. These results are given for different thresholds, where the threshold represents the percentage of dynamic loads covered during counter allocation. Here the percentage is with respect to all dynamic loads that can derive some benefit from our technique if enough counter-registers were available. In Table 2 we present the number of counter-registers that are needed for each program in following cases: (APS+IS) with both Across-Procedure and Intraprocedural Sharing, (IS) only intraprocedural sharing, and (Unopt) without sharing. In addition, the number of counter-registers used with full sharing as a percentage of counter-registers needed without any sharing is also given. As we can see, this number is computed by threshold settings of 100%, 99% and 98%. From the data in Table 2 we draw three conclusions. First the counter sharing strategy is highly effective. For example, for threshold of 100%, the number of counter-registers used after sharing ranges from 4% to 23% of the number used without sharing (on an average we have a 5 fold reduction). Second we observe that even if we set the threshold to 99%, in all cases 16 counter-registers are sufficient. Third we observe that both intraprocedural and across-procedure Table 3. Number of Counter-ids Used APS:Unopt Sharing Benchmark

Threshold of Dynamic Loads Covered 100% 99% 98% bitcount 62:106 (58%) 29:40 (73%) 13:16 (81%) sha 98:161 (61%) 16:25 (64%) 15:24 (63%) adpcm 32:58 (55%) 25:46 (54%) 24:44 (44%) fft 109:191 (57%) 69:108 (64%) 65:102 (65%) stringsearch 43:69 (62%) 30:48 (63%) 29:45 (64%) crc 89:137 (65%) 52:88 (59%) 31:54 (57%) dijkstra 95:174 (55%) 40:76 (53%) 38:67 (57%) rijndael 148:226 (65%) 100:140 (71%) 91:130 (70%) jpeg 338:560 (60%) 130:217 (60%) 115:178 (65%) lame 705:1344 (52%) 468:811 (58%) 377:660 (57%) qsort 109:164 (66%) 31:52 (60%) 45:45 (100%)

38

V. Nagarajan, R. Gupta, and A. Krishnaswamy

sharing optimizations contribute significantly although intraprocedural sharing contributes more than across-procedure sharing. In Table 3 the number of counter-ids used with and without sharing are given. In addition, the number of counters used with sharing as a percentage of counters needed without sharing, is also given. As we can see, across-procedure sharing results in use of 55% of number of counters needed without sharing (i.e., we have roughly 2 fold reduction). Note that Intraprocedural sharing does not cause reduction in counter-ids. Although the counter-ids do not represent a hardware resource, reducing the number of counter-ids is beneficial as the size of counter-id allocated can be reduced. 4.2

Execution Time Overhead

Next we conducted experiments to study the effectiveness of our strategy in reducing execution time overhead. We present the overheads of the two techniques: (Encrypted Optimized) which is our compiler-assisted strategy; and (Encrypted Unoptimized) which is the version in which memory encryption was performed using simply the global counter (i.e., this corresponds to [13]). Execution times were normalized with respect to the execution time of the (Unencrypted) configuration, i.e. the configuration that does not perform memory encryption. The results of this experiment are given in Fig. 8 for low-end and medium-end processor configurations respectively. For each benchmark the three bars correspond to the three cache sizes. Each bar is stacked to allow comparison of the overheads of the (Encrypted Optimized) and (Encrypted Unoptimized). As we can see, for a low-end processor configuration, while the average overhead of (Encrypted Unoptimized) method is 60% (0 KB), 12.4% (2 KB), and 9.9% (4 KB), the overhead of our (Encrypted Optimized) method is 12.5% (0 KB), 2.3% (2 KB), and 1.9% (4 KB). Thus, the benefits of using our technique are substantial for the low-end processor configuration. As expected, we observe that the benefit of our technique is greatest for processors that support no on-chip L1 cache. Moreover, our technique is beneficial for all benchmarks in this case. However, when an on-chip L1 data cache is provided, due to very low data cache miss rates, some of the benchmarks (bitcount, sha, adpcm, and fft) do not benefit

Fig. 8. Overhead of Memory Encryption - Low and Medium

Compiler-Assisted Memory Encryption for Embedded Processors

39

from our technique. In case of the medium-end configuration the savings are substantial again (excluding the first four benchmarks that have very low miss rates). While the average overhead of (Encrypted Unoptimized) method is 20.1% (8 KB), 14.3% (16 KB), and 12.3% (32 KB), the overhead of our (Encrypted Optimized) approach is 5.0% (8 KB) 2.8% (16 KB), and 2.1% (32 KB). In the above experiment we always used 8 counter-registers and up to 32 generated counter-ids. Next we wanted to see if more counter-ids and counter-registers would further produce significant savings in the execution time overhead. Therefore we conducted an experiment in which we compared the execution time overhead for (8,32) configuration with (16,64) and (4,16) configurations. We considered the low-end configuration with 0 KB cache for this experiment because this is the configuration that needs the most counter-registers and counter-ids. As we can see from the results in in Fig. 9, the (8,32) configuration performs nearly as well as (16,64) configuration but the (4,16) configuration shows a significant drop in performance. Therefore we conclude that 8 counter registers and up to 32 generated counter-ids are well suited across the set of benchmarks used. Finally, we present the distribution of dynamic loads in terms of how they are treated by our technique. We divide the dynamic loads into three categories: (unoptimized) dynamic loads to which our technique could not be applied; (prediction correct) dynamic loads that benefited from our technique; and (prediction wrong) dynamic loads to which our technique was applied but it failed as the value loaded was produced by a different store than the one to which the load was matched by the compiler. The results of this experiment for the low-end configuration with 0 KB cache size are shown in Fig. 10. As we can see, a substantial percentage of dynamic loads benefit from the application of our technique. Only in case of sha, lame, and jpeg the compile time matches that failed to produce correct pads account for over 10% of dynamic loads. For other programs we observe that our profile-guided approach of matching loads to stores turns out to be highly accurate.

Unoptimized conf(16,64) conf(8,32) conf(4,16)

2 1.5 1

Benchmark

Fig. 9. Overhead for 8 vs. 16 Counter-registers

mean

qsort

lame

jpeg

rijndael

dijkstra

crc

string_search

fft

adpcm

0

sha

0.5

bitcount

Normalized Execution Time

2.5

V. Nagarajan, R. Gupta, and A. Krishnaswamy prediction wrong prediction correct unoptimized

80% 60% 40%

mean

lame

qsort

jpeg

dijkstra

rijndael

stringsearch

fft

sha

0%

adpcm

20%

bitcount

Dynamic Loads (%)

100%

crc

40

Benchmark

Fig. 10. Loads that Benefit

4.3

Static Code Size Increase

Since static code size is an important concern in embedded systems, we also measured the increase in static code size due to the introduction of additional instructions. The results of this experiment are given in Fig. 11 for the two configurations of our architecture. As we can see, with aggressive application of our approach, i.e. for (16,64) configuration, the maximum increase in static code size is only 4%. For the (8,32) configuration the maximum code size increase is around 3% while for many programs this increase is 1% or less. Thus, we conclude that code size increases that result from our technique are quite small. 4.4

Counter Sizes

conf(16,64) conf(8,32)

5% 4% 3% 2%

Benchmark

Fig. 11. Code Size Increase

mean

qsort

lame

jpeg

rijndael

dijkstra

crc

string_search

fft

adpcm

0%

sha

1%

bitcount

Code Size Increase (%)

Recall that our technique requires that we store both the counter-id and counter value used to derive the pad used in encryption along with encrypted data. We use 1 word of storage to store both counter-id and counter value while the

Compiler-Assisted Memory Encryption for Embedded Processors

41

Table 4. Maximum Counters Values Observed Benchmark Global Only Compiler-assisted (Million) (Million) Global Allocated (Max. of 8) bitcount 2.8 1.50 0.900 sha 12.5 2.90 0.005 adpcm 9.6 8.60 0.394 fft 13.5 8.20 0.222 stringsearch 4.2 1.97 0.064 crc 54.0 0.12 0.006 dijkstra 23.1 11.17 0.221 rijndael 15.5 3.5 0.120 jpeg 53.6 20.20 0.287 lame 68.2 32.18 0.315 qsort 10.2 1.80 0.007

technique in [13] uses 1 word to store only the counter value. We present the maximum counter values reached if one global counter is used versus if one global counter and 8 additional allocated counters are used in Table 4. As we can see, the values of counters used by our technique are lower than that of one global counter used in [13]. The data in Table 4 confirms that using 31 bit global counter instead of a 32 bit counter is not a problem as our global counter is incremented significantly fewer times for nearly all the benchmarks. Also three byte additional counters are enough because their maximum value is quite small (the number given in the table is the maximum of the highest values reached by all 8 counters). From these observations we can conclude that although the use of multiple counter-ids caused us to use smaller counters, overall there was no adverse affect as use of multiple counters ensures that smaller sized counters are sufficient. Finally we would like to point out that if a counter overflows, data in off-chip memory must be re-encrypted using a new key and the counter is reinitialized. However, as shown in [13], although memory re-encryption is expensive, need for it arises rarely. In fact in all our program runs we did not encounter counter overflow situation. In our technique the allocated counter values are quite small so they are not expected to overflow. The maximum value of 31 bit global counter in our technique is much lower than the maximum value of 32 bit counter in case of [13]. Thus, re-encryptions due to global counter overflow in our technique are not expected to be more frequent than observed in [13].

5

Related Work

We have already shown the benefits of using a small number of additional compiler controlled counter-registers over the basic technique of [13] which only uses a single global counter. As mentioned earlier, hardware enhancements to [13] have been proposed in [16] and [11]. However, significant on-chip resources are devoted for caching [16] or prediction [11] which makes these solutions unattractive for embedded processors. Memory predecryption [10] is also a technique used to hide the latency of the decryption. The basic idea here is to prefetch

42

V. Nagarajan, R. Gupta, and A. Krishnaswamy

the L2 cache line. Prefetch can increase workload on the front side bus and the memory controller. Moreover in the absence of on-chip L1 cache prefetch would be needed for every load making this approach too expensive. A recent trend in embedded processors is to provide on-chip compiler managed scratch-pad memory as an alternative to on-chip caches [1,8,15]. Our technique is also applicable in presence of scratch-pad memories. Since the presence of these memories will reduce traffic to off-chip memory, the cost of memory encryption will be reduced further. Our paper deals essentially with support for memory encryption in embedded processors which is useful for among other things for protecting the privacy of code and data in off-chip memory. Other types of attacks have also been considered by researchers which we briefly discuss next. Work has been carried out to detect tampering of information stored in off-chip memory [9,6]. The hardware cost of detecting tampering is very high. In context of embedded processors where on-chip resources are limited, it is more appropriate to follow the solution proposed by the commercial DS5002FP processor [26]. Tamper-detection circuitry is provided that prevents writes to be performed to off-chip memory. However, off-chip memory can be read; hence techniques are still needed to protect privacy of code and data in off-chip memory. Address leakage problem has been studied and techniques have been proposed for its prevention in [17,18]. However, this is orthogonal to the problem we have studied. The solutions proposed in [17,18] are still applicable. Defense against code injection attacks is also an important problem which is being extensively studied (e,g., see [4,14]). Memory encryption techniques, such as what we have described in this paper, are also a critical component in building a defense against remote code injection attacks [4].

6

Conclusions

In this paper, we argued that existing techniques for caching [16] and predicting [11] counter values for reducing memory encryption overhead, although suitable for high-performance processors, are not suitable for low- and medium-end embedded processors for which on-chip hardware resources are not plentiful. Therefore we developed a strategy in which a small number of additional counterregisters are allocated in a manner that enables that the counter-register to be used at each load is determined at compile-time. The specified counter-register is expected to contain the correct counter value needed for decryption. The only hardware cost is due to small number of counter-registers that must be supported on-chip. Our experiments show that the proposed technique reduces average execution time overhead of memory encryption for low-end (mediumend) embedded processor with 0 KB (32 KB) L1 cache from 60% (13.1%), with single counter, to 12.5% (2.1%) by additionally using only 8 compiler controlled counter-registers that accommodate 32 different counters.

Compiler-Assisted Memory Encryption for Embedded Processors

43

Acknowledgments We would like to thank the anonymous reviewers for providing useful comments on this paper. This work is supported by a grant from Intel and NSF grants CNS-0614707, CCF-0541382, and CCF-0324969 to the University of Arizona.

References 1. Banakar, R., Steinke, S., Lee, B.-S., Balakrishnan, M., Marwedel, P.: Scratchpad Memory: Design Alternative for Cache On-chip Memory in Embedded Systems. In: Tenth International Symposium on Hardware/software Codesign (May 2002) 2. Bellare, M., Desai, A., Jokipii, E., Rogaway, P.: A Concrete Security Treatment of Symmetric Encryption: Analysis of the DES Modes of Operation. In: 38th Symposium on Foundations of Computer Science. IEEE, Los Alamitos (1997) 3. Burger, D., Austin, T.M.: The Simplescalar Tool Set, Version 2.0. In: Computer Architecture News, pp. 13–25 (June 1997) 4. Cowan, C., Beattie, S., Johansen, J., Wagle, P.: Pointguard: Protecting Pointers from Buffer Overflow Vulnerabilities. In: 12th USENIX Security Symposium (August 2003) 5. De Bus, B., De Sutter, B., Van Put, L., Chanet, D., De Bosschere, K.: LinkTime Optimization of ARM Binaries. In: ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES 2004) (2004) 6. Gassend, B., Edward Suh, G., Clarke, D.E., van Dijk, M., Devadas, S.: Caches and Hash Trees for Efficient Memory Integrity. In: Ninth International Symposium on High-Performance Computer Architecture (HPCA), pp. 295–306 (2003) 7. Guthaus, M.R., Ringenberg, J.S., Ernst, D., Austin, T.M., Mudge, T., Brown, R.B.: MiBench: A Free, Commercially Representative Embedded Benchmark Suite. In: IEEE 4th Annual Workshop on Workload Characterization (December 2001) 8. Kandemir, M., Ramanujam, J., Irwin, M.J., Vijaykrishnan, N., Kadayif, I., Parikh, A.: Dynamic Management of Scratch-pad Memory Space. In: 38th Design Automation Conference (DAC), pp. 690–695 (June 2001) 9. Lie, D., Thekkath, C., Mitchell, M., Lincoln, P., Boneh, D., Mitchell, J., Horowitz, M.: Architectural Support for Copy and Tamper Resistant Software. In: Ninth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 168–177 (November 2000) 10. Rogers, B., Solihin, Y., Prvulovic, M.: Memory Predecryption: Hiding the Latency Overhead of Memory Encryption. In: Workshop on Architectural Support for Security and Anti-Virus (2004) 11. Shi, W., Lee, H.-H.S., Ghosh, M., Lu, C., Boldyreva, A.: High Efficiency Counter Mode Security Architecture via Prediction and Precomputation. In: 32nd Annual International Symposium on Computer Architecture (ISCA) (June 2005) 12. Suh, G.E., O’Donell, C.W., Sachdev, I., Devadas, S.: Design and Implementation of the AEGIS Single-Chip Secure Processor using Physical Random Functions. In: 32nd Annual International Symposium on Computer Architecture (ISCA) (June 2005) 13. Suh, G.E., Clarke, D., Gassend, B., van Dijk, M., Devadas, S.: Efficient Memory Integrity Verification and Encryption for Secure Processors. In: 36th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), December 2003, pp. 339–350 (2003)

44

V. Nagarajan, R. Gupta, and A. Krishnaswamy

14. Tuck, N., Calder, B., Varghese, G.: Hardware and Binary Modification Support for Code Pointer Protection From Buffer Overflow. In: 37th Annual International Symposium on Microarchitecture (MICRO), pp. 209–220 (2004) 15. Udayakumaran, S., Barua, R.: Compiler-decided Dynamic Memory Allocation for Scratch-pad based Embedded Systems. In: International Conference on Compilers, Architectures and Synthesis for Embedded Systems (CASES) (2003) 16. Yang, J., Zhang, Y., Gao, L.: Fast Secure Processor for Inhibiting Software Piracy and Tampering. In: 36th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), December 2003, pp. 351–360 (2003) 17. Zhuang, X., Zhang, T., Pande, S.: HIDE: An Infrastructure for Efficiently Protecting Information Leakage on the Address Bus. In: 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), October 2004, pp. 72–84 (2004) 18. Zhuang, X., Zhang, T., Lee, H.-H.S., Pande, S.: Hardware Assisted Control Flow Obfuscation for Embedded Processors. In: International Conference on Compilers, Architecture, and Synthesis for Embedded Systems (CASES) (September 2004) 19. Zhang, C., Vahid, F., Najjar, W.: A Highly Configurable Cache Architecture for Embedded Systems. In: 30th Annual International Symposium on Computer Architecture (ISCA), p. 136 (2003) 20. DIABLO, http://www.elis.ugent.be/diablo/ 21. http://www.arm.com/products/CPUs/embedded.html 22. http://www.arm.com/products/CPUs/securcore.html 23. Benchmark reports, http://www.eembc.org/ 24. http://www.opencores.org/projects.cgi/web/ aes core/overview/ 25. Intel XScale, http://www.intel.com/design/intelxscale/ 26. DS5002FP Secure Microprocessor Chip, Dallas Semiconductor, MAXIM, http://www.maxim-ic.com/

Branch Predictor Warmup for Sampled Simulation through Branch History Matching Simon Kluyskens and Lieven Eeckhout ELIS Department, Ghent University Sint-Pietersnieuwstraat 41, B-9000 Gent, Belgium [email protected]

Abstract. Computer architects and designers rely heavily on simulation. The downside of simulation is that it is very time-consuming — simulating an industry-standard benchmark on today’s fastest machines and simulators takes several weeks. A practical solution to the simulation problem is sampling. Sampled simulation selects a number of sampling units out of a complete program execution and only simulates those sampling units in detail. An important problem with sampling however is the microarchitecture state at the beginning of each sampling unit. Large hardware structures such as caches and branch predictors suffer most from unknown hardware state. Although a great body of work exists on cache state warmup, very little work has been done on branch predictor warmup. This paper proposes Branch History Matching (BHM) for accurate branch predictor warmup during sampled simulation. The idea is to build a distribution for each sampling unit of how far one needs to go in the pre-sampling unit in order to find the same static branch with a similar global and local history as the branch instance appearing in the sampling unit. Those distributions are then used to determine where to start the warmup phase for each sampling unit for a given total warmup length budget. Using SPEC CPU2000 integer benchmarks, we show that BHM is substantially more efficient than fixed-length warmup in terms of warmup length for the same accuracy. Or reverse, BHM is substantially more accurate than fixed-length warmup for the same warmup budget.

1 Introduction Architectural simulations are extensively used by computer architects and designers for evaluating various design tradeoffs. Unfortunately, architectural simulation is very time-consuming. Even on today’s fastest machines and simulators, simulating an industry-standard benchmark easily takes several weeks to run to completion. As such, simulating entire benchmark executions is infeasible for exploring huge microarchitecture design spaces. Therefore, researchers have proposed sampled simulation [1,2,3,4]. Sampled simulation takes a number of so called sampling units that are simulated in detail. Statistics or appropriate weighting is then applied to the simulation results of the various sampling units for predicting the performance of the overall benchmark execution. An important issue with sampled simulation is the microarchitecture state at the beginning of each sampling unit, i.e., the microarchitecture state at the beginning of a P. Stenstr¨om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 45–64, 2009. c Springer-Verlag Berlin Heidelberg 2009 

46

S. Kluyskens and L. Eeckhout

sampling unit is unknown during sampled simulation. This is well known in the literature as the cold-start problem. A solution to the cold-start problem is to warmup various microarchitecture structures prior to each sampling unit. A large amount of work has been done on cache structure warmup. However, the amount of work done on branch predictor warmup is very limited. This paper proposes Branch History Matching (BHM) as a novel branch predictor warmup method. The basic idea is to inspect the pre-sampling unit, i.e., the instructions in the dynamic instruction stream prior to the sampling unit, for branch instances of the same static branch with similar global and local histories as the branch instances in the sampling unit. A BHM distribution is then built for all sampling units that quantifies the locality in the branch execution stream taking into account both the global and local histories of the branches. As a final step, the appropriate warmup length is then determined for each sampling unit taking into account the BHM distributions as well as the total warmup budget. In other words, the total warmup budget is distributed across the various sampling units according to the BHM distribution. Sampling units that show good locality are given a small warmup length; sampling units that show poor locality are given a larger warmup length. BHM is microarchitecture-independent, i.e., the warmup lengths are computed once and are then reused across branch predictors during design space exploration. An appealing way of using BHM in practice for sampled processor simulation is to use (i) checkpointed sampling [5,6] maintaining reduced checkpoints of architecture state (registers and memory) along with (ii) checkpointed cache warmup [5,7,8,9] and (iii) compressed branch traces [10] that are reduced through BHM. In other words, instead of having branch traces of full benchmark executions as proposed in [10], BHM limits the length of the compressed branch traces. This would result in a reduction in required disk space as well as a reduction in overall simulation time while pertaining the advantage of compressed branch traces of being branch predictor independent. In addition, this would allow for the parallel simulation of the various sampling units. This paper makes the following contributions: – First, we show that branch predictor warmup is an issue when it comes to guaranteeing an accurate hardware state at the beginning of a sampling unit. We show that for small sampling unit sizes, branch predictor warmup is required in order to achieve an accurate estimate of the hardware state at the beginning of the sampling unit. We provide results showing that even for (fairly large) 1M instruction sampling units branch predictor warmup is required. – Second, we propose Branch History Matching (BHM) as a novel branch predictor warmup approach. Using the SPEC CPU2000 integer benchmarks and 10K-instruction sampling units, we show that BHM is 39% more accurate than fixed-length warmup for the same warmup length. Or reverse, BHM achieves the same accuracy as fixed-length warmup with a 1.6X shorter warmup length. Compared to Memory Reference Reuse Latency (MRRL) [17,20] applied to branch predictor warmup, BHM is 87% more accurate. This paper is organized as follows. We first revisit sampled simulation and cover the main issues related to sampled simulation. We then present BHM as a branch predictor

Branch Predictor Warmup for Sampled Simulation

47

warmup method. We subsequently evaluate BHM and compare it against fixed-length warmup and MRRL. And finally, we conclude.

2 Sampled Simulation Background In sampled simulation, a number of sampling units are chosen from a complete benchmark execution. Those sampling units are then simulated in detail; the pre-sampling units, i.e., the instructions prior to a given sampling unit, are skipped. The performance of the complete benchmark is then estimated by simply aggregating or weighting the performance numbers from the various sampling units. There are basically three issues with sampled simulation. First, the sampling units need to be chosen in such a way that the sampling units are representative for the entire program execution. Various researchers have proposed approaches for achieving this, such as random sampling [1], periodic sampling as done in SMARTS [3] and targeted sampling based on program phase behavior as done in SimPoint [2]. The second issue is how to get to those sampling units. In other words, the architecture state (register and memory state) needs to be reconstructed so that all sampling units can be functionally simulated in a correct way. This is trivial in case of trace-driven simulation; the simulator just needs to jump to the right sampling unit. For executiondriven simulation on the other hand, guaranteeing a correct architecture state is slightly more difficult. The most straightforward way is to fast-forward until the beginning of each sampling unit, i.e., the benchmark is functionally simulated from the beginning of the benchmark execution until the beginning of the sampling unit. This is easy to implement in a simulator, however, it is very time-consuming in terms of simulation time, especially for sampling units deep down the dynamic instruction stream. Therefore, researchers have proposed efficient checkpointing techniques [5,9]. The idea is to store the touched memory regions for a sampling unit. Upon simulation of a given sampling unit, the checkpoint is loaded from disk, the touched memory regions are written into memory and the simulation of the sampling unit proceeds. Given that the checkpoints are very small in size, they load quickly and overall simulation time is reduced substantially compared to fast-forwarding. This is especially beneficial for the parallel simulation of sampling units [11,12]. The third issue with sampled simulation is to estimate the microarchitecture state at the beginning of each sampling units. The microarchitecture structures that suffer the most from the cold-start problem are cache structures and branch predictors. We will discuss warmup approaches tailored towards these types of hardware structures in the following two subsections. 2.1 Cache Warmup Given the fact that caches have the largest state in a microprocessor, they are likely to suffer the most from inaccurate microarchitecture warmup. In fact, most of the prior research on the cold-start problem has been done on cache warmup. Various approaches have been proposed such as no warmup, stale state (also called stitch) [11,13], fixed warmup [1], cache miss rate estimators [14], no-state-loss [12,15], minimal subset

48

S. Kluyskens and L. Eeckhout

evaluation (MSE) [16], memory reference reuse latency (MRRL) [17,20], boundary line reuse latency (BLRL) [8,18], self-monitored adaptive cache warmup (SMA) [19], memory hierarchy state (MHS) [5], memory timestamp record (MRT) [7], etc. 2.2 Branch Predictor Warmup Compared to the amount of work done on cache warmup, very little work has been done on branch predictor warmup. The first paper dealing with branch predictor warmup was by Conte et al. [1]. They proposed two approaches to branch predictor warmup, namely stale state and fixedlength warmup. Stale state (or stitch) means that the branch predictor state at the end of the previous sampling unit serves as an approximation for the branch predictor state at the beginning of the current sampling unit. An important disadvantage of stale state is that it serializes the simulation of the various sampling units, i.e., it is impossible to simulate the current sampling unit without having finalized the simulation of the previous sampling unit. Fixed-length warmup is a simple-to-implement method that achieves good accuracy if sufficiently long warmup lengths are chosen. The second paper mentioning branch predictor warmup is by Haskins and Conte [17,20] in which they propose memory reference reuse latency (MRRL). The idea of MRRL is to look in the pre-sampling unit how far one needs to go in order to encounter the same static branch as the one in the sampling unit. MRRL computes the reuse latency, i.e., the number of instructions between the branch instance in the pre-sampling unit and the one in the sampling unit, for all branch instances in the presampling unit and sampling unit. For a given target cumulative probability, for example 99.5%, it is then determined where warmup should start in the pre-sampling unit. During this warmup period, the branch predictor is warmed up but no misprediction rates are computed. A number of papers have proposed checkpointed sampling techniques [5,7,9] in which the architecture state is stored on disk, as mentioned above. These techniques typically use checkpointed microarchitecture warming for warming cache state, such as memory timestamp record [7], live-points [9] and memory hierarchy state (MHS) [5]. They suggest to store the branch predictor state as part of the microarchitecture state for the various branch predictors one may be interested in during design space exploration. This can be space-inefficient in case multiple branch predictors need to be stored, and in addition, it prevents from simulating a branch predictor that is not contained in the microarchitecture warmup. For addressing this problem, Barr and Asanovic [10] propose branch trace compression. They store a compressed branch trace on disk and upon branch predictor warming they simply decompress the compressed branch trace and use the decompressed trace for branch predictor warming. This approach is branch predictor independent and can be used to warm any branch predictor during sampled simulation. The branch trace compression scheme by Barr and Asanovic [10] however does not address the issue of how far one needs to go back in the pre-sampling unit. They assume that the entire branch trace from the beginning of the benchmark execution up to the current sampling unit needs to be compressed and decompressed. This can be time-consuming in practice, especially for sampling units deep down the benchmark execution. BHM

Branch Predictor Warmup for Sampled Simulation

49

as proposed in this paper can be used to cut down the branch traces that need to be compressed. This saves both disk space and simulation time, while keeping the benefit of the warmup approach to be branch predictor independent.

3 The Need for Branch Predictor Warmup Branch predictors need to be warmed up during sampled simulation. This is illustrated in Figures 1 and 2 where the number of branch mispredictions per thousand instructions (MPKI) is shown for gcc and mcf, respectively, for four sampling unit sizes: 10K, 100K, 1M and 10M instruction sampling unit sizes. Note this is in the range of sampling units used in contemporary sampled simulation environments such as SMARTS [3,9] (sampling unit size of 10K instructions) and SimPoint [2,5,21] (sampling unit sizes from 1M to 100M instructions). Each graph shows the MPKI for four (fairly aggressive) branch predictors: a 128Kbit gshare predictor, a 256Kbit local predictor, a 128Kbit bimodal predictor and a 192Kbit hybrid predictor — more details about the experimental setup and the branch predictors are given in section 5. The various bars correspond to various branch predictor warmup strategies: no warmup, stale state and perfect warmup. The no warmup approach assumes an initialized branch predictor at the beginning of a sampling unit, i.e., the branch predictor content is flushed at the beginning of the sampling unit — two-bit saturating counters in adjacent entries are initialized in alternate ‘01’ and ‘10’ states. The stale state approach assumes that the branch predictor at the beginning of the sampling unit equals the branch predictor state at the end of the previous sampling unit. Note that the stale state approach assumes that sampling units are simulated sequentially — this excludes parallel sampled simulation. The perfect warmup approach is an idealized warmup scenario where the branch predictor is perfectly warmed up, i.e., the branch predictor state at the beginning of the sampling unit is the state as if all instructions prior to the sampling unit were simulated. Figures 1 and 2 clearly show that the no warmup and stale state approaches fail in being accurate, especially for small sampling unit sizes. For example for 10K instruction sampling units, the ΔM P KI can be very high for both no warmup and stale state. Even for 1M instruction sampling units, the error can be significant, more than 1.5 ∆MPKI for gcc and more than 3 ∆MPKI for mcf. Note that the error varies across branch predictors. The error is typically higher for the gshare and local predictors than for the bimodal predictor, which is to be understood intuitively, the reason being the fact that the hashing in the gshare and local predictor tables typically results in more entries being accessed in the branch predictor table than the bimodal predictor does. As a result of the non-uniform warmup error across branch predictors, incorrect design decisions may be taken. For example, for gcc, using the no warmup approach, a computer architect would conclude that the local predictor achieves a better accuracy (a lower MPKI) than the gshare predictor. This is the case for 10K, 100K and even 1M instruction sampling units. For mcf, using the no warmup approach, the conclusion would be that the hybrid branch predictor outperforms all other branch predictors; this is the case for all sampling unit sizes considered here (including the 10M sampling unit size). However, these incorrect conclusions are just an artifact of the inadequate warmup approach. For gcc, perfect warmup shows that the gshare predictor outperforms the

50

S. Kluyskens and L. Eeckhout 100K insn sampling unit

10K insn sampling unit 12

6

no warmup

5

stale state

no warmup 10

stale state

4

MPKI

8

MPKI

perfect warmup

perfect warmup

6

3

4

2

2

1

0

0

gshare

local

bimodal

gshare

hybrid

1M insn sampling unit 5 4

4 3.5

3.5

hybrid

no warmup stale state perfect warmup

3

3

MPKI

MPKI

bimodal

10M insn sampling unit 4.5

no warmup stale state perfect warmup

4.5

local

2.5 2

2.5 2 1.5

1.5 1

1

0.5

0.5

0

0 gshare

local

bimodal

gshare

hybrid

local

bimodal

hybrid

Fig. 1. No warmup, stale state and perfect warmup MPKI results for gcc and 4 branch predictors and 4 sampling unit sizes 10K insn sampling unit 24 22

stale state

20

20

no warmup

18

stale state perfect warmup

perfect warmup 16

18

MPKI

MPKI

100K insn sampling unit

no warmup

16 14

14 12

12 10

10 8

8 gshare

local

bimodal

hybrid

gshare

1M insn sampling unit 18 17

bimodal

hybrid

10M insn sampling unit 18

no warmup stale state perfect warmup

15

MPKI

MPKI

16

local

14

17

stale state

16

perfect warmup

15 14

13

13

12

12

11

11

10

no warmup

10 gshare

local

bimodal

hybrid

gshare

local

bimodal

hybrid

Fig. 2. No warmup, stale state and perfect warmup MPKI results for mcf and 4 branch predictors and 4 sampling unit sizes

Branch Predictor Warmup for Sampled Simulation

51

d2 d1 pre-sampling unit

7

100

111

100

011

100

010

101

010

101

010

BHMS = 5

6

8

9

d=0

BHMS = 0

BHMS = 4

5

BHMS = 6

001

BHMS = 4

111

100

sampling unit

4

BHMS = 0 BHMS = 0

100

BHMS = 0

011 110

BHMS = 3

global history 001 local history 011

BHMS = 2

3

BHMS = 0

2

BHMS = 2

1

1

BHM cumulative distribution

2/3 1/3 d

d1

d2

0

Fig. 3. An example illustrating how the cumulative Branch History Matching distribution is computed

local predictor; for mcf, perfect warmup shows that the gshare predictor outperforms all other branch predictors. As a conclusion, no warmup may lead to incorrect design decisions for sampled branch predictor simulation. The stale state warmup approach only solves this problem for the 1M and 10M instruction sampling unit sizes, however, it does not solve the problem for smaller sampling unit sizes and it cannot be used for parallel sampled simulation. As such, there is a need for accurate warmup strategies for sampled branch predictor simulation.

4 Branch History Matching This paper proposes Branch History Matching (BHM) as a novel branch predictor warmup approach. Computing the branch predictor warmup length through Branch History Matching (BHM) is done in two steps. First, we compute the BHM distribution for all sampling units. In a second phase, we then determine the warmup length for each sampling unit for a given total warmup length budget using the BHM distributions for all sampling units. 4.1 Computing the BHM Distribution Computing the BHM distribution for a given sampling unit is illustrated in Figure 3. At the top of Figure 3, a sampling unit along with its pre-sampling unit is shown. The bullets represent a single static branch being executed multiple times in the pre-sampling unit as well as in the sampling unit. Instructions with labels ‘1’ thru ‘6’ are part of the

52

S. Kluyskens and L. Eeckhout

pre-sampling unit; instructions labeled ‘7’, ‘8’ and ‘9’ are part of the sampling unit. A white bullet represents a non-taken branch; a black bullet shows a taken branch. Figure 3 also shows the global and local history for each dynamic instance of the given static branch; the example assumes three global history bits and three local history bits. Note that the most recent branch outcome is shifted in on the right hand side of the history register; for example, a non-taken branch changes the local history from ‘011’ to ‘110’. In order to compute the BHM distribution, we first compute the BHM histogram. The BHM histogram is computed by scanning all the branch instances in the sampling unit and proceeds as follows. – Searching the sampling unit. We first determine whether there is a perfect match for the local and global history of the given branch instance in the sampling unit versus the local and global histories of all the preceding branch instances of the same static branch in the sampling unit. A perfect match means that both the local and global histories are identical for the two respective branch instances. For the example given in Figure 3, the local and global histories of branch instance ‘9’ in the sampling unit show a perfect match with the local and global history of branch instance ‘7’ in the sampling unit. This case increments the count for d = 0 in the BHM histogram. – Searching the pre-sampling unit. In case there is no perfect match with a preceding branch instance in the sampling unit, we search the pre-sampling unit for the most recent branch instance that shows the highest match with the local and global history for the given branch instance. This is done by computing the Branch History Matching Score (BHMS) between the given branch instance in the sampling unit with all the branch instances of the same static branch in the pre-sampling unit. The BHMS between two branch instances is computed as the number of bit positions that are identical between the local and global histories of the respective branch instances. When computing the number of identical bit positions we count from the most recent bit to the least recent bit and we stop counting as soon as there is disagreement for a given bit, i.e., we count the matching most recent history bits. This is done for both the global and local histories; the overall BHMS then is the sum of the global and local BHMSs. Computed BHMSs are shown in Figure 3 for the first and second branch instances of the sampling unit. For example, the BHMS for branch instance ‘8’ with relation to branch instance ‘4’ equals 4, i.e., 2 (compare global histories ‘011’ versus ‘111’) plus 2 (compare local histories ‘101’ versus ‘001’). The first branch instance (with label ‘7’) achieves a perfect match (BHMS equals 6) for the branch instance with label ‘5’. The idea is then to update the BHM histogram reflecting the fact that in order to have an accurate warmup for instruction ‘7’ we need to go back to instruction ‘5’ in the pre-sampling unit. For this purpose, the BHM histogram is incremented at distance d1 with ‘d1’ being the number of instructions between the branch instance with label ‘5’ and the beginning of the sampling unit — this is to say that branch predictor warmup should start at branch instruction ‘5’. For the second branch instance (with label ‘8’) in the sampling unit, the highest BHMS is obtained for the branch instance with label ‘6’; the number

Branch Predictor Warmup for Sampled Simulation

53

/* this function computes the current warmup length */ int current_warmup_length (int* d) { for (i = 0; i < n; i++) sum += d[i]; return sum; } /* main algorithm */ /* initialize warmup length for each sampling unit */ for (i = 0; i < n; i++) d[i] = 0;

/* iterate as long as the user defined total warmup length L_w is not reached */ while (current_warmup_length (d) < L_w) { /* find the sampling unit max_j that faces the maximum slope */ max_prob = 0.0; max_i = -1; for (i = 0; i < n; i++) { if ((P[i][d[i] + b] - P[i][d[i]])/b > max_prob) { max_prob = (P[i][d[i] + b] - P[i][d[i]])/b; max_i = i; } } /* update warmup length for sampling unit facing the maximum slope */ d[max_i] += d[max_i] + b; }

Fig. 4. The algorithm in pseudocode for determining the warmup length per sampling unit using BHM distributions

of instructions between that branch instance and the sampling unit starting point is denoted as d2 in Figure 3. We then increment the BHM histogram at distance d2. Dividing the BHM histogram with the number of branch instances in the sampling unit, we then obtain the BHM distribution. Figure 3 shows the cumulative BHM distribution for the given sampling unit: since there are three branch instances in our example sampling unit, the cumulative distribution starts at 1/3 for distance d = 0, reaches 2/3 at distance d = d2 and finally reaches 1 at distance d = d1. 4.2 Determining Warmup Length Once the BHM distribution is computed for each sampling unit we determine the warmup length per sampling unit for a given total warmup length budget. The goal is to partition a given warmup length budget over a number of sampling units so that accuracy is maximized. In other words, sampling units that do not require much warmup, are granted a small warmup length; sampling units that require much more warmup are given a much larger warmup length. The algorithm for determining the appropriate warmup length per sampling unit works as follows, see also Figure 4 for the pseudocode of the algorithm. We start from n BHM distributions, with n being the number of sampling units. In each iteration, we determine the sampling unit i out of the n sampling units that faces the maximum slope in the BHM distribution. This means that the sampling unit i (called max i in the

54

S. Kluyskens and L. Eeckhout Table 1. The branch predictors considered in this paper predictor configuration gshare 16-bit history gshare predictor, 128Kbit total state local 16-bit local predictor, 8K entries at first level, 64K entries at second level 256 Kbit total state bimodal 64K-entry bimodal predictor, 128Kbit total state hybrid hybrid predictor consisting of a 32K-entry bimodal predictor, a 15-bit history gshare predictor and a 32K-entry PC-indexed meta predictor; 192Kbit total state

Pi (di ) , with pseudocode in Figure 4) is determined that maximizes the slope Pi (di +b)− b Pi (d) being the probability for distance d in the cumulative BHM distribution for sampling unit i, and di being the warmup length granted to sampling unit i in the current state of the algorithm. For the sampling unit i that maximizes the slope, we increase the granted warmup length di to di + b. This algorithm is iterated until the total warmup length n over all sampling units equals a user-defined maximum warmup length Lw , i.e., i=1 di = Lw . By doing so, we effectively budget warmup to samples that benefit the most from the granted warmup. Note that this algorithm is only one possible design point in BHM warmup. More in particular, this algorithm heuristically determines to increase the warmup length for the sampling unit that faces the maximum slope in the BHM distribution. The algorithm does not take into account the distance over which this slope is observed; taking this distance into account for determining appropriate warmup lengths would be an interesting avenue for future work though.

5 Experimental Setup We use SPEC CPU2000 integer benchmarks with reference inputs in our experimental setup. We include all integer benchmarks except for perlbmk because its branch misprediction rate is very low; in fact, no warmup is very accurate for perlbmk. The binaries which were compiled and optimized for the Alpha 21264 processor, were taken from the SimpleScalar website. All measurements presented in this paper are obtained using the binary instrumentation tool ATOM [22]. The branch predictors considered in this paper are shown in Table 1. We consider four fairly aggressive branch predictors: a gshare predictor, a local predictor, a bimodal predictor and a hybrid predictor [23,24]. Our primary metric for quantifying the accuracy of the branch predictor warmup approaches proposed in this paper is ΔM P KI which is defined as the absolute difference between the number of misses per thousand instructions under perfect warmup (M P KIperfect ) versus the number of misses per thousand instructions under the given branch predictor warmup approach (M P KIwarmup ). In other words, ΔM P KI = M P KIwarmup − M P KIperfect  and thus the smaller ΔM P KI, the better. Our second metric, next to accuracy, is warmup length which is defined as the number of instructions required by the given warmup technique. This is a smaller-is-better metric: the smaller the warmup length, the smaller the total simulation time.

Branch Predictor Warmup for Sampled Simulation

55

6 Evaluation We now evaluate the accuracy and warmup length of BHM compared to fixed-length warmup; section 6.1 covers accuracy and section 6.2 covers warmup length. Throughout this evaluation we consider a sampling unit size of 10K instructions. The reason is that, as mentioned in section 3, small sampling unit sizes suffer most from the lack of warmup; small sampling unit sizes will stress our warmup approach the most. Note also that 10K instructions is the sampling unit size used in the SMARTS sampled simulation framework [3]. All the results presented in this paper are for 50 sampling units. Further, we assume that the number of global and local history bits equals 16 for the BHM approach in sections 6.1 and 6.2. Section 6.3 then studies the impact of the BHM history length on accuracy and warmup length. 6.1 Accuracy Comparison against fixed-length warmup. Figure 5 evaluates the accuracy of BHM compared to fixed-length warmup. Both warmup techniques are budgeted a 1M warmup length per sampling unit, i.e., both warmup techniques use the same warmup length. The four graphs in Figure 5 represent four different branch predictors, namely the gshare, local, bimodal and hybrid branch predictors. The ΔM P KIs are shown for both warmup techniques. We observe that BHM substantially outperforms fixed-length warmup. Over all four branch predictors, the average ΔM P KI decreases from 0.48 (under fixed-length warmup) to 0.29 (under BHM) which is 39% more accurate. Figure 6 gives a more detailed comparison between BHM and fixed-length warmup on a per sampling unit basis for mcf and the gshare branch predictor. The top graph shows the warmup length per sampling unit; the bottom graph shows the MPKI per sampling unit. The fixed-length warmup technique budgets 1M warmup instructions to all sampling units. The BHM approach on the other hand, budgets a smaller warmup length for some sampling units, see for example sampling units 1 thru 20. However, although the warmup length is smaller, BHM is as accurate as fixed-length warmup, see the bottom graph. In other words, BHM finds that for those sampling units, a long warmup length is not required for achieving a given level of accuracy. For other sampling units on the other hand, BHM budgets a longer warmup — this is the case for most of the sampling units with IDs 30 thru 50, see the top graph in Figure 6. By taking a longer warmup length for those sampling units, BHM achieves a better accuracy. In other words, BHM is capable of finding for which sampling units it needs to take a longer warmup. The end result is that BHM budgets warmup efforts in a better way than fixed-length warmup does. BHM budgets more warmup to sampling units that require more warmup; other sampling units that require less warmup are budgeted less warmup. Comparison against MRRL. Figure 7 compares BHM against MRRL [17,20]. As mentioned before, MRRL looks how far one needs to go back in the pre-sampling unit for encountering branch instances of the same static branch for all branch instances in the sampling unit. The results in Figure 7 show that BHM clearly outperforms MRRL. Over all four branch predictors, the average ΔM P KI decreases from

56

S. Kluyskens and L. Eeckhout gshare predictor

local predictor

1.6

2.5 fixed 1M

1.4

DMPKI

DMPKI

fixed 1M 2.0

BHM 1M

1.2 1.0 0.8

BHM 1M

1.5 1.0

0.6 0.4

0.5

vpr

avg

twolf

vortex

mcf

parser

gcc

fixed 1M

vpr

avg

twolf

vortex

mcf

parser

gzip

gcc

gap

eon

bzip2

BHM 1M

crafty

vpr

0.50 0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

avg

twolf

vortex

mcf

parser

gzip

gcc

gap

eon

DMPKI

BHM 100K

crafty

gzip

hybrid predictor

fixed 100K

bzip2

DMPKI

bimodal predictor 0.50 0.45 0.40 0.35 0.30 0.25 0.20 0.15 0.10 0.05 0.00

gap

eon

bzip2

crafty

0.0

avg

twolf

vortex

mcf

parser

gcc

gzip

gap

eon

bzip2

crafty

0.0

vpr

0.2

Fig. 5. ΔM P KI results for fixed 1M warmup and BHM for the gshare, local, bimodal and hybrid branch predictors warmup length warmup length (millions)

8 7

fixed 1M

6

BHM 1M

5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

0 sampling unit ID

MPKI

accuracy 40

fixed 1M

35

BHM 1M

30

perfect warmup

25 20 15 10 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

0 sampling unit ID

Fig. 6. Warmup length (top graph) and accuracy (bottom graph) on a per sampling unit basis for mcf and the gshare predictor

Branch Predictor Warmup for Sampled Simulation

local predictor 6 MRRL 100%

MRRL 100%

5 DMPKI

BHM 1M

BHM 1M

4 3 2

vpr

avg

twolf

vortex

mcf

parser

5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0

MRRL 100%

vpr

avg

twolf

vortex

mcf

parser

gzip

gcc

gap

eon

crafty

BHM 1M

bzip2

vpr

avg

twolf

vortex

mcf

parser

gcc

gzip

gap

eon

DMPKI

BHM 100K

bzip2

gcc

hybrid predictor MRRL 100%

crafty

DMPKI

bimodal predictor 5.0 4.5 4.0 3.5 3.0 2.5 2.0 1.5 1.0 0.5 0.0

gzip

gap

eon

bzip2

crafty

vpr

0

avg

twolf

vortex

mcf

parser

gcc

gzip

gap

eon

bzip2

1 crafty

DMPKI

gshare predictor 10 9 8 7 6 5 4 3 2 1 0

57

Fig. 7. ΔM P KI results for MRRL and BHM for the gshare, local, bimodal and hybrid branch predictors. For MRRL, we consider all branch instances in the sampling unit, hence the ‘MRRL 100%’ labels.)

2.13 (under MRRL) to 0.29 (under BHM) which is 87% more accurate. The important difference between MRRL and BHM is that BHM, in contrast to MRRL, takes into account branch histories; this results in significantly more accurate branch predictor state warmup for BHM compared to MRRL. Note that MRRL also performs worse than fixed 1M warmup, compare Figure 5 against Figure 7. The reason is that, because of the fact that MRRL does not take into account branch history, MRRL is unable to come up with long enough warmup lengths for accurately warming up the branch predictors. We expect other warmup techniques, such as Boundary Line Reuse Latency (BLRL) [8,18], to be equally inaccurate for this reason. The average warmup length through MRRL is only 200K instructions per sampling unit; according to our results, much larger warmup lengths are required to accurately warmup branch predictors. 6.2 Warmup Length In order to quantify the reduction in warmup length through BHM compared to fixedlength warmup, we have measured the average ΔM P KI across all benchmarks and all branch predictors as a function of the warmup length under the fixed-length warmup strategy, see Figure 8; Figure 9 shows the same info for the individual gshare, local and hybrid branch predictors. The average ΔM P KI is shown for fixed-length warmup with the warmup budget varying between 1M and 2M instructions per sampling unit. The ΔM P KI for BHM with a 1M warmup length budget per sampling unit is shown on the right. We observe that fixed-length warmup achieves about the same average

58

S. Kluyskens and L. Eeckhout

0.45

DMPKI

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05

BHM 1M

fixed 2.0M

fixed 1.9M

fixed 1.8M

fixed 1.7M

fixed 1.6M

fixed 1.5M

fixed 1.4M

fixed 1.3M

fixed 1.2M

fixed 1.1M

fixed 1M

0

Fig. 8. Average ΔM P KI across all branch predictors as a function of warmup length for the fixed-length warmup approach compared to BHM 1M

accuracy as BHM for a warmup length of 1.6M instructions per sampling unit. In other words, BHM with a 1M warmup budget per sampling unit results in a 1.6X reduction in warmup length on average compared to fixed-length warmup while achieving the same accuracy. Comparing the graphs for the individual branch predictors in Figure 9, it is interesting to observe that the benefit of BHM is larger for the gshare branch predictor than for the local branch predictor; BHM achieves the same accuracy as fixed-length warmup with a 2M warmup budget for the gshare predictor compared to a 1.4M warmup budget for the local predictor. The reason is that the gshare predictor is more susceptible to aliasing in the pattern history table. Figure 10 shows MPKI versus warmup length for the gcc benchmark and the four branch predictors. Note that the horizontal axes are shown on a log scale. The two curves in each graph represent fixed-length warmup and BHM warmup, respectively; and the various points in these curves represent different warmup budgets. This graph clearly shows that BHM achieves the same accuracy with substantially shorter warmup lengths, or reverse, BHM achieves better accuracy for the same warmup length. 6.3 Impact of BHM History Length Note that the amount of branch history used by three of the four branch predictors, namely the gshare, local and hybrid predictors, equals 16 bits. The number of BHM history bits used for computing the warmup length also equals 16 bits. The question however is how sensitive BHM’s accuracy is to the BHM history length. Figure 11 explores the impact of the BHM history length. The average ΔM P KI over all benchmarks is shown on the vertical axis versus the warmup length on the horizontal axis. The five curves represent the five branch predictors. The different points on each curve represent different BHM history lengths. We varied the BHM history length from 0, 2, 4, 8 to 16; when varying the history length we simultaneously vary the global and local BHM history lengths. A zero BHM history length means that no global and local history is taken into account for building the BHM distribution. In other words, the BHM warmup method then simply looks for the last occurrence of the

59

fixed 1.6M

fixed 1.7M

fixed 1.8M

fixed 1.9M

fixed 2.0M

BHM 1M

fixed 1.6M

fixed 1.7M

fixed 1.8M

fixed 1.9M

fixed 2.0M

BHM 1M

fixed 1.7M

fixed 1.8M

fixed 1.9M

fixed 2.0M

BHM 1M

fixed 1.4M

fixed 1.3M

fixed 1.2M

fixed 1.1M

fixed 1.5M

gshare

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 fixed 1M

DMPKI

Branch Predictor Warmup for Sampled Simulation

local

0.8 0.7

DMPKI

0.6 0.5 0.4 0.3 0.2 0.1 fixed 1.5M

fixed 1.4M

fixed 1.3M

fixed 1.2M

fixed 1.1M

fixed 1M

0

hybrid

0.25

DMPKI

0.2 0.15 0.1 0.05

fixed 1.6M

fixed 1.5M

fixed 1.4M

fixed 1.3M

fixed 1.2M

fixed 1.1M

fixed 1M

0

Fig. 9. ΔM P KI for the gshare, local and hybrid predictors as a function of warmup length for the fixed-length warmup approach compared to BHM 1M

same static branch for updating the BHM distribution. In all of these experiments, we budgeted a warmup length to 1M instructions per sampling unit. There are two interesting observations to be made from this graph. First, accuracy improves or ΔM P KI decreases with increasing BHM history lengths. This is to be expected because the more history is taken into account, the better BHM will be able to determine how far it needs to go back in the pre-sampling unit for appropriate warmup.

60

S. Kluyskens and L. Eeckhout local predictor

gshare predictor 3.5

4.0 fixed warmup

fixed warmup

2.5

3.0

2.0

2.5

1.5

2.0

1.0 1K 1000

10K 10000

100K 100000

1M 1000000

BHM

3.5

BHM

MPKI

MPKI

3.0

1.5 1K 1000

10M 10000000

10K 10000

100K 100000

warmup length

bimodal predictor

10M 10000000

hybrid predictor 2.4

4.2

fixed warmup

fixed warmup

4.1

2.2

BHM

BHM

2.0

MPKI

4.0

MPKI

1M 1000000

warmup length

3.9

1.8

3.8

1.6

3.7

1.4

3.6

1.2

3.5 1K 1000

10K 10000

100K 100000

1M 1000000

1.0 1K 1000

10M 10000000

10K 10000

100K 100000

warmup length

1M 1000000

10M 10000000

warmup length

Fig. 10. Comparing BHM versus fixed warmup in terms of MPKI versus warmup length for gcc

4.0

gshare

history length = 0

local

3.5

history length = 2

3.0

bimodal

DMPKI

hybrid 2.5

history length = 4 2.0

history length = 8

1.5

history length = 16

1.0 0.5 0.0 0.2

0.4

0.6 0.8 warmup length (millions)

1

Fig. 11. Evaluating the impact of the BHM history length on accuracy and warmup length

Second, small BHM histories are unable to budget the warmup lengths so that the average warmup length per sampling unit effectively equals the 1M instruction warmup budget. For example, a zero BHM history only yields slightly more than 200K instructions of warmup per sampling unit. In other words, it is impossible for BHM with limited history to fully exploit the available warmup budget. By increasing the BHM history length, BHM is better able to approach the target 1M warmup length per sampling unit. (Note that the MRRL approach [17,20] corresponds to a zero BHM history

Branch Predictor Warmup for Sampled Simulation perceptron predictor

neural path-based predictor

2.5

4.5 fixed warmup

2.3

BHM

3.5

1.9 1.7

MPKI

MPKI

fixed warmup

4

BHM

2.1

1.5 1.3

3 2.5 2

1.1

1.5

0.9

1

0.7 0.5 1000

61

10000

100000

1000000 10000000

warmup length

1E+08

0.5 1000

10000

100000

1000000 10000000

1E+08

warmup length

Fig. 12. Warming up a 368KB perceptron branch predictor (on the left) and neural path-based branch predictor (on the right): M P KI versus warmup length for BHM versus fixed-length warmup for gcc

length.) We further observe that an 8 bit and a 16 bit BHM history length yields approximately the same accuracy. From this experiment, we thus conclude that in order to achieve accurate warmup for branch predictors, the BHM history length needs to be set to an appropriate value, for example, to the maximum history length one would look at during the branch predictor design space exploration. 6.4 Feasibility of BHM for More Advanced Branch Predictors Most branch predictors in today’s commercial processors use a global and/or local branch history and are variations or combinations of the basic branch predictors considered in this paper. Because BHM is also based on global and local branch history matching, it is to be expected that BHM will be an appropriate warmup technique for most commercial branch predictors considered today. However, branch predictors proposed in the more recent literature are more advanced. Examples are the perceptron predictor [25], the O-GEHL predictor [26], the piecewise linear branch predictor [27], etc. Some of these branch predictors use extremely long branch histories, much longer than the 16 bit branch history lengths considered in this paper. Moreover, some of these branch predictors use information that is different from the global and local histories considered in the current implementation of BHM. For example, some branch predictors are path-based and use a sequence of recent branch addresses as the branch history. As such, it is an open question whether the version of BHM presented in this paper that uses limited global/local history is adequate for warming up more advanced branch predictors. In order to do a preliminary evaluation on the applicability of BHM for the sampled simulation of more advanced branch predictors, we consider our running example gcc and two aggressive advanced branch predictors. The first branch predictor that we consider is a perceptron branch predictor that features a 66-bit global history and a 24-bit local history and 3060 perceptrons following [25] — a total of 368KB hardware state. This perceptron predictor uses a much larger global and local branch history than the 16 bit BHM history that was used for computing the warmup lengths. The second branch predictor is a neural path-based branch predictor [28] that uses a 32 entry global path history. Figure 12 show MPKI versus warmup length for gcc for the perceptron and

62

S. Kluyskens and L. Eeckhout

neural path-based branch predictors. Both graphs show that BHM substantially outperforms fixed-length warmup. BHM achieves the same accuracy as fixed-length warmup with a warmup budget that is roughly an order of magnitude smaller, similar to what we observed for the other branch predictors considered in this paper, see Figure 10.

7 Conclusion Sampled simulation is a well known approach to speed up architectural simulations that are heavily used by computer architects and designers. An important issue with sampled simulation however is the cold-start problem, i.e., the microarchitecture state is unknown at the beginning of each sampling unit. Although a great deal of work has been done on cache structure warmup, very little research has been done on branch predictor warmup. This paper proposed Branch History Matching (BHM) as a novel branch predictor warmup method. The idea is to analyze the sampling unit as well as the pre-sampling unit for recurring branch instances of the same static branch with similar global and local branch histories. By doing so, BHM builds a distribution for each sampling unit that characterizes the branch locality behavior. BHM then budgets its total warmup budget to the various sampling units. Sampling units that are warmup-sensitive are budgeted more warmup; sampling units that are warmup-insensitive are budgeted less warmup. Compared to fixed-length warmup, BHM achieves better accuracy for the same total warmup budget, or reverse, BHM achieves the same accuracy with a shorter total warmup budget. This paper is only a first step towards accurate and efficient branch predictor warmup for sampled simulation. We believe more research is needed in this area in at least two directions. First, BHM needs to be evaluated for a much broader range of branch predictors than the ones considered in this paper. Second, BHM may need to be extended for accomodating different types of branch predictors. For example, using path-based history information instead of global/local history information may improve the efficacy of BHM warmup for path-based branch predictors.

Acknowledgements The authors would like to thank the anonymous reviewers for their valuable feedback. Lieven Eeckhout is a Postdoctoral Fellow with the Fund for Scientific Research— Flanders (Belgium) (FWO—Vlaanderen). This research is also supported by Ghent University, IWT, HiPEAC and the European SARC project No. 27648.

References 1. Conte, T.M., Hirsch, M.A., Menezes, K.N.: Reducing state loss for effective trace sampling of superscalar processors. In: Proceedings of the 1996 International Conference on Computer Design (ICCD 1996), pp. 468–477 (1996) 2. Sherwood, T., Perelman, E., Hamerly, G., Calder, B.: Automatically characterizing large scale program behavior. In: Proceedings of the Tenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-X), pp. 45–57 (2002)

Branch Predictor Warmup for Sampled Simulation

63

3. Wunderlich, R.E., Wenisch, T.F., Falsafi, B., Hoe, J.C.: SMARTS: Accelerating microarchitecture simulation via rigorous statistical sampling. In: Proceedings of the 30th Annual International Symposium on Computer Architecture (ISCA-30), pp. 84–95 (2003) 4. Yi, J.J., Kodakara, S.V., Sendag, R., Lilja, D.J., Hawkins, D.M.: Characterizing and comparing prevailing simulation techniques. In: Proceedings of the 11th International Symposium on High-Performance Computer Architecture (HPCA-11), pp. 266–277 (2005) 5. Van Biesbrouck, M., Eeckhout, L., Calder, B.: Efficient sampling startup for sampled processor simulation. In: Conte, T., Navarro, N., Hwu, W.-m.W., Valero, M., Ungerer, T. (eds.) HiPEAC 2005. LNCS, vol. 3793, pp. 47–67. Springer, Heidelberg (2005) 6. Wenish, T., Wunderlich, R., Falsafi, B., Hoe, J.: TurboSMARTS: Accurate microarchitecture simulation in minutes. In: Proceedings of the 2005 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pp. 408–409 (2005) 7. Barr, K.C., Pan, H., Zhang, M., Asanovic, K.: Accelerating multiprocessor simulation with a memory timestamp record. In: Proceedings of the 2005 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 66–77 (2005) 8. Van Ertvelde, L., Hellebaut, F., Eeckhout, L., De Bosschere, K.: NSL-BLRL: Efficient cache warmup for sampled processor simulation. In: Proceedings of the 29th Annual International Simulation Symposium (ANSS), pp. 168–175 (2006) 9. Wenisch, T.F., Wunderlich, R.E., Falsafi, B., Hoe, J.C.: Simulation sampling with live-points. In: Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 2–12 (2006) 10. Barr, K.C., Asanovic, K.: Branch trace compression for snapshot-based simulation. In: Proceedings of the International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 25–36 (2006) 11. Girbal, S., Mouchard, G., Cohen, A., Temam, O.: DiST: A simple, reliable and scalable method to significantly reduce processor architecture simulation time. In: Proceedings of the 2003 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, pp. 1–12 (2003) 12. Lauterbach, G.: Accelerating architectural simulation by parallel execution of trace samples. Technical Report SMLI TR-93-22, Sun Microsystems Laboratories Inc. (1993) 13. Kessler, R.E., Hill, M.D., Wood, D.A.: A comparison of trace-sampling techniques for multimegabyte caches. IEEE Transactions on Computers 43, 664–675 (1994) 14. Wood, D.A., Hill, M.D., Kessler, R.E.: A model for estimating trace-sample miss ratios. In: Proceedings of the 1991 SIGMETRICS Conference on Measurement and Modeling of Computer Systems, pp. 79–89 (1991) 15. Conte, T.M., Hirsch, M.A., Hwu, W.W.: Combining trace sampling with single pass methods for efficient cache simulation. IEEE Transactions on Computers 47, 714–720 (1998) 16. Haskins Jr., J.W., Skadron, K.: Minimal subset evaluation: Rapid warm-up for simulated hardware state. In: Proceedings of the 2001 International Conference on Computer Design (ICCD 2001), pp. 32–39 (2001) 17. Haskins Jr., J.W., Skadron, K.: Memory Reference Reuse Latency: Accelerated warmup for sampled microarchitecture simulation. In: Proceedings of the 2003 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2003), pp. 195–203 (2003) 18. Eeckhout, L., Luo, Y., De Bosschere, K., John, L.K.: BLRL: Accurate and efficient warmup for sampled processor simulation. The Computer Journal 48, 451–459 (2005) 19. Luo, Y., John, L.K., Eeckhout, L.: Self-monitored adaptive cache warm-up for microprocessor simulation. In: Proceedings of the 16th Symposium on Computer Architecture and High Performance Computing (SBAC-PAD 2004), pp. 10–17 (2004) 20. Haskins, J.W., Skadron, K.: Accelerated warmup for sampled microarchitecture simulation. ACM Transactions on Architecture and Code Optimization (TACO) 2, 78–108 (2005)

64

S. Kluyskens and L. Eeckhout

21. Perelman, E., Hamerly, G., Calder, B.: Picking statistically valid and early simulation points. In: Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques (PACT 2003), pp. 244–256 (2003) 22. Srivastava, A., Eustace, A.: ATOM: A system for building customized program analysis tools. Technical Report 94/2, Western Research Lab, Compaq (1994) 23. McFarling, S.: Combining branch predictors. Technical Report WRL TN-36, Digital Western Research Laboratory (1993) 24. Yeh, T.Y., Patt, Y.N.: Alternative implementations of two-level adaptive branch prediction. In: Proceedings of the 19th Annual International Symposium on Computer Architecture (ISCA-19), pp. 124–134 (1992) 25. Jimenez, D., Lin, C.: Neural methods for dynamic branch prediction. ACM Transactions on Computer Systems (TOCS) 20, 369–397 (2002) 26. Seznec, A.: Analysis of the O-GEometric history length branch predictor. In: Proceedings of the 32nd Annual International Symposium on Computer Architecture (ISCA), pp. 394–405 (2005) 27. Jimenez, D.A.: Piecewise linear branch prediction. In: Proceedings of the 32nd Annual International Symposium on Computer Architecture (ISCA), pp. 382–393 (2005) 28. Jimenez, D.A.: Fast path-based neural branch prediction. In: Proceedings of the 36th Annual International Symposium on Microarchitecture (MICRO), pp. 243–252 (2003)

Data Cache Techniques to Save Power and Deliver High Performance in Embedded Systems Major Bhadauria1, Sally A. McKee1 , Karan Singh1 , and Gary S. Tyson2 1

Computer Systems Lab School of Electrical and Computer Engineering Cornell University {major,sam,karan}@csl.cornell.edu 2 Department of Computer Science Florida State University [email protected]

Abstract. Minimizing power consumption continues to grow as a critical design issue for many platforms, from embedded systems to CMPs to ultrascale parallel systems. As growing cache sizes consume larger portions of the die, reducing their power consumption becomes increasingly important. Voltage scaling reduces leakage power for cache lines unlikely to be referenced soon. Partitioning reduces dynamic power via smaller, specialized structures. We introduce a reuse distance (RD) drowsy caching mechanism that exploits temporal locality, delivers equivalent or better energy savings than the best policies from the literature, suffers little performance overhead, is simple to implement, and scales with cache size and hierarchy depth 1 .

1

Introduction

Minimizing power consumption continues to grow as a critical design issue for many platforms, from embedded systems to CMPs to ultrascale parallel systems. Cache sizes have grown steadily in an attempt to mask the widening gap between main memory latency and core clock frequency and to avoid the large energy costs of off-chip memory accesses [11]. Caches thus consume increasing portions of die area (approximately half for current general-purpose chips) and account for larger percentages of total system power. Most of the increase stems from exponential growth in leakage due to shrinking feature sizes and their decreasing transistor voltage thresholds. The bottom line is large caches cause significant current leakage. We investigate power issues for embedded and higher performance systems and present solutions tailored for each domain. Previous research targets both static (leakage) power and dynamic switching power. Leakage power can be reduced by making caches smaller or shutting off portions. Another option is to reduce the operating voltage for those portions, which reduces leakage current but increases access latency. Such drowsy 1

This paper extends an article from the HiPEAC 2007 conference.

P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 65–84, 2009. c Springer-Verlag Berlin Heidelberg 2009 

66

M. Bhadauria et al. Virtual Address

region select

Stack RD Buffer

Addr DeMux TLB Xlation

Stack Cache

Global Cache

Global RD Buffer

Heap Cache (DL1)

Heap RD Buffer

L2 Cache or Memory

Fig. 1. Organization of Region-Based L1 Caches

caches [5] usually reduce voltage on all cache lines periodically, “waking up” (bringing back to normal voltage) lines as they are accessed. Since most lines remain idle most of the time, drowsy caching techniques can help reduce leakage power. Some mechanism must implement a policy to control when to turn lines off and on. Two main policies have been studied previously: the simple policy indiscriminately puts all lines to sleep after a specified number of clock cycles (and is thus clock-frequency dependent with respect to some performance aspects), and the noaccess policy only turns off lines that have not been accessed within a predefined number of cycles (also clock-frequency dependent). Simple performs almost identically to the more sophisticated noaccess, and thus is most often used in drowsy cache organizations [5]. Petit et. al [19] propose a Reuse Most Recently used On (RMRO) drowsy cache that behaves much like a noaccess policy adapted for associativity, using update intervals to calculate how often a set is accessed. The MRU way remains awake, but other ways of infrequently used sets are made drowsy. RMRO requires more hardware per set and uses dynamic power every cycle. These policies all count elapsed clock cycles to determine when cache lines should be put to sleep, making them inherently architecturespecific, and preventing prediction of leakage savings before individual workload evaluation, which we find to be a drawback. Switching power can also be minimized by reducing the size of the cache. Unlike leakage power, which is determined by the total area of all powered portions of the cache, switching power is only consumed by the portion of the cache being accessed. Partitioning can thus reduce switching power. For instance, a single cache can be partitioned into multiple sub-cache structures that are accessed independently. When a reference is made, only one of the cache partitions is accessed; the selection of which cache to access depends on the partitioning strategy. Region caches [18,16,17,6] partition the data cache into separate stack, global and heap caches that service accesses to the corresponding memory regions. The heap cache serves as a general L1 Dcache for data not assigned to other caches. A simple address decoder (using only a few address bits) identifies the region and routes the reference to the appropriate cache. Our research builds on both drowsy caching and region caching. We develop a new region cache model that maintains original hit rates while reducing

Data Cache Techniques to Save Power and Deliver High Performance

67

leakage via smaller multi-access column associative (CA) [1] and MRU [10] caches. These structures exhibit excellent implementation cost, performance, and energy tradeoffs for the target applications. We employ a drowsy policy that is simple to implement, scales better than previous policies, and allows finer control over cache power management. This Reuse Distance (RD) policy tracks lines accessed, keeping a limited (potentially configurable) number of lines awake. Fig. 1 illustrates our organization (RD buffers are much smaller than depicted). The RD buffer records IDs of awake cache lines. When the buffer is full and a new cache line is accessed, the LRU line is made drowsy and the new line’s ID is entered. A buffer of N entries needs N counters of log2 N bits, since they’re incremented every memory access (not per cycle, as in most previous drowsy policies). Power consumption is a first order design criteria for most embedded systems. Therefore, we adapt techniques originally proposed to improve cache performance to develop robust cache organizations to conserve energy in high performance embedded systems. On the MiBench suite, our multiple-access associative caches with RD drowsiness deliver IPCs of more complicated drowsy designs while using 16% less power. Compared to non-drowsy designs, we reduce power by 65% on average, while remaining within about 1% of the original mean IPC. We then examine using RD for higher performance systems with more traditional cache organizations. For these systems, we solely examine drowsy behavior (previous studies have examined region caches in similar settings). We compare RD and simple drowsy policies for 25 SPEC CPU 2000 benchmarks and a range of hierarchies. For modest clock rates of 0.5-1.5 GHz we observe IPC decreases of 2.4% on average over non-drowsy caches, in exchange for power savings of 93% for 32KB L1 caches and 95% for 512KB L2 caches, respectively. The latter represents a percentage of a much larger number: a significant energy savings. As L2 grows relative to L1, we maintain both performance and power savings by keeping fewer L2 lines awake. Finally, we enable larger overall cache sizes and/or higher clock rates, delivering better performance while adhering to a strict power budget. Ultimately, we propose data cache organizations that better match reference behavior and provide finer control for dynamic power management.

2

Related Work

Inoue et al. [11] survey techniques for reducing memory access energy while maintaining high performance. We discuss those techniques most related to our investigations: partitioned, drowsy, and multi-access cache organizations. Breaking monolithic memories into separate components enables optimizing those components to achieve better performance or conserve energy. Early “horizontal” partitioning broke L1 caches into separate instruction (Icache) and data (Dcache) designs, and “vertical” partitioning added multiple levels to the hierarchy. Modern vertical partitionings may create an L0 memory level between the processor and L1, as in line buffers [12,8] and filter caches [15]. These small structures reduce average access energy for hits at the expense of increased access

68

M. Bhadauria et al.

latency for misses (which increases average L1 latency). Horizontal partitioning reduces dynamic power by directing accesses at the same hierarchy level to different structures or substructures, as in cache subbanking [8], allowing smaller structures. Specialized loop caches and scratchpad memories can improve performance for scientific or embedded applications, for instance. Since different data references exhibit different locality characteristics, breaking the L1 data cache into separate, smaller horizontal structures for stack, global, and heap accesses [16,6] better exploits locality behavior of the different data regions. This can improve hit rates and cut dynamic and static energy consumption. Turning off dead portions of cache after writing dirty lines back to the next level of memory helps control leakage energy. Such decay caches [20,13] often trade performance for reduced power consumption. By predicting dead lines, Kaxiras et al. [13] reduce L1 leakage energy by up to a factor of five with little performance impact. A two-bit counter per line tracks the current working set, and at each adaptive decay interval they set line states based on counter values. Even when cache lines are still live, they may be read infrequently. Placing idle cache lines in a dormant state-preserving (drowsy) condition reduces static power consumption by dynamically decreasing supply voltage of the wordlines. Drowsy lines must be brought back to normal voltage before being loaded in the sense-amplifiers, thus access latency to these lines is increased. Repeatedly changing state causes high performance and power overheads. Drowsy wordlines consist of a drowsy bit, voltage control mechanism, and wordline gating circuit [14]. The drowsy bit controls switching between high and low voltages, and the wordline gate protects data from being accessed in drowsy mode. Flautner et al. [5] investigate simple and noaccess strategies (described in Sect. 1) to keep most lines drowsy and avoid frequent state changes. Geiger et al. make their heap cache drowsy [7] and explore a small, non-drowsy hot heap cache to hold lines with high temporal locality [6]. Since the stack and global caches service most references, an aggressive drowsy policy in the heap cache has negligible performance effects. The RMRO policy of Petit et al. [19] (described in Sect. 1) makes associative cache ways drowsy depending on their usage (this attempts to reduce leakage similarly to Albonesi’s [2] selective cache ways for dynamic power reduction). These policies cannot place a hard limit on total cache leakage. A two-way associative cache delivers similar performance to a direct mapped cache twice the size, but even on accesses that hit, the associative cache wastes power on the way that misses: two banks of sense-amplifiers are always charged simultaneously. Multiple-access or Hybrid Access caches address the area and power issues of larger direct mapped or conventional two-way associative caches by providing the benefits of associativity—allowing data to reside at more places in the cache, but requiring subsequent lookups at rehashed address locations on misses—to trade a slight increase in complexity and in average access time for lower energy costs. Smaller structures save leakage power, and charging shorter bitlines or fewer sets saves dynamic power. Examples include the hash-rehash (HR), column associative (CA) [1], MRU [10], skew associative [21], and predictive sequential associative [4] caches. HR caches swap lines on rehash hits to

Data Cache Techniques to Save Power and Deliver High Performance

69

Table 1. Baseline Processor Configuration Process Operating Voltage Operating Temperature Frequency Fetch Rate Decode Rate Issue Rate Commit Rate Functional Units Load/Store Queue Branch Prediction Base Memory Hierarchy Parameters L1 (Data) Size L1 (Data) Latency L1 (Instruction) Size L1 (Instruction) Latency L2(Unified) Size L2 Latency Main Memory

70nm 1.0V 65o C 1.7 GHz 1 per cycle 1 per cycle 1 per cycle 1 per cycle 2 Integer, 2 FP 4 entries not taken 32B line size 32KB, direct mapped 2 cycles 16KB 32-way set associative 1 cycle (pipelined) 512KB 4-way set associative 12 cycles 88 cycles

Table 2. Region Drowsy Cache Configuration Parameters L1 (Data) Size L1 (Data) Latency L1 (Stack)Size L1 (Stack) Latency L1 (Global) Size L1 (Global) Latency Drowsy Access

16KB, CA or 2-Way Set Associative 1 cycle 4KB, direct-mapped 1 cycle 4KB, direct-mapped 1 cycle 1 cycle

move most recently accessed lines to original lookup locations. CA caches avoid thrashing between original and rehash locations by adding a bit per tag to indicate whether a given line’s tag represents a rehashed address. Adding an MRU bit for way prediction reduces power consumption with minimal performance effects. Ease of implementation and good performance make CA and MRU waypredictive structures attractive choices for low power hierarchies; other multipleaccess designs focus on performance over energy savings.

3

Experimental Setup

We use SimpleScalar with HotLeakage [24] for the ARM and Alpha ISAs. For the embedded platform we simulate the MiBench suite [9], which represents a range of commercial embedded applications. For the high performance sector, we evaluate SimPoints [22] for 25 SPEC CPU 2000 applications with reference inputs on a four-issue Alpha architecture. 3.1

Embedded Architecture

Our simulator models region caching [6], and we incorporate column associativity and MRU way prediction, along with our RD drowsy policy. The embedded processor is single-issue, in-order, and five-stage pipelined, (see Table 1). We calculate static power consumption via HotLeakage, using CACTI 3.2 [23] and Wattch [3] to calculate dynamic power. We calculate memory latency for single accesses via CACTI. For baseline comparison with Geiger et al.’s work [7] we use their direct-mapped cache configurations (see Table 2).

70

M. Bhadauria et al.

HotLeakage calculates static cache leakage as a function of process technology and operating temperature. Operating temperature is 65o C, suitable for embedded systems for personal electronics. Dynamic consumption for typical logic circuits is 21 CV 2 f , a function of frequency, capacitance and operating voltage. CACTI accurately calculates numbers of sense amplifiers in associative caches. We convert CACTI energy numbers to power values based on parameters in Table 2. CACTI dynamic power values combined with HotLeakage static values compute total power, scaling CACTI results for frequency to add accurate numbers from different tools. Drowsiness or IPC changes do not affect dynamic power (e.g., from scaling clock rates), since it is independent of runtime. We assume negligible dynamic power increases due to clock switching for lower IPCs. Cache Accesses Check Drowsy Misses Increment These Bits These Bits 0 1 2 3 4 5 6 7

124 11 325 804 806 125 803 805

3 4 7 0 2 6 5 1

Fig. 2. Organization of Drowsy LRU Structure for an RD of Eight

Fig. 2 illustrates organization of a Reuse Distance drowsy mechanism. RD tracks lines being accessed, keeping a limited number of lines awake. The RD buffer stores IDs corresponding to awake cache lines. When the buffer is full and a new cache line is accessed, the LRU line is made drowsy and its ID is overwritten by the new line’s. Counters track the LRU buffer entry. RD circuitry never dictates what lines to awaken, but only lines to make drowsy, which keeps it off the critical path (making RD timing irrelevant). Power consumption and silicon area are accounted for (but negligible). An RD buffer of N entries only needs N counters of log2 N bits that are updated every memory access (and not every cycle). Assuming a reuse distance of eight and 1024 cache lines (as for a 32KB 32-Byte line baseline cache), storing awake cacheline IDs and LRU counts requires 104 bits ([log2 1024 + log2 8 bits] * 8), of which only a single buffer’s count (three bits) is reset every cache access. Power consumption for this extra circuitry is akin to the simple policy’s single counter’s dynamic power, since more bits are used but are accessed less frequently. An alternative implementation replaces counters with timestamps (incremented every cycle) to record when cache lines are accessed, and the lowest indicates the LRU line. To keep error of such an LRU scheme low, sufficient timestamp bits are needed (32 or 64 bits). 3.2

High Performance Alpha Architecture

Table 3 lists parameters of our Alpha 21264 model. For a valid baseline comparison with Flautner et al.’s work, we use their cache parameters, instead of the real 21264’s. All simulations use separate, single-cycle access, 32KB direct-mapped instruction and 32KB four-way associative data L1 caches. The unified L2 is

Data Cache Techniques to Save Power and Deliver High Performance

71

Table 3. High Performance Architectural Parameters Technology Frequency Temperature Voltage Issue/Decode/Commit Width Instruction Fetch Queue Size INT/FP ALU Units Physical Registers LSQ Branch Mispredict Latency Branch Type L1 Icache

L1 Dcache

L2 Cache

Main Memory

70 nm 1.5 GHz 80 C 1V 4 8 4/2 80 40 2 Tournament 32KB 4-Way Associative 1-cycle Access 32B Lines 32KB 4-Way Associative 1-cycle Access 32B Lines 256KB/512KB/1MB/2MB 4-way Associative 4/10/27/32 cycles 32B Lines 97 cycles

four-way set-associative and 256KB, 512KB, 1MB or 2MB in size (with appropriate memory latencies). We model drowsy L1 data and L2 caches. Switching from between sleep and wake-up modes incurs a one-cycle transition penalty. Tag lines stay awake, so only hits to drowsy cache lines suffer extra latency. Leakage current is a function of process technology (due to the transistor voltage threshold) and is largely dependent on temperature (doubling approximately every 10 degrees). We model an operating temperature of 80o C (typical for personal electronics), and a 70nm2 process technology. These are sufficiently state-of-the-art to support future cores with large L2 caches. Table 4 reports power parameters extracted via Wattch for modest clock frequencies of 500MHz and 1.4GHz. RD5 indicates that we keep five lines awake (in the L1 Dcache); RD1 indicates that we keep a single line awake (here, in the L2 cache). Using RD5 (for L1) and RD1 (for L2) policies reduces leakage by 92% and 94%, respectively. This configuration lets us maintain constant power while increasing core frequency from 500MHz to 1.4GHz. We account for power consumed by all cache circuitry, including mechanisms to implement RD drowsy caching.

4

Evaluation

We apply simple and RD policies to L1 data caches, examining leakage and power performance for embedded and high performance architectures. We extend the embedded architectures with multiple-access caches (half size) for the heap region. For the high performance architecture, we apply drowsiness to L1 and L2 caches to maximize energy reductions with minimal performance degradation. We compare performance as L2 scales and demonstrate how the power envelope can be used for scaling frequency. We compare RD and simple drowsy policies for different architectures, modifying parameters to obtain best energy-delay tradeoffs for each architecture. We characterize reuse distance for all benchmarks, finding the appropriate RD to maximize temporal locality for each suite. 2

This is the smallest feature size for which CACTI has an accurate spice model.

72

M. Bhadauria et al.

Table 4. Benchmark-Independent Power Parameters (Watts) for Frequency Scaling Leakage Power Non-Drowsy Caches L1 D-Leakage (32KB) L2 D-Leakage (2MB) Ireg I-Cache Core Processor Leakage Assumed Total Leakage

0.134 8.827 0.002 0.131 4 9.960

Drowsy Caches L1 D-Leakage (32KB) (Drowsy RD5) L2 D-Leakage (2MB) (Drowsy RD1) Ireg I-Cache Core Processor Leakage Assumed Total Leakage

0.011 0.530 0.002 0.130 4 1.663

Dynamic Power 500MHz Core Clock Frequency Total Chip Dynamic Power 7.502 Total Chip Power 20.461 1.4GHz Core Clock Frequency Total Chip Dynamic Power 16.159 Total Chip Power 20.822

4.1

Embedded Architecture

We compare our baseline region caches to organizations where the heap cache is replaced with a multiple-access cache (CA or MRU) of half the size. Access patterns for stack and global caches make their direct mapped organizations work well, thus they need no associativity. We study drowsy and non-drowsy region caches, comparing simple, noaccess, and RD policies. Keeping all lines drowsy incurs an extra cycle access penalty lowering IPC by up to 10% for some applications. If performance is not crucial, completely drowsy caches are attractive design points. The noaccess drowsy policy always uses slightly more power than the simple policy, but it yields higher average IPC by about 0.6%. Given the complexity of implementation, low payoff in terms of performance, and lack of energy savings, we use the simple policy in our comparisons. Keeping only three to five lines awake at a time yields good results. Many applications reuse few lines with high temporal locality, while others have such low temporal locality that no drowsy policy permits line reuse: our policy works well in both cases. The RD drowsy policy is implemented with a buffer (per region cache) that maintains an N-entry LRU “cache” of the most recently accessed set IDs. The RD buffer LRU information is updated on each cache access, and when a line not recorded in the buffer is accessed and awakened, the LRU entry is evicted and put to sleep. Unlike the simple and noaccess policies, RD requires no counter updates or switching every clock cycle, and thus consumes no dynamic power between memory accesses. We approximate RD’s dynamic power consumption as the number of memory accesses multiplied by the switching power of a number of registers equal to the buffer size. Static power overhead is negligible relative to cache sizes. RD is essentially a simplified implementation of the noaccess policy, but is based on the last unique N lines, not lines accessed during an arbitrary update interval. We experiment with buffers of three, five, 10, and 20 entries, finding that buffers of only three entries reduce power

Data Cache Techniques to Save Power and Deliver High Performance adpcm.decode adpcm.encode basicmath bitcount blowfish.decode blowfish.encode CRC32 dijkstra FFT.inverse FFT ghostscript gsm.decode gsm.encode ispell jpeg.decode jpeg.encode mad patricia quicksort rijndael.decode rijndael.encode rsynth sha stringsearch susan.corners susan.edges susan.smoothing tiff2bw tiff2rgba tiffdither tiffmedian typeset

100

Percentage of Normalized Total Accesses

73

80

60

40

20

0 1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Reuse Distance

Fig. 3. Reuse Distances for Last 15 Cache Lines

significantly over those with five, while only negligibly decreasing IPC. Larger buffers suffer larger cache leakage penalties because more lines are kept awake, in return for reducing the number of drowsy accesses. Fig. 3 illustrates the percentage of total accesses that are accessed within the last N or fewer memory references. On average, the last three and four memory accesses capture 43.5% and 44% of the total accesses to the heap cache. The increase is not linear, as the last 15 unique memory line accesses only capture 45.5% of all heap accesses. Update window size for the other drowsy policies is 512 cycles for the heap and stack cache, and 256 cycles for the global cache3 . We find RD’s power and performance to be highly competitive with the simple policy for the caches we study. Note that our small update windows are aggressive in their leakage energy savings: windows of 4K cycles, as in Flautner et al. [5] and Petit et al. [19], suffer 20% more leakage energy. Unlike RD, the simple drowsy policy is dependent on update window size and CPU frequency. This means RD scales well with number and sizes of caches: hardware overhead is minimal and fixed. To provide intuition into simple drowsy policy performance and motivation for our RD configuration, we track the number of awake lines during update intervals. Fig. 4 shows that for all benchmarks, on average, 66% of the intervals access fewer than four lines. These data are normalized to the total number of intervals per benchmark so that detail for shorter-running benchmarks is 3

These values were found to be optimal for simple drowsy policies and region caches [7].

74

M. Bhadauria et al. adpcm.decode adpcm.encode basicmath bitcount blowfish.decode blowfish.encode CRC32 dijkstra FFT.inverse FFT ghostscript gsm.decode gsm.encode ispell jpeg.decode jpeg.encode mad patricia quicksort rijndael.decode rijndael.encode rsynth sha stringsearch susan.corners susan.edges susan.smoothing tiff2bw tiff2rgba tiffdither tiffmedian typeset

1.0

Normalized Number of Intervals

0.8

0.6

0.4

0.2

0.0

0

1

2

3

4

5

6

7

>=8

Heap Cache Lines Referenced

Fig. 4. Number of Heap Cache Lines Accessed During simple Intervals (512 Cycles)

preserved. Benchmarks such as jpeg.decode use eight or more lines during 42% of their intervals, but larger active working sets mean increased leakage. RD’s performance on jpeg.decode is within 2% of simple’s, and RD saves 5% more leakage energy by limiting the number of lines awake at a time. These examples indicate that a configurable RD buffer size would allow software to trade off performance and energy savings: systems or applications would have fine-grain control in enforcing strict performance criteria or power budgets. Although CA organizations potentially consume higher dynamic power on a single access compared to a direct mapped cache, this slight cost is offset by significant leakage savings since the CA cache is half the capacity. The CA organization consumes less dynamic power than a conventional two-way set associative cache that charges two wordlines simultaneously (the two-way consumes the same power on hits and misses). In contrast, a CA cache only charges a second line on a rehash access. The second lookup requires an extra cycle, but rehash accesses represent an extremely small percentage of total accesses. Fig. 5 shows percentages of accesses that hit on first lookup, hit on rehash lookup, or miss the cache: on average, 99.5% of all hits occur on the first access. MRU associative caches use a one-bit predictor per set to choose which way to charge and access first. This performs well because most hits occur to the way last accessed. On a miss to both ways, the prediction bit is set to the way holding the LRU line to be evicted. On an incorrect guess or miss, MRU caches suffer an extra cycle of latency over a normal two-way associative cache. This is offset by significant power savings on correct predictions: since the MRU cache

Data Cache Techniques to Save Power and Deliver High Performance Misses Rehash Hit First Hit

100

% Total Accesses

80

60 40

60 40

20

20

0

0

ad adpc pcm. m de bl ba.encod o bl wf sic code ow is b m e fish.ditcoath h. ec un en o t cd FF CR odee T. dijkC3 in st 2 v r g gs ho ersa gsm. sts FF e m de cri T .e co pt jp nc d jpeg. isode eg de p e .e co ell nc d ode rij n rij da qupatmae nd e ic ric d l . ae d ks ia l.eec or ncod t oe sustri rsy de su ssanngs snth sa us .c ea ha n. an or rc sm .e ne h oodg rs th es tiftiff2ing f2 b t tif iffdrgbw fm it a h tyedi er pe an se t ad adpc m pc . m de bl ba.encod blowf sic code ow is b m e fish.ditcoath h. ec un en o t cd FF CR odee T. dijkC3 in st 2 v r g gs ho ersa gsm. sts FF e m de cri T .e co pt jp nc d jpeg. isode eg de p e .e co ell nc d ode rij n rij da qupatmae nd e ic ric d l ae .d ks ia l.eec or ncod t oe sustri rsy de su ssanngs snth sa us .c ea ha n. an or rc sm .e ne h oodg rs th es tiftiff2ing f2 b t tif iffdrgbw fm it a h tyedi er pe an se t

% Total Accesses

Misses Rehash Hit First Hit

100

80

75

(a) Column Associative Cache

(b) MRU Cache

Fig. 5. Heap Accesses Broken Down by Category

0.8

simple 2-way associative simple column associative simple MRU RD3 direct mapped RD3 column associative RD3 MRU

0.6 0.4

e

od

nc jp

jp

eg

.d eg

.e

ec

od

e

ll

e

pe is

e od

nc

.d

m

gs

gs

m

.e

ec

ts os gh

T. FF

od

t cr

FF

ip

T

e rs

ra in

di

h. fis ow bl

bl

ve

jk

C

st

32

de co

C

en

co de

ow

fis

h.

bi

ba

R

de

nt

h at

ou tc

cm si

nc .e m pc

ad

ad

pc

m

.d

ec

od

e

0.0

e

0.2 od

Normalized IPC

1.0

0.8 0.6 0.4

ty

pe

ia

se

t

n

er

ed fm tif

ba

ith fd tif

rg f2

bw

sa

n.

tif

sm

f2

in th oo

ed n. sa su

su

tif

g

s ge

s er

h n. sa su

st

rin

gs

co

ea

rn

rc

a sh

yn

th

e od

od

nc l.e ae nd

rij

rs

e

t or

ec l.d

rij

nd

ae

ci

ks ic qu

tri pa

m

0.0

a

0.2 ad

Normalized IPC

1.0

Fig. 6. IPCs Normalized to simple Drowsy Direct Mapped Region Caches

is physically partitioned into two sequential sets, it only charges half the bitline length (capacitance) of a same size direct mapped or CA cache. Fig. 5 shows percentages of hits in the predicted way, hits in the second way, and misses. On average, 98.5% of all hits are correctly predicted, resulting in a 50% reduction in dynamic power. The remaining accesses (the other 1.5% that hit plus the misses) consume the same dynamic power as a two-way associative cache. 4.2

Sustaining Performance

Fig. 6 graphs IPCs relative to direct mapped caches with the best update windows from Geiger et al. [7]. For most benchmarks, two-way set associative, CA, and MRU caches match or exceed the performance of their direct mapped counterparts, but do so at half the size. Hit rates are competitive, and MRU caches afford smaller access latencies from correct way predictions. Exceptions are ghostscript and the susan image processing codes, which have higher hit

su

t

se

n

ia

pe

ty

er

ba

ith

fm ed

tif

fd

tif

bw

rg

f2

tif

g

in

f2

s

ge

th

tif

oo

sm

h

a

rs

rn e

ed

co

n.

sa

n.

sa

su

n.

e

th

rc

ea

t

e

sh

yn

rs

od

nc

od

ec

gs

rin

sa

su

st

l.e

l.d

ae

rij nd

or

ks

ic

qu

a

tri ci

jp eg .

t

rip

el l

de

en co

de

de co

is p

od e

od e

nc

.e

ec

.d

jp eg .

gs m

gs m

FF T

sc

st

gh o

e

ra

ve rs

T. in

FF

st

di jk

R C 32

C

od e

en c

od e

de c

h.

bl ow fis

h

ou nt

bi tc

h.

bl ow fis

at

si cm

od e

od e

.e nc

ba

pc m

ad

t

n

se

pe

ty

er

ia

ed

fm

tif

ith

fd

tif

ba

rg

f2

tif

g

bw

f2

tif

in

s

ge

th

oo

sm

n.

sa

su

ed

s

h

a

er

rn

co

n.

sa

su

n.

sa

su

rc

ea

gs

rin

st

th

e

sh

yn

rs

t

e

od

nc

l.e

ae

nd

od

ec

l.d

a

or

ks

ic

qu ae

nd

rij

rij

ci

tri

pa

ad

m

Normalized Leakage Power 0.00

ae

rij nd

0.0

ad

Normalized Total DL1 Power

.d ec

0.0

m

pc m

ad

ll

e

e co d

en

g.

jp e

co d

de

g.

is pe

e

e nc od

.e

gs m

t

T

FF

se

ec od

.d

gs m

jp e

e

32

os ts cr ip

gh

C

di jk st ra

R

in ve r

T.

C

e

co d

en

co d

de

fis h.

h

nt

bi tc ou fis h.

FF

bl ow

bl ow

at

e

e

nc od

si cm

ba

Normalized Leakage Power ec od

.e

pc m

ad

.d

pc m

ad 0.00

pa

Normalized Total DL1 Power

ad Normalized pc ad m.d pc ec m od .e n e ba cod si cm e bl ow bi ath tc f i o s bl ow h.d unt fis eco h. en de co C de R C 32 FF dijk T. str in a ve rs e gh F o F gs sts T m cri p . gs dec t m od .e nc e od e jp eg isp e . jp dec ll eg o .e de nc od e m pa ad tri rij qu ci nd ic a k a rij el. sor nd de t ae co l.e de nc od rs e yn th st ri s su ngs ha sa ea n rc s .co h su usa rne sa n r n. .ed s sm g oo es th in tif g f2 tif bw f2 rg tif ba fd tif ithe fm r ed i ty an pe se t

Dynamic Power

76 M. Bhadauria et al.

1.0

0.8

0.6

0.4

0.2

0.0

0.10

0.05

0.6

0.4

0.2

simple 2-way associative simple column associative simple MRU

Fig. 7. Dynamic Power Consumption of Associative Heap Caches Normalized to a Direct Mapped Heap Cache

0.15

simple direct mapped heap RD direct mapped heap simple direct mapped stack RD direct mapped stack simple direct mapped global RD direct mapped global

0.15

0.10

0.05

Fig. 8. Static Power Consumption of Different Drowsy Policies with Direct Mapped Region Caches Normalized to Non-Drowsy Direct Mapped Region Caches

1.0

0.8

simple 2-way associative simple column associative simple MRU RD3 direct mapped RD3 column associative RD3 MRU

1.0

0.8

0.6

0.4

0.2

Fig. 9. Total Power of Drowsy Policies (for All Region Caches) and Heap Cache Organizations Normalized to Direct Mapped Region Caches

Data Cache Techniques to Save Power and Deliver High Performance

77

rates with associativity, but lower IPCs.4 Overall, IPCs are within 1% of the best baseline case, as with the results of Flautner et al. [5]. These differences fall within the range of modeling error. 4.3

Reducing Dynamic Power

Fig. 7 shows heap cache dynamic power normalized to a direct mapped baseline (note that dynamic power is independent of drowsy policy). CACTI indicates that the two-way associative cache uses 19% more power, the CA cache uses 7.3% less power, and the MRU cache uses 50% less power on a single lookup. Fig. 7 illustrates this for CA and MRU organizations: rehash checks and way mispredictions increase power consumption for some scenarios. 4.4

Reducing Leakage Current

Access pattern has little effect on static leakage. However, static power consumption is higher for benchmarks for which associative organizations yield lower IPCs than the direct mapped baseline. Reducing sizes of associative caches reduces leakage on average by 64% over the baseline direct mapped caches. Although IPC degrades by 1% between non-drowsy and drowsy caches, the leakage reduction in drowsy organizations is substantial, as shown in Fig. 8. Performance for the RD policy is slightly worse, but differences are sufficiently small as to be statistically insignificant. The noaccess policy has highest IPCs, but suffers greatest leakage, dynamic power, and hardware overheads; we exclude it in from our graphs to make room for comparison of the more interesting design points. Simple and RD exhibit similar savings, but the RD mechanism is easier to implement and tune, without requiring window size calibration for each workload. Software-configurable update window sizes for simple drowsy policies and RD buffer sizes achieve better power-performance tradeoffs among different workloads. No single setting will always yield best results. The effects of heap cache dynamic energy savings shown in Fig. 7 represent a small contribution to the total L1 Dcache power consumption shown in Fig. 9. Implementing drowsy policies across all memory regions plays a significant role in the power reductions we observe, with RD again having lowest power consumption of the different policies. MRU caches implementing RD drowsiness yield the lowest power consumption of the organizations and policies studied. This combination delivers a net power savings of 16% compared to the best baseline region organization implementing simple drowsiness, and 65% compared to a typical non-drowsy, partitioned L1 structure. These significant power savings come at a neglible performance reduction of less than 1.2%. 4.5

High Performance Architecture

For high performance architectures, the simple policy uses an optimal 4000-cycle execution window [5] before all cache lines expire and are turned off. Fig. 10 4

We find these results to be anomalous: on a single-issue, in-order, architecturally similar Alpha simulator, the way-associative caches have higher IPCs for these benchmarks.

78

M. Bhadauria et al. 15

bzip2 crafty eon gap gcc gzip mcf parser twolf vortex vpr

10

5

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

percentage normalized to total access

percentage normalized to total access

15

ammp applu apsi art equake facerec fma3d galgel lucas mesa mgrid sixtrack swim wupwise

10

5

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

reuse distance

reuse distance

(a) Integer Applications

(b) Floating Point Applications

Fig. 10. L1 Temporal Locality

and Fig. 11 show histograms of the percentages of total cache accesses having reuse distances from one to 24 line accesses. We use this to find good powerperformance tradeoffs for our RD mechanism. Temporal locality drops significantly after the most recent five unique line references. Temporal locality for the L2 is an order of magnitude lower than the L1, most likely due to the larger size, which results in many more lines to which data may map (lines ×associativity, i.e., 4096 ×4). After the last five cache lines are accessed, locality is very low for most benchmarks (vortex, vpr and mesa being the exceptions). This indicates that larger RD buffers will provide little benefit. Based on our histogram analysis, we partition the RD to keep 15 lines awake in L1 and one in L2 (i.e., the most recently accessed line). A drowsy L1 access increases latency significantly (by 50%) compared to a drowsy L2 access. We therefore spend the majority of our power budget masking performance degradation at the L1 level. We simulate an RD of size one, five and 15 and find 15 to give the best ratio of performance to active lines. Differences between RD policies are accentuated in Fig. 12, which shows numbers of drowsy accesses normalized to an RD of one, averaged for all the benchmarks. An RD of five decreases drowsy accesses by 35%, and an RD of 15 by 60%. The rate of return decreases after an RD of 15, so we use that as our basis for comparison with other drowsy policies. Fig. 13 through Fig. 15 compare IPCs and leakage for RD versus simple and RMRO [19] for a 512KB L2. Both deliver significant leakage savings with almost indiscernible performance degradation (less than 3%). RD delivers substantial savings at the L1 level (55.2% more than simple and 69.6% more than RMRO), since the number of awake lines is capped at 15 at any time. RD achieves lower power consumption than simple with the L2 cache, but the improvement is not as high as in the L1, since the L2 is not accessed as often: most lines are dormant the majority of the time, allowing both policies to perform favorably. Additionally,

Data Cache Techniques to Save Power and Deliver High Performance 5

4

bzip2 crafty eon gap gcc gzip mcf parser twolf vortex vpr

3

2

1

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

percentage normalized to total access

percentage normalized to total access

5

79

ammp applu apsi art equake facerec fma3d galgel lucas mesa mgrid sixtrack swim wupwise

4

3

2

1

0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

reuse distance

reuse distance

(a) Integer Applications

(b) Floating Point Applications

Fig. 11. L2 Temporal Locality

RD performs consistently across all L2 sizes we study, thus it scales well. Our results indicate that the most cost-effective design would be to construct the L2 cache out of high threshold transistors, and only keep the L1 caches with low latency high leakage components. As L2 sizes scale, ideally L1 cache performance and power would remain independent. Unfortunately, the simple policy update window parameter relies on the number of clock cycles: memory behavior for the L1 changes due to different L2 latencies for different sizes. Fig. 16 shows how numbers of drowsy accesses with the simple policy can differ by up to 50% from the largest L2 to the smallest. Drowsy access rates improve for some benchmarks and degrade for others. For example, increasing cache size from 256KB to 2MB reduces drowsy accesses by 47% for art, but a larger cache increases drowsy accesses by 66% for vortex. In contrast, we find RD drowsy accesses to change by less than 1%, thus application performance for L1 will be consistent across changes in L2 size. Furthermore, the RD scheme retains leakage savings for the L1 as L2 scales from 256KB to 2MB. The increases in L2 cache size result in increases in the number of drowsy accesses, since reuse distances increase. This affects both the simple and RD policies, but the RD policy ensures that leakage remains controlled by capping the number of awake lines. Note that increasing cache size improves IPC, which results in faster running times and a net reduction in leakage (since applications finish faster). Increased cache size does not yield significant increases in leakage, though, since the majority of cache lines are already drowsy. One advantage of RD is that the power envelope is guaranteed not to be exceeded. The power saved from reducing the leakage can be used for increasing the frequency without regard to potentially having too many cache lines awake at the higher frequency, thereby keeping the original power envelope. The frequency was increased from 500MHz to 1400MHz, while the total power remained

M. Bhadauria et al. % of Drowsy Accesses

80

1.0 0.8 0.6 0.4 0.2 0.0 0

10

20

30

RD Size

Fig. 12. Drowsy Accesses Normalized to an RD of One RMRO 256 simple RD15/1 L1/L2

r

is e

vp w

up

w

f

x

ol

rte

tw

vo

pa rs er si xt ra ck sw im

a

id

es m

m

gr

s

cf

ca

m

lu

c gc

gz ip

n eq ua ke fa ce re c fm a3 d ga lg el ga p

fty

eo

cr a

t

p2

ar

bz i

u pl

ap

am

m

p

0.0

si

0.5

ap

Normalized IPC

1.0

SPEC Benchmark

Fig. 13. Drowsy IPCs for 512KB L2 RMRO 256 simple RD15/1 L1/L2

0.3 0.2

e

r

is

vp w

up

w

rte

x

f ol tw

vo

im

ck

xt

ra

rs

sw

er

id pa

si

m

m

gr

es

a

cf m

s ca lu

ip

c gc

gz

n

eq ua ke fa ce re c fm a3 d ga lg el ga p

af cr

eo

ty

2

t ar

ip bz

si

pl

ap

am

m

p

0.0

u

0.1

ap

Normalized Leakage

0.4

SPEC Benchmark

Fig. 14. Drowsy DL1 Leakage for 512KB L2

RMRO 256 simple RD15/1 L1/L2

0.08 0.06 0.04

e

r

is

vp

w

up w

rte

x

f ol tw

vo

im

ra ck

sw

xt

rs er

pa

si

gr

id

a m es

m

m cf

s lu ca

c

ip gz

p

gc

el lg ga

ga

c

d

re

a3 fm

ce fa

n

ke ua

eq

eo

ty af cr

2

t ar

ip bz

ap

0.00

si

0.02

am m p ap pl u

Normalized Leakage

0.10

SPEC Benchmark

Fig. 15. Drowsy 512KB L2 Leakage

constant. Table 4 outlines the power consumption parameters of a an Alpha 21264 processor running at 500MHz on 90nm technology. The net power for the application decreases since the application completes faster. Fig. 17 shows the

Data Cache Techniques to Save Power and Deliver High Performance

81

ammp applu apsi art bzip2 crafty eon equake facerec fma3d gcc gzip lucas mcf mesa mgrid parser sixtrack swim twolf vortex vpr wupwise

L1 Drowsy Accesses Normalized to 256KB

1.5

1.0

0.5

0.0 512KB

1MB L2 Size

2MB

(a) DL1 Leakage for 2MB L2

0.8 0.6 0.4 0.2 0.0

simple 1.4GHz RD5/1 L1/L2 1.4GHz none 1.4GHz

am apmp p aplu si bz art cr ip2 a eq e fty o fa ua n ce ke fm re c a ga 3d lg gael gp g cc lu zip ca s mmc e m saf g p si arsrid xt e ra r sw ck twim vo o r lf w tex up v w pr is e

simple 1.4GHz RD5/1 L1/L2 1.4GHz none 1.4GHz

Normalized Leakage

0.8 0.6 0.4 0.2 0.0

am apmp p aplu si bz art cr ip2 af eq eoty fa uakn ce e fm rec a3 gcd gz c lu ip ca ms m cf m esa p gr si ars id xt e ra r sw ck twim vo ol r f w tex up vp w r is e

Normalized Leakage

Fig. 16. Drowsy Accesses for Simple Policy as L2 Scales

(b) 2MB L2 Leakage

Fig. 17. Leakage Normalized to 500MHz Non-Drowsy Caches

leakage improvement of a 1.4GHz processor compared to a 500MHz processor with non-drowsy caches. HotLeakage does not model frequency, thus leakage of the 1.4GHz processor has been scaled to yield an approximate representation for the 500MHz processor. Performance improves, but power remains the same, and net energy consumed is reduced. By finely controlling leakage via a stringent drowsy policy, the designer can channel power savings into increasing clock frequency and improving performance. The RD policy retains performance as frequency scales from 500MHz to 1.4GHz. Number of drowsy accesses remains constant in both L1 and L2. In contrast, with the simple policy, number of drowsy accesses increases with increasing frequency, depending on the application.

5

Conclusions

We investigate power reduction techniques for embedded and multi-level cache systems. For embedded systems, we adapt techniques developed to improve cache

82

M. Bhadauria et al.

performance, using them to address both dynamic and leakage power. We revisit multiple-access caches in the arena of embedded systems, finding that they generally achieve hit rates equal to larger direct mapped caches while a) reducing static power consumption compared to direct mapped caches, and b) reducing dynamic power consumption compared to normal associative caches. We employ multiple-access region caches with drowsy wordlines to realize further reductions in both dynamic and static power. With respect to drowsy caching, a simple three- or five-entry Reuse Distance buffer maintaining a few awake (recently accessed) lines performs as well as more complex policies. Our RD drowsy mechanism is easy to implement, scales well with different cache sizes and with number of caches, and enables finer control of power and performance tradeoffs than other published drowsy policies. Results for most competing drowsy caching solutions are highly dependent on update window sizes. These execution-window based solutions generally employ different intervals for different types of caches. Performance and power properties of all such policies are intimately tied to CPU speed, which means that intervals must be tuned for every microarchitectural configuration (and could be tuned for expected workloads on these different configurations). In contrast, behavioral properties of the RD drowsy mechanism depend only on workload access patterns. Combining multiple-access “pseudo-associativity” with region caching and our RD drowsy policy reduces total power consumption by 16% on average compared to a baseline direct mapped cache with a simple drowsy policy for embedded systems. This savings comes with less than 1% change in IPC. Compared to a direct mapped, non-drowsy region caching scheme, we remain within 1.2% of IPC while realizing power reductions of 65%. We also apply the RD policy to high performance systems with multiple levels of cache hierarchy. We find the L1, drowsy prediction rate is consistently high regardless of changes to the L2. Having an upper bound allows one to divert energy saved from reduced leakage to higher processor frequency in high performance systems. The RD policy provides similar performance (within 1%) to the simple and RMRO policies, while incurring 55% and 70% less leakage, respectively. A future feature could dynamically change reuse distance sizes depending on device operating mode. Often, handheld electronics have three main modes of operation. A high performance mode, with intensive computation, where the reuse distance is larger or infinite (no drowsy); a standard usage scenario, where reuse distance is optimized for a balance between speed and power; and a third mode, akin to standby, where the device is not being actively used. Such a standby mode of operation would only need a small reuse distance for significant reductions in leakage with no degradation in quality of service. Although process technology could potentially solve the leakage problem, future research will investigate steering data to dumb caches that don’t require drowsy circuitry (statically assigned high and low threshold cache banks, e.g.) depending on cache access patterns. Well managed drowsy caches will be important to a range of CMP systems, and thus we are beginning to study combinations of energy saving approaches to memory design within that arena. For shared cache

Data Cache Techniques to Save Power and Deliver High Performance

83

resources within a CMP, drowsy policies relying on specified windows of instruction execution become more difficult to apply, making the CPU-agnostic RD mechanism more attractive. In addition, we believe our RD drowsy mechanism to be particularly well suited to asynchronous systems.

References 1. Agarwal, A., Pudar, S.: Column-associative caches: A technique for reducing the miss rate of direct-mapped caches. In: Proc. 20th IEEE/ACM International Symposium on Computer Architecture, pp. 169–178 (May 1993) 2. Albonesi, D.: Selective cache ways: On-demand cache resource allocation. In: Proc. IEEE/ACM 32nd International Symposium on Microarchitecture, pp. 248–259 (November 1999) 3. Brooks, D., Tiwari, V., Martonosi, M.: Wattch: A framework for architecturallevel power analysis and optimizations. In: Proc. 27th IEEE/ACM International Symposium on Computer Architecture, pp. 83–94 (2000) 4. Calder, B., Grunwald, D., Emer, J.: Predictive sequential associative cache. In: Proc. 2nd IEEE Symposium on High Performance Computer Architecture, pp. 244–253 (February 1996) 5. Flautner, K., Kim, N., Martin, S., Blaauw, D., Mudge, T.: Drowsy caches: Simple techniques for reducing leakage power. In: Proc. 29th IEEE/ACM International Symposium on Computer Architecture, pp. 147–157 ( May 2002) 6. Geiger, M., McKee, S., Tyson, G.: Beyond basic region caching: Specializing cache structures for high performance and energy conservation. In: Conte, T., Navarro, N., Hwu, W.-m.W., Valero, M., Ungerer, T. (eds.) HiPEAC 2005. LNCS, vol. 3793, pp. 102–115. Springer, Heidelberg (2005) 7. Geiger, M., McKee, S., Tyson, G.: Drowsy region-based caches: Minimizing both dynamic and static power dissipation. In: Proc. ACM Computing Frontiers Conference, pp. 378–384 (May 2005) 8. Ghose, K., Kamble, M.: Reducing power in superscalar processor caches using subbanking, multiple line buffers and bit-line segmentation. In: Proc. IEEE/ACM International Symposium on Low Power Electronics and Design, pp. 70–75 (August 1999) 9. Guthaus, M., Ringenberg, J., Ernst, D., Austin, T., Mudge, T., Brown, R.: MiBench: A free, commercially representative embedded benchmark suite. In: Proc. IEEE 4th Workshop on Workload Characterization, pp. 3–14 (December 2001) 10. Inoue, K., Ishihara, T., Murakami, K.: Way-predicting set-associative cache for high performance and low energy consumption. In: Proc. IEEE/ACM International Symposium on Low Power Electronics and Design, pp. 273–275 (August 1999) 11. Inoue, K., Moshnyaga, V., Murakami, K.: Trends in high-performance, low-power cache memory architectures. IEICE Transactions on Electronics E85-C(2), 303–314 (2002) 12. Kamble, M., Ghose, K.: Analytical energy dissipation models for low power caches. In: Proc. IEEE/ACM International Symposium on Low Power Electronics and Design, pp. 143–148 (August 1997) 13. Kaxiras, S., Hu, Z., Martonosi, M.: Cache decay: Exploiting generational behavior to reduce cache leakage power. In: Proc. 28th IEEE/ACM International Symposium on Computer Architecture, pp. 240–251 (June 2001)

84

M. Bhadauria et al.

14. Kim, N., Flautner, K., Blaauw, D., Mudge, T.: Circuit and microarchitectural techniques for reducing cache leakage power. IEEE Transactions on VLSI 12(2), 167–184 (2004) 15. Kin, J., Gupta, M., Mangione-Smith, W.: Filtering memory references to increase energy efficiency. IEEE Transactions on Computers 49(1), 1–15 (2000) 16. Lee, H.: Improving Energy and Performance of Data Cache Architectures by Exploiting Memory Reference Characteristics. PhD thesis, University of Michigan (2001) 17. Lee, H., Smelyanski, M., Newburn, C., Tyson, G.: Stack value file: Custom microarchitecture for the stack. In: Proc. 7th IEEE Symposium on High Performance Computer Architecture, pp. 5–14 (January 2001) 18. Lee, H., Tyson, G.: Region-based caching: An energy-delay efficient memory architecture for embedded processors. In: Proc. 4th ACM International Conference on Compilers, Architectures and Synthesis for Embedded Systems, pp. 120–127 (November 2000) 19. Petit, S., Sahuquillo, J., Such, J., Kaeli, D.: Exploiting temporal locality in drowsy cache policies. In: Proc. ACM Computing Frontiers Conference, pp. 371–377 (May 2005) 20. Powell, M., Yang, S.-H., Falsafi, B., Roy, K., Vijaykumar, T.: Gated-Vdd: A circuit technique to reduce leakage in deep-submicron cache memories. In: Proc. IEEE/ACM International Symposium on Low Power Electronics and Design, pp. 90–95 (July 2000) 21. Seznec, A.: A case for two-way skewed-associative cache. In: Proc. 20th IEEE/ACM International Symposium on Computer Architecture (May 1993) 22. Sherwood, T., Perelman, E., Hamerly, G., Calder, B.: Automatically characterizing large scale program behavior. In: Proc. 10th ACM Symposium on Architectural Support for Programming Languages and Operating Systems, pp. 45–57 (October 2002) 23. Shivakumar, P., Jouppi, N.: CACTI 3.0: An integrated cache timing, power, and area model. Technical Report WRL-2001-2, Compaq Western Research Lab (August 2001) 24. Zhang, Y., Parikh, D., Sankaranarayanan, K., Skadron, K., Stan, M.: Hotleakage: A temperature-aware model of subthreshold and gate leakage for architects. Technical Report CS-2003-05, University of Virginia Department of Computer Science (March 2003)

Combining Edge Vector and Event Counter for Time-Dependent Power Behavior Characterization Chunling Hu, Daniel A. Jim´enez, and Ulrich Kremer Department of Computer Science, Rutgers University, Piscataway, NJ 08854 USA {chunling, djimenez, uli}@cs.rutgers.edu

Abstract. Fine-grained program power behavior is useful in both evaluating power optimizations and observing power optimization opportunities. Detailed power simulation is time consuming and often inaccurate. Physical power measurement is faster and objective. However, fine-grained measurement generates enormous amounts of data in which locating important features is difficult, while coarse-grained measurement sacrifices important detail. We present a program power behavior characterization infrastructure that identifies program phases, selects a representative interval of execution for each phase, and instruments the program to enable precise power measurement of these intervals to get their time-dependent power behavior. We show that the representative intervals accurately model the fine-grained time-dependent behavior of the program. They also accurately estimate the total energy of a program. Our compiler infrastructure allows for easy mapping between a measurement result and its corresponding source code. We improve the accuracy of our technique over previous work by using edge vectors, i.e., counts of traversals of control-flow edges, instead of basic block vectors, as well as incorporating event counters into our phase classification. We validate our infrastructure through the physical power measurement of 10 SPEC CPU 2000 integer benchmarks on an Intel Pentium 4 system. We show that using edge vectors reduces the error of estimating total program energy by 35% over using basic block vectors, and using edge vectors plus event counters reduces the error of estimating the fine-grained time-dependent power profile by 22% over using basic block vectors.

1 Introduction Research in power and energy optimizations focuses not only on reducing overall program energy consumption, but also on improving time-dependent power behavior. Evaluating such optimizations requires both accurate total energy consumption estimation and precise detailed time-dependent power behavior. Simulators are often used for power and performance evaluation, but detailed power simulation is very time-consuming and often inaccurate. While physical measurement is much faster, fine-grained power measurement requires proper measurement equipment and a large amount of space to store measurement results. An example optimization that requires fine-grained, time-dependent power behavior information for its experimental evaluation is instruction scheduling for peak power and P. Stenstr¨om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 85–104, 2009. c Springer-Verlag Berlin Heidelberg 2009 

86

C. Hu, D.A. Jim´enez, and U. Kremer

step power (dI/dt problem) reduction, for instance in the context of VLIW architectures [1,2,3]. This previous work relies on simulation to evaluate the impact of the proposed optimizations. The dI/dt problem is caused by large variations of current in a short time. Such variations in CPU current may cause undesired oscillation in CPU supply voltage, which may results in timing problems and incorrect calculations [4]. In this paper, we introduce a new strategy to enable time-dependent power behavior characterizations based on physical measurements. 1.1 Characterizing Phases with Representative Slices Program phase behavior shows that many program execution slices have similar behavior in several metrics, such as instructions-per-cycle (IPC), cache miss rate, and branch misprediction rate. Phase classification makes it easier to measure the fine-grained program behavior. A representative slice from each phase instead of the whole program execution is measured and analyzed, and then the whole program behavior can be characterized based on the analysis result. Using this whole program behavior characterization method in power behavior analysis, we can obtain fine-grained power behavior with significant savings in both time and storage space. 1.2 Illustrating Time-Dependent Power Behavior Figure 1 shows the measured CPU current of 256.bzip2 from SPEC CPU 2000 measured using an oscilloscope. Figure 1(a) shows that the program execution can be roughly partitioned into 4 phases based on its power behavior. One representative slice from each phase can be measured to characterize the detailed power behavior of the benchmark. Figure 1(b) is the measured power behavior of half of a second in the first phase with a resolution that is 100 times higher than the one used for Figure 1(a). There is a repeated power behavior period of 300 milliseconds. Figure 1(c) shows the detailed power behavior of a piece of 0.05 second, from 0.1 second to 0.15 second in Figure 1(b). It shows repeated power behavior periods of less than 5 milliseconds, indicating possible finer phase classification than Figure 1(b). Also, finer measurement gives more information of time-dependent CPU power due to the resolution of the oscilloscope that we use for power measurement. The oscilloscope reports the average power for a given time granularity. This is the reason why the difference between the observed peak power (peak current) in Figure 1(a) and (c) is almost 6 Watts (0.5 amperes). 1.3 An Infrastructure for Characterizing Time-Dependent Power Behavior In this paper, we present our infrastructure for program time-dependent power behavior characterization and optimization evaluation. Our Camino compiler statically instruments the assembly code of a program for profiling and physical measurement. A SimPoint-like [5] method is used for phase classification. SimPoint identifies several intervals, or simpoints, of program execution that characterize the behavior of the entire program execution. It is often used to speed up simulation by simulating only the simpoints and estimating, for instance, IPC, by taking a weighted average of the IPCs of each simpoint.

Combining Edge Vector and Event Counter

87

(a) Very coarse granularity CPUcurrent (A)

6 5 4 3 2 0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

time (second)

(b) A slice in phase 1 of (a) CPUcurrent (A)

6 5 4 3 2 0.10

0.11

0.12

time (second)

0.13

0.14

(c) Detailed CPU power behavior of a small slice in (b) Fig. 1. f Measured power behavior of bzip2 with different granularity

SimPoint uses the Basic Block Vector (BBV), i.e., a vector of counts of basic block executions, as the feature for classification. We introduce the edge vector (EV), i.e., a vector of counts of control-flow-graph edge traversals, as the fingerprint of each interval of the program execution. Instead of using a fixed number of instructions as interval length, we use infrequently executed basic blocks to demarcate intervals. This results in variable interval length, but much lower instrumentation overhead for physical power measurement of a representative interval. The selected simpoints are weighted based on the number of instructions executed in each phase, instead of number of intervals. We show that our method enables us to do power measurement for simpoints with very low interference to program execution. To demonstrate the improved accuracy of using edge vectors for classification, we show that our infrastructure estimates the total energy of a program with an average error of 7.8%, compared with 12.0% using basic block vectors, an improvement of 35%. More importantly, we want to find representative intervals that represent the fine-grained time-dependent power profile of a phase. We develop a metric for measuring the accuracy of estimating a power profile and show that using edge vectors with event counter information improves accuracy by 22%. Unlike simulation, physical measurement is sensitive to the overhead for identification of simpoints during program execution. So this low instrumentation overhead is very important. This infrastructure can be used to evaluate optimizations for energy consumption or time-dependent power behavior, for example, the impact on power behavior of pipeline gating [6] or dynamic voltage/frequency scaling [7]. We evaluate our infrastructure by measuring 10 SPEC CPU2000 integer benchmarks on a Pentium 4 machine, and we present the error rates in whole program energy

88

C. Hu, D.A. Jim´enez, and U. Kremer

consumption estimation as well as fine-grained power behavior estimation based on the measurement result of the selected simpoints. This paper makes the following contributions: 1) We show that using edge vectors significantly improves accuracy over using basic block vectors for estimating total program energy as well as fine-grained power behavior. 2) We show that classification accuracy can further be improved by combining control-flow information such as edge vectors with event counter information. 3) We present our infrastructure that uses edge vectors and event counters to select representative intervals and to measure efficiently their power profiles with minimal perturbation of the running program.

2 Related Work Several techniques have been proposed to identify program phases. Some of them use control-flow information [8,5,9,10,11], such as counts of executed instructions, basic blocks, loops, or functions, as the fingerprint of program execution. This fingerprint depends on the executed source code. Some methods depend on run-time event counters or other metrics [12,13,14,15], such as IPC, power, cache misses rate and branch misprediction, to identify phases. Our infrastructure uses the edge vector of each interval, a vector that gives a count for each control-flow edge in the program, along with the measured IPC. This set of features allows for a precise characterization of power phases. SimPoint [8,5] partitions a program execution into intervals with the same number of instructions and identifies the phases based on the BBV of each interval. One interval, called a simpoint, is selected as the representative of its phase. These simpoints are simulated or executed to estimate the behavior of the whole program execution. Sherwood et al. apply SimPoint to SPEC benchmarks to find simpoints and estimate the IPC, cache miss rate, and branch misprediction rate. The error rates are low and the simulation time saving is significant. A new version of SimPoint supports variable length intervals. Lau et al. [11] shows a hierarchy of phase behavior in programs and the feasibility of variable length intervals in program phase classification. They break up variable length intervals based on procedure call and loop boundaries. We use infrequent basic blocks to break up intervals and at the same time use a pre-defined length to avoid too long or too short intervals. This satisfies our requirement for low-overhead instrumentation and accurate power behavior measurement. Besides phase classification, we also generate statically instrumented executables for physical measurement of simpoints and CPU peak power control on a dual core machine. Shen et al. [9] propose a data locality phase identification method for run-time data locality phase prediction. A basic block that is always executed at the beginning of a phase is identified as the marker block of this phase, resulting in variable interval lengths. They introduce the notion of a phase hierarchy to identify composite phases. We also use variable interval lengths, but the basic block that marks a phase is not necessary to uniquely mark the phase. It might be the mark for other phases. Phases are identified by the execution times of the infrequent basic blocks that demarcate the intervals, such that we implement precise physical measurement.

Combining Edge Vector and Event Counter

89

PowerScope [16] maps energy consumption to program structure through runtime system power measurement and system activity sampling. System components responsible for the bulk of energy consumption are found and improved. The delay between power sampling and activity sampling results in possible imprecise attribution of energy consumption to program structure. Compared to the power measurement granularity used by PowerScope, which is 1.6ms, our infrastructure measures CPU current with much higher granularity. 1000 samples are collected for each 4ms. Precise mapping between power measurement and program structure is achieved through measuring the selected representative intervals. Isci and Martonosi [17] show that program power behavior also falls into phases. Hu et al. propose using SimPoint to find representative program execution slices to simplify power behavior characterization, and validate the feasibility of SimPoint in power consumption estimation through power simulation of some Mediabench benchmarks [18]. Isci and Martonosi [19] compare two techniques of phase characterization for power and demonstrate that the event-counter-based technique offers a lower average power phase classification errors. Our goal is to characterize the time-dependent power behavior, instead of power consumption, of programs. Our method causes negligible overhead for identification of an interval during program execution, and the measurement result is very close to the real time-dependent power behavior of the interval. Furthermore, through the combination of edge vector and event counters, we get better phase characterization than using only control flow information as well as the mapping between observed power behavior and the source code. The latter is difficult for an event-counter-based technique by itself.

3 Phase Classification Based on Edge Vectors and Event Counters Our phase classification infrastructure is based on the ability to demarcate the start and end of a particular interval of execution with infrequently executed basic blocks. We instrument these infrequent basic blocks so that our instrumentation minimally perturbs the execution of the program. Phase classification and power measurement of representative intervals for programs is implemented as an automatic process. The threshold for determining whether a basic block is infrequent, the minimum number of instructions in each interval, and the number of phases are the input to this process. The flowchart in Figure 2 illustrates its steps. The implementation of each step will be presented in the following sections. 3.1 Instrumentation Infrastructure for Profiling, Measurement, and Optimization Camino [20] is a GCC post-processor developed in our lab. We use it to implement the static instrumentation for profiling and physical power measurement. Camino reads the assembly code generated by GCC, parses it into a control-flowgraph (CFG) intermediate representation, performs transformations including instrumentation, and then writes the modified assembly language to a file to be assembled and linked.

90

C. Hu, D.A. Jim´enez, and U. Kremer

Instrument assembly code for basic block execution frequency profiling

Instrument assembly code for EV profiling

Instrument final infrequent basic blocks for each simpoint

Compile and run instrumented code Compile and run instrumented code Find infrequently executed basic blocks

Phase classification based on profiled EVs Refine phases using profiled IPC Select simpoints. Find final infrequently basic blocks, calculate their execution frequency for future simpoint identification

Infrequent basic block determination

Compile and run instrumented code, measuring CPU power behavior for the simpoint

Edge Vector profiling and phase classification

Characterize whole program power behavior using the measured power of each simpoint and the corresponding weight

Power measurement and characterization

Fig. 2. Infrequent basic block-based phase classification and power measurement of simpoints

Instrumentation using Camino is simple and minimally intrusive. Only two routines are required: an instrumentation routine that inserts a call to the analysis routine, and and an analysis routine that does profiling or generates special signals. Note that our infrastructure does two kinds of instrumentations: 1) profiling for all basic blocks to identify infrequent basic blocks and gathering features used to do phase classification, and 2) infrequent basic block instrumentation for signaling the start and end of a representative interval to our measurement apparatus. The first kind of instrumentation results in a moderate slowdown, but the second kind results in no slowdown so that the measured program’s behavior is as close as possible to that of the uninstrumented program. 3.2 Infrequent Basic Blocks Selection Instrumentation is done through Camino to collect the execution frequency of each basic block. Each basic block makes a call to an execution frequency counting library function. The distinct reference value of the basic block is passed to the function that increments the frequency of this basic block. During the first profiling pass, we collect counts for each basic block. A threshold is needed to determine which basic blocks are infrequently executed and can be used to demarcate intervals. An absolute value is infeasible, since different program/input pairs execute different number of basic blocks. Instead, we consider a basic block to be infrequent if it account for less than a certain percentage of all executed basic blocks. Intuitively, when a low threshold is used, the selected infrequent basic blocks will be distributed sparsely in program execution and there is more variance in interval size than when a higher threshold is used. We investigate 4 different threshold values, 0.05%, 0.1%, 1%, and 5%, to explore the trade-off between interval size variance and instrumentation overhead. 3.3 Program Execution Interval Partitioning and Edge Vector Profiling We use the edge vector (EV) of all edges as the fingerprint of an interval used for the clustering phase of our SimPoint-like phase classification method. This vector is the absolute count for each control-flow edge traversed during the execution of an interval. Compared to basic block vectors (BBV), EVs give us more information about the

Combining Edge Vector and Event Counter

1

1

2

4

3

2

3

5

6

4

5 7

6

91

8

BBV: EV:

……

Fig. 3. Several EVs are possible for the same BBV

control behavior of the program at run-time. BBVs contain information about what parts of a program were executed, but EVs tell us what decisions were made in arriving at these parts of the program. This extra information allows a classification of phases that more accurately reflects program behavior. For the same BBV, it is possible that there are several EVs depending on the dynamic paths taken during program execution. An example is shown in Figure 3. Partitioning program execution just based on the execution times of infrequent basic blocks will results in intervals with a variable number of instructions. Acquiring detailed information from very large intervals to characterize program behavior is inefficient. Moreover, a large variance in interval size affects the accuracy of the phase classification result. In order to make use of our physical measurement infrastructure to characterize the whole program behavior more correctly and efficiently, we use a pre-specified interval size to avoid largely variable intervals. Instrumentation for EV profiling is similar to that for basic block execution frequency profiling. An pre-defined interval size of 30 million instructions is used to avoid too large or too small intervals. All basic blocks are instrumented so that we can get the complete fingerprint of an interval. The library function remembers the last executed basic block and knows the taken edge based on the last and the current executed basic blocks. It counts each control flow edge originating in a basic block that ends in a conditional branch. It counts the total number of executed instructions for the current interval as well. When an infrequent basic block is encountered, if the count is larger than or equal to 30 million, this basic block indicates the end of the current interval and it is the first basic block of the next interval. Note that, because we only have coarse control over where the demarcating infrequent basic blocks will occur, the actual interval might be somewhat longer than 30 million instructions; thus, the intervals are variable-length. Figure 4 illustrates the interval partition using the combination of infrequent basic block and interval size. Here A, B, C, and D are basic blocks. C and D are infrequent and used to demarcate intervals. Since we use a pre-defined interval size, 30 million, only the occurrences of C and D in shadow mark intervals. Other occurrences do not mark intervals because the interval size is smaller than 30 million when they are encountered. We get intervals of similar size by using this method. An execution frequency counter of C and D can be used to identify the exact execution of an interval. For example, the fourth interval starts when the counter is 5 and ends when the counter is 8.

92

C. Hu, D.A. Jim´enez, and U. Kremer

basic block sequence C …….DABBBAABCBBCAAACAAAAAAABCAABBBDBADDACBBBCBAAABBABDACCBBA D C C D C D C ABAC…… 30M

30M

30M

30M

30M

30M

30M

30M

30M

30M

30M

number of instructions

Fig. 4. Interval partitioning using infrequent basic blocks and interval length

3.4 Phase Classification for Power Behavior Characterization Intervals profiled in Section 3.3 are classified into phases based on their EVs. K-Means clustering is used to cluster the intervals with similar EVs and select a representative for each phase. The EV of each interval is projected to a vector with much smaller dimension. Then k initial cluster centers are selected. The distance between a vector and each center is calculated and each vector is classified into the cluster with the shortest distance. A cluster center is changed to the average of the current cluster members after each iteration. The iteration stops after the number of vectors in each cluster is stable. The simpoint of a phase is the one that is closest to the center of the cluster [5]. Since the intervals are demarcated by infrequently executed basic blocks and have variable number of instructions, weighting a simpoint with just the number of intervals in its phase cannot reflect the real proportion of this phase in whole program execution. In our method, each simpoint has two weights. One is based on the percentage of the number of executed instructions of the corresponding phase in that of the whole program, the other is based on the number of intervals in the corresponding phase as the one used in [5]. A recent version of the SimPoint tool also supports variable-length phases [11]. Given the number of phases, K-Means clustering is performed for different number of clusters and different cluster seeds. The BIC (Bayesian Information Criterion) score of each clustering is calculated and used to choose the clustering with the best trade-off between BIC score and number of phases. The BIC score calculation in our method is changed to use the number of executed instructions in each phase such that phases with longer intervals have larger influence. To identify an interval during program execution, we need to find the beginning and end of the interval. We use the execution frequencies of one or two infrequent basic blocks that demarcate the interval. Infrequent basic blocks that mark the start or end of a desired representative interval are chosen as final infrequent basic blocks. Their execution frequencies in each interval are recorded, so that we know how many times a basic block has executed before the start of an interval. We instrument these final infrequent basic blocks with simple assembly code to increment a counter and trigger power measurement when the count indicates the beginning of the interval, or turn off measurement when the end of an interval is reached. The combination of infrequent basic blocks and static instrumentation enables us to identify the execution of an interval at run-time with negligible overhead. 3.5 Finer Phase Classification Using IPC Two intervals that execute the same basic blocks may generate different time-dependent power behavior due to run-time events, such as cache misses and branch mispredictions.

Combining Edge Vector and Event Counter

93

Phase classification only based on control flow information cannot precisely differentiate these intervals, so the resulting simpoints may not really be representative in terms of power behavior. Our infrastructure combines EV and instructions-per-cycle (IPC) as measured using performance counters provided by the architecture to take the run-time events into account. IPC Profiling. Profiling IPC is easy to do in our infrastructure. After the program execution is partitioned into intervals, all of the infrequent basic blocks that demarcate the resulting intervals are instrumented to collect the number of clock cycles taken by each interval. By running the instrumented program once, we can get the IPC values of all intervals by dividing the number of instructions by the number of cycles. We already have the number of instructions executed from the edge vector profiling. This technique very slightly underestimates IPC because of system activity that is not profiled, but we believe this has no impact on the accuracy of the classification since IPC tends to vary significantly between phases. Since we identify intervals based on infrequent basic block counts, the overhead is low and has a negligible impact on the accuracy of the profiling result. Combining EV Clustering with IPC Clustering. For a program execution, we first perform the phase classification in Section 3.4 to group intervals with similar EVs together. Then we do another phase classification based on the profiled IPC values. KMeans clustering is also used in the second phase classification. Then we combine the results from the two classifications and get a refined phase classification for power behavior characterization through refining the classification result of the first one using that of the second one. The mechanism in the next section performs more control on the number of the resulting phases without a significant loss in accuracy. Our experiment result shows that after applying the controlling mechanism, if the number of phases identified based on IPC is 10, the number of the resulting phases after the classification refinement is expanded to less than 3 times of the number after the first classification, instead of around 10 times. Controlling Unnecessarily Fine Phase Classification. Using a constant K value for the IPC-based phase classification of all programs results in unnecessarily fine partitioning and more simpoints to simulate or measure when the IPC values of the intervals in the same phase are already very close to each other. We control the number of resulting phases based on IPC in two steps. The first step controls the selection of the initial centers based on the maximum and minimum IPC of the program. A percentage of the minimum IPC value is used as the distance d between the initial centers. This ensures that intervals with very close IPCs need no further partitioning and the final number of simpoints does not explode with little benefit. This percentage is adjustable in our infrastructure. The maximum value is divided by d. The value of quotient plus 1 is then compared with the given k. The smaller one is used as number of clusters. This value may be 1, meaning that the IPC values of all of the intervals are very close and no finer partitioning is necessary. The second step maintains the distance between centers during the initialization of the centers in case there is a IPC much higher than others, but there are only two different IPC values during program execution. The first step does not know this and the

94

C. Hu, D.A. Jim´enez, and U. Kremer

number of clusters will be k which results in unnecessarily more simpoints. This step is similar to the construction of a minimum spanning tree except that we use the largest values in each step to choose the next initial center. The first initial center is selected randomly. During the generation of the other initial centers, each time the value with largest distance to the existing centers is the candidate. If this distance value is less than half of d, no more initial centers are generated. This prevents intervals with the similar EVs and very close IPCs from being partitioned into different clusters.

4 Experimental Setup We validate our infrastructure through physical power measurement of the CPU of a Pentium 4 machine. This machine runs Linux 2.6.9, GCC 3.4.2 and GCC 2.95.4. Benchmarks are from the members of SPEC CPU2000 INT that can be compiled by Camino successfully. The back-end compiler for gzip, vpr, mcf, parser and twolf is GCC 3.4.2. The back-end compiler for the other benchmarks is GCC 2.95.4 because the combination of Camino and GCC 3.4.2 fails to compile these programs correctly. We measure the current on the separate power cable to the CPU using a Tektronix TCP202 DC current probe, which is connected to a Tektronix TDS3014 oscilloscope. The experimental setup is shown in Figure 5. The data acquisition machine is a Pentium 4 Linux machine that reads data from the oscilloscope when a benchmark is running on the measured system. Simultaneous benchmark execution and power data acquisition on different machines eliminates interference with the measured benchmark. The picture on the right of Figure 5 is our experimental setup, data acquisition machine is not shown in the picture. The oscilloscope has a TDS3TRG advanced trigger module. When it is in trigger mode, it accepts trigger signals from one of its four channels. We use its edge trigger. It starts measurement only after the voltage or current on the trigger channel increases to some predefined threshold and stops when its window fills to its capacity. The data points stay in the buffer until the next trigger signal. We generate the trigger signal by controlling the numlock LED on the keyboard. A voltage probe is connected to the circuit of the keyboard to measure the voltage on the LED as shown in Figure 5. The voltage difference between when the light is on and off is more than 3.0V, which is

Data Acquisition Machine

Measured System

Power data CPU current

Keyboard

trigger Oscilloscope

Fig. 5. The physical measurement infrastructure used in the experiments

Combining Edge Vector and Event Counter

95

enough to trigger the oscilloscope. The voltage on the trigger channel is set to high by instrumentation code to trigger the oscilloscope at the beginning of the program slice to measure. This voltage is consistently high until it is set to low at the end of this slice. It is easy to identify the power behavior of the measured slice. 4.1 Instrumentation Overhead In order to get the power behavior close to the real program power behavior, the instrumentation overhead should be as low as possible to reduce its impact on the measured power behavior. We instrument all of the infrequent basic blocks that demarcate the final simpoints to evaluate the overhead. The instrumented code does the same thing as it does to generate signals before and after each simpoint, but controls another LED. Thus, we get the same overhead as when the CPU power of a simpoint is measured, and still can use the numlock to generate signals to get the precise measurement of each program. If we measure the simpoints one by one, the overhead is even lower than the one measured in this experiment, since only one or two basic blocks are instrumented. We use the auto mode of the oscilloscope to measure the power behavior of the whole benchmark execution and still identify the exact power data points for the benchmark by setting the voltage on the trigger channel to high and low before and after the execution of each benchmark. However, no instrumentation is needed to generate signals during program execution. The oscilloscope records power data points continuously, and the data acquisition program running on another machine collects the data points. We adjust the data acquisition to read the data in each window without losing data points or reading duplicated data points due to a data reading period that is too long or too short, respectively. This is validated through the comparison of the real benchmark execution time and the one obtained from the measurement result. To evaluate the instrumentation overhead, we also measure the power consumption of the 10 benchmarks without any instrumentation. 4.2 Energy Consumption Estimation Based on Simpoints The first step to verify that this infrastructure is useful in power behavior characterization is to calculate the error rate when the measurement result of the selected simpoints is used to estimate the power consumption of the whole program. Although we use EVs as the fingerprint of an interval in our infrastructure, we also measured the CPU power of the simpoints using BBVs for comparison. The energy consumption of each simpoint is measured using the trigger mode of the oscilloscope. We generate an executable for each simpoint and measure the simpoints one by one so we can get very high resolution as well as the lowest possible instrumentation overhead. Program execution and data acquisition are on the same machine. Reading data from the oscilloscope is scheduled after the measurement of a simpoint is done. Data acquisition does not interfere with the running program. We implement an automatic measurement and data acquisition process to measure any number of simpoints as a single task.

96

C. Hu, D.A. Jim´enez, and U. Kremer

4.3 Power Behavior Similarity Evaluation Even though we can get low error rates in estimating whole program energy consumption, energy consumption is the average behavior of an interval. Intervals that are classified into the same phase may have different time-dependent power behavior. If intervals in the same phase have largely different power behavior, we cannot characterize the time-dependent power behavior of the whole program execution using the measurement result of the simpoints. Comparing in the Frequency Domain. Our power measurements come in the form of discrete samples in the time domain. Power behavior is characterized by periodic activity, so a comparison in the frequency domain is more appropriate for determining whether two intervals are similar. Fast Fourier Transform (FFT)is a computationally fast way to calculate the frequency, amplitude and phase of each sine wave component of a signal. Thus, we compare the power behavior similarity of two intervals by comparing their discrete Fourier transforms computed using FFT. After the FFT calculation of a power curve, each frequency is represented by a complex number. In power curve similarity comparison, the phase offset of the same frequency should not affect the similarity of two curves. For instance, two power curves might be slightly out of phase with one another, but have exactly the same impact on the system because they exhibit the same periodic behavior. So when we compare two power curves, we calculate the absolute value of the complex number for each frequency, the distance between two corresponding absolute values, and the Root Mean Square (RMS) of the distances for all frequencies. The equation is given in a following section. Figure 6 shows the FFT distance between the sine curves with different values in amplitude, frequency and phase offset, calculated using our method mentioned above. We generate 4096 samples for each curve. Ideally, there is only one frequency in the FFT output of each sine curve. But we get multiple frequencies due to the discrete data samples. This is the reason why the calculated distance values are not 0’s in Figure 6 (c). The three curves in Figure 6 (a) have the same frequency and phase offset, but different amplitude, which determines the simmilarity of two curves. Figure 6 (b) shows the effect of frequency in our similarity calculation. The small(compared to the values in (a) and (b)) distance between the curves in Figure 6 (c) demonstrate that the effect of phase offset is eliminated. A More Robust Sampling Approach for Verification. Measuring every interval in a long-running program is infeasible because of time and space constraints (indeed, this fact motivates our research). Thus, we use a more robust sampling methodology to verify that power behavior is consistent within a phase. We choose 20 intervals at random for each phase of each program to compare the FFT results of their curves. If the number of intervals in some phase is less than 20, all of the intervals are selected. The selected intervals for each phase are selected from a uniformly random distribution among all the intervals in the phase. Instrumenting for Verification. Infrequent basic blocks demarcating the intervals from the same phase are instrumented to measure each interval in the same way we

Combining Edge Vector and Event Counter

97

6 current1 current2 current3

5

Current

4

3

2

1

0 0

500

1000

1500

2000 time

2500

3000

3500

4000

(a) same frequency and phase offset, different amplitude. dist(1,2)=22.4, dist(1,3)=89.8, dist(2,3)=67.3 6 current1 current2 current3

5

Current

4

3

2

1

0 0

500

1000

1500

2000

2500

3000

3500

4000

time

(b) same amplitude and phase offset, different frequency. dist(1,2)=119.6, dist(1,3)=115.6, dist(2,3)=120.8 6 current1 current2 current3

5

Current

4

3

2

1

0 0

500

1000

1500

2000 time

2500

3000

3500

4000

(c) same amplitude and frequency, different phase offset. dist(1,2)=6.5, dist(1,3)=8.1, dist(2,3)=3.0 Fig. 6. Power curve distances calculated using our similarity calculation method

measure a simpoint. Each selected interval is measured separately. Then the FFT is performed on the measured power curve of each interval. The Root Mean Square (RMS) error of the FFT results is used to evaluate the variation of the power behavior of the intervals in this phase. For each phase, we calculate the arithmetic average over the frequencies in the FFT result of all measured intervals as the expected FFT of the phase. The distance between an interval i and the expected FFT is:

Di =

  N   2   ( cj 2 + dj 2 − aj 2 + b j 2 )   j= 1 N

98

C. Hu, D.A. Jim´enez, and U. Kremer

cj and dj are the real and imaginary part of the jth frequency of interval i, respectively. aj and bj are the real and imaginary part of the jth frequency of the expected FFT respectively. N is the number of frequencies in the output of Fast Fourier Transform. Then the FFT RMS of a phase is calculated as:   M   Di 2   i=1 F F TRMS = M M is the number of measured intervals in the phase. The lower F F TRMS is, the high the similarity among the time-dependent power behavior of the intervals in the phase. The F F TRMS for each phase is then weighted by the weight of the corresponding phase to get the RMS for the whole benchmark. We evaluated the weighted F F TRMS for all of the 10 benchmarks in two cases: when phase classification is based on EV only, and when IPC is used to refine phase classification. 4.4 Interval Length Variance Using infrequent basic blocks to partition program execution into intervals results in variable interval length. We use a pre-specified interval size to avoid intervals that are too small. Intervals of large size are still possible due to the distribution of the infrequent basic blocks during program execution. We analyze the resulting size for each interval of each benchmark to show the distribution of the interval sizes. We evaluate the interval length variance of a benchmark as the weighted RMS of the interval lengths in each phase. If this value is high, intervals that are of largely different number of instructions are classified into the same phase, the simpoint for the phase can not be representative of the phase in terms of power behavior.

5 Experimental Results and Evaluation Using the power measurement infrastructure described in Section 4, we measured the CPU power curves for the instrumented benchmarks, the ones with all final infrequent basic blocks instrumented, the simpoints, and the selected intervals from each phase. 5.1 Instrumentation Overhead Figure 7 shows the overhead of the instrumentation using different thresholds. It is normalized to the measured energy consumption of the uninstrumented benchmarks. A positive value means the measured energy consumption for this configuration is larger than that of the uninstrumented one. A negative value means the opposite. For some benchmarks, for example, perlbmk and gap, the energy consumption of the instrumented program is slightly lower than the uninstrumented program. One possible reason is that inserting instructions somewhere might accidentally improve the performance or power consumption, possibly due to a reduction in conflict misses in the cache

Normalized Energy Overhead

Combining Edge Vector and Event Counter

99

threshold=0.05 threshold=0.1 threshold=1

0.1

0.05

0

gzip

vpr

gcc

mcf

parser

perlbmk

gap

vortex

bzip2

twolf

avg

-0.05

-0.1

Fig. 7. Normalized instrumentation overhead in energy consumption. The difference between the energy consumption of the instrumented and uninstrumented benchmark divided by the energy consumption of the latter.

because of different code placement. Overhead in execution time when different thresholds are used follow the same trend. Instrumentation overhead for power measurement of a single simpoint is even lower because only one or two of the final infrequent basic blocks are instrumented. 5.2 Total Energy Consumption Estimation We investigate both BBV and EV as the fingerprint of intervals in phase classification. A maximum number of clusters, 30, is used to find the best clustering in both cases. Simpoints are measured and the whole program energy consumption is estimated as Eest =

k 

Ei ×Wi

i=1

Ei is the measured energy consumption of the ith simpoint, Wi is its weight, and k is the number of phases. Although intervals have variable sizes, we estimate the total energy consumption using the weight based on the number of intervals in each phase. For BBV-based phase classification, we use three percentage values 0.1%, 1%, and 5% to get the threshold for infrequent basic blocks. The measured energy consumption of simpoints are used to estimate the whole program energy consumption. The error rate is the lowest when threshold is 1% due to the trade-off between uniform interval size and instrumentation overhead. Then we use 1%, 0.1% and 0.05% as threshold in EVbased phase classification. Energy consumption of a measured benchmark or simpoint is calculated as:  E = U × (I ×t) where U is the voltage of the measured CPU power cable, I is the measured current on the CPU power cable, t is the time resolution of the power data points. The sum is over all of the data points for one benchmark or simpoint. Energy estimation error rate is calculated as: error =

|energy estimated −energy measured| energy measured

Execution time estimation is similar to energy estimation.

100

C. Hu, D.A. Jim´enez, and U. Kremer

Figure 8 shows the error rates of the infrequent basic block-based phase classification method using different program execution fingerprints. The error reported is that of the estimate using the threshold that delivered the minimum overall error for each method: 1% for BBVs, and 0.1% for EVs. The figure shows that EV performs better than BBV for almost all of the benchmarks. EV improves the estimation accuracy on average by 35%. One possible reason for the higher error rate of EV for some benchmarks is that we only record conditional edges taken during program execution. Some benchmarks have many unconditional edges, such as jmp, so it is possible that some information is lost in EV, although we significantly reduce the edge vector size. For example, method sort basket of mcf is called 14683023 times and many of its edges are non-conditional edges. We can improve the phase classification accuracy through recording executiion frequency of all edges, at the cost of largger edge vectors and slower phase classification. All of the following analysis and evaluation are for the experimental results of EV-based phase classification if there is no specification. 5.3 Time-Dependent Power Behavior Similarity As mentioned in Section 4.3, we use the distance between the FFT results of their power curves to evaluate the similarity of two intervals in terms of power behavior. We use 4096 points in the Fast Fourier Transform. The maximum number of data points for a curve is 10,000 when the oscilloscope is in trigger mode. If the measured data points for the curve of an intervals is less than 4096, the curve is repeated to reach the number of frequencies. Figure 9 (a) shows the measured CPU current curves of two intervals from the same identified phase, while (b) shows that of two intervals from two different phases. Distance between the FFT values is included to show the relation between timedependent power behavior similarity and FFT distance. In Figure 9 (a), the upper curve uses the left y axis, while the other one use the right y axis, to avoid overlapping curves. 60

error rate(%)

50

BBV

EV

40 30 20 10

vp r gc c m c pa f rs pe er rlb m k ga p vo rt e x bz ip 2 tw ol f av g

gz ip

0

Fig. 8. Error rates of energy consumption estimation using different fingerprints

Combining Edge Vector and Event Counter

101

5 interval 712 interval 1744

CPU Current (A)

4.5

4

3.5

3

2.5

2 0

1

2

3

4

5

6

7

8

9

10 11 time (msec)

12

13

14

15

16

17

18

19

(a) Power curves of intervals from the same phase(distance=5.4). 5 interval 712 interval 160

CPU Current (A)

4.5

4

3.5

3

2.5

2 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

time (msec)

(b) Power curves of intervals from different phases(distance=55.1). Fig. 9. Similarity between measured CPU current of intervals

RMS of FFT

30 24

bbv ev ev+ipc

18 12 6 0

gzip

vpr

gcc

mcf

parser

perlbmk

b

h

gap

vortex

bzip2

twolf

avg

k

Fig. 10. Root Mean Squared error of the FFT calculated based on RMS of FFT and the weight of each phase

The second column of each group in Figure 10 is the weighted F F TRMS for each benchmark when EV is used for phase classification. We measure the IPC using performance counters for each interval and do phase classification based on IPC to refine the EV-based phase classification. The third column in each group in Figure 10 is the weighted F F TRMS for each benchmark when EV+IPC is used for phase classification. The similarity among the intervals is improved by 22% over using BBVs. Compared to the FFT distance between an interval and another interval from a different phase, the distance inside a phase is much smaller. This shows that the combination of EV and IPC enables us to classify intervals into phases in which the intervals have similar power behavior. Thus the power behavior of the whole program can be characterized by the measured behavior of the simpoints.

C. Hu, D.A. Jim´enez, and U. Kremer

RMS of interval length

102

0.015

0.04

0.03

0.02

0.05

0.01

bbv-0.1 ev-0.1 bbv-1 ev-1 bbv-5 ev-5

0.005 0

gzip

vpr

gcc

mcf

parser

perlbmk

gap

vortex

bzip2

twolf

avg

Fig. 11. Weighted average of the RMS error of interval length in the same phase

5.4 Interval Length Variance Figure 11 shows the weighted average of interval length variance of each phase for each benchmark when BBV and EV is used in phase classification respectively. A smaller number means the intervals of the same phase have very close interval size. Again it shows that EV is better for our infrastructure because, on average, it causes much lower interval length variance than BBV no matter which threshold is used. Again One possible reason for the higher RMS of EV for some benchmarks is that we only record conditional edges taken during program execution, which results in information loss. Although the possible reason is the same as in Section 5.2, the higher error rate or RMS happens to different benchmarks in these two set of experiments. The reason is that total power consumption is an average metric, if the energy consumption of the selected representatiove interval is close to the average energy consumption of all of the intervals in the same phase, the error rate should be low. While RMS of interval length is used to evaluate the similarity among intervals in the same phase, low error rate in total energy consumption does not mean this RMS value is small. This also applies to time-dependent power behavior and is also one of the motivation to use FFT to evaluate the time-dependent power behavior similarity among intervals in the same phase.

6 Conclusion This paper introduced our infrastructure for efficient program power behavior characterization and evaluation. We presented a new phase classification method based on edge vectors combined with event counters. We described the physical measurement setup for precise power measurement. By demarcating intervals using infrequently executed basic blocks, we find intervals with variable lengths and negligible instrumentation overhead for physical measurement of simpoints. Through experiments on a real system, we demonstrated that our new phase classification method can find representative intervals for energy consumption with an accuracy superior to using basic block vectors. More importantly, we demonstrated the ability of our infrastructure to characterize the fine-grained time-dependent power behavior of each phase in the program using a single representative interval per phase. The ability of instrumenting programs on various levels, identifying phases, and obtaining detailed power behavior of program execution slices makes this infrastructure useful in power behavior characterization and optimization evaluation.

Combining Edge Vector and Event Counter

103

References 1. Yun, H.S., Kim, J.: Power-aware modulo scheduling for high-performance VLIW. In: International Symposium on Low Power Electronics and Design (ISLPED 2001), Huntington Beach, CA (August 2001) 2. Toburen, M., Conte, T., Reilly, M.: Instruction scheduling for low power dissipation in high performance microprocessors. In: Power Driven Microarchitecture Workshop, Barcelona, Spain (June 1998) 3. Su, C.L., Tsui, C.Y., Despain, A.: Low power architecture and compilation techniques for high-performance processors. In: IEEE COMPCON, San Francisco, CA, February 1994, pp. 489–498 (1994) 4. Hazelwood, K., Brooks, D.: Eliminating voltage emergencies via microarchitectural voltage control feedback and dynamic optimization. In: International Symposium on Low-Power Electronics and Design, Newport Beach, CA (August 2004) 5. Sherwood, T., Perelman, E., Hamerly, G., Calder, B.: Automatically characterizing large scale program behavior. In: Proceedings of the 10th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2002) (2002) 6. Manne, S., Klauser, A., Grunwald, D.: Pipeline gating: speculation control for energy reduction. In: Proceedings of the 25th Annual International Symposium on Computer Architecture (ISCA 1998), pp. 132–141 (1998) 7. Hsu, C.H., Kremer, U.: The design, implementation, and evaluation of a compiler algorithm for cpu energy reduction. In: Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation (PLDI 2003), pp. 38–48 (2003) 8. Sherwood, T., Perelman, E., Calder, B.: Basic block distribution analysis to find periodic behavior and simulation points in applications. In: Proceedings of the 2001 International Conference on Parallel Architectures and Compilation Techniques (PACT 2001), pp. 3–14 (2001) 9. Shen, X., Zhong, Y., Ding, C.: Locality phase prediction. In: Proceedings of the 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2004), pp. 165–176 (2004) 10. Iyer, A., Marculescu, D.: Power aware microarchitecture resource scaling. In: Proceedings of the Conference on Design, Automation and Test in Europe (DATE 2001), pp. 190–196 (2001) 11. Lau, J., Perelman, E., Hamerly, G., Sherwood, T., Calder, B.: Motivation for variable length intervals and hierarchical phase behaviour. In: The Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2005), pp. 135– 146 (2005) 12. Chi, E., Salem, A.M., Bahar, R.I.: Combining software and hardware monitoring for improved power and performance tuning. In: Proceedings of the Seventh Annual Workshop on Interaction between Compilers and Computer Architectures (INTERACT 2003) (2003) 13. Duesterwald, E., Cascaval, C., Dwarkadas, S.: Characterizing and predicting program behavior and its variability. In:Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques (PACT 2003), p. 220 (2003) 14. Srinivasan, R., Cook, J., Cooper, S.: Fast, accurate microarchitecture simulation using statistical phase detection. In: Proceedings of The 2005 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS 2005) (2005) 15. Isci, C., Martonosi, M.: Identifying program power phase behavior using power vectors. In: Proceedings of the IEEE International Workshop on Workload Characterization (WWC-6) (2003)

104

C. Hu, D.A. Jim´enez, and U. Kremer

16. Flinn, J., Satyanarayanan, M.: Powerscope: A tool for profiling the energy usage of mobile applications. In: Proceedings of the Second IEEE Workshop on Mobile Computer Systems and Applications, p. 2 (1999) 17. Isci, C., Martonosi, M.: Runtime power monitoring in high-end processors: Methodology and empirical data. In: Proceedings of the 36th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO 2003), p. 93 (2003) 18. Hu, C., Jim´enez, D.A., Kremer, U.: Toward an evaluation infrastructure for power and energy optimizations. In: 19th International Parallel and Distributed Processing Symposium (IPDPS 2005, Workshop 11), CD-ROM / Abstracts Proceedings (April 2005) 19. Isci, C., Martonosi, M.: Phase characterization for power: Evaluating control-flow-based and event-counter-based techniques. In: 12th International Symposium on High-Performance Computer Architecture (HPCA-12) (Feburary 2006) 20. Hu, C., McCabe, J., Jim´enez, D.A., Kremer, U.: The camino compiler infrastructure. SIGARCH Comput. Archit. News 33(5), 3–8 (2005)

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors Woojin Choi1, Seok-Jun Park2, and Michel Dubois1 1

Department of Electrical Engineering, University of Southern California {woojinch, dubois}@usc.edu 2 System LSI Division, Samsung Electronics Corporation [email protected]

Abstract. The role of the instruction scheduler is to supply instructions to functional units in a timely manner so as to avoid data and structural hazards. Current schedulers are based on the broadcast of result register numbers to all instructions waiting in the issue queue and on a global arbiter to select ready instructions from that queue. This approach called broadcast scheduling does not scale well due to its complexity. To reduce the complexity of the broadcast schedulers, data-flow pre-scheduling has been proposed. The basic idea is to predict the issue time of instructions based on the availability of operands and then time them down until they are ready to issue. However, resource conflicts for issue slots and functional units delay the issue time of conflicted instructions, and cause a large amount of replays. We propose to add instruction pre-selection to dataflow pre-schedulers for accurate instruction pre-scheduling. Our pre-scheduler keeps track of the allocation status of resources so that resource conflicts are eliminated. Pre-scheduled instructions are stored in an issue buffer until their issue delay elapses and then issue automatically. Our analysis shows that preschedulers with pre-selection result in performance improvements of 60% over current broadcast schedulers in pipeline designs where the scheduler is the bottleneck. In future technologies we expect this result to hold as logic intensive designs with short wires will be preferable to designs with long wire delays.

1 Introduction The performance of microprocessors can be improved by issuing more instructions per cycle and/or raising the clock frequency. However, issuing more instructions per cycle results in very complex hardware so that, in the past decade, clock frequency has been the major source of sustained performance improvements in single-chip microprocessors. Improvements in process technology yield smaller and faster gates but expose wire delays, which do not scale with technology [1, 2]. Deeper pipelines also reduce the logic delay per stage and raise the clock rate. However, some functions cannot be pipelined because this can lead to severe IPC (Instruction Per Clock) degradation. Instruction scheduling is currently one of these pipeline bottlenecks [3]. Dynamically scheduled microprocessors execute instructions out of program order to expose more instruction level parallelism (ILP) to the functional units. The role of the instruction scheduler is to supply as many independent instructions to the functional units as possible in each clock. Current schedulers broadcast the result register number of each instruction scheduled for execution to all instructions pending in P. Stenström (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 107–127, 2009. © Springer-Verlag Berlin Heidelberg 2009

108

W. Choi, S.-J. Park, and M. Dubois

the issue queue and this register number must be matched with the operand tags of all instructions in the issue queue (wake-up broadcast). Ready instructions are then selected by an arbiter which is aware of resource conflicts (selection). This approach called broadcast scheduling does not scale well due to its complexity. Observing that data dependencies and operation latencies are known well before the scheduling stage, we can pre-schedule instructions, i.e., submit them to the scheduler in an order that simplifies the scheduler or even eliminates it. One possibility is to submit instructions to the issue queue when they are close to their issue time, thus avoiding cluttering the issue queue and extending its effective size [4, 5, 6, 7]. This can be done by predicting the issue time of instructions based on operand availability (data-flow pre-scheduling) and storing the instructions in a pre-scheduling array so that they reach the issue queue based on the predicted issue time. Another possibility is to eliminate the broadcast altogether and issue the instructions directly from the pre-scheduling array [8, 9, 10]. The problem with data-flow pre-scheduling is that instructions prescheduled based on operand availability only may conflict for resources such as issue slots or functional units. When a conflict occurs, the issue of one of the conflicted instructions (i.e., the younger instruction) is delayed. This delay causes the dependent instructions to be issued before their operands are ready. These instructions issued prematurely must be replayed, causing large overheads because of a cascade effect. To reduce the impact of resource conflicts, we propose to improve the accuracy of data-flow pre-schedulers by adding instruction pre-selection in the pre-scheduler. Along with operand availability, the pre-scheduler manages the reservation status of resources to avoid structural hazards. Resource reservations are tracked by resource availability bit vectors which precisely represent the reservation information of each resource in future cycles. Resource conflicts are predicted and eliminated early to pre-schedule the following instructions accordingly. Therefore, we can maintain a more accurate preschedule. After predicting the issue delay, i.e., the cycle when an instruction can be issued without data dependency violation or resource conflicts, the pre-scheduler inserts each instruction with its issue delay in an issue buffer. The issue delays of all instructions in the buffer are clocked down and, when the issue delay of an instruction reaches zero, the instruction is automatically sent for execution without any additional check. A basic problem of pre-schedulers with pre-selection is the finite length of their resource availability bit vectors. Longer resource availability bit vectors lead to more complex logic and lower clock rates. We explore various solutions to this problem. We compare the IPC of various pre-schedulers with pre-selection to the IPC of a classical broadcast scheduler by simulating SPEC2000 SimPoints on an upgraded SimpleScalar simulator. Additionally we have designed the logic for the various scheduling schemes to estimate their delays in a 0.13um 1.8V technology. Our analysis shows that pre-schedulers with pre-selection results in average performance improvements of 60% over broadcast schedulers when both the IPC and the attainable frequency are considered in designs where the scheduler is the pipeline bottleneck. The rest of this paper is organized as follows. Section 2 gives some background information on broadcast scheduling and data-flow pre-scheduling. Section 3 introduces the concept and shows the implementation of instruction pre-selection, and combines it with data-flow pre-scheduling. Section 4 explains the evaluation methodology. Section 5 explores solutions to the problem of finite length resource availability bit vectors, and Section 6 presents an overall performance comparison of all scheduling

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

109

schemes taking into account both IPC and clock cycle. Section 7 reviews related work, and Section 8 is the conclusion.

2 Speculative Instruction Scheduling The baseline pipeline is shown in Figure 1. After the Fetch and Decode stages, instructions are renamed and are assigned physical resources, such as entries in the issue queue, load/store queue and reorder buffer in the Allocate stage. Instructions stay in the issue queue until they can be issued to a functional unit with the help of the instruction scheduler. After leaving the scheduler, instructions read their register operands, execute, update their destination register, and then wait in the reorder buffer until they commit in process order. The branch predictor at the front of the pipeline speculates the program counter of the next instruction. On a mispredicted branch the pipeline is flushed and restarted. Typically, instructions are scheduled speculatively. If instructions were issued after their parent instructions have produced their results, instructions in a dependency chain could not be scheduled in consecutive cycles [11]. The instruction scheduler speculatively schedules an instruction after its parent instructions so that the operation latencies of the parent instructions are met. This works well for all instructions with fixed latencies. However the latency of LOADs is variable because of cache misses and memory disambiguation delays. The latency of LOADs can be predicted by a cache hit/miss predictor [12]. When a LOAD is predicted to hit but actually misses, all its dependent instructions which have already issued must replay. A typical replay-queue based recovery mechanism [13] is shown in Figure 1. Every instruction leaving the issue queue is inserted into the replay queue where the availability of its register operands is verified by a checker just before its execution. If one of its operands is not available, a scheduling miss occurs and miss-scheduled instructions are reissued in dependency order. Instructions may replay several times until their operands are available. Independently from predicting the latency of LOADs, predicting the data dependency between STOREs and LOADs is critical for scheduling LOADs. Unlike data dependencies between registers, data dependencies between STOREs and LOADs are not always known at schedule time because they can only be detected after the calculation

FETCH DECODE RENAME RENAME

ALLOCATE

SCHEDULE

DISPATCH

DISPATCH

REG READ

REG READ

EX

WRITE COMMIT BACK

(a) Store Set Predictor Branch Predictor

Instruction Cache

Hit/Miss Predictor

Schedule Fetch/ Decode

Register Rename

Allocate

Issue Queue

Dispatch

Register File

Select Logic

ALU

Replay Queue

Verification Signal Checker

(b)

Fig. 1. Baseline Processor. (a) Processor Pipeline. (b) Processor Architecture.

110

W. Choi, S.-J. Park, and M. Dubois

of memory addresses. However, if the issue of a LOAD must wait until the addresses of all prior STOREs are known, a large amount of ILP will be squandered. Therefore, memory dependencies are typically predicted and mispredictions are recovered from. A good predictor is the store set predictor [14]. The store set predictor consists of the store set identifier table (SSIT) and the last fetched store table (LFST). LFST holds the information of the latest STORE in each store set. Each LOAD indexes LFST with the store set id obtained from SSIT and, if a dependency with a prior STORE is predicted, the issue of the LOAD is delayed until that STORE is issued. More details on the design of store set predictors can be found in [14]. On a memory dependency violation, the scheduling miss triggers a recovery mechanism to repair the schedule [11, 15, 16, 17]. In this paper, the violating LOAD and all subsequent instructions are squashed and reinserted into the Allocate stage from the reorder buffer in program order [11]. 2.1 Broadcast Scheduling Instruction schedulers in current microprocessors are based on broadcast. Figure 2 illustrates a speculative broadcast scheduler. Whenever an instruction is issued, its result register tag (tagDest) is broadcast (after a delay equal to the predicted operation latency) to all the instructions in the issue queue in order to notify dependent instructions. Wake-up logic matches the register tags on the wake-up buses with all the operand register tags (tagL, tagR) in the issue queue. If there is a match, the ready bit of the matching operand (readyL, readyR) is set. When all the operands of an instruction are ready, the instruction is ready and sends a request signal to the select logic. The select logic acts as an arbiter to choose the next instructions for execution and returns grant signals to the selected instructions. The select logic picks a number of ready instructions up to the issue width while avoiding conflicts for functional units. Increasing the issue queue size exposes more independent instructions that can be issued at the same time leading to higher IPC [18]. However, as the issue queue size grows, the scheduling delay also increases [19] and does not scale well with technology [2]. Unfortunately, wake-up and selection must be done atomically in broadcast scheduling [3] because, if the wake-up and select logic is pipelined, parent instruction and child instruction cannot be issued in consecutive cycles, severely degrading IPC. 2.2 Data-Flow Pre-scheduling To address the scaling problem plaguing broadcast schedulers, data-flow pre-schedulers [4, 5, 6, 7, 8, 9, 10] exploit the information on data dependencies and instruction Wake-up Bus tag.IW

Select Logic

tag.1

OR

=

=

OR

readyL

tagL

tagR

readyR

tagDest

Grant Request

Grant

Wake-up Bus Delay

readyL

tagL

tagR

readyR

tagDest

Fig. 2. Broadcast Instruction Scheduler

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors Register Timing Table Delay Value r1

Cycle #n Operation Latency

Issue Delay +

t1 MAX

t2

r2

111

r3

Prescheduling Array

...

6

5

4

3

2

i3

1

0

i2 i0

Issue Delay = MAX (t1, t2)

i1

...

(a) Program Order

Cycle #(n+1)

i0: i1: i2: i3:

ADD MUL ADD MUL

r1 r2 r3 r4

r9, r9, r1, r1,

r10 r10 r9 r2

Scheduling Group 1

i4: i5: i6: i7:

ADD MUL MUL ADD

r5 r6 r7 r8

r1, r4, r1, r3,

r1 r10 r9 r5

Scheduling Group 2

(b)

Issue Delay ...

6

5

4

3

i5

2

1

0

i3 i7 i2 i4 i6

(c)

Fig. 3. Data-flow Instruction Pre-scheduler. (a) Issue Delay Calculation. (b) Example Instruction Sequence. (c) Pre-scheduling Array.

latencies known before the scheduling stage to reorder instructions in an execution order predicted by the time of their operand availability. In data-flow pre-schedulers a register timing table (RTT) indexed with a source register number keeps the delay value of each register indicating the number of cycles until the register value becomes available. The delay values in the RTT are decremented in every cycle. A register becomes ready when its delay value reaches zero. The issue delay of an instruction, i.e., the number of cycles until an instruction becomes ready to issue, is the largest delay value of its operands. Issue delay calculation is illustrated in Figure 3(a). If the delay values of the operands of an instruction are t1 and t2, its issue delay is MAX(t1, t2). The sum of the predicted issue delay and the operation latency of an instruction becomes the delay value of its result register. Each instruction is inserted into a pre-scheduling array according to its issue delay. Instructions in the pre-scheduling array move to the right in every cycle and the instructions in the rightmost column are issued in each cycle. Figure 3(b) shows an instruction sequence and Figure 3(c) shows the states of the pre-scheduling array for this instruction sequence. We assume that all the registers are initially ready, that the operation latency of ADDs is one cycle and that the operation latency of MULs is three cycles. The issue delay of each instruction is computed from the delay values in program order. At first, instructions i0 and i1 are placed in the rightmost column of the pre-scheduling array with zero-issue delay because all operands of instructions i0 and i1 are available and the delay values of register r1 and r2 are updated, using operation latencies. Instruction i2 is dependent on instruction i0 and can be issued after register r1 is ready. The maximum delay value is one, and instruction i2 is pre-scheduled with one issue delay. Instruction i3 is dependent on both instructions i0 and i1, and the maximum delay value is three. Therefore, i3 is placed in the third column to the left of the rightmost column. In the next cycle, the instructions in the rightmost column (i0 and i1) are issued and the other instructions in the pre-scheduling array move to the right. The instructions from the second scheduling group are reordered with the same procedure. Data-flow pre-scheduling is weak on dynamic events, such as cache misses, LOAD/ STORE dependency violations, and resource conflicts [5]. The latency of LOADs is variable depending on memory behavior. A cache hit/miss predictor [12]

112

W. Choi, S.-J. Park, and M. Dubois

can forecast the latency of a LOAD. When the prediction is wrong, instructions are replayed through the replay queue. Data-flow pre-schedulers must also deal with the violations of LOAD/ STORE dependencies. A modified store set predictor [4] is used to predict the issue delay of LOADs. On a LOAD/STORE dependency violation, the violating LOAD and all subsequent instructions are squashed. Resource conflicts occur whenever the number of ready instructions exceeds the number of available resources, such as issue slots and functional units. These conflicts must be sorted out dynamically. In the Cyclone scheduler [9] for example, pre-scheduled instructions move through a countdown queue for the first half of their issue delay, and then move to a main queue for the second half of their issue delay. However, if no entry is available in the main queue when the instruction tries to switch queues, an issue slot conflict occurs and the instruction has to move into a subsequent issue slot in the main queue. It is ultimately issued later than the cycle at which it was pre-scheduled and its dependent instructions are pre-scheduled incorrectly because they are pre-scheduled according to the information in the RTT. If they are issued before all of their operands are ready they must be replayed. Dealing with resource conflicts is an important problem in data-flow preschedulers [9, 10]. To prevent resource conflicts, we propose to add the reservation information of resources in the pre-schedule. We call this phase “pre-selection.”

3 Instruction Pre-selection In this section, we describe the concept and show the implementation of pre-selection to reduce the number of replays caused by resource conflicts. When the latency of a LOAD is mispredicted, we use the modified replay queuebased recovery mechanism shown in Figure 4. Instead of immediately being reissued, miss-scheduled instructions are reinserted into the Allocate stage. By doing so, the replayed instructions are pre-scheduled again to repair the pre-schedule even though the replay loop is longer. 3.1 Counter-Based Pre-scheduler We first describe a pre-scheduler with pre-selection in which resource availability is maintained by counters. The counter-based pre-scheduler contains three tables – register timing table (RTT), issue slot allocation table (ISAT), and functional unit allocation table (FUAT). RTT is the same table as the one used in data-flow pre-scheduling [9]. ISAT and FUAT indicate when and how many resources of each type are available using counters. Each counter in these tables keeps track of the number of available resources of a given type in a given cycle. The maximum value of an ISAT counter is the issue width. An ISAT counter of zero means that all the issue slots for the corresponding cycle are already reserved. Each row in FUAT is indexed with the instruction opcode and corresponds to a type of functional unit. If the processor has two multiply units, the maximum value of a MUL counter is two. The MUL row in FUAT keeps track of the number of available multiply units in each cycle. In the pre-scheduler an instruction accesses RTT to obtain the delay values of its operands, t1 and t2. Then the minimum delay value n until all operands of the instruction are available is computed (MAX). ISAT and FUAT are then accessed with n to find the next cycle closest to n when both types of resources are available, and this cycle becomes the issue delay of the instruction. The delay value of the result register

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

Indexed with register tag

Register Timing Table

t

1

t

2

MAX Delay Value r1 r2 r3

UUU

Indexed with opcode

Functional Unit Allocation ALU Table

Instruction Cache

Fetch/ Decode

Register Rename

Allocate

n = MAX(t , t ) 1

#0

MUL

Issue Slot Allocation Issue Table Slot

Pre-schedule

113

#0

2

...

#n #(n+1) ...

...

...

...

...

...

#n #(n+1) ...

...

...

Issue Buffer

Dispatch

Register File ALU

Replay Queue

Verification Signal Checker

Fig. 4. Counter-based Instruction Pre-scheduler

is updated with the sum of issue delay and operation latency, and the appropriate counters in ISAT and FUAT are decremented. The pipeline for the counter-based pre-scheduler with pre-selection is shown in Figure 4. It uses an issue buffer instead of a pre-scheduling array. The issue buffer is similar to the issue queue in broadcast schedulers and stores the instructions not yet issued with their issue delay. The issue delay of each instruction in the issue buffer is counted down in every cycle. When its issue delay reaches zero, an instruction issues automatically. Because each counter in the allocation tables maintains the number of available resources at each cycle and instructions are pre-scheduled based on this information, resource conflicts never occur and instructions are correctly issued without the help of conventional select logic. If all operands and resources of an instruction are available at the Pre-schedule stage, its issue delay is zero and the instruction is immediately issued, bypassing the issue buffer. Counters in the ISAT and FUAT are shifted to the left in every clock and the maximum count of every type of resources is inserted into the rightmost column of the tables in every cycle. The counter-based pre-scheduler keeps an accurate record of the reservation information of each type of resources. However, it is very complex to implement in practice because of counter conflicts occurring when several instructions from the same prescheduling group, i.e., the group of instructions entering the pre-scheduler in the same cycle, try to reserve the same type of resource for the same future cycle. Because of this complexity, we have not attempted to evaluate its logic design. To simplify the pre-selection logic, we propose to replace the counters with resource availability bit vectors to keep track of the reservation information of each individual resource. 3.2 Resource Availability Bit Vectors A resource availability bit vector is assigned to each resource (either issue slot or functional unit). Each bit indicates the reservation status of the resource in a future cycle. Availability bit 1 means the resource is available, and availability bit 0 means the resource is already reserved in a given cycle. FUAT and ISAT are now arrays of bits instead of arrays of counters. FUAT has as many functional unit availability bit vectors as the total number of functional units.

114

W. Choi, S.-J. Park, and M. Dubois

Figure 5 shows a FUAT for an execution unit with 4 add and 2 multiply pipelines. In this example, ALU#3 is available in every cycle except for the first and third cycles (i.e., previous instructions have already reserved ALU#3 for these cycles). Within the same class of functional units, functional unit availability bit vectors are assigned to instructions in a round-robin manner. ISAT has as many issue slot availability bit vectors as the issue width. Issue slot availability bit vectors are also assigned round-robin. All resource availability bit vectors are shifted to the left with 1-fill in every clock. Performance may be affected negatively when counters are replaced by bit vectors in FUAT and ISAT. With counters an instruction is allocated the earliest possible available cycle for any one of the resources of same type. By contrast with bit vectors an instruction is allocated the earliest possible available cycle for the particular resource it is assigned in a round robin manner. The size of the bit vectors is a critical design parameter. With longer bit vectors, we can keep track of resource usage farther ahead in the future, but the logic design of the scheduler is more complex. In Section 5, we evaluate the impact of bit vector size on performance. In the following sections, we set the length of bit vectors to eight unless otherwise specified. 3.2.1 Issue Delay Calculation Figure 6(a) illustrates the calculation of issue delay. The pre-scheduler is made of three pipeline stages, PRESCH1, PRESCH2 and PRESCH3 (pre-selection). Because RTT is indexed with physical register numbers, RTT accesses are pipelined in stages PRESCH1 and PRESCH2 in a way similar to accesses to a physical register file. When an instruction enters the pre-scheduler, it starts accessing RTT in the PRESCH1 stage. It then accesses the other tables in the PRESCH2 stage to obtain the bit vectors of its resources. The issue delay for the instruction is calculated in the PRESCH3 stage. Each register delay value is converted into an operand availability bit vector by number-to-bit (NtoB) logic. The operand availability bit vector consists of a series of zeros followed by a series of ones, and the number of zeros in the operand availability bit vector indicates the number of cycles needed until the operand becomes ready1. In parallel with number-to-bit conversions, each resource availability bit vector is shifted left by one to reflect elapsed cycle.

Functional Unit Allocation Table

Assigned roundrobin

#0

#1

#2

#3

#4

...

ALU#0

0

0

1

1

1

...

ALU#1

0

0

1

1

1

...

ALU#2

1

1

0

1

1

...

ALU#3

1

0

1

0

1

...

MUL#0

0

0

1

1

1

...

MUL#1

0

0

1

1

1

...

Left-shift with 1-fill

Fig. 5. An Example of Functional Unit Allocation Table with Resource Availability Bit Vectors 1

If the RTT or the issue buffer contained bit vectors instead of numbers (i.e., delay value or issue delay), we could eliminate the logic delay introduced by NtoB and BtoN logics. However, the representation of delay values by bit vectors is very inefficient and is wasteful of transistors and power in the RTT or in the issue buffer. After evaluating their logic delays, we decided to adopt NtoB and BtoN transformations in the PRESCH3 stage.

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors PRESCH1 & PRESCH2

PRESCH3

Register Timing Table

PRESCH1 & PRESCH2

NtoB Functional Unit Allocation Table

shift-1

Issue Slot Allocation Table

shift-1

PRESCH3 BtoN

Register Timing Table

NtoB

AND

(a)

BtoN

Issue Delay Bit Vector

Issue Delay

115

SHIFT

NtoB NtoB

Functional Unit Allocation Table

shift-1

Issue Slot Allocation Table

shift-1

AND

BtoN

XOR KILL

XOR

(b)

Fig. 6. (a) Issue Delay Calculation. (b) Table Update.

After number-to-bit conversions, all bit vectors are ANDed and the result is the issue delay bit vector. The number of zeros to the left of the first 1 in the issue delay bit vector is the issue delay for the instruction. For example, assume that an instruction reads operand delay values 2 and 3, and is assigned issue slot availability bit vector 01001011 and functional unit availability bit vector 00111111. In the next cycle, the instruction moves into the PRESCH3 stage and the delay values are converted to 01111111 and 0011111 by NtoB logic, and resource availability bit vectors are shifted left by 1. The result of ANDing the bit vectors is 00010111 and the instruction can be issued three cycles later free of data and structural hazards. Finally, the issue delay bit vector is converted into issue delay by bit-to-number (BtoN) logic. BtoN logic is a priority encoder and its output is the issue delay kept with the instruction in the issue buffer. After BtoN logic, instructions with zero issue delay value bypass the issue buffer and are immediately issued. Instructions with nonzero issue delay are inserted in the issue buffer and the issue delay of each instruction is counted down in every cycle. When their issue delay reaches zero, instructions issue automatically. 3.2.2 Table Updates After predicting the issue delay of an instruction, allocation tables are updated to maintain the latest resource reservation status. The logic for table updates is shown in Figure 6(b). To update the delay value of the result register, the issue delay bit vector is shifted to the right by the operation latency of the pre-selected instruction, and a number of zeros equal to the operation latency are inserted from the leftmost bit. This task is accomplished with SHIFT logic. Then, the shifted issue delay bit vector is converted into delay value by BtoN logic. The RTT is then updated with the resulting delay value. We note that the maximum delay value in RTT must be larger than the length of bit vectors. Because the delay value of the result register is the sum of issue delay and operation latency, an RTT entry must be able to express the sum of the largest issue delay and operation latency. At the same time the RTT is updated, all bits of the issue delay bit vector to the right of the first bit set to 1 are cleared by KILL logic, and the resulting bit vector is XORed with each original resource availability bit vector to reserve the corresponding cycle of the resource. 3.3 Multiple Issue Processors One problem overlooked so far in the description of our pre-scheduler with pre-selection is how to handle dependent instructions in the same pre-scheduling group. The

116

W. Choi, S.-J. Park, and M. Dubois

issue delay of a child instruction can be calculated only after the issue delay of its parent instruction is known. Pre-scheduling dependent instructions in the same cycle would enlarge the delay of the PRESCH3 stage. In our implementation, we preschedule only independent instructions in the same clock. If parent and child instructions enter the pre-scheduler in the same cycle, the child and following instructions are delayed by one cycle. After predicting the delay value of the parent’s result register, the value is forwarded to the child instruction via bypassing logic. We do not expect a significant performance impact from this choice. The goal is to minimize the issue interval between parent and child instructions, not to minimize the interval between the pre-selections of parent and of child instructions. Even though parent and child instructions are not pre-selected in the same cycle, their execution in back-toback cycles is possible and happens most of the time. In multiple issue processors it is also very complex to simultaneously assign two instructions to the same functional unit availability vector. Thus, in all implementations of instruction pre-selection we consider, whenever the number of instructions in the same pre-scheduling group which need the same type of functional unit exceeds the number of functional units of that type, reservations of the functional units are processed in process order up to the number of functional units of that type. The remaining instructions are stalled at the PRESCH1 stage. 3.4 Bit Vector Overflow One of the critical obstacles of our pre-scheduler with pre-selection is bit vector overflow. The length of the bit vectors is one of the important design parameters affecting the IPC, clock frequency, and power consumption. Bit vector overflow occurs when it is impossible to correctly predict the issue delay due to the limited length of the bit vectors. The delay value of an operand may exceed the length of the bit vectors. In this case, we cannot update the resource availability bit vectors for the cycle when the operand is available. An instruction whose delay value of an operand causes a bit vector overflow is said to be operand-overflowed and cannot be pre-selected until its delay value becomes less than the length of the bit vector. Bit vector overflow can be detected at the PRESCH2 stage. In Section 5 we explore solutions to the problem of bit vector overflow. But before we can do this, we first describe our evaluation methodology.

4 Evaluation Methodology In this section, we present the evaluation method used to derive the results in the remainder of this paper. 4.1 Processor Models Our simulator used for performance evaluation is derived from the SimpleScalar/pisa version 3.0 tool set [21], a suite of functional and timing simulation tools. We expanded SimpleScalar’s sim-outorder to implement the 13-stage dynamically scheduled superscalar processor shown in Figure 1. The baseline machine separates issue queue and reorder buffer, and models the speculative broadcast scheduler described in Section 2. For the scheduling of LOADs, a cache hit/miss predictor [12] and a store

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

117

Table 1. Machine Configuration Parameter Machine Buffers Machine Width Branch Prediction (number of entries) Memory Prediction (number of entries) Functional Units (operation latency) Memory Systems (line size, latency)

Configuration 128-entry ROB, 64-entry IQ, 64-entry LSQ 4-wide fetch/decode/issue/commit Combined bimodal (8K) /gshare (8K) with selector (8K), 32-entry RAS, 4-way BTB (2K), at least 13-cycles taken for misprediction recovery Cache hit/miss predictor: bimodal predictor (8K), STORE set predictor: SSIT (8K), LFST (256) 4 INT ALU (1), 2 INT MULT/DIV (3/20), 2 FP ALU (2), 2 FP MULT/DIV (4/24), 2-read ports and 1-write port 64KB 4way IL1 (64B, 2), 64KB 4way DL1 (64B, 2), 2MB 8way unified L2 (128B, 15), main memory (150)

set predictor [14] are used. Store set prediction follows the store set assignment rule from [14] and uses the cyclic clear algorithm on every two million cycles. A replay queue-based recovery mechanism [13] is used on LOAD latency misprediction. If a memory dependency violation occurs, the violating LOAD and all subsequent instructions are squashed and re-fetched from the reorder buffer [11]. The primary machine parameters are shown in Table 1. The machines with pre-scheduler and pre-selection are derived from the baseline machine and incorporate the design alternatives described in Section 5. 4.2 Benchmark Programs A subset of SPEC2000 benchmark with reference input data sets is used for all simulation results [22]. Each benchmark was compiled with the gcc-pisa compiler using -O3 optimization. The simulations were run for 100-million instructions for each benchmark after fast-forwarding to the single SimPoint of each benchmark [23]. Table 2 shows the benchmarks, the number of instructions fast-forwarded, and the baseline IPC. Table 2. Benchmarks Benchmarks 164.gzip 176.gcc 181.mcf 197.parser 255.vortex 256.bzip2 300.twolf 177.mesa 179.art 183.equake 301.apsi

SimPoint 100M 960M 369M 1030M 378M 184M 11M 2703M 42M 5496M 3M

Base IPC 1.4278 1.4067 0.6949 1.1786 1.8651 1.3342 2.1598 2.4309 0.7831 1.4676 1.6000

118

W. Choi, S.-J. Park, and M. Dubois

5 Design Alternatives This section presents design alternatives for instruction pre-schedulers with pre-selection. The difference between the designs is in the way they manage bit vector overflow and operand-overflowed instructions. All performance numbers shown in the graphs are IPCs normalized to the baseline IPC in Table 2. 5.1 Stalling Pre-scheduler The simplest solution for bit vector overflow is the stalling scheme. In the stalling pre-scheduler, if a bit vector overflow occurs, an operand-overflowed instruction and the following instructions are stalled and wait until they can be pre-scheduled with no overflow. Figure 7 shows the normalized IPC of stalling pre-schedulers with various bit vector sizes. The normalized IPCs of counter-based pre-schedulers are shown for comparison. The counter-based pre-schedulers also adopt the stalling policy and the maximum value of the counter is equal to the number of bits in the corresponding bit vector scheme. The suffix indicates how many future cycles each scheme can track. For example, both bit-vector_08 and counter_08 can track 7 cycles in the future. The performance differences observed between the counter and the bit vector schemes come from the fact that resources are assigned round robin in the bit vector scheme, whereas any resource of each type can be assigned in the counter scheme. We observe little performance loss by assigning resources round robin instead of with a counter. To explore the effect of bit vector sizes, we have simulated stalling pre-schedulers with bit vector sizes of 8, 16, and 32. As the length of the bit vector increases, we can pre-schedule farther in the future. Therefore, the stalling due to the bit vector overflow is reduced and the IPC increases. As the bit vector size increases from 8 to 32, the normalized IPC increases from 64.17% to 70.23% on average. The mean IPC gap between the baseline processor and a processor with stalling pre-scheduler is significant (3040%). The operand-overflowed instruction itself causes little IPC degradation but it also blocks the following issuable instructions, which is the major cause of IPC degradation. The performance of mcf and art is severely degraded under the stalling scheme. These benchmarks experience lots of cache misses and very long bit vector sizes are required to deal with the long latency of LOADs. To overcome this weakness of stalling pre-schedulers we propose to add an overflow queue to the pre-scheduler.

bit-vector_08

counter_08

bit-vector_16

counter_16

bit-vector_32

counter_32

1.0

Normalized IPC

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 gzip

gcc

mcf

parser

vortex

bzip2

twolf

mesa

art

equake

Fig. 7. Normalized IPC of Stalling Pre-scheduler

apsi

G.Mean

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

119

5.2 Overflow Queue with Busy-Waiting The scheme using an overflow queue with busy-waiting is illustrated in Figure 8(a). In this scheme, each entry in the RTT has an additional bit, the waiting-bit. A waiting-bit of 1 means that the operand is overflowed, i.e., unavailable due to bit vector overflow. Whenever an instruction finds that one of its operands is overflowed, it sets the waiting-bit of its result register to inform child instructions, and moves into the overflow queue to avoid blocking following instructions. The overflow queue is a simple first-in first-out (FIFO) queue. If there is any instruction in the overflow queue, the head instruction in the overflow queue re-accesses the RTT. In Figure 8(a), instruction iold from the overflow queue competes with the youngest instruction in the current pre-scheduling group (inew) to re-access the RTT. iold always has priority to ensure forward progress. If an instruction finds that the waiting-bit of any of its operands is set (i.e., its parent instruction is in the overflow queue), it also moves into the overflow queue to maintain dependency order and it sets the waiting-bit of its result register. Every time an instruction moves to the PRESCH3 stage, it clears the waiting-bit of its result register. Operand-overflowed instructions can be reinserted into the overflow queue many times. When the overflow queue is full, the front-end is stalled. Sometimes, child instructions in the overflow queue fill up the overflow queue and block the parent instructions from reentering the overflow queue. In this situation, the parent instruction is stalled at the PRESCH2 stage as was the case in the stalling pre-scheduler. This situation can degrade performance. Figure 9 shows the normalized IPC of busy-waiting pre-schedulers. We vary the bit vector size and the overflow queue size and label the configurations “x_y,” where x and y indicate the bit vector size and the number of entries in the overflow queue, respectively. The busy-waiting pre-scheduler outperforms the stalling pre-scheduler by 17.97% on average. We note that the performance of both mcf and art is dramatically improved as compared to the stalling scheme. Some benchmarks such as gzip experience some performance loss as the number of entries in the overflow queue is increased. This is due to the FIFO characteristics of the overflow queue. If the number of instructions in the overflow queue increases, a newly inserted instruction must wait until all the previous instructions re-access RTT, which can degrade the performance if this newly inserted instruction is on the critical path. Additionally, the re-circulated instructions occupy scheduler bandwidth. We now propose to time accesses to RTT so that instructions do not re-access it multiple times. Overflow Queue Overflow Queue Head instruction re-accesses RTT

Register Timing Table

Overflow Queue

Delay Value

Register Timing Table

Delay Waiting Value Bit

iold inew

Round-robin selection

Register Timing Table

Delay Waiting Value Bit

Delay Value

r1

r1

r1

r2

r2

r2

r3

r3

Queue Index

r3

UUU

UUU

UUU

(a)

(b)

(c)

Fig. 8. (a) Overflow Queue with Busy-Waiting. (b) Overflow Queue with Delay Value. (c) Dependency-based Overflow Queue.

120

W. Choi, S.-J. Park, and M. Dubois

08_08

08_32

16_08

16_32

32_08

32_32

1.0

Normalized IPC

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 gzip

gcc

mcf

parser

vortex

bzip2

twolf

mesa

art

equake

apsi

G.Mean

Fig. 9. Normalized IPC of Busy-Waiting Pre-scheduler

5.3 Overflow Queue with Delay Value Figure 8(b) illustrates the overflow queue with delay value. In this scheme, the overflow queue keeps operand-overflowed instructions with their delay value. An operand-overflowed instruction moves into the overflow queue with its delay value and sets the waiting-bit of its result register. The delay value in the overflow queue is counted down in every cycle. If the waiting-bit of one of its operand is set, an instruction is inserted in the overflow queue with zero-delay value and sets the waiting-bit of its result register. Only the head instruction with zero-delay value can re-access RTT. Every time an instruction moves to the PRESCH3 stage, it clears the waiting-bit of its result register. When the overflow queue is full, the front-end is stalled. Figure 10 shows the normalized IPC of the delay-value pre-scheduler with various bit vector sizes and overflow queue sizes. The IPC of delay-value pre-schedulers is slightly worse than the IPC of busy-waiting pre-schedulers. The delay-value pre-scheduler prevents operand-overflowed instructions from being reinserted into the overflow queue several times. However, if the head instruction of the overflow queue has a large delay value, it blocks the following instructions in the overflow queue from re-accessing RTT and eventually stalls the front-end when the overflow queue is full. If there are several operand-overflowed instructions in the overflow queue, an instruction with a larger delay value could block the child instruction of a prior instruction. To solve this problem we propose to allocate one queue to each dependency chain. 08_08

08_32

16_08

16_32

32_08

32_32

1.0 0.9

Normalized IPC

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 gzip

gcc

mcf

parser

vortex

bzip2

twolf

mesa

art

equake

apsi

Fig. 10. Normalized IPC of Delay-Value Pre-scheduler

G.Mean

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

121

5.4 Dependency-Based Overflow Queue In the dependency-based overflow queue scheme, several overflow queues are each assigned to a dependency chain. Figure 8(c) illustrates a dependency-based pre-scheduler with two overflow queues. Instead of a waiting-bit, each RTT entry has a queue index. An operand-overflowed instruction is inserted into the assigned overflow queue with its delay value. The index of the queue is written in the queue index field of its result register. If an instruction finds that the queue index of one of its operand is non-zero, it is inserted into the appropriate overflow queue with zero-delay value and copies the queue index into the queue index field of its result register. When both operands have a valid queue index, the overflow queue indexed by the left operand is chosen. The head instruction of each overflow queue with zero-delay value can re-access RTT. If instructions from two overflow queues compete to access the RTT, a round-robin policy is used. Every time an instruction moves to the PRESCH3 stage, it clears the queue index of its result register. When an overflow queue is full or if there is no overflow queue left for a newly operand-overflowed instruction, the front-end is stalled until this situation is resolved. Figure 11 shows the normalized IPC of dependency-based pre-schedulers with various configurations. We label the configurations “x_y_z,” where x, y, and z indicate the bit vector size, the number of overflow queues, and the number of entries in each overflow queue, respectively. The dependency-based pre-scheduler has the best performance overall among our pre-scheduler alternatives. 5.5 Discussion Overall, the IPC extracted by pre-schedulers with pre-selection is less than the IPC of broadcast schedulers, even in the case of dependency-based pre-schedulers, which are the most complex and the best pre-schedulers we have considered. There are several reasons for this IPC degradation: • Because we insert three more pipeline stages in the front-end, the cost of replays caused by branch misprediction, memory dependency misprediction, and cache hit/miss misprediction is higher. • After events such as branch misprediction or memory dependency misprediction, we do not reset the bits of the resource availability bit vectors for the instructions that were squashed. • The limited size of bit vectors may constrain ILP. 08_04_16

08_08_08

16_04_16

16_08_08

32_04_16

32_08_08

1.0 0.9

Normalized IPC

0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 gzip

gcc

mcf

parser

vortex

bzip2

twolf

mesa

art

equake

apsi

G.Mean

Fig. 11. Normalized IPC of Dependency-based Pre-scheduler

122

W. Choi, S.-J. Park, and M. Dubois

• The number of overflow queues and the number of entries in each overflow queue are both limited. • The round-robin assignment of resource availability bit vectors may not be optimal. • The round-robin selection of head instructions from overflow queues may be sub-optimal. • The stalling condition related with functional unit availability vector assignment may create false structural hazards. It is possible that the IPCs could be improved in future research by solving some of these problems. Nevertheless, the average loss in IPC of the dependency-based prescheduler (as compared to a broadcast scheduler) is only around 20% and is quite uniform across all benchmarks. Moreover the instruction pre-scheduler can be pipelined and is free of long wires transmitting broadcast signals. This leads to higher clock frequencies as compared to traditional broadcast schedulers. To compare the schedulers in a fair way, we must also consider their impact on the clock rate.

6 Overall Comparison with Broadcast Scheduling 6.1 Logic Delay In this section we combine the effects of IPC and clock rate to compare the true effectiveness of pre-scheduling algorithms. To understand the efficiency of our pre-scheduling algorithms, we must estimate the delays of the critical circuits used in the traditional broadcast scheduler and in our proposed pre-schedulers with pre-selection. We have designed all circuits in Verilog HDL and synthesized them with Synopsys’ Design Compiler [24] targeting Samsung Electronics Corporation 0.13um 1.8V fabrication process with stdh 150hde library [25]. The broadcast scheduler has a 64-entry issue queue and conventional wake-up logic and select logic. The critical path for all pre-schedulers with pre-selection is made of the number-tobit conversion, AND operation, SHIFT logic, and bit-to-number conversion. NtoB logic is a decoder with high-order bits selection. It decodes the number to select a bit position and selects all high-order bits above that bit as well. BtoN logic is a basic priority encoder. In this design, the SHIFT logic and BtoN logic are combined to minimize the path delay. The circuit design for these functions used in our instruction pre-schedulers can be found in [26]. The timing results for each configuration are shown in Table 3. NtoB logic and the combined SHIFT and BtoN logic are the dominant components in the critical path. As the length of the bit vector increases, the total delay also increases because the load capacitance driven by each input increases in both the NtoB and BtoN circuits. The delay of the AND circuit is not easily explained as it decreases with the bit vector size. The design compiler tries to optimize the total delay. Sometimes, it uses different drive length in the critical path to decrease the total delay and to increase driving capability in the synthesis process. As the drive length increases, the cell delay increases but the wire delay decreases. The AND operation delay with each bit vector size shows anomalies because different drive lengths have been adopted by the compiler for each bit vector size. From Table 3, we can validate the efficiency of the instruction pre-scheduler. By pipelining instruction pre-scheduling, the delay of prescheduling is radically reduced, as compared to the delay of the broadcast scheduler.

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

123

Table 3. Logic Delays Pre-scheduler with Pre-selection

Delay (ps) NtoB AND SHIFT+BtoN Total

8-bit 86.56 87.83 303.20 477.59

16-bit 128.38 59.15 439.37 626.90

Broadcast Scheduler

32-bit 188.00 75.32 495.73 759.05

926.31

6.2 Overall Performance In Section 5, we showed that the IPC of a pipeline equipped with our pre-schedulers cannot be better than the IPC of a pipeline with a broadcast scheduler. However, in [9, 19] the instruction scheduler is identified as one of the major bottlenecks in out-of-order processors, which limits the clock rate. To evaluate the potential benefit of our prescheduler, we must include the clock rate in the evaluations. Here we combine IPC and clock rate data to compare the overall performance of several scheduling schemes assuming that the scheduler remains the bottleneck. The performance metric is the number of instructions per nanosecond (IPns) normalized to the IPns of the pipeline with broadcast scheduling. In Figure 12, the prefixes “st”, “bw”, “dv”, and “dep” refer to stalling pre-scheduler, busy-waiting pre-scheduler, delay-value pre-scheduler, and dependencybased pre-scheduler, respectively. Geometric means for all the benchmarks are shown. All 8-bit vector schemes have better average performance than broadcast scheduling. Even though the IPC of the 32-bit vector scheme is the highest in each pre-scheduler, its IPns is relatively poor because of its long logic delays. This may improve with future technologies. For the selected technology, the average IPns of the dependency-based pre-scheduler with 8-bit vector, 8 overflow queues, and 8 queue entries is roughly 60% better than the IPns of the broadcast scheduler. The values displayed in Figure 12 can be seen as upper bounds reached where the scheduler is the bottleneck. 6.3 The Number of Replays Finally, Table 4 shows the number of replays and circulations. Arithmetic means for all the benchmarks are shown. The second column in the table gives the number of replays. st_08 dv_16_32

bw_08_32 dep_16_08_08

dv_08_32 st_32

dep_08_08_08 bw_32_32

st_16 dv_32_32

bw_16_32 dep_32_08_08

1.6

Normalized IPns

1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8

Fig. 12. Normalized Average IPns of the Pre-schedulers

124

W. Choi, S.-J. Park, and M. Dubois Table 4. The Number of Circulations A.Mean (Million) st_08 st_16 st_32 bw_08_32 bw_16_32 bw_32_32 dv_08_32 dv_16_32 dv_32_32 dep_08_08_08 dep_16_08_08 dep_32_08_08 broadcast

Num.Replays 1.657 1.955 4.598 1.688 2.069 4.019 1.337 2.633 2.818 2.881 2.666 3.723 34.148

Num.OFQ Circulations 0 0 0 47.156 39.703 32.913 30.246 23.127 20.965 27.130 22.491 16.451 0

Note that a given instruction may replay multiple times. With shorter bit vectors, the number of front-end stalls due to the bit vector overflow increases. This frequent stalling prevents instructions from entering the processor back-end and being replayed. Therefore, the number of replays increases with longer bit vectors. The broadcast scheduler experiences a huge number of replays due to busy-waiting [13]. The replayed instructions keep re-circulating until their operands become available. By inserting some delays between availability check and reissue, the number of replays could be reduced. However, this lengthens the replay loop and may introduce performance degradation. The third column gives the number of circulations through the overflow queue. The overflow queue contains instructions which might otherwise be replayed (in the case of broadcast scheduler) or stalled (in the case of stalling pre-scheduler). Contrary to the number of replays, the number of overflow queue circulations increases with shorter bit vectors. Shorter bit vectors increase the number of operand-overflowed instructions. By using delay value instead of busy-waiting, we prevent instructions from re-accessing overflow queue multiple times.

7 Related Work Dynamically scheduled processors are a topic of active research and the instruction scheduler is one of the most researched components of the processor because it is a critical bottleneck. Several scheduling schemes based on data-flow pre-scheduling have been proposed to reduce the complexity of traditional broadcast schedulers. In this section we contrast prior pre-scheduling proposals with our approach. In some data-flow pre-scheduling schemes, instructions are reordered in a prescheduling array and moved into a small issue queue after their issue delay has elapsed [4, 5, 6, 7]. Michaud and Seznec [4] described a pre-scheduling logic that predicts the issue delay of each instruction and rearranges instructions in the order of their issue delay. Instructions whose issue delay elapses move to a traditional broadcast-based issue

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

125

queue from the pre-scheduling array. By storing instructions that are likely to be issued soon, issue queue entries are occupied by short-lived instructions, which increases the effective size of the issue queue. In this scheme, conflicts for issue slots and functional units are resolved in the issue queue with conventional select logic. If a resource conflict occurs, the issue time of the conflicted instruction is delayed. However, this delay is not reflected in the pre-schedule of instructions in the pre-scheduling array. Also, the RTT is not updated with this delay. Therefore, the scheduling information in the RTT becomes obsolete and the issue queue could fill up with instructions which will not be issued soon. Raasch, et al. [5] pipelined the pre-scheduling array to adjust the issue delay on scheduling misses. The flow of instructions from segment to segment is managed by logic similar to wake-up logic. Liu, et al. [6] suggested differently sized pre-scheduling queues to precisely adjust the issue delay of each instruction with a sophisticated LOAD latency predictor. Other data-flow pre-scheduling schemes issue the instruction directly from the prescheduling array [8, 9, 10]. Canal and González [8] proposed deterministic latency issue logic, which comprises the RTT, pre-scheduling array, and issue queue. After predicting issue delay with the RTT, an instruction is inserted into the pre-scheduling array and is issued directly from the pre-scheduling array when its predicted issue delay elapses (i.e., it becomes the head of pre-scheduling array). If an issue delay is mispredicted, the incorrectly pre-scheduled instruction is moved to the issue queue and reissued from the queue by broadcast scheduling logic. This scheme gives priority to the issue queue. So, if the instructions in the issue queue take its issue slot or functional unit, an instruction at the head of the pre-scheduling array cannot be issued and is also moved into the issue queue. This delayed issue time is not reflected to the pre-schedule of the following instructions and the delay value in the RTT. So, the issue queue might be full with instructions which are incorrectly pre-scheduled due to resource conflicts. In the Cyclone scheduler advocated by Ernst et al. [9], instructions are inserted into a countdown queue after their issue delay has been predicted. Instructions move through a countdown queue during the first half of their issue delay, and then move to the main queue for the second half. Instructions are issued directly from the main queue. Missscheduled instructions are re-circulated through the queues. Structural hazards break the pre-schedule if the target entry in the main queue is already occupied by a previous instruction when an instruction tries to switch queues [9, 10] and get behind schedule. Dependent instructions are issued too early, and must be re-circulated. These issue slot conflicts have a severe impact on performance, as is shown in [10]. Hu, et al. [10] propose wake-up free scheduling to overcome the weakness of Ernst’s scheduler. Their WF-Precheck scheme consists of RTT, an issue buffer similar to ours, register read-bit checker, and conventional select logic. A pre-scheduled instruction stays in the issue buffer until its issue delay elapses, and then periodically checks its operand availability. When its operands are all available, the instruction sends a request signal to the select logic and select logic picks up the ready instructions based on the issue width and function unit availability. Even though this scheme uses pre-scheduling logic, instructions can send request signals to the select logic only after their operand ready bit is set with the help of a register ready-bit checker, which takes the role of conventional wake-up logic. Therefore, their IPC results are almost identical to the broadcast scheduler. However, the complexity of WF-Precheck scheme could be a bottleneck in frequency scaling because the register ready-bit checking and the instruction selection need to be done within a single cycle.

126

W. Choi, S.-J. Park, and M. Dubois

In previously proposed data-flow pre-schedulers, resource conflicts are detected when the instructions try to be selected in the issue queue [4], to be issued from the prescheduling array [8], or to move to the main queue [9]. By delaying the issue of one of the conflicted instructions (i.e., the younger instruction), the resource conflicts could be easily resolved. However, the RTT or the pre-schedule of instructions in the pre-scheduling array is not updated with this schedule change, and the RTT provides incorrect delay values. Dependent instructions are pre-scheduled with obsolete delay values and miss their operands, which either clogs the issue queue with incorrectly pre-scheduled instructions (in the case of [4, 8]) or causes lots of replays (in the case of [9]).

8 Conclusions The major contribution of our paper is the design of pre-schedulers with pre-selection which monitor the reservation status of individual resources with resource availability bit vectors. The resource conflicts are detected and eliminated early enough to preschedule the following instructions with accurate scheduling information. We have identified key problems of pre-schedulers with pre-selection, and developed simple solutions. We have compared the logic delay of our pre-schedulers with that of broadcast schedulers. Our analysis shows that accurate pre-schedulers can be built with today’s technology and can result in performance improvements of up to 60% over broadcast schedulers in pipeline designs where the scheduler is the bottleneck. As a part of future work, it would be interesting to study the design complexity and power consumption of our pre-schedulers. More research may lead to IPC improvements by solving some of the issues raised in section 5.5. Because the accurate pre-scheduler knows most of the time when resources will be active, it can be used for power management schemes such as clock gating. We have assumed that the initiation interval of the functional units was one. Since we use bit vectors to represent availability it would be interesting to extend our pre-schedulers so that they can apply to non-linear pipelines using well-known approaches such as reservation tables and collision vectors.

References [1] Agarwal, V., Hrishikesh, M., Keckler, S., Burger, D.: Clock Rate versus IPC: The End of the Road for Conventional Microprocessors. In: Proceedings of the 27th International Symposium on Computer Architecture (2000) [2] Hrishikesh, M., Jouppi, N., Farkas, K., Burger, D., Keckler, S., Shivakumar, P.: The Optimal Logic Depth per Pipeline Stage is 6 to 8 FO4 Inverter Delays. In: Proceedings of the 29th International Symposium on Computer Architecture (2002) [3] Stark, J., Brown, M., Patt, Y.: On Pipelining Dynamic Instruction Sched¬uling Logic. In: Proceedings of the 33rd International Symposium on Microarchitecture (2000) [4] Michaud, P., Seznec, A.: Data-Flow Prescheduling for Large Issue queues in Out-ofOrder Processors. In: Proceedings of the 7th International Symposium on High Performance Computer Architecture (2001) [5] Raasch, S., Binkert, N., Reinhardt, S.: A Scalable Instruction Queue De¬sign Using Dependence Chains. In: Proceedings of the 29th International Symposium on Computer Architecture (2002)

Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors

127

[6] Liu, Y., Shayesteh, A., Memik, G., Reinman, G.: Scaling the Issue Window with LookAhead Latency Prediction. In: Proceedings of the 18th Annual ACM International Conference on Supercomputing (2004) [7] Liu, Y., Shayesteh, A., Memik, G., Reinman, G.: Tornado Warning: the Perils of Selective Replay in Multithreaded Processors. In: Proceedings of the 19th Annual ACM International Conference on Supercomputing (2005) [8] Canal, R., González, A.: Reducing the Complexity of the Issue Logic. In: Proceedings of the 15th International Conference on Supercomputing (2001) [9] Ernst, D., Hamel, A., Austin, T.: Cyclone: A Broadcast-Free Dynamic Instruction Scheduler with Selective Replay. In: Proceedings of the 30th International Symposium on Computer Architecture (2003) [10] Hu, J., Vijaykrishnan, N., Irwin, M.: Exploring Wakeup-Free Instruction Scheduling. In: Proceedings of the 10th International Symposium on High Performance Computer Architecture (2004) [11] Kim, I., Lipasti, M.: Understanding Scheduling Replay Schemes. In: Proceedings of the 10th International Symposium on High Performance Computer Architecture (2004) [12] Yoaz, A., Erez, M., Ronen, R., Jourdan, S.: Speculation Techniques for Improving Load Related Instruction Scheduling. In: Proceedings of the 26th International Symposium on Computer Architecture (1999) [13] Merchant, A., Sagar, D.: Computer Processor Having a Checker. United States Patent #6,212,626, assigned to Intel Corporation, issued April 3 (2001) [14] Chrysos, G., Emer, J.: Memory Dependence Prediction Using Store Sets. In: Proceedings of the 25th International Symposium on Computer Architecture (1998) [15] Kessler, R.: The Alpha 21264 Microprocessor. IEEE Micro. 19(2), 24-36(1999) [16] Tendler, J., Dodson, S., Fields, S., Le, H., Sinharoy, B.: Power4 System Microarchitecture. IBM Journal of Research and Development 46(1), 5–26 (2002) [17] Hinton, G., Sager, D., Upton, M., Boggs, D., Carmean, D., Kyker, A., Roussel, P.: The Microarchitecture of the Pentium 4 Processor. Intel Technology Journal, Q1 (2001) [18] Lebeck, A., Koppanalil, J., Li, T., Patwardhan, J., Rotenberg, E.: A Large, Fast Instruction Window for Tolerating Cache Misses. In: Proceedings of the 29th International Symposium on Computer Architecture (2002) [19] Palacharla, S., Jouppi, N., Smith, J.: Complexity-Effective Superscalar Processors. In: Proceedings of the 24th International Symposium on Computer Architecture (1997) [20] Allan, A., Edenfeld, D., Joyner, W., Kahng, A., Rodgers, M., Zorian, Y.: 2001 Technology Roadmap for Semiconductors. IEEE Computer 35(1), 42–53 (2002) [21] Austin, T., Larson, E., Ernst, D.: SimpleScalar: an Infrastructure for Computer System Modeling. IEEE Computer 35(2), 59–67 (2002) [22] Standard Performance Evaluation Corporation, http://www.specbench.org [23] Sherwood, T., Perelman, E., Hamerly, G., Calder, B.: Automatically Char¬acterizing Large Scale Program Behavior. In: Proceedings of the 10th International Conference on Architectural Support for Programming Languages and Operating Systems (2002) [24] Synopsys Inc., http://www.synopsys.com/products/logic/design_compiler.html [25] Samsung Electronics Corporation, http://www.samsung.com/products/semiconductor/ ASIC/StandardCellLibraries/STDH150E/STDH150E.htm [26] Choi, W., Park, S., Dubois, M.: Accurate Instruction Pre-scheduling in Dynamically Scheduled Processors,” Technical Report #CENG-2007-3, Depart¬ment of Electrical Engineering - Systems, University of Southern California (March 2007)

Fetch Gating Control through Speculative Instruction Window Weighting Hans Vandierendonck1 and Andr´e Seznec2 1

Ghent University, Department of Electronics and Information Systems/HiPEAC, B-9000 Gent, Belgium [email protected] 2 IRISA/INRIA/HiPEAC Campus de Beaulieu, 35042 Rennes Cedex, France [email protected]

Abstract. In a dynamic reordering superscalar processor, the front-end fetches instructions and places them in the issue queue. Instructions are then issued by the back-end execution core. Till recently, the front-end was designed to maximize performance without considering energy consumption. The front-end fetches instructions as fast as it can until it is stalled by a filled issue queue or some other blocking structure. This approach wastes energy: (i) speculative execution causes many wrong-path instructions to be fetched and executed, and (ii) back-end execution rate is usually less than its peak rate, but front-end structures are dimensioned to sustained peak performance. Dynamically reducing the frontend instruction rate and the active size of front-end structure (e.g. issue queue) is a required performance-energy trade-off. Techniques proposed in the literature attack only one of these effects. In previous work, we have proposed Speculative Instruction Window Weighting (SIWW) [21], a fetch gating technique that allows to address both fetch gating and instruction issue queue dynamic sizing. SIWW computes a global weight on the set of inflight instructions. This weight depends on the number and types of inflight instructions (non-branches, high confidence or low confidence branches, ...). The front-end instruction rate can be continuously adapted based on this weight. This paper extends the analysis of SIWW performed in previous work. It shows that SIWW performs better than previously proposed fetch gating techniques and that SIWW allows to dynamically adapt the size of the active instruction queue.

1

Introduction

Dynamic reordering superscalar architectures are organized around an instruction queue that bridges the front-end instruction delivery part to the back-end execution core. Typical performance-driven designs maximize the throughput of both the front-end and the back-end independently. However, it has been noted [3,4,14] that such designs waste energy as the front-end fetches instructions as fast as it can up to the point where the back-end fills up and the front-end necessarily stalls. All of this fast, aggressive work may be performed at a lower P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 128–148, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Fetch Gating Control through Speculative Instruction Window Weighting

129

and more power-efficient pace, or it may even turn out to be unnecessary due to control-flow misspeculation. Historically, the first step to lower the front-end instruction rate relates to fetching wrong-path instructions. By assigning confidence to branch predictions, Manne et al [17] gate instruction fetch when it becomes likely that fetch is proceeding along the wrong execution path. However, with the advent of highly accurate conditional [8,12,19] and indirect branch predictors [6,20], the impact of wrong-path instructions on energy decreases [18]. Besides fetching wrong-path instructions, it has been shown that the frontend flow rate may well exceed the required back-end rate [3,4]. Closely linked to this flow-rate mismatch is a mismatch between the required issue queue size and the available issue queue size. Consequently, fetch gating mechanisms are combined with dynamic issue queue adaptation techniques to increase energy savings [4]. This paper contributes to this body of work by analyzing a new fetch gating algorithm built on these principles. Our fetch gating algorithm simultaneously tracks branch confidence estimation and the set of already inflight and unissued instructions. In effect, it modulates branch confidence estimation by issue queue utilization: as issue queue utilization is higher, uncertainty in the control-flow speculation weighs stronger to limit the front-end flow rate. Hereby, our technique avoids both wrong-path work and it matches the front-end flow rate to the back-end flow rate. To illustrate the advantage of our technique, let us consider the two following situations. In example (A), 50 instructions, a low-confidence and 6 highconfidence branches have already been fetched. In example (B), 5 instructions and a single low-confidence branch have been fetched. If the next instruction is a low-confidence branch then a fetch gating control mechanism based only on branch confidence estimation and boosting [17] will take exactly the same decision for the two situations. A simple analysis of pipeline operation shows that for (A), delaying the next instruction fetch for a few cycles (but maybe not until the low confidence branch resolves) is unlikely to degrade performance while for (B), delaying it is very likely to induce a few cycles loss if the two low-confidence branches are correctly predicted. The first contribution of this paper is Speculative Instruction Window Weighting (SIWW). Instead of only considering confidence on inflight branches for controlling fetch gating, we consider the overall set of already inflight and unissued instructions, i.e. the speculative instruction window. SIWW tries to evaluate whether or not the immediate fetch of the next instruction group will bring some extra performance. When the expected benefit is low, then fetch is gated until the benefit has increased or a branch misprediction has been detected. This expected performance benefit increases when (i) branch instructions are resolved or (ii) instructions execute and the number of unissued instructions in the issue queue drops. Our experiments show that fetch gating based on SIWW easily outperforms fetch gating schemes based on confidence boosting [17], fetch throttling [2] as well as issue queue dynamic sizing techniques [4].

130

H. Vandierendonck and A. Seznec

A second contribution of this paper is to show that fetch gating control through SIWW can be efficiently implemented without any extra storage table for confidence estimation. Current state-of-the-art branch predictors such as O-GEHL [19] and piecewise linear branch prediction [12] provide a confidence estimate for free. We show that this estimate does not work very well for fetch gating control through boosting, but it works well with SIWW fetch gating and instruction queue dynamic sizing. This paper extends earlier work [21] by presenting more detailed analysis of Speculative Instruction Window Weighting. The remainder of the paper is organized as follows. Section 2 reviews related work on fetch gating or throttling and dynamic sizing of instruction queues. Section 3 describes our proposal to use SIWW for fetch gating. Our experimental framework is presented in Section 4. Section 5 presents the performance of SIWW and compares it to confidence boosting, the previous state-of-the-art approach. Finally, Section 6 presents possible future research directions and summarizes this study.

2

Related Work

Gating the instruction fetch stage on the first encountered low-confidence branch results in significant performance loss. By delaying gating until multiple lowconfidence branches are outstanding – a technique called boosting – it is possible to limit this performance loss while still removing extra work [17]. Fetch throttling slows down instruction fetch by activating the fetch stage only once every N cycles when a low-confidence branch is in-flight [2]. This reduces the performance penalty of pipeline gating, but sacrifices energy reduction by allowing additional extra work. Recently, Lee et al. [15] proposed wrong path usefulness predictors. These are confidence estimators that take into account the positive or negative effects of fetching wrong-path instructions. The rationale is that fetching wrong-path instructions may be beneficial for instruction or data cache prefetching. Decode/commit-rate fetch gating is an instruction flow-based mechanism that limits instruction decode bandwidth to the actual commit bandwidth [3]. This technique saves energy even for correct-path instructions, as only the required fetch bandwidth is utilized. Buyuktosunoglu et al [4] combine fetch gating with dynamic issue queue adaptation in order to match front-end and back-end instruction flow rates and to match the issue queue size to its required size. They propose to gate instruction fetch based on the observed parallelism in the instruction stream. Fetch is gated during one cycle when instruction issue occurs mostly from the oldest halve of the reorder buffer and the issue queue is more than half full. Several techniques to dynamically adapt the issue queue size have been proposed in the literature. Folegnani and Gonzalez [7] divide the reorder buffer into portions of 8 instructions. The reorder buffer grows and shrinks by units of a portion. The reorder buffer is dimensioned by monitoring the number of instructions that issue from the portion of the reorder buffer holding the youngest instructions.

Fetch Gating Control through Speculative Instruction Window Weighting

131

Just-in-time (JIT) instruction delivery [14] applies a dynamic reconfiguration algorithm to adapt the reorder buffer size. It determines the smallest reorder buffer size that yields a performance degradation less than a preset threshold. In [5], the issue queue is also divided into portions of 8 instructions but the queue size is determined by its average utilization over a time quantum.

3

Speculative Instruction Window Weighting

Instead of only considering confidence on inflight branches for controlling fetch gating, we consider the overall set of already inflight instructions, i.e. the speculative instruction window. Speculative Instruction Window Weighting (SIWW) tries to evaluate whether or not the immediate fetch of the next instruction group will bring some performance benefit. Our thesis is that this benefit decreases with the number of already inflight instructions, with the number of branches and with the quality of the branch prediction (i.e. with the confidence in the predictions). The performance benefit may also depend on the precise type of already inflight instructions and parameters such as latency (e.g. divisions, multiplications, loads that are likely to miss), etc. For this purpose, a global Speculative Instruction Window (SIW) weight is computed on the overall set of unexecuted inflight instructions. The SIW weight is intended to “evaluate” the performance benefit that immediately fetching new instructions would deliver. The SIW weight is constantly changing. It increases when instructions are fetched and it decreases as instructions are executed. When the SIW weight exceeds a pre-set threshold, instruction fetch is halted. As soon as the SIW weight drops below the threshold, the instruction fetch is resumed (Figure 1). 3.1

Computing the SIW Weight: Principle

The SIW weight is computed from the overall content of the instruction window. To obtain a very accurate indicator, one should take into account many factors, such as dependencies in the instruction window, instruction latency, etc. However, realistic hardware implementation must also be considered. Therefore, we

exceeds threshold?

SIW weight Add instruction’s SIW weight contribution

Gate decode

Fetch Decode DispatchSchedule Reg.File 9 cycles I-cache

Subtract instruction’s SIW weight contribution when instruction executes

Issue

Execute Writeback Commit

7 cycles

4-cycle backward edge latency for branch mispredictions

Fig. 1. Block diagram of a pipeline with speculative instruction window weighting

132

H. Vandierendonck and A. Seznec Table 1. SIW weight contributions Instruction type Contrib. high-confidence conditional branches 8 low-confidence conditional branches 40 returns 8 high-confidence indirect branches 8 low-confidence indirect branches 40 unconditional direct branches 1 non-branch instructions 1

propose to compute the SIW weight as the sum of individual contributions by the inflight instructions. These contributions are determined at decode time. As an initial implementation of SIWW, we assign a SIW weight contribution to each instruction by means of its instruction class. The instruction classes and SIW weight contributions used in this paper are listed in Table 1. The weight contributions reflect the probability of a misprediction. Thus, lowconfidence branches are assigned significantly higher weight contributions than high-confidence branches. High-confidence branches are assigned higher weight contributions than non-branch instructions because high-confidence branches too are mispredicted from time to time. Return instructions have a small weight contribution because they are predicted very accurately. Unconditional direct branches have the same weight contributions as nonbranch instructions as their mispredict penalty is very low in the simulated architecture. Mispredicted targets for unconditional direct branches are captured in the decode stage. Fetch is immediately restarted at the correct branch target. The weight contributions depend on the accuracy of the conditional branch predictor, branch target predictor and return address stack and their confidence estimators. The weight contributions may have to be tuned to these predictors. The weight contributions describe only the speculativeness of the in-flight instructions and are therefore independent of other micro-architectural properties. The confidence estimators too may be tuned to maximize the performance of SIWW. In particular, it is helpful to maximize the difference of prediction accuracy between high-confidence branches and low-confidence branches, such that the corresponding weights can be strongly different. Ideally, high-confidence branches are always correctly predicted (predictive value of a positive test or PVP is 100% [9]) and have a weight of 1, while low-confidence branches are always incorrectly predicted and have an infinitely large weight (predictive value of a negative test or PVN is 100%). In practice, however, the PVP and PVN values of confidence estimators are traded-off against each other and cannot be close to 100% at the same time. Consequently, the confidence estimator has to be carefully constructed such that PVP and PVN are both relatively large. When finding such a balance, it is important to keep in mind that the fraction of lowconfidence branches that is detected (SPEC) also has an important influence, since a smaller SPEC implies less fetch gating.

Fetch Gating Control through Speculative Instruction Window Weighting

3.2

133

A Practical Implementation of SIW Weight Computation

In principle, the SIW weight is computed from the old SIW weight by adding the contributions of all newly decoded instructions and substracting the contributions of all executed instructions. Initially, the SIW weight is zero. However, when a branch misprediction is detected, the SIW weight represents an instruction window with holes (Figure 2): some of the instructions that were fetched before the mispredicted branch are still waiting to be executed. Restoring the SIW weight to its correct value while resuming instruction fetch after the mispredicted branch would require to retrieve the contributions of these instructions and perform an adder tree sum. To sidestep a complex adder tree, we approximate the SIW weight by setting it to zero on a misprediction. The SIW weight then ignores the presence of unexecuted instructions in the pipeline. However, the SIW weight contribution of these instructions may not be substracted again when they execute. To protect against substracting a contribution twice, we keep track of the most recently recovered branch instruction. Instructions that are older (in program order) should not have their SIW weight contribution substracted when they execute. Experimental results have shown that this practical implementation performs almost identical to the exact scheme. In most cases, when a mispredicted branch is detected, the instruction window will be largely drained, causing the exact SIW weight to drop to the range 20–50 (compare this to the SIW threshold of 160). Most of these remaining instructions are executed before the first corrected path instructions reach the execution stage. At this time, the approximate SIW weight is already very close to its maximum value, minimizing the impact of the temporary underestimation. 3.3

Dynamically Adapting the SIW Weight Contributions

The weight contributions proposed in Table 1 are based on the prediction accuracy of particular types of branches (low-confidence vs. high-confidence, conditional vs. indirect, etc.). However, the prediction accuracy varies strongly from benchmark to benchmark, so the weight contributions should reflect these differences. To improve the SIWW mechanism, we investigated ways to dynamically adapt the weight contributions based on the prediction accuracy. last insn. fetched

mispredicted branch detected

some instructions have already left the SIW

last insn. committed

SIW weight close to zero

instructions in program order insn. inflight, executed insn. inflight, not executed

Fig. 2. The set of inflight instructions is a contiguous slice of instructions from the fetched instruction stream. Some of these inflight instructions have executed and have left the speculative instruction window, while others are waiting for execution and are still part of the speculative instruction window.

134

H. Vandierendonck and A. Seznec

We dynamically adjust the weight contribution of each instruction class in Table 1 where the baseline contribution differs from 1. Each contribution is trained using only the instructions in its class. The contribution is increased when the misprediction rate is high (high probability of being on the wrong execution path) and is decreased when the misprediction rate in its instruction class is low. To accomplish this, we use two registers: a p-bit register storing the weight contribution and a (p + n)-bit register storing a counter. Practical values for p and n are discussed below. The counter tracks whether the weight contribution is proportional to the misprediction rate. For each committed instruction in its class, the counter is incremented with the weight. If the instruction was mispredicted, it is also decremented by 2p . Thus, the counter has the value c − f 2p where c is the current weight contribution and f is the misprediction rate. As long as the counter is close to zero, then the contribution is proportional to the misprediction rate. When the counter deviates strongly from zero, then the weight contribution needs adjustment. When the counter overflows, the weight contribution is decremented by 1 because it was higher than the misprediction rate. When the counter underflows, the weight contribution is incremented by 1. At this point, the counter is reset to zero to avoid constantly changing the weight contribution. When computing the overall SIW weight, the weight contributions for branch instructions are no longer constants but are read from the appropriate register. The values for p and n used in this paper are 7 and 8, respectively. Note that the size of the counter (n) determines the learning period. In total, we need 5 7-bit registers, 5 15-bit registers and a small number of adders and control to update these registers. This update is not time-critical because these registers track the average over a large instruction sequence and change slowly over time. 3.4

Selecting the SIW Threshold

The SIW threshold remains fixed. Selecting the SIW threshold involves a tradeoff between reducing wrong-path instructions (smaller thresholds) and execution speed (larger thresholds). The SIW threshold also depends on the weight contributions: larger weight contributions lead to a larger SIW weight, so to gate fetch under the same conditions a larger SIW threshold is required too. Finally, the SIW threshold depends on branch prediction accuracy too. We analyze SIWW using multiple SIW thresholds in order to quantify this trade-off.

4

Experimental Environment

Simulation results presented in this paper are obtained using sim-flex1 with the Alpha ISA. The simulator is modified and configured to model a future deeply pipelined processor (Table 2). The configuration is inspired by the Intel Pentium 4 [10], but at the same time care is taken to limit the extra work that 1

http://www.ece.cmu.edu/~simflex

Fetch Gating Control through Speculative Instruction Window Weighting

135

the baseline model performs for wrong-path instructions. Amongst others, we use conservative fetch, decode and issue widths of 4 instructions per cycle because this is a good trade-off between power consumption and performance and it is a more realistic number if power efficiency is a major design consideration. Gating control resides in the decode stage because the instruction type, confidence estimates and SIW contributions are known only at decode. To improve the effectiveness of the gating techniques, the fetch stage and the decode stage are simultaneously gated. Two different branch predictors are considered in this study: gshare and OGEHL. These predictors feature 64 Kbits of storage. For gshare, we considered 15 bits global history, a JRS confidence estimator [11] with 4K 4 bit counters and 15 as the confidence threshold. Power consumption in the JRS confidence estimator is modeled by estimating power dissipation in the JRS table. The O-GEHL predictor selects a signed weight from each one of eight tables, depending on the global history. The sum of these weights determines the predicted branch direction: taken if the sum if positive or zero. We simulated the baseline configuration presented in [19]. The sum of weights lends itself very well to obtain confidence estimation: a branch is high-confidence if the absolute value of the sum of weights is larger than the confidence threshold. We call this self confidence estimation as in [13,1]. Self confidence estimation consumes no additional power. Table 2. Baseline Processor Model Processor core 4 instructions 96 48 7 cycles Fetch Unit Fetch width 4 instructions, 2 branches/cycle Instruction fetch queue 8 instructions Fetch-dispatch delay 9 cycles Cond. branch predictor gshare or O-GEHL Cond. branch confidence estimator JRS (gshare) or self confidence (O-GEHL) Return address stack 16 entries, checkpoint 2 Branch target buffer 256 sets, 4 ways Cascaded branch target predictor 64 sets, 4 ways, 8-branch path history Indirect branch confidence 2-bit saturating counter associated to estimator stored branch targets Memory Hierarchy L1 I/D caches 64 KB, 4-way, 64B blocks L2 unified cache 256 KB, 8-way, 64B blocks L3 unified cache 4 MB, 8-way, 64B blocks Cache latencies 1 (L1), 6 (L2), 20 (L3) Memory latency 150 cycles Issue width ROB, issue queue Load-store queue Dispatch-execute delay

136

H. Vandierendonck and A. Seznec

Note that there is a direct relation between the confidence threshold and the update threshold of the predictor. If the confidence threshold is larger than the update threshold then one may enter situations where the predictions are always correct but the predictor is not updated: the branches will be classified low-confidence for ever. On the other hand, if the confidence threshold is smaller than or equal to the update threshold, then low-confidence implies that the predictor will be updated, therefore if the (branch, history) pair is O-GEHL predictable then it will become high-confidence. As the O-GEHL predictor dynamically adapts its update threshold, the confidence threshold is adapted in the same manner. A cascaded branch target predictor [6] is implemented. Confidence is estimated as follows. Each entry is extended with a 2-bit resetting counter. The counter is incremented on a correct prediction and set to zero on an incorrect prediction. An indirect branch is assigned high confidence when the counter is saturated in the highest state. We measure the benefits of pipeline gating using extra work metrics [17], i.e. the number of wrong-path instructions that pass through a pipeline stage divided by the number of correct-path instructions. We simulate SPEC CPU 2000 benchmarks executing the reference inputs.2 Traces of 500 million instructions are obtained using SimPoint3 .

5

Evaluation

We evaluate the performance of speculative instruction window weighting for the SPEC benchmarks. We have used all SPECint benchmarks that work correctly in our simulation infrastructure, as well as 4 SPECfp benchmarks that exhibit distinct branch behaviors, ranging from almost perfectly predictable to highly predictable. Table 3 displays the characteristics of our benchmark set considering gshare and OGEHL as branch predictors. Column CND and IND represents the misprediction rate in mispredicts per 1000 instructions for conditional branches and indirect branches. Notwithstanding high prediction accuracy, the extra fetch work (EFW) represents between 15.5% and 93.6% extra work on the SPECint benchmarks when using the O-GEHL predictor. The SPECfp benchmarks exhibit less than 10% extra fetch work. Using gshare instead of O-GEHL as branch predictor reduces the overall base performance by 5.65%. It also induces more work on the wrong path: the average extra instruction fetch work is increased from 39.2% to 52.4%. In Table 3, we also illustrate performance as instruction per cycle (IPC) and power consumption as energy per instruction (EPI) using the SimFlex technological parameters. EPI is represented for the base configuration and an oracle configuration assuming that fetch is stopped as soon as a mispredicted branch 2

3

Our simulation infrastructure cannot handle the perlbmk inputs, so we resort to the SPEC’95 reference scrabble input. http://www.cs.ucsd.edu/~calder/SimPoint/

Fetch Gating Control through Speculative Instruction Window Weighting

137

Table 3. Statistics for the benchmarks executing on the baseline processor model with two different branch predictors. The columns show: IPC, mispredicts per kilo instructions (MPKI) for conditional (CND) and indirect branches (IND), fetch extra work (EFW), energy per instruction (EPI) and EPI as obtained with an oracle confidence estimator (ORA). O-GEHL Benchmark bzip2 crafty gap gcc gzip mcf parser perlbmk twolf vortex ammp apsi swim wupwise average

gshare

IPC CND IND EFW EPI ORA IPC CND IND EFW EPI ORA 1.89 5.17 0.00 55.31% 17.47 16.1 1.81 6.29 0.00 64.38% 17.67 16.0 2.16 3.42 0.84 47.79% 16.26 14.8 1.98 5.43 0.85 63.36% 16.74 14.8 1.89 0.37 0.13 15.52% 16.46 16.2 1.83 1.22 0.19 24.78% 16.54 16.0 1.93 4.05 1.50 62.89% 16.64 14.8 1.68 7.61 1.58 89.41% 17.72 15.1 1.56 5.16 0.00 60.95% 18.80 16.5 1.51 6.09 0.00 71.44% 18.92 16.3 0.32 6.99 0.00 93.61% 39.95 36.6 0.31 8.23 0.00 107.16% 39.79 36.1 1.65 4.13 0.33 53.02% 16.86 15.1 1.54 6.00 0.42 70.81% 17.47 15.1 2.38 0.66 1.86 37.60% 15.88 14.9 2.20 1.98 2.20 52.69% 16.31 14.8 1.26 7.82 0.00 81.12% 19.58 17.1 1.17 10.62 0.00 106.43% 20.49 17.2 2.49 0.13 0.03 18.55% 14.49 14.4 2.42 0.73 0.05 23.72% 14.50 14.2 1.67 0.69 0.00 8.70% 17.11 16.9 1.61 1.77 0.00 18.70% 17.29 16.8 2.52 0.00 0.00 4.24% 14.66 14.7 2.44 0.76 0.00 11.86% 14.76 14.5 0.92 0.05 0.00 2.93% 21.48 21.5 0.92 0.05 0.00 2.94% 21.16 21.2 2.10 0.02 0.00 6.29% 15.10 15.1 1.95 2.40 0.00 25.69% 15.60 15.0 1.77 2.76 0.33 39.18% 18.62 17.5 1.67 4.23 0.38 52.38% 18.92 17.4

is decoded. The overall objective of fetch gating in terms of power consumption can be seen as reducing as much as possible the extra EPI over the oracle configuration while inducing performance loss as small as possible compared with the base configuration. 5.1

Fetch Gating

We compare SIWW with pipeline gating by boosting the confidence estimate and by throttling. First, Figure 3 compares the three gating techniques on a per benchmark basis on configurations favoring a small performance reduction rather than a large extra fetch reduction. The O-GEHL predictor is used here. SIWW (label “SIWW+CE”) incurs less performance degradation than boosting: 0.31% on average compared to 0.79% for boosting. Furthermore, extra fetch work is reduced from 39.2% to 24.2% for SIWW vs. 28.4% for boosting level 2. In total, SIWW removes 38.1% of the extra fetch work. Throttling is known to perform better than boosting. When a low-confidence branch is inflight, fetch is activated once every two cycles. This improves performance slightly over boosting at the expense of a little extra fetch work. However, throttling may be very ineffective for particular benchmarks, e.g. mcf, where hardly any improvement over the baseline is observed.

H. Vandierendonck and A. Seznec 100 90 80 70 60 50 40 30 20 10 0

avg avg

swim

wupw wupw

apsi

baseline boosting throttling SIWW no CE SIWW+CE

2 Slowdown (%)

ammp

twolf

2.5

vortex

perl

mcf

parser

gcc

gzip

gap

crafty

baseline boosting throttling SIWW no CE SIWW+CE

bzip2

Fetch extra work (%)

138

1.5 1 0.5 0

swim

apsi

ammp

twolf

vortex

perl

mcf

parser

gzip

gcc

gap

crafty

bzip2

-0.5

Fig. 3. Comparison between SIWW and boosting for the O-GEHL predictor. The boosting level is 2 low-confidence branches. Throttling fetches only once every two cycles when a low-confidence branch is inflight. The SIWW threshold is 224 (“SIWW no CE”) or 160 (“SIWW+CE”).

5.2

Analysis of SIWW

SIWW works correctly even without using any confidence estimator. We run an experiment without using any confidence estimator i.e. assigning the same weight to each indirect or conditional branch. 16 is the assigned weight. On Figure 3, the considered SIW threshold is 224. This configuration of SIWW (“SIWW no CE”) achieves average extra fetch work and slowdown very similar to throttling. This is explained by the fact that there is still a correlation between the number of inflight branches and the probability to remain on the correct path. Since the weight of a branch is higher than the weight of a non-branch instruction, SIWW enables fetch gating when the number of inflight branches is high. SIWW allows to fully exploit the self confidence estimator. Figure 4 illustrates SIWW versus boosting when varying boosting levels and SIWW thresholds. Decreasing the boosting level to 1 significantly decreases the performance by 5.9% and reduces the extra fetch work from 39.2% to 17.7%. Therefore with fetch gating based on confidence, the designer has the choice between a limited extra work reduction, but small performance reduction with boosting level 2 or higher, or a large performance degradation, but also a larger extra work reduction with no boosting. This limited choice is associated with intrinsic properties of

Fetch Gating Control through Speculative Instruction Window Weighting 6

base boosting throttling 1/2 throttling 1/4 SIWW no CE SIWW+CE

5 Slowdown (%)

139

4 3 2 1 0 -1 15

20

25

30

35

40

Fetch extra work (%)

Fig. 4. Varying boosting levels (1 to 5), throttling parameters (threshold 1 and 2, frequency 1/2 and 1/4) and SIWW (thresholds “SIWW no CE” 192 to 320, “SIWW+CE” 128 to 256)

the self confidence estimator. Manne et al. [17] pointed out that a good trade-off for a confidence estimator for fetch gating based on boosting is high coverage (SPEC) of mispredicted branches (e.g. 80% or higher) and medium predictive value of a negative test (PVN)(10-20%). The JRS estimator applied to gshare can be configured to operate in such a point (Table 4). The self confidence estimator for O-GEHL exhibits medium SPEC and PVN metrics. The SPECint benchmarks show SPEC values in the range of 40%–57% and PVN values above 30%, except for the highly predictable benchmarks gap and vortex. It is not possible to configure the self confidence estimator in the operating point advised by Manne et al. because the confidence threshold may not exceed the update threshold, as explained earlier in this document. On the other hand, this property of the self confidence estimator is not a handicap for SIWW. In addition to providing better performance-extra work trade-off than boosting or throttling, SIWW offers the possibility to choose the SIW threshold in function of the desired performance/extra work trade-off. For Table 4. Statistics on confidence estimators. The columns show predictive value of a negative test (PVN) and specificity (SPEC) for the self confidence estimator of OGEHL and the JRS estimator applied to gshare. Benchmark bzip2 gap gzip parser twolf ammp swim average

O-GEHL/self PVN SPEC 36.1% 56.5% 26.4% 35.9% 33.7% 50.9% 33.2% 49.7% 33.4% 50.3% 26.3% 35.7% 5.0% 5.2% 33.0% 49.3%

gshare/JRS PVN SPEC 22.8% 92.2% 21.2% 92.4% 27.1% 95.1% 21.5% 90.1% 20.4% 91.5% 13.4% 86.2% 0.9% 13.1% 20.5% 90.3%

Benchmark crafty gcc mcf perlbmk vortex apsi wupwise

O-GEHL/self PVN SPEC 32.2% 47.4% 31.8% 46.6% 32.0% 47.1% 31.1% 45.1% 28.6% 40.1% 16.7% 21.6% 3.3% 3.4%

gshare/JRS PVN SPEC 15.9% 83.3% 17.1% 85.2% 20.4% 90.1% 23.9% 94.3% 25.6% 90.1% 16.1% 90.6% 37.7% 100.0%

140

H. Vandierendonck and A. Seznec

instance, with SIWW theshold 128, one sacrifices 1.3% performance but reduces the extra fetch work from 39.2% to 19.9%.

100 90 80 70 60 50 40 30 20 10 0

avg avg

swim

wupw wupw

apsi

SIWW+CE, 128 SIWW+CE, 160 SIWW+CE, 192 baseline

2.5 Slowdown (%)

ammp

twolf

3

vortex

perl

mcf

parser

gcc

gzip

gap

crafty

SIWW+CE, 128 SIWW+CE, 160 SIWW+CE, 192 baseline

bzip2

Fetch extra work (%)

The SIW Threshold is Invariant across Benchmarks. The SIW thresholds 128 and 160 are good choices for every benchmark in our analysis. This conclusion is supported by Figure 5 showing extra fetch work reduction and slowdown for 3 SIW thresholds. Even with SIW threshold 128, slowdown is limited to 3% for all benchmarks. SIW threshold 160 presents a sweet spot across all benchmarks as it reduces slowdown strongly compared to SIW threshold 128, but sacrifices little extra fetch work. As the slowdown incurred with SIW threshold 160 does not exceed 1% for any of the benchmarks, it does not make much sense to use still larger thresholds.

2 1.5 1 0.5 0

swim

apsi

ammp

twolf

vortex

perl

mcf

parser

gzip

gcc

gap

crafty

bzip2

-0.5

Fig. 5. Comparison of 3 SIW thresholds. SIWW uses the confidence estimators.

SIWW versus Cache Miss Rates. Long latency cache misses have a pronounced impact on IPC as they tend to stall the processor pipeline. When pipeline stalls become more dominant, the impact of fetch gating mechanisms changes. An instruction miss results in an immediate stall of the instruction fetch, therefore automatically limiting the number of inflight instructions. On applications or sections of applications featuring high instruction miss rates, fetch gating mechanisms such as SIWW have a reduced impact. E.g., when reducing the instruction cache in the baseline processor to 4 KB, gcc sees a 26% reduction in baseline IPC while extra fetch work reduces from 63% to 42%. SIWW fetch

Slowdown (%)

Fetch Gating Control through Speculative Instruction Window Weighting 10 9 8 7 6 5 4 3 2 1 0

141

base boosting throttling 1/2 throttling 1/4 SIWW no CE SIWW+CE

20

25

30

35

40

45

50

55

Fetch extra work (%)

Fig. 6. Gshare branch predictor. Varying boosting levels (2 to 5), throttling parameters (threshold 1 and 2, frequency 1/2 and 1/4) and SIWW thresholds (both cases 128 to 224).

gating with threshold 192 reduces EFW to 43% in the baseline processor against 33% in the processor with 4 KB instruction cache, yielding a smaller gain both in absolute and relative terms. On the other hand, data misses and particularly L2 and L3 data misses tend to delay the execution of instructions and therefore to increase the number of inflight instructions. In this case, SIWW allows to limit this number at execution time, much more efficiently than traditional confidence boosting. This effect is confirmed by experimentation on parser and reducing the level-1 data cache size to 4 KB. This modification causes the extra fetch work to rise from 53% to 58% in the absence of fetch gating. SIW fetch gating (threshold 192) was already more efficient than confidence boosting in the baseline processor (EFW of 45% vs. 47%). In the processor with reduced level-1 data cache, the benefit of SIWW (EFW 47%) over confidence boosting (EFW 50%) increases. SIWW Works for all Branch Predictors. In order to show that fetch gating control through SIWW works for all branch predictors, we analyze SIWW assuming a gshare branch predictor. Figure 6 shows that SIWW performs better than boosting and throttling both in terms of fetch extra work and in terms of slowdown. For instance SIWW removes more than half of the extra fetch work at a slowdown of 2.1% while boosting level 3 only removes 43% of the extra fetch work but involves a slowdown of 2.6%. 5.3

Dynamic Adaptation of SIW Weight Contributions

The SIWW mechanism is improved by dynamically adapting the SIW weight contributions depending on the predictability of branch conditions and targets in each benchmark. We find that dynamically adapting the SIW weight yields only small reductions in extra work. On the other hand, it is useful to limit slowdown because decreasing the SIW weight contribution for highly predictable instruction classes avoids unnecesary fetch gating. The lower slowdown also translates into energy savings.

142

H. Vandierendonck and A. Seznec 1.2

SIWW+CE SIWW+CE+DWC

Slowdown (%)

1 0.8 0.6 0.4 0.2 0 -0.2

avg

swim

wupw

apsi

ammp

twolf

vortex

perl

mcf

parser

gcc

gzip

gap

crafty

bzip2

-0.4

Fig. 7. Slowdown obtained with dynamic weight contributions (DWC). SIW thresholds are 160 (SIWW+CE) and 96 (SIWW+CE+DWC). Both schemes have almost equal extra work metrics. The O-GEHL predictor is used. Table 5. Dynamic weight contributions averaged over a run of the benchmarks conditional return indirect high low high low bzip2 2.67 23.89 1.01 - 5.00 crafty gap 1.07 15.84 1.01 1.01 4.85 gcc gzip 3.54 22.33 1.01 5.00 5.00 mcf parser 2.09 21.88 1.73 5.00 5.00 perlbmk twolf 3.52 21.97 1.01 - vortex ammp 1.00 17.15 1.19 - apsi swim 1.01 5.03 5.00 5.00 5.00 wupwise

conditional return indirect high low high low 2.27 21.18 1.01 6.61 31.65 2.08 20.99 1.03 2.02 26.49 2.14 21.25 1.00 - 5.00 1.00 20.30 1.01 1.01 26.88 1.00 16.26 1.01 4.02 16.13 1.00 5.17 1.20 1.00 4.56 1.02 -

Figure 7 compares SIWW with fixed weight contributions from the previous section (SIWW+CE) to SIWW with dynamically adapted weight contributions. The dynamic adaptation algorithm is effective in reducing slowdown. E.g. slowdown is reduced to almost zero for gap and wupwise. The slowdown is reduced from an average of 0.31% to 0.13%. We found that the extra work metrics change little, e.g. extra fetch work reduces slightly from 24.2% to 24.0%. However, we will see in the following section that dynamic adaptation is effective at reducing energy consumption. Analysis of the trained weight contributions (Table 5) shows large variations across benchmarks. Furthermore, the difference between the weight contribution for low-confidence branches and high-confidence branches also varies strongly. This difference is small when the PVN of the confidence estimator is small. Then, low-confidence branches are still likely to be predicted correctly and are assigned a lower weight contribution. We attempted to select fixed weight contributions based on the trained values. This, however, yields only small benefits over the fixed weight contributions used throughout this paper.

Fetch Gating Control through Speculative Instruction Window Weighting

5.4

143

SIWW Control for Fetch Gating Is Energy-Efficient

Figure 8 illustrates the trade-off between the EPI (energy per committed instruction) reduction and the slowdown. The graph shows the baseline architecture without fetch gating and an oracle fetch gating scheme that gates fetch for all mispredicted instructions. The oracle scheme defines an upper bound on the total energy savings obtainable with fetch gating, which is 5.8% in our architecture. The fact that this upper bound on total energy savings is quite low simply means that we did a good job at selecting a baseline architecture that is already power-efficient. This is achieved by limiting fetch and issue width to four instructions, by using the highly accurate O-GEHL predictor and by adding history-based branch target prediction. Within this envelope, SIWW is most effective in realizing an energy reduction. The three variations of SIWW reduce energy in the range of 40–70% for a limited slowdown (< 1%). Previously known techniques, such as throttling and pipeline gating realize no more than 26% of the envelope for the same slowdown. Note that boosting with level 1 does not save more energy than boosting level 2, this is due to a particularly high loss of performance on a few benchmarks where both performance and power consumption are made worse. 6

base oracle boosting throttling 1/2 throttling 1/4 SIWW no CE SIWW+CE SIWW+CE+DWC

Slowdown (%)

5 4 3 2 1 0 -1 0

20

40 60 Energy Savings (% of oracle)

80

100

Fig. 8. Reduction of EPI relative to the oracle scheme; the O-GEHL predictor is used 30

Fetch EW Decode EW Execute EW ROB occupancy

Reduction (%)

25 20 15 10 5 0

avg

swim

wupw

apsi

vortex

ammp

perl

twolf

mcf

parser

gzip

gcc

gap

crafty

bzip2

-5

Fig. 9. Reduction of fetch, decode and execute work and reduction of reorder buffer occupancy, using the O-GEHL predictor

144

H. Vandierendonck and A. Seznec

5.5

SIWW and Flow Rate Matching

In the previous sections, we evaluated SIWW for the purpose of gating-off wrongpath instructions. Using the same parameters as in Figure 3, Figure 9 illustrates that SIWW reduces the activity in all pipeline stages. The reduction of activity 10

baseline PAUTI SIWW+CE SIWW+CE+DWC

Slowdown (%)

8 6

ADQ PAUTI+ADQ SIWW+CE+ADQ SIWW+CE+DWC+ADQ

4 2 0 avg

wupw

swim

apsi

ammp

vortex

twolf

perl

parser

mcf

gzip

gcc

avg

wupw

swim

apsi

ammp

vortex

twolf

perl

mcf

gzip

gcc

gap

crafty

parser

Average ROB size reduction (%)

bzip2 60 50 40 30 20 10

avg

wupw

vortex

vortex

swim

twolf

twolf

apsi

perl

perl

ammp

parser

parser

mcf

gzip

gap

gcc

bzip2

0 crafty

Extra Fetch Work (%)

100 90 80 70 60 50 40 30 20 10 0

12 Energy Reduction (%)

gap

bzip2

crafty

-2

10 8 6 4 2 avg

wupw

swim

apsi

ammp

mcf

gzip

gcc

gap

crafty

bzip2

0

Fig. 10. Slowdown, extra fetch work, reorder buffer size reduction and total energy savings. The O-GEHL predictor is used.

Fetch Gating Control through Speculative Instruction Window Weighting

145

in the execute stage is small. However, SIWW exhibits the potential to reduce power consumption in the schedule, execute and wake-up stages as the occupancy of the reorder buffer is strongly reduced compared to the baseline, up to 27% for gcc. This property can be leveraged to reduce power further by dynamically scaling the size of the reorder buffer or the issue queue [4,7]. Comparison to PAUTI Flow-Rate Matching. We compare SIWW to PAUTI [4], a parallelism and utilization-based fetch gating mechanism. We assume a non-collapsing issue queue in our baseline processor because of its energyefficiency [7]. SIWW is not dependent on the issue queue design but PAUTI is specified for a collapsing issue queue [4]. We adapted PAUTI to a non-collapsing issue queue in the following way. During each cycle, PAUTI decides on gating fetch for one cycle based on the issued instructions. If more than half of the issued instructions are issued from the oldest half of the issue queue and the number of issuable instructions in the issue queue exceeds a preset threshold, then fetch is gated during one cycle. Otherwise, fetch is active. We use an issue queue fill threshold of 48 instructions. The energy-efficiency of a flow-rate matching technique (e.g. PAUTI or SIWW) is amplified by dynamically adapting the issue queue size [4]. The issue queue is scaled using a method we refer to as ADQ [5]. The usable issue queue size is determined at the beginning of a time quantum (e.g. 10000 cycles), depending on the average issue queue utilization during the previous quantum. If the average occupancy is less than the usable queue size minus 12, then the usable queue size is reduced by 8. If the average occupancy exceeds the usable queue size during the last quantum minus 8, then the usable queue size for the next quantum is increased by 8. The thresholds are chosen such that reducing the issue queue size further would cause an unproportionally large slowdown. Figure 10 shows slowdown, extra fetch work, average reorder buffer size reduction (down from 96 instructions) and total energy reduction for the baseline processor, fetch gating with PAUTI, SIWW (threshold 160) and SIWW with dynamic weight contributions (threshold 80). We show each fetch gating technique with and without issue queue adaptation (ADQ). We selected configurations that limit slowdown to about 1% on average. Slowdown is usually larger with ADQ than without. Otherwise, PAUTI incurs the largest slowdown for some benchmarks while the SIWW schemes incur the larger slowdown for others. SIWW Removes more Extra Fetch Work. Due to the use of confidence estimates, the SIWW schemes provide a stronger reduction of fetch extra work compared to the PAUTI and ADQ schemes (Figure 10). As was established above, the SIWW schemes almost remove half of the fetch extra work by themselves, but by adding issue queue adaptation, fetch extra work is reduced by more than half. SIWW Enhances Dynamic Issue Queue Scaling. The ADQ scheme by itself reduces issue queue size by 19.6% on average, but fails to scale the queue for some benchmarks. SIWW co-operates symbiotically with dynamic issue queue scaling as the reduced front-end flow-rate allows to reduce issue queue size by 25.9% and 28.8% on average for the fixed and dynamic weights, respectively.

146

H. Vandierendonck and A. Seznec 2.5

Slowdown (%)

2.0 1.5 base PAUTI SIWW+CE SIWW+CE+DWC ADQ PAUTI+ADQ SIWW+CE+ADQ SIWW+CE+DWC+ADQ

1.0 0.5 0.0 -0.5 0

2

4

6

8

10

Energy Savings (%)

Fig. 11. Energy reduction vs. slowdown for several configurations of each scheme. The issue queue fill thresholds for PAUTI are 40, 48, 56 and 64. The SIWW thresholds are 128, 160, 192 and 224 with fixed weights and 72, 80, 96, 112, 128 and 160 with dynamically adapted weights.

PAUTI allows to reduce the issue queue size by 29.8% on average, which is only slightly more than the SIWW schemes. PAUTI outperforms SIWW only on benchmarks with highly predictable control flow (crafty, gap and the floatingpoint benchmarks). SIWW Is more Energy-Efficient than PAUTI. The last graph in Figure 10 shows the total energy savings. We have shown that PAUTI and SIWW achieve their energy savings mostly in different areas (fetch vs. issue stage), so the total energy savings depend on how much the front-end and the issue queue contribute to total energy. For the architecture modeled by sim-flex, it turns out that total energy savings average out to the same values for PAUTI and SIWW with fixed weight contributions (4.88% and 4.94%, respectively). SIWW with dynamic weight contributions obtains a significantly higher energy reduction (6.5% of total energy) because it removes more fetch extra work than SIWW with fixed weight contributions and it allows for almost the same reduction in the issue queue size as PAUTI. A different trade-off between slowdown and energy consumption is obtained depending on the configuration of the fetch gating scheme (issue queue fill threshold for PAUTI or SIWW threshold). Figure 11 shows that, regardless of the configuration, the SIWW methods achieve higher energy savings for the same slowdown.

6

Conclusion

Fetch gating improves power-efficiency because of (i) eliminating energy consumption on wrong-path instructions and (ii) matching the front-end instruction rate to the back-end instruction rate. Previous proposals for wrong-path fetch gating relied only on branch confidence estimation, i.e. counting the number of inflight low-confidence branches.

Fetch Gating Control through Speculative Instruction Window Weighting

147

These proposals were not taking into account the structure of the remainder of the speculative instruction window (number of instructions, number of inflight high-confidence branches, . . . ). SIWW takes this structure into account and therefore allows more accurate decisions for fetch gating. Fetch gating control through SIWW allows to reduce extra work on the wrong path in a more dramatic fashion than fetch gating through confidence boosting and throttling. Fetch gating mechanisms have been proposed that focus on matching the front-end and back-end instruction flow-rates, neglecting to filter out wrong-path instructions. The SIWW method combines both: by weighting control transfers heavily, wrong-path instructions are gated-off and the front-end flow rate is limited during phases with many hard-to-predict control-transfers. Future directions for research on SIWW include new usages of SIWW, e.g., optimizing thread usage in SMT processors. We have shown that SIWW limits resource usage by wrong-path instructions, which is very important for SMT processors [16]. Furthermore, by setting a different SIW threshold per thread, different priorities can be assigned to each thread.

Acknowledgements Hans Vandierendonck is a Post-doctoral Research Fellow with the Fund for Scientific Research-Flanders (FWO-Flanders). Part of this research was performed while Hans Vandierendonck was at IRISA, funded by FWO-Flanders. Andr´e Seznec was partially supported by an Intel research grant and an Intel research equipment donation.

References 1. Akkary, H., Srinivasan, S.T., Koltur, R., Patil, Y., Refaai, W.: Perceptron-based branch confidence estimation. In: HPCA-X: Proceedings of the 10th international symposium on high-performance computer architecture, pp. 265–275 (February 2004) 2. Arag´ on, J.L., Gonz´ alez, J., Gonz´ alez, A.: Power-aware control speculation through selective throttling. In: HPCA-9: Proceedings of the 9th international symposium on high-performance computer architecture, pp. 103–112 (February 2003) 3. Baniasadi, A., Moshovos, A.: Instruction flow-based front-end throttling for poweraware high-performance processors. In: ISLPED 2001: Proceedings of the 2001 international symposium on low power electronics and design, pp. 16–21 (August 2001) 4. Buyuktosunoglu, A., Karkhanis, T., Albonesi, D.H., Bose, P.: Energy efficient coadaptive instruction fetch and issue. In: ISCA 2003: Proceedings of the 30th Annual International Symposium on Computer Architecture, pp. 147–156 (June 2003) 5. Buyuktosunoglu, A., Schuster, S.E., Brooks, M.D., Bose, P., Cook, P.W., Albonesi, D.H.: A circuit level implementation of an adaptive issue queue for power-aware microprocessors. In: Proceedings of the 11th Great Lakes Symposium on VLSI, pp. 73–78 (March 2001)

148

H. Vandierendonck and A. Seznec

6. Driesen, K., Holzle, U.: The cascaded predictor: Economical and adaptive branch target prediction. In: Proceeding of the 30th Symposium on Microarchitecture (December 1998) 7. Folegnani, D., Gonz´ alez, A.: Energy-effective issue logic. In: Proceedings of the 28th Annual International Symposium on Computer Architecture, pp. 230–239 (June 2001) 8. Gao, H., Zhou, H.: Adaptive information processing: An effective way to improve perceptron predictors. 1st Journal of Instruction-Level Parallelism Championship Branch Prediction, 4 pages (December 2004) 9. Grunwald, D., Klauser, A., Manne, S., Pleszkun, A.: Confidence estimation for speculation control. In: ISCA 1998: Proceedings of the 25th annual international symposium on Computer architecture, pp. 122–131 (June 1998) 10. Hinton, G., Sager, D., Upton, M., Boggs, D., Carmean, D., Kyker, A., Roussel, P.: The microarchitecture of the Pentium 4 processor. Intel Technology Journal 5(1) (2001) 11. Jacobsen, E., Rotenberg, E., Smith, J.: Assigning confidence to conditional branch predictions. In: MICRO 29: Proceedings of the 29th Annual ACM/IEEE International Conference on Microarchitecture, pp. 142–152 (December 1996) 12. Jim´enez, D.: Piecewise linear branch prediction. In: ISCA 2005: Proceedings of the 32nd Annual International Symposium on Computer Architecture, pp. 382–393 (June 2005) 13. Jim´enez, D.A., Lin, C.: Composite confidence estimators for enhanced speculation control. Technical Report TR-02-14, Dept. of Computer Sciences, The University of Texas at Austin (January 2002) 14. Karkhanis, T., Smith, J., Bose, P.: Saving energy with just in time instruction delivery. In: Intl. Symposium on Low Power Electronics and Design, pp. 178–183 (August 2002) 15. Lee, C.J., Kim, H., Mutlu, O., Patt, Y.: A performance-aware speculation control technique using wrong path usefulness prediction. Technical Report TR-HPS-2006010, The University of Texas at Austin (December 2006) 16. Luo, K., Franklin, M., Mukherjee, S.S., Seznec, A.: Boosting SMT performance by speculation control. In: Proceedings of the 15th International Parallel & Distributed Processing Symposium (IPDPS 2001) (April 2001) 17. Manne, S., Klauser, A., Grunwald, D.: Pipeline gating: speculation control for energy reduction. In: ISCA 1998: Proceedings of the 25th Annual International Symposium on Computer Architecture, pp. 132–141 (June 1998) 18. Parikh, D., Skadron, K., Zhang, Y., Barcella, M., Stan, M.R.: Power issues related to branch prediction. In: HPCA-8: Proceedings of the 8th International Symposium on High-Performance Computer Architecture, pp. 233–246 (February 2002) 19. Seznec, A.: Analysis of the O-GEometric History Length branch predictor. In: ISCA 2005: Proceedings of the 32nd Annual International Symposium on Computer Architecture, pp. 394–405 (June 2005) 20. Seznec, A., Michaud, P.: A case for (partially) TAgged GEometric history length branch prediction. Journal of Instruction-Level Parallelism (February 2006) 21. Vandierendonck, H., Seznec, A.: Fetch gating control through speculative instruction window weighting. In: De Bosschere, K., Kaeli, D., Stenstr¨ om, P., Whalley, D., Ungerer, T. (eds.) HiPEAC 2007. LNCS, vol. 4367, pp. 120–135. Springer, Heidelberg (2007)

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers* Minwook Ahn and Yunheung Paek** Center for SoC Design Technology, Seoul National University, School of Electrical Engineering and Computer Science, Korea [email protected], [email protected]

Abstract. Many embedded processors have complex, irregular architectures resulting from the customization for the maximum performance and energy efficiency of target applications. One such example is the heterogeneous register architecture, which has fast, small-sized register files, for their specific uses, distributed over the data paths between different functional units. Although this architectural design may be good at achieving the H/W design goal of high speed, small area and low power, it requires highly expensive algorithms for optimal code generation. This is primarily because multiple registers contained in each file come with many different constraints subject to their design purposes, and often their names are aliased with each other; thus the final code quality is very sensitive to how properly such aliased, heterogeneous registers are utilized in every instruction. In this work, we propose a code generation approach to attack this complex problem. The experiments reveal that our approach is fast, practically running in polynomial time. In comparison with the related work, it achieves approximately 13% of code size reduction and 16% of speed increase. Keywords: compiler, heterogeneous register architecture, code generation, register allocation, register aliasing, register coalescing.

1 Introduction Strict performance requirements in an embedded processor often compel its architecture designers to distribute the registers into multiple register files which are scattered, for their specific purposes, over the datapaths between different functional units, thereby often forming complex, irregular datapaths in the processor. This irregular architecture with multiple register files, called the heterogeneous register architecture (HRA) [3], helps to improve the performance since it generally reduces chip size and enables fast access to the value stored in each file. Unfortunately, this improvement comes at a cost: an HRA usually has many peculiar hardware features [9] that forcefully correlate the code generation subtasks, which have traditionally long been processed in separate, *

An extension of [1] with more detailed description of our register allocation and coalescing algorithms with extended experimental results to show our effects on code generation. ** Corresponding author. P. Stenström (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 149–172, 2009. © Springer-Verlag Berlin Heidelberg 2009

150

M. Ahn and Y. Paek

sequential phases. In particular, the HRA creates a strong relationship between instruction selection and register allocation. Thus, when we select an instruction, we should be able to simultaneously decide from which register files the instruction uses registers for its operands. Typically, this correlation impacts compilers detrimentally because they need to use more expensive and sophisticated algorithms in order to solve the related subtasks at the same time. To tackle this issue, phase-coupling, a technique to cleverly couple these related phases, has been the norm to most compilers for HRAs. For the past decade, diverse forms of phase-coupling have been proposed for HRAs [3][4][8][14]. Due to the intractable complexity of phase coupling, they mostly employ exponential-time algorithms, hence exposing themselves to a high possibility of consuming an insurmountable amount of compilation time for large applications. Therefore, there have been several efforts made to develop faster algorithms. A noticeable example is SPAM[3], The SPAM code generator consists of two phases: local code generation and global register allocation. For local code generation, it uses a polynomial-time algorithm based on the dynamic programming method in [11], where a register file is determined for each operand of an instruction at the same time the instruction is selected. Fixing one register file for each operand greatly simplifies the register allocation problem. If the target is an HRA with only one register in each register file, this early binding of physical files to instructions can come along with the best register utilization since all register files have only one register. However, in the case of general HRAs where each register file has more than one register, such decision often causes us to lose some efficiency in code generation. For instance, consider Fig. 1 where we show a code generated for DSP563xx from FreeScale Co. as a typical example of general HRAs. Like in Fig. 2(a), it has two register files in ALU: FAB={A,B} and FXY={X0,X1,Y0,Y1}. The two versions of DSP563xx code in Fig. 1(b) and Fig. 1(c) are translated from the same C code in Fig. 1(a). They have three basic blocks. For clarity of our explanation, let fi, 1 ≤i ≤3, denote the i-th block from the top of the code in Fig. 1(b) and pi denote that in Fig. 1(c). In fact, there is one-toone correspondence between fi and pi.

Fig. 1. C source code example and its two different versions of target code produced

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

151

Fig. 2. DSP563xx core data-path and its partial ISA

The first version in Fig. 1(b) is obtained by SPAM. As can be expected, all three blocks f1, f2 and f3 are locally optimal. Notice that f2 uses two registers A and B. This is because SPAM early binds the physical register file FAB(i.e., A or B) to the instructions in f2 for optimal local code generation. Therefore, later in their global register allocation phase, SPAM cannot use registers from FXY(i.e., X0, X1, Y0 or Y1) in the final code. In contrast, it can be designed not to choose between FAB and FXY for the example in Fig. 1 during its local code generation. Instead, it will bind a logical register set FABXY={A,B,X0,X1,Y0,Y1} to the instructions in p2. As a result, our global register allocator later can have more freedom to use register resources in p2. Notice that p2 uses only B. By saving the use of register A in p2 as shown in Fig. 1(c), we will be able to better use the register in the other blocks, consequently allowing us to avoid extra spills unlike SPAM in Fig. 1(b). To show this, observe the global usage of registers A and B across all the blocks in the code. In both f1 and p1, A is allocated for variable t3 in Fig. 1(a) since t3 stores the result of mpy (see Fig. 1(b) and (c)). Note that the two extra instructions for spill operations just before and after f2 are mandatory since A is used inside f2. This example motivated us to develop a new code generation approach as an alternative to SPAM. In this approach, we couple more loosely instruction selection and register allocation in the first phase to the extent that we can provide the global register allocator with enough freedom to choose best registers. For this, the idea of logical register sets is essential in our work. Also, as backed by the analysis of [12], we deem that SPAM would not properly handle the registers in the HRAs since its register allocator is based on the conventional graph coloring algorithm[5][17]. Thus, in our work, we have adopted the register allocation algorithm from [12] that can deal efficiently with those registers. Particularly in this work, we have tested diverse register coalescing schemes that are essential to remove excessive data moves in the HRA during register allocation, and identified the best scheme specifically for our approach. In Section 2, as a previous work, we will explain the code generation method of SPAM compiler in detail. Our approach mainly differs from SPAM’s in that we

152

M. Ahn and Y. Paek

delay the early binding of physical register files to instructions until the second phase. Instead in our first phase, we assign logical register sets, which we formally define in Section 3. In Sections 4 and 5, we respectively describe each phase of our code generation approach based on these logical registers. In Section 6, we introduce a new register coalescing scheme that helps us to remove redundant moves during register allocation. In Section 7, we experimentally measure the performance of our code generator on a commercial HRA. The experiments show that we outperform SPAM roughly by 13% in terms of code size and by 16% in terms of code speed, yet achieving comparably as fast compilation times as SPAM for all programs.

2 Related Work In this section, we describe the SPAM code generation approach as our related work. In their work, HRAs are classified by the number of registers in a register file and the number of storage cells in their memory. They say an HRA is of the [n,∞] form to mean that the machine has n registers in each file and an (practically) infinite number of storage cells in the memory. The simplest form of [n,∞] HRAs is defined when n=1. They claim that for this limited form of HRAs, their code generator virtually fully-couples instruction selection and register allocation in a single phase, yet running in polynomial time. This is mainly because their global register allocation problem becomes trivial for this architecture; that is, when SPAM designates a register file for each instruction in its local code generation, the physical register is in fact already allocated to the instruction in the same phase. The SPAM approach is workable for very early HRAs like TI TMS320C25, which followed the [1,∞] form. But, with the advance of IC technologies, the more transistors squeezed onto a chip, the more registers available in a processor. As a result, more recent off-the shelf HRAs, such as DSP16xx, ADSP21xx and DSP563xx, follow the [n,∞] form with n > 1. This general [n,∞] HRA form promises better performance than its [1,∞] sibling. It has multiple registers per register file, which thus may provide the compiler better chance to utilize registers in the code generation. But, the architecture substantially complicates the code generation problem because when the compiler selects instructions, it must choose not only the best register file for each instruction but also the best register among the multiple candidates in the same file assigned to the instruction. Since the full description about SPAM compiler is available in [3], we will highlight only its essential concepts needed to explain our algorithm in the subsequent sections. 2.1 Target [n,∞] HRA with Register Aliasing Fig. 2(a) shows the ALU datapath of DSP563xx from FreeScale Co. as a typical [n,∞] HRA. The processor has two register files in ALU: FAB={A,B} and FXY={X,Y}. As illustrated in Fig. 2(b), the instruction ‘add’ has two operands D and S, indicating that it can take as its destination D one register from FAB (i.e., A or B), and as its source S one from FAB or FXY (i.e., A, B, X, or Y). It commonly occurs in a [n,∞] HRA that registers are not only heterogeneous but also aliased[12]. Two registers are called aliases if they have different names but

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

153

share the whole or a part of the same physical register. For instance, Fig. 2(a) shows that the double-word registers X and Y are divided into two sub-registers, each of which can be individually referenced as an operand. So, the same physical register X is accessible via three different names X0, X1 and X. In Fig. 3, we summarize the register aliasing relation of DSP563xx. Obviously, the register allocation problem would become more complicated for [n,∞] HRAs in the presence of register aliasing since the compiler should be aware of the aliased relation of registers in order to avoid assigning two aliased registers respectively to two different variables with overlapping lifetimes.

Fig. 3. Alias relation of DSP563xx registers

2.2 SPAM Code Generation Fig. 4 displays an AST example and the DSP563xx code generated from it. Each operation node v in the tree is annotated with a cost list Cv: Cv = (Cv(1),Cv(2),…,Cv(n)), where every field Cv(l), 1 ≤ l ≤ n, records the minimal cost that is paid to store the computation result of v into the l-th location of storage (i.e., memory or register). The example shows that the cost list has three fields (m,r,a) for DSP563xx where m stands for memory, and r for input registers and a for accumulators. SPAM provides multiple register fields for instruction selection to separately handle heterogeneous registers because heterogeneous registers are dedicated to different groups of instructions, as in the case of DSP563xx. In this example, two register fields r and a are provided to represent two ALU register files FXY and FAB, respectively. SPAM uses a two-step dynamic programming for code generation. In the first bottom-up step, the compiler collects the computation cost Cv of every node v from the leaves to the root. Each field Cv(l) in the cost list records the minimal cost that must be paid to store the result of its node into its location l (a register or memory in the target architecture). Since for every child node vj of v, Cv has already been recursively calculated bottom-up, the cost Cv (l) at the location l of vj will be available before evaluating cv(i), which is the minimal cost to compute an instruction i matched on v. HRAs often come with irregular datapaths where move instructions are required for the value transfer between heterogeneous registers, so we consider the transfer cost t(d,l) from the destination d of i to l for Cv(l). To explain this, suppose that register files F1 and F2 are dedicated to instructions i1 and i2, respectively. Then, even if i1 need take as its source operand the result value of i2, it cannot read the value directly from the destination register of i2, thus necessitating an extra transfer of the data from j

j

154

M. Ahn and Y. Paek

F2 to F1. This irregular structure renders the exact computation of t(d,l) a bit challenging since it requires a careful routing from one register to another; this means that heedless routing would cause the contents of some live registers to be excessively spilled in the routing path. Therefore Cv(l) is computed by evaluating cv() and t() for every instruction i matched to v: C v (l ) =

min

∀ i matched to v

(c v ( i ) + t ( d ( i ), l ) )

(1)

Provided that the computation result of i is stored into its destination operand d(i), t(d(i),l) returns the cost to transfer the data from d(i) to the l-th storage location in Cv. In order to evaluate cv(i), we must first consider e(i), the actual execution time of i: c v ( i ) = e ( i ) + ∑ min C v j ( l ) + t ( l , s j ( i )) ∀v j

1≤ l ≤ n |

(

)

(2)

The transfer cost t(l,sj(i)) is needed to forward the data from the location l of node vj to the j-th source operand sj(i) of node v executing i.

Fig. 4. Example of an AST and its target code for DSP563xx

Fig. 4(a) shows the resulting cost lists for the AST after being computed using the above functions. Let us assume one cycle for a load, and t(l,l′) = 1 for l ≠ l′. When the calculation is complete, instructions are selected starting from the root. In the cost list of the root node 7, we see that two versions of sub instructions (I3-AB and I3-XY) are candidates which can be selected at the node. As for their source operands, I3-AB allows the physical register file FAB and I3-XY allows FXY. As you can see, it is optimal to store the result of I3-AB into location a (i.e., register file FAB). This is marked by the circle around C7(a) in Fig. 4(a). Again, this is marked by the circle around C6(a). Likewise, notice that all storage locations selected as the operands between nodes are circled.

3 Register Class In other work, the term register class has been frequently used [6][13][15][16]. Although the name is the same, its definition differs depending on its usage in each work. To avoid confusion with others, in this section, we formally define our notion of register classes. Our definition of register class is rather logical, as being classified according to their usages in the instruction set rather than physical layouts in the hardware.

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

155

Definition 1. Given a processor P, let I = {i1, i2, ..., in} be a set of all instructions defined on P, and R = {r1, r2, ..., rm} be a set of all its registers. For instruction ij I, we define a set of all its operands, op(ij) = {Oj1,Oj2, ...,Ojk}. Assume φl(ij) is a set of all the registers that can appear at the position of some operand Ojl, 1 ≤ l ≤ k. Then we say that φl(ij) forms a register class for ij.

∈

By Definition 1, any member in a register class for a given instruction can be referenced interchangeably as an operand of the instruction. For instance, the ARM machine has an instruction, add reg1,reg2,reg3, where any of its 16 registers (r0,r1,…,r15) can appear as any operand in the instruction; that is, φ1(add) = φ2(add) = φ3(add) = {r0,r1,…,r15}. So, this set forms a single register class for add according to Definition 1. As another example, consider the instruction mpy s1,s2,d of DSP563xx, which multiplies the first two sources and places the product in the destination. DSP563xx restricts s1 and s2 to be four registers X0, X1, Y0 or Y1, and d to be a subregister of A or B (i.e., A1 or B1). Consequently, as can seen in Fig. 2(b), two register classes XY and AB1 (see Fig. 5) are dedicated to instruction I5(multiplication instruction), respectively at s1/s2 and at d. From this, we see that unlike in the case of add, not all registers in DSP563xx are equally usable by instruction I5. Definition 2. Using Definition 1, we define Φ(i), a collection of distinct register classes for instruction i as follows: k

Φ (i ) = U l =1{φl (i )}.

We say that two instructions i and j have disjoint register classes if Φ(i) ∩ Φ(j) = ∅. For processor P with instruction set I = {i1,i2,..,in}, we define the whole collection of register classes, denoted by ΦP, as follows: ΦP = U

n j =1

Φ(i j )

By Definition 2, we have Φ(add) = {{r0,…,r15}} for instruction add in ARM. Virtually for all other ARM ALU instructions i, we will also have Φ(i) = Φ(add). This equivalently means that ΦARM contains only one register class consisting of the whole 16 registers. In the case of DSP563xx, however, registers can usually be assigned differently to the machine instructions. For example, recall that even a single instruction I5(multiplication instruction) in Fig. 2(b) has two different sets of registers. In fact, the subset of ΦDSP563xx is listed in Fig. 5, from which we can recognize that the register architecture of DSP563xx is heterogeneous.

Fig. 5. Register Classes of DSP563xx

156

M. Ahn and Y. Paek

4 Instruction Selection and Register Class Allocation In the fist phase for local code generation, SPAM handles heterogeneous registers by directly managing each individual register file. But, as seen in Section 1, this integration of physical files inside instruction selection limits the global utilization of registers during the global register allocation phase for [n,∞] HRAs. To avoid this loss of efficiency, we relax this tight relation in our approach by allocating all registers logically first (i.e., regardless of their physical layouts or file structures) through register classes during instruction selection. Thereby, we can cut the physical bindings between instructions and registers in the first phase of our code generation. For this, we first modify the original design of SPAM for local code generation such that in this phase, we bind each operand of the instructions with its register class, postponing the decision on specific register and register files until the next phase where a register allocator can determine globally the most appropriate register file and its specific register for every operand. In this section, we will first describe our implementation of the first phase for local code generation. Our first phase uses the same dynamic programming algorithm as SPAM. However, to reflect the above-stated modification, we use different cost functions. First, the function in Equation 1 is redefined with register classes instead of register files: C v (l ) =

min

∀ i matched

to v

(c v ( i ) + t (φ d ( i ), l ) )

(3)

where φd(i) represents the register class used as the destination operand of instruction i. Likewise, Equation 2 is refined as: cv (i) = e(i ) + ∑ min C v j (l ) + t (l , φ s j (i)) ∀v j

1≤ l ≤ n|

(

)

(4)

where φs (i) represents the register class used as the j-th source operand of instruction i. Equations 3 and 4 provide the new basis for our local code generation in which the original tight integration of instruction selection and register file allocation now become partially decoupled via register classes. To accommodate the changes in these equations, we extend the original cost list for node v of Fig. 4 into the list with four fields: j

Cv = (m,XY,AB1XY,AB1). As shown in Fig. 6, we recompute the cost lists for the AST in Fig. 4(a) with these extended fields. As can be seen in the figure, the changes from Fig. 4 are not only the number of cost fields, but also the way the field values are evaluated. In both Equations 1 and 3, to evaluate the field value Cv(l), we need to compute the transfer cost t. In Equation 1, it is somewhat simple since the cost to transfer data between two register files is usually fixed (probably available from the ISA user’s manual). In Equation 3, however, it requires more discretion since the transfer cost between logical register classes is not always straightforward. For instance, suppose for instructions i and j that φs(i) and φd(j) denote two register classes respectively for i’s source operand and j’s destination operand. Now, if i reads the data from j through the class φs(i), then the result stored in a register of φd(j) must be transferred to the one of φs(i). When φs(i) and φd(j) are identical or mutually disjoint, t would evaluate simply to:

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

157

t(φd(j),φs(i)) = 0 if φs(i) = φd(j), and t(φd(j),φs(i)) = 1 if φs(i) ∩ φd(j) = ∅, where we assume for simplicity that the latency of all transfers between storage locations is 1 cycle, as in Section 2.2. For the other cases, the exact cost cannot be known before actual registers are allocated, which is happening later in the second phase. That is, if the same register r ∈ φs(i) ∩ φd(j) is allocated for i and j, then t = 0. On the other hand, if the two instructions have different registers, we need an extra cycle for data transfer. Thus, to handle this issue, we use a probabilistic approach where we predict the possibility of the actual data transfer among the two register classes after register allocation: 1 ⎛ | φ ( j ) ∩ φ s (i) | | φ d ( j ) ∩ φ s (i) | ⎞ ⎟⎟ t (φ d ( j ), φ s (i)) = 1 − × ⎜⎜ d + 2 ⎝ | φd ( j) | | φ s (i) | ⎠

(5)

Notice that the above two boundary cases are in fact special cases of Equation 5. For example, if two classes are identical, the transfer cost become zero since their sizes are equal to the size of their intersection, that is, |φs(i)| = |φd(j)| = |φs(i) ∩ φd(j)|. Equation 5 implies that the transfer cost will decrease as the two register classes have more physical registers in common. This would be true for most cases because the more registers both storage locations have in common, the more likely they are allocated to the same register, thereby eliminating the need for an extra data move. We apply Equation 5 to compute all transfer costs between the memory and seven register classes in Fig. 5, and present them in Fig. 7.

Fig. 6. The AST from Fig. 4 with new cost lists Fig. 7. Transfer costs bet’n all storage locations including memory and register classes in Fig. 5

Using Fig. 7, we will show an example how a cost list is computed in the first pass of our code generation. Let ΦR denote a subset of ΦDSP563xx such that ΦR = {XY,AB1XY,AB1}. Now, consider again node 6 in Fig. 6. In a similar way we did in Section 2.2, we first compute the two nodes 4 and 5 from which node 6 takes its source operands as follows: C4(m) = C5(m) = 0, and C4(l) = C5(l) = 1 for l ∈ ΦR. Since node 6 is matched to instruction mpy, its source operands come from register class XY and send its result to register class AB1. Using the cost list values of its child nodes, node 6 now obtains its cost list as follows:

158

M. Ahn and Y. Paek c6 ( mpy) = e( mpy) + minR (C4 (l ) + t (l , XY) ) + minR (C5 (l ) + t (l , XY) ) l∈Φ

C6 (l ) = c6 ( mpy ) + t ( AB1, l )

l ∈Φ

for l ∈ {m} ∪ Φ R

where e(mpy) = 1 as stated earlier. From the table in Fig. 7, we can easily see that c6(mpy) = 1+1+1 = 3. Using this cost result, we can compute every field cost of C6: C 6 ( m) = C 6 ( XY ) = c 6 ( mpy) + 1 = 4 C 6 ( AB1XY) = c 6 ( mpy) + 0.33 = 3.33 C 6 ( AB1) = c 6 ( mpy) = 3

Notice that C3 = C6 since their nodes both are matched to mpy and read source operands from the leaves. These cost lists are in turn used to compute C7 at the root as follows: c7 (sub) = e(sub) + minR (C3 (l ) + t (l , AB1) ) + minR (C6 (l ) + t (l , AB1XY) ) l∈Φ

C7 (l ) = c7 (sub) + t ( AB1, l )

l∈Φ

for l ∈ {m} ∪ Φ R

Fig. 2(b) shows that instruction sub reads the first source operand from both register files FXY and FAB. So, register class AB1XY is dedicated to the first operand. The second source and the destination share the same file FAB. Therefore, register class AB1 is dedicated to the operands. Since C6 and C3 are symmetric, we arbitrarily choose to read the first source operand from C6 and the second from C3 as shown above. The following shows the computation of each field in C7: C7 (m) = C7 (XY) = c7 (sub) + 1 = 8.33 C7 (AB1XY) = c7 (sub) + 0.33 = 7.66 C7 ( AB1) = c7 (sub) = 7.33

From this cost list at the root node, we now traverse the AST down to select an optimal set of instructions just as we did in the SPAM code generation. However, unlike the SPAM approach, we do not fix a physical register file for each instruction. Fig. 8 shows the resulting code generated from the C code in Fig. 1 after our local code generation. It also shows the output code for the AST in Fig. 6 encircled with a dotted line. Notice that the operands of all instructions are bound to register classes. For

Fig. 8. Intermediate target code for the C code in Fig. 1(a), produced after local code generation

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

159

instance, the instruction, move t3(AB1),x:y, means that the temporary value t3 currently residing in one of the registers in class AB1 moves to the memory location y in the xmemory of DSP563xx. Our global register allocator in the second pass will determine which one of the registers in class AB1 should store the value t3 before the move instruction is executed. In our code generation, due to the lack of detailed decision on actual register files from each class in the first pass, we often have many move operations between register classes. For instance, consider the instruction in Fig. 8: move t1(AB1),t1’(AB1XY). Although the classes AB1XY and AB1 share the same registers A1 and B1, they are assumed to contain different registers. This conservative assumption creates a move to explicitly transfer the data between the classes. In Section 0, we will decide whether we can eliminate most of such moves or not in the subsequent pass.

5 Physical Register Allocation Smith et. al. [12] proposed an extended algorithm of Chaitin’s graph-coloring to cope with aliased, heterogeneous registers in register allocation. To implement the register allocator in our work, we have borrowed most of their formulation with some extra techniques for coupling it with our code generator in the first phase. In our register allocation for HRAs, we allow a colorability for each node, which is different from the k colorability in the Chaitin’s algorithm. For instance, if φx denote a register class bound to variable x, we see from Fig. 9 that φt3 contains two registers. This implies that node t3 is 2-colorable. On the other hand, node t1 is 4-colorable since φt1 has four registers. Also in ordinary k-coloring, every node must share the k resources of the same types, which is not always true for HRAs. As an example, notice that the colors of t1 are {A,B,X,Y} while the colors of t4 are {A1,B1,X0,X1,Y0,Y1}. Another complexity imposed by HRAs on register allocation is that each node has different impacts on the colorability of its neighbors in the interference graph (IG) [12]. For instance, suppose that node t1 is assigned register X. Then this coloring decision will strip two colors off node t4 since t4 can be no longer assigned register X0 or X1. However, at the same time, it strips no color off node t3.

. Fig. 9. Interference graph and the calculation of squeeze

Based on all these observations on the complexities of register allocation for HRAs, Smith et al [12] introduce a new measure, called the squeeze, that supplements the notion of k-colorability in graph coloring. To describe the register allocation algorithm with the squeeze, suppose that there is a variable node x in an interference graph G and a register class φx is bound to x. Then, the squeeze for x, denoted by σ*x, is defined as the maximum number of registers from φx that cannot be allocated to x due

160

M. Ahn and Y. Paek

to an assignment of registers to all its neighbors. To formally define σ*x, assume that φx has k registers, implying that x is initially k-colorable. Now, suppose that a neighbor y of x is assigned a register r from its class φy. Then, if r aliases with k′ registers in φx, the initial colorability of x will reduce by k′ due to its neighbor y because by the definition of alias, the k′ registers could not be allocated to x anymore. As a result, x would become at most (k-k′)-colorable after r is allocated to y. This can be summarized as follows: Reduction of colorability of x due to y = k′ = |A(r) ∩ φx|, where A(r) denotes the set of all aliases of r. Likewise, the remaining neighbors are all assigned registers from their classes, forming a set of registers assigned to x’s neighbors, say R, as a result. Now, let k″ be the total amount of reduction of x’s colorability due to all its neighbors. Then, obviously we have: k" = A ( R ) ∩ φ x

for A ( R ) =

∪

A(r ) .

∀ r∈ R

For every coloring of x’s neighbors, we would have many different configurations for R. The squeeze σ*x is defined for the set ℜ of all colorings of x’s neighbors as follows: σ

* x

= max

(A(R ) ∩

φ

x

).

(6 )

∀ R∈ℜ

From the result, we conclude that node x would always be at least (k-σ*x)-colorable with any coloring of its neighbors. This finding induces a heuristic that x can be removed immediately from G as long as k > σ*x since it is then trivially colorable under any circumstance. For instance, to compute the squeeze σ*t4 for node t4 in Fig. 9, we identify all possible colorings R* of its neighbor t1. From Fig. 5, we obtain four possible colorings: ℜ = {{A},{B},{X},{Y}}. Then, from Fig. 3, we find the aliases of each coloring as follows: A(A) = {A,A0,A1}, A(B) = {B,B0,B1}, A(X) = {X,X0,X1} and A(Y) = {Y,Y0,Y1}. Using this information, we compute for φt4 = {A1,,B1,X0,X1,Y0,Y1}:

σ*t4 = max(|A(A) ∩ φt4|, |A(B) ∩ φt4|, |A(X) ∩ φt4|, |A(Y) ∩ φt4|) = max(1, 1, 2, 2) = 2. Node t4 is initially 6-colorable since |φt4| = k = 6. As a result, we have k – σ*t4 = 4. This means that even after t1 is assigned any register from its class φt1, t4 would remain at least 4-colorable. Here, we therefore safely remove node t4 from the IG of Fig. 9. In the same manner, we identify the remaining nodes are trivially colorable, hence removing all from the graph. Similar to the degree-less-than-k heuristic[5], this new squeeze-based heuristic is also effective to simplify the register allocation for HRAs by precluding many trivially-colorable nodes.[12] Unfortunately, finding an ideal squeeze σ* is virtually impossible for a large IG because the set ℜ has exponential size complexity in the number of nodes. For instance, in Fig. 9, to compute σ*t1 for node t1, we should consider 24 (=2×2×6) cases of colorings for its three neighbors even for such a small graph. So, in practice, an approximated squeeze σ [12] is sought to prevent the compilation time from increasing dramatically. To compute σx for node x in a graph G, assume the followings:

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

161

1. All register classes bound to x’s neighbors are classified into m distinct register classes πi, 1 ≤ i ≤ m. 2. ni = |{y | y ∈ N ∧ φy = πi}| where N is the set of x’s neighbors. Then, we define σx: σx =

∑

∀π i ,1≤ i ≤ m

ni ⋅ max (| φ x ∩ A(r ) |) ∀r∈π i

(7 )

This formula guarantees that σ is a safe approximation of σ*. However, it is sometimes too conservative, thus hindering some trivially-colorable nodes from being removed from G. In Fig. 9 for example, σ*t1 is 3,implying that t1 is always at least 1-colorable since it is initially 4-colorable. But, σt1 evaluates to: σ t1 = max (| φt1 ∩ A( r ) |) + 3 ⋅ max (| φt1 ∩ A(r ) |) = 1 + 3 = 4 ∀r∈AB1XY

∀r∈AB1

This estimated value σt1 falsely informs us that t1 is not colorable. To bar σ from being overly estimated as in this case, other safeguards such as upper bounds are applied[12].

6 Eliminating Moves between Register Classes A main disadvantage of our strategy to assign register classes in the first phase is that it introduces many more move instructions in the code. This is because we have to regard any classes with different names to be also physically different; therefore, when two different register classes φx and φy are assigned respectively to two dependent variables x and y, we insert a move instruction to transfer the value in x to y, or vice versa. This conservative decision is clearly disadvantageous especially when a move is inserted between the two classes sharing almost identical registers. Fortunately, our experiment has exhibited that our register coalescing scheme was able to successfully eliminate most of these moves in the second phase when actual registers are assigned. In this section, we will first describe existing coalescing schemes, and show how we have altered one of the schemes for our code generation specifically targeting HRAs. 6.1 Existing Register Coalescing Schemes Register coalescing is a compiler technique that removes a move instruction by assigning the same register to its source and destination operands. Two most well-known schemes of coalescing are iterated coalescing[7] and optimistic coalescing[10]. Iterated coalescing employs a conservative coalescing scheme where two move-related nodes x and y in the IG are coalesced only when the original colorability of the IG is not affected by the coalescing. This can be ensured by testing if their coalesced node xy is colorable. As seen in Fig. 10(a), iterated coalescing repeatedly applies both the conservative coalescing and simplification phases. The simplification phase prunes the colorable, non-move-related nodes from the IG, and reduces the degree of move-related nodes, yielding more opportunities for coalescing during conservative coalescing. These two phases are repeated until there are left only noncolorable or move-related nodes in the IG. If neither phase can be applicable, colorable move-related nodes are frozen, which means giving up a further chance of its coalescing by removing its move-related edges and marking it as a non-move-related node. As frozen nodes are no more a move-related nodes, they may be pruned in the simplification phase.

162

M. Ahn and Y. Paek

Fig. 10. (a) Iterated Coalescing and (b) Optimistic Coalescing

Fig. 11. Examples to show (a) the positive and (b) the negative impact of coalescing

One critical drawback of iterated coalescing is that it gives up the chances of coalescing too early even when a coalescible node violating the colorability criterion is not necessarily spilled. If a coalesced node violates the criterion without being actually spilled, the decision for spill should be delayed, which provides room for further coalescing. Moreover, iterated coalescing is too conservative to enjoy the positive impact of coalescing. For instance, see the IGs in Fig. 11, where nodes a and c are move-related. Note that in Fig. 11(a), these nodes have a common neighbor b, while in Fig. 11(b), they has different neighbors b and d, respectively. As shown here, if moverelated nodes have a common neighbor, coalescing brings positive impact on coloring, but otherwise, it brings negative impact. In Fig. 11(a), see that the IG remains 2-colorable with fewer nodes after coalescing. In contrast, in Fig. 11(b), the IG turns from 2-colorable to 3-colorable. The aggressive coalescing scheme attempts to overcome this weakness of conservative coalescing by fully exploiting this positive impact of coalescing. The aggressive scheme usually colors the IG better than the conservative one. As shown in Fig. 10(b), optimistic coalescing is based on the aggressive scheme. We have seen that taking such advantage of aggressive coalescing, optimistic coalescing outperforms iterated coalescing for most cases. After merging all move-related nodes, optimistic coalescing prunes nodes from the IG if the nodes are trivially colored. If none of the nodes left in the IG are trivially colorable, the register allocator selects one as a candidate of spill, called a potential spill. The decision for a potential spill is based on the spill metric M(v) of every node v1 in the IG. The spill metric is calculated from the following equation: 1

A node in the IG represents a live range of a symbolic variable in the code. So we use the terms ‘node in the IG’, ‘symbolic variable’ and ‘live range’ interchangeably without clearly differentiating their meanings.

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

163

M (v ) = C (v ) / D ( v ) (8) where C(v) is the spill cost of v, and D(v) is the degree of v. The spill cost is computed as: C (v) = wd × ∑ 10 d ( def ) + wu × ∑ 10 d ( use ) def ∈v

use∈v

where wd is the relative weight of the definition for v, wu is the relative weight of the use for v, and d(x) is the depth of the loop nest at the location of x in the code. C(v) indicates how much the performance would be degraded if v is spilled. So does M(v) since by Equation 8, it is proportional to C(v). But notice that M(v) is inversely proportional to D(v) . This is due to the simple fact that the more interfering nodes a node has in the IG, the greater chance of spilling the node opens up for coloring other nodes. Thereupon, we select a node with the minimum spill metric for a potential spill. Notice from Fig. 10(b) that in optimistic coalescing, the decision of spills for coalesced nodes is delayed after aggressive coalescing. Although the negative impact of aggressive coalescing is negligible[10], the decision should be made with caution because the coalesced nodes tend to represent variables with long live ranges in the code; thus, once a coalesced node is actually spilled, it is very likely to trigger many other spills. This undesirable circumstance can be avoided by live range splitting[10], which splits a long live range into short ones by inserting move or load/store instructions. If the live range of a variable becomes shorter via live range splitting, we will more likely avoid spills since the shorter live range would less likely conflict with other ranges. Registers are allocated for the split live ranges of the variable. Normally, live range splitting corresponds to the undo of coalescing since it splits the coalesced node back to its original move-related nodes. For instance, see that the IGs in Fig. 11 are transformed to the new IGs by coalescing two nodes a and c into a node ac. Also see that the original IGs are restored after live range splitting that undoes the coalescing effect by splitting ac back into a and c. Note that coloring split nodes a and c is easier than coloring ac since their degrees are lower than or equal to that of ac. 6.2 Modified Optimistic Coalescing for HRAs The original coalescing schemes in Section 6.1 are designed to work with the traditional register allocation algorithm based on graph coloring like the one employed by SPAM. So, without further adjustments, they would not be directly applicable to our squeeze-based allocation algorithm for register classes. For instance, the original schemes use the strategy that move-related nodes are merged only when they are in the same register file. However, unlike register files, register classes are not always disjoint. So, if this strategy is applied to our algorithm, very few move-related nodes might be left for coalescing. Thus in [12], Smith et al have modified the original coalescing scheme for their squeeze-based register allocator such that move-related nodes in different register classes can be merged as long as the classes are not disjoint. Likewise, for our register allocator, we have also used a modified coalescing scheme. But, a main difference between these two modified schemes is that ours is based on optimistic coalescing while theirs is on iterated coalescing. Recall that in an earlier study[10] with the original schemes, optimistic coalescing has been proven generally more powerful than iterated coalescing. In this section, we will discuss two major changes made to modify the original optimistic coalescing scheme.

164

M. Ahn and Y. Paek

6.2.1 Modified Coalescing Criteria for Move-Related Nodes In our register allocation, if move-related nodes are bound to different register classes, then the register class of their coalesced node is to be the intersection of their register classes. To compute the intersection of two classes, we must know the relationship between the register classes defined in the target HRA. For fast reference, we store this relationship in a table, called the Register Class Relation Table (RCRT), as displayed in Fig. 12. This table represents the inclusion relationship between register classes. For instance, Fig. 12 indicates that the intersection of register classes AB1XY and AB1 is the class AB1. Besides the intersection of two classes, the table shows two special attributes d and a, where the former denotes that the two register classes are disjoint, and the latter denotes that each class of the two contains the registers whose aliased registers belong to the other class.

Fig. 12. RCRT for DSP563xx constructed from the register class information in Fig. 3 and Fig. 5

Fig. 13. An example of coalescing and live range splitting

Remind that in optimistic coalescing, all move-related nodes are aggressively coalesced before simplification. There are three cases in this decision of whether or not the given move-related nodes are coalescible. First, if their register classes are disjoint, they are not coalescible since there is no common register to allocate in the source and destination operands of the move instruction. We can check this immediately from the RCRT. Second, the move-related nodes whose register classes are aliased are not coalescible. Of course, among the aliased register classes, there must

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

165

be a physical register to be commonly allocated. But this register may not usually be assigned all the move-related nodes. For example, Fig. 12 indicates that two classes AB1 and ABL are aliased because the registers {A1,B1} in AB1 are respectively the upper halves of the registers {A,B} in ABL. Now, assume that the move-related nodes a and c in Fig. 11 are bound to these aliased classes, respectively. By Definition 1, node a cannot be assigned either A or B, and similarly, node c cannot be assigned either A1 or B1. Consequently, these move-related nodes should not be coalesced. The third case arises when there is an intersection of the register classes for the moverelated nodes. In this case, they will be coalesced, and the register class of their coalesced node will be the intersection of their classes[12]. We see in Fig. 13(a) that the move-related nodes b and f have different register classes. The register class for b is AB1XY and that for f is XY. The RCRT in Fig. 12 gives that the intersection of AB1XY and XY is XY. As a result, nodes b and f are coalesced and bound to the register class XY, creating the new IG in Fig. 13(b). 6.2.2 Modified Spill Metrics for Live Range Splitting Recall that our register allocator determines the colorability of node v by the value of its squeeze σv. As σv is the maximum number of registers from the register class φv of node v that could be denied to v due to an assignment of registers to the current neighbors of v, node v should be trivially colorable if σv < |φv|. Likewise, the decision for a potential spill is influenced by the degree of a node in the IG as well as the register class bound to the node. For instance in Fig. 13(b), after two nodes a and e were pruned from the IG, all remaining nodes (bf,c,d,g,h) cannot be pruned because there is no node whose squeeze is less than the size of its register class. Optimistic coalescing prefers a coalesced node to a non-coalesced one for a potential spill because spills can be minimized due to live range splitting[10]. Thus as shown in Fig. 13(b), the coalesced node bf will be selected for a potential spill and pruned from the IG. Now, the remaining four nodes (c,d,g,h) can be pruned. As we see from Fig. 13(c), the pruned nodes are piled inside a stack. When being popped from the stack later for coloring, each node is assigned a register among the registers in the register class bound to the node. Assume that as the nodes (c,d,g,h) are first popped (see Fig. 13(d)), they are assigned the registers {X0,X1,Y0,Y1}, respectively. Then, the coalesced node bf would have no register available for allocation since its squeeze σbf (= 4) is identical to the size of the register class |φXY| (= 4) bound to bf. To escape from this dead end, we follow the original scheme of optimistic coalescing in Section 6.1; that is, we try a second chance for the coloring of bf by restoring the original IG of Fig. 13(a) via live range splitting. Now, nodes b and f in the IG are colorable since both σb and σf are less than the size of their register classes AB1XY and XY. Therefore, we color one of the split nodes immediately, and delay the coloring of the other node by placing it at the bottom of the stack, as in the case of either Fig. 13 (d) or (e).2 The question here is which node should be selected for immediate coloring. Among all split nodes, the original optimistic coalescing scheme would select a node v with the maximum value of the spill metric 2

After live range splitting in optimistic coalescing scheme, one of the split nodes tries to be colored immediately, not to ruin the coloring of other node. Details are in [10].

166

M. Ahn and Y. Paek

M(v) computed from Equation 8. With traditional register allocators for homogeneous registers, this simple metric should be workable since any register can be interchangeably assigned to anyone among the split nodes. But, with our register allocator for heterogeneous registers, a more elaborate spill metric that reflects the heterogeneity of registers allocatable to the split nodes is required; otherwise, unnecessary spills would be created in the code. To illustrate this, assume C(b)=C(f) in Fig. 13. Then, in the original optimistic coalescing scheme, the spill metrics of nodes b and f are obviously the same. So the scheme would like to break the tie by coloring arbitrarily first either b or f. Suppose that b is first colored with A1 or B1, and so f is placed at the bottom of the stack (see Fig. 13(d). Then, a or e becomes an actual spill. In contrast, if f is first colored with X0 as seen in Fig. 13(e), there would be no spill since any registers in AB1XY except those assigned to c, a and e can be allocated to b. The different outcome of register allocation between these choices is caused by the fact that both the split nodes are bound to different register classes with different sizes, which does not likely occur in homogeneous register architectures. To manage this characteristic of heterogeneous registers, we suggest a heuristic that gives the first priority for coloring to a node bound to a register class with the smallest size. This is of course based on our observation that the other split node with the larger-sized register class usually has a more chance to be later colored. In the above example, the register class XY bound to node f is a proper subset of the class AB1XY bound to node b, so we have |φXY| = 4 < |φAB1XY| = 6, which implies that two more registers are allocatable to b than to f. Thus, coloring f first would likely increase the chance to avoid extra spills. This surely is the case for this example. In fact, as you can see in experiments, we can reduce many spills using this heuristic in our spill metric. Motivated by this example, we have introduced to Equation 8 a new value, called the register class influence (φv), which measures the amount of influence that a node v bound to a register class φv has on the coloring of its neighboring nodes. From the fact that σx is proportional to |φy| where y is a neighbor of node x in an IG, we can derive a relation of the register class influences between two nodes u and v: (φu) ≤ (φv) for φu ⊆ φv, where φu and φv are the register classes bound to these nodes. For instance in Fig. 13 (a), since the register class XY bound to node f is a subset of the class AB1XY bound to node b, the neighbors of b would more likely have larger squeezes than those of f. This means that the colorability of b’s neighbors should be more limited than that of f’s neighbors, thus resulting in (φf) < (φb). This relation is visualized by the Register Class Tree (RCT) in which the nodes are all register classes of the target machine, and the edges represent the inclusion relation between the classes. In the tree, a register class φ is a descendant of another class φ′ if φ ⊂ φ′. If there is no other class φ″ such that φ ⊂ φ″ ⊂ φ′, then there will be an edge between φ and φ′ in the RCT, representing that φ is an immediate descendant of φ′. Fig. 14 shows the RCT built from the RCRT in Fig. 12. Notice that two register classes ABL and XYL are the children of class ABXYL since they are its subsets. We can see from the example that the RCT built for the target machine actually forms a forest.

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

167

Fig. 14. Register Class Tree (RCT) of DSP563xx

The depth (φ) of a register class φ in an RCT is the distance from the root node r down to φ, assuming (r) = 1. For instance in Fig. 14, the depth of XY is 3. Based on the argument above, we conclude that (φ) is inversely proportional to (φ): ε (φv ) =

1

δ (φv )

where φv is the register class of a node v in an IG. In the example of Fig. 14, we have ABXYRN = 1 and XY = 1/3. Now, we modify the original spill metric in Equation 8 by taking into account the influence ε of the register class φv of each node v on the coloring. The degree D(v) in Equation 8 represents how many nodes are interfered with the node v in the IG. For a HRA, it is in fact insufficient to show the interference relationship between two nodes since the nodes may be bound to different register classes. We must consider two additional factors here. First, the more registers a node has in its register class, the more interfering nodes it has in the IG. Second, even when there is an edge between two nodes in the IG, there might be no actual interference between their live ranges if their register classes are disjoint. To reflect into the spill metric this interference relationship between nodes bound to different register classes, we devise a new interference degree I(v): I (v) = ε (φ v ) ×

∑ f (v, n )

n∈adj ( v )

where adj(v) is a set of the neighboring nodes of v. For the register classes φv and φn bound to two nodes v and n, we have f(v,n) = 1 if φv ∩ φn = ∅, and f(v,n) = 0 otherwise. I(v) represents how many neighboring nodes are actually interfered with v. Using I(v), we formulate a new spill metric S(v): S (v ) = C (v ) / I (v )

(9)

In Section 7, we will experimentally prove that this new metric improves the coloring of the IG, and also reduces the spills considerably. In the example of Fig. 13, the original optimistic coalescing scheme could not efficiently break the tie between the spill metrics for nodes b and f when C(b) = C(f). So in the experiment, it often colors b first after live range splitting, resulting in an unnecessary spill. However, our modified coalescing scheme with S(v) evaluates I(b) = 3 and I(f) = 1, and consequently produces S(b) = C(b)/3 and S(f) = C(f). Thus, even for C(b) = C(f) > 0, we have S(b) < S(f), which places node f in the first priority for coloring, preventing the spill in the end.

168

M. Ahn and Y. Paek

7 Experiments In our experiments, the following four versions of code generators are tested: v0 : SPAM code generation approach; v1 : our code generation approach with the modified iterated coalescing; v2 : our code generation approach with the original optimistic coalescing; v3 : our code generation approach with the modified optimistic coalescing. The primary goal of our experiments is to compare between our code generation approach and the SPAM approach shown in Section 2.2. For this goal, we have implemented both code generation approaches in the same compiler platform targeting DSP563xx. We tried our best to minimize the difference between our version, namely v0, of the SPAM code generator and that of the actual SPAM compiler in [3] by thoroughly studying their literature and looking into the real implementation of their compiler. However, we admit that there still exists some discrepancy. For instance, the original implementation of SPAM explicitly included no coalescing scheme. So for fair comparison of their main code generation modules (i.e., those described in Sections 4 and 5) with theirs, we have added the same coalescing scheme to both versions v0 and v1. Even within our code generation approach, we have three different variations v1, v2 and v3, all of which share almost the same algorithm for code generation itself. But, they are different with regard to their coalescing schemes mentioned in Section 6. By implementing version v1, we attempt to test the modified iterated coalescing scheme proposed by Smith et al[12] in comparison with optimistic coalescing. Both versions v2 and v3 are virtually identical except our modifications made for HRAs in Section 6.2. Thus by comparing them, we try to measure the amount of performance improvement enabled by the modifications.

Fig. 15. Twelve benchmark programs collected according to their types and sizes

We conduct the experiments on 3.2GHz PentiumIV with 2GB RAM. Fig. 15 lists the benchmark programs tested in this work. The programs are of various sizes from different benchmark suites. The smallest ones such as iir_biquad_one_section and lms from DSPStone are the kernel codes frequently used in DSP applications. The larger ones such as viterb, fft and g721_encoder/decoder are typical applications in the DSP domain.

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

169

Fig. 16 presents the total numbers of move instructions in three versions v1, v2 and v3 after the global register allocation. We normalize the numbers with respect to that in version v0. Fig. 16 reveals that version v1 reduces move instructions by 14% on average, as being compared to SPAM. It is because much of move instructions generated in version v1 can be coalesced by delaying the decision of register files until the second phase of code generation. On the other hand, in version v0, register files are fixed for each node in the IG early in the first phase, and thus in the second phase, some move instructions prefixed to different register files could not be coalesced. Fig. 16 also reveals that both versions v2 and v3 with optimistic coalescing are surely more effective to reduce move instructions than version v1 with iterated coalescing.

Fig. 16. Ratio of move instructions of three code generator versions after global register allocation with respect to SPAM(version v0). The numbers at the bottom stands for the code numbers in Fig. 15.

Fig. 17. Code size ratios of three code generator versions with respect to that of SPAM

` Fig. 18. Run time ratios of three code generator versions with respect to that of SPAM

170

M. Ahn and Y. Paek

We have observed that the reduction of move instructions shown in Fig. 16 also has a considerable impact on the code sizes as well as speeds. Fig. 17 and Fig. 18 show the sizes and run times of the code generated by all three versions of our code generators after being normalized to those of version v0. Version v1 reduces the code sizes by 10% and reduces the run times by 7% as compared to SPAM. The code size reduction of version v2 is almost as the same as that of v1, but its run time is reduced by 5% from v1. Overall, version v3 shows the best performance; it reduces the code size by 3%, and reduces the runtime by 7% in comparison with version v1. Notice that for the code viterb, version v2 generates larger code size than SPAM although its performance is comparable to that of SPAM. We found that this is because the original optimistic coalescing scheme in version v2 produces too many spills during register allocation. Fortunately, our modifications for optimistic coalescing successfully enable version 3 to reduce the final code size of viterb.

Fig. 19. Compile time overhead ratios of three code generator versions with respect to that of SPAM

Not surprisingly, the improvement in our code generation approaches comes at the extra cost of the compilation time. Fig. 19 shows the compilation times of our code generator versions (v1, v2 and v3) normalized to that of SPAM. Although version v1 uses the same iterated coalescing as SPAM, its compilation time is approximately 30% higher than that of version v0 because as shown earlier, the register classes of DSP563xx outnumber its physical register files, and therefore, the compiler needs more time to deal with the larger number of register classes during code generation. Luckily, in versions v2 and v3, the overheads in compilation time are reduced for most cases. The reason is that the optimistic coalescing scheme implemented in these versions usually runs faster than iterated coalescing in version v1. This is mainly due to the crucial drawback of iterated coalescing which needs to iterate the simplification, coalescing and freeze phases many times.

8 Conclusion In this work, we propose a fast code generation approach for HRAs, where instruction selection and register allocation are coupled in a single phase. Since coupling the two code generation subtasks completely in the same phase is enormously complex, we in this work try to couple the two subtasks more loosely through the logical relationship

Fast Code Generation for Embedded Processors with Aliased Heterogeneous Registers

171

based on register classes. In our first phase of code generation, we simultaneously select instructions and registers for their operands. But, unlike the previous work in SPAM[3], we attempt to designate not a physical register for each operand, but a logical class of registers for the operand, among which any register can be chosen for the operand. Capitalizing on this freedom of choice, our global register allocator in the following phase tries to find a globally optimal allocation of registers from these predetermined register classes. We have discovered through the experiments that our approach generally outperforms the SPAM approach in various performance aspects. Also, the experiments reveal that the register allocation scheme proposed in [12] is well workable in our code generation approach. However, by adopting the optimistic coalescing scheme with some modifications, we were able to even further improve the performance on the target HRA. Finally, we empirically exhibit that the compilation time of our code generator is almost as fast as the original SPAM, from which we glean the plausible evidence that our compiler runs in near polynomial time even with such improvement of the code quality. Acknowledgments. This work was supported by the Korea Science and Engineering Foundation (KOSEF) NRL Program grant funded by the Korea government (MEST) (No. R0A-2008-000-20110-0) and the Engineering Research Center of Excellence Program of Korea Ministry of Education, Science and Technology (MEST) / Korea Science and Engineering Foundation (KOSEF), grant number R11-2008-007-01001-0.

References [1] Ahn, M., Lee, J., Jung, S., Yoon, J.W., Paek, Y.: A Code Generation Approach for Heterogeneous Register Architectures. In: Proceedings of Workshop on the Interaction between Compilers and Computer Architecture (February 2007) [2] Ahn, M., Lee, J., Paek, Y.: Optimistic Coalescing for heterogeneous register architectures. ACM SIGPLAN Notices 42(7), 93–102 (2007) [3] Araujo, G., Malik, S.: Code Generation for Fixed-Point DSPs. ACM Transactions on Design Automation of Electronic Systems (TODAES) 3(2), 136–161 (1998) [4] Bashford, S., Leupers, R.: Constraint driven code selection for fixed-point DSPs. In: Proceedings of Design Automation Conference, pp. 817–822 (1999) [5] Chaitin, G.: Register allocation and spilling via graph coloring. SIGPLAN Notices 17(6), 98–105 (1982) [6] Feuerhahn, H.: Data Flow Driven Resource Allocation in a Retargetable Microde Compiler. In: Proceedings of International Symposium on Microarchitecture, pp. 105–107 (1988) [7] George, L., Appel, A.: Iterated Register Coalescing. ACM Transactions on Programming Languages and Systems 18(3), 300–324 (1996) [8] Lorenz, M., Marwedel, P.: Phase Coupled Code Generation for DSPs Using a Genetic Algorithm. In: Proceedings of Conference on Design, Automation and Test in Europe, vol. 2, p. 21270 (2004) [9] Marwedel, P., Goossens, G. (eds.): Code Generation for Embedded Processors. Kluwer Academic Pub., Dordrecht (1995)

172

M. Ahn and Y. Paek

[10] Park, J., Moon, S.-M.: Optimistic Register Coalescing. In: Proceedings of PACT (1998) [11] Coffman, E.G., Sethi, R.: Instruction Sets for Evaluating Arithmetic Expressions. Journal of the ACM 30(3) (1983) [12] Smith, M., Ramsey, N., Holloway, G.: A Generalized Algorithm for Graph-coloring register allocation. In: Proceedings of the ACM SIGPLAN conference on Programming language design and implementation PLDI, vol. 39(6) (2004) [13] Stallman, R.: Using and Porting GNU CC, Free Software Foundation (June 1993) [14] Wilson, T., Grewal, G., Halley, B., Banerji, D.: An Integerated Approach to Retargetable Code Generation. In: Proceedings of seventh International Symposium on High Level Synthesis, pp. 70–75 (1994) [15] Liem, C., May, T., Paulin, P.: Register Assignment through Resource Classification for ASIP Microcode Generation. In: Proceedings of the IEEE/ACM international conference on Computer-aided design (1994) [16] Paulin, P., Liem, m.C., May, T., Sutarwala, S.: DSP Design Tool Requirements for Embedded Systems: A Telecommunications Industrial Perspective. Journal of VLSI Signal Processing (1994) [17] Briggs, P., Cooper, K.D., Torczon, L.: Improvements to graph coloring register allocation. ACM TOPLAS 16(3), 428–455 (1994)

Linux Kernel Compaction through Cold Code Swapping Dominique Chanet1 , Javier Cabezas2 , Enric Morancho2, Nacho Navarro2, and Koen De Bosschere1 1

Department of Electronics and Information Systems Ghent University B-9000 Ghent, Belgium {dchanet,kdb}@elis.UGent.be 2 Department of Computer Architecture Technical University of Catalonia E-08034 Barcelona, Spain {jcabezas,enricm,nacho}@ac.upc.edu

Abstract. There is a growing trend to use general-purpose operating systems like Linux in embedded systems. Previous research focused on using compaction and specialization techniques to adapt a general-purpose OS to the memory-constrained environment presented by most embedded systems. However, there is still room for improvement: it has been shown that even after application of the aforementioned techniques more than 50% of the kernel code remains unexecuted under normal system operation. We introduce a new technique that reduces the Linux kernel code memory footprint through on-demand code loading of infrequently executed code, for systems that support virtual memory. In this paper, we describe our general approach, and we study code placement algorithms to minimize the performance impact of the code loading. A code size reduction of 68% is achieved, with a 2.2% execution speedup of the system-mode execution time, for a case study based on the MediaBench II benchmark suite.

1

Introduction

In recent years, embedded systems have become increasingly complex. For example, mobile phones have evolved from relatively simple devices that provide phone calls and text messaging to veritable multi-media devices that also take pictures, play music and movies, surf the Internet and have extensive contact management and calendaring functionality. Due to this trend, the complexity of the software running on these devices has risen exponentially. Developers turn more and more to pre-built components and high-level programming languages in order to meet the functionality requirements and time-to-market pressure in highly competitive markets. This also concerns the operating system used on these devices: there is a growing trend to use general-purpose operating systems. Most of the attention has gone into Linux, as it is freely available, and its P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 173–200, 2009. c Springer-Verlag Berlin Heidelberg 2009 

174

D. Chanet et al.

open source nature gives developers full control, allowing them to adapt the OS in any way they see fit. General-purpose operating systems offer a large number of functionalities that are unneeded in embedded devices, as they have been developed for desktop or server computers that must support a very wide range of applications and peripheral devices. The use of such systems can vary widely over their lifetime. Embedded systems, by contrast, usually have a well-defined and limited functionality, with software and hardware configurations that do not change over the device’s lifetime. As embedded systems typically have strict memory constraints, it is desirable to remove as much of the overhead incurred by the unnecessary features of general-purpose operating systems as possible. Part of the overhead can be avoided by configuring the OS kernel appropriately at build time to exclude unnecessary drivers and features. However, this configuration facility is usually not fine-grained enough to remove all of the unneeded code and data. Recent research on link-time compaction and specialization of general-purpose OS kernels [7,19] has shown that even on a fully-configured Linux kernel significant amounts of code can be removed if the hardware and software configuration of the target system are known and fixed. However, in [8] it is shown that, even after application of the aforementioned compaction and specialization techniques, less than half the code in the kernel is executed during normal system operation. Part of the unexecuted code is there to handle unexpected situations, like hardware failures. The other unexecuted code is in effect unreachable, but is not detected by the aforementioned specialization techniques due to the limitations of static analysis. The authors of [8] propose to store all the unexecuted code (henceforth called frozen code, a term introduced by Citron et al. [9]) in memory in a compressed form, and later decompress only those parts that are needed at run time. However, under their approach it is impossible to determine a hard upper bound for the amount of memory the kernel’s code memory footprint, as once-decompressed code cannot be removed from memory any more due to concurrency issues that arise from the inherent multithreadedness of the Linux kernel. In this paper, we propose a novel approach to solve this problem for systems that have support for virtual memory. While virtual memory support is not yet available for all embedded devices, it is already supported by several important embedded processor families, such as the Intel XScale [2], the Texas Instrument OMAP [4], and the MIPS 4K [3]. Based on profile information, our technique selects code that will be put aside from the kernel’s resident memory image and loaded on demand whenever it is needed. To avoid the high latencies of loading code from disk, the removed code will be stored in a fast off-line memory (e.g. Flash memory). Contrary to the aforementioned approach [8], our technique allows to determine an upper bound on the kernel’s code memory footprint. In this paper, the technique is evaluated for the Linux kernel on the i386 architecture, but it is easily portable to other architectures and operating systems. For this paper, we have chosen to focus on Flash memory as the secondary memory in which the not-loaded code is stored. We believe Flash memory is a

Linux Kernel Compaction through Cold Code Swapping

175

good fit for this technique, as it is sufficiently fast, and it is typically already available in embedded systems to store the device’s firmware. Some typical devices that can be targeted by our technique are the Linksys WRT54GL wireless internet router (16 MiB of RAM, 4 MiB of Flash ROM), the Linksys NSLU2 network storage server for home networks (32 MiB of RAM, 8 MiB of Flash ROM) and the Devon NTA 6010A thin client (64 MiB of RAM, 64 MiB of Flash ROM). All of these devices have a fixed function, and all of them run the Linux kernel as their operating system. It is important to note that the technique proposed in this paper is not the ultimate solution to the memory woes of embedded systems. Rather, it is a building block in a total solution. The OS kernel is only a part of the software that runs on the device. It is equally important to reduce the memory footprint of the user space programs through techniques such as link-time compaction [10] or code compression [11]. The remainder of this paper is organized as follows. In the next section, we give a general overview of our approach. Section 3 details the code selection and layout algorithms that are instrumental to limiting the performance impact of our technique. In Section 4 we discuss the implementation. In Section 5 the technique is evaluated. Section 6 reviews related work, and we draw conclusions and present future work in Section 7.

2

General Approach

In the most general sense, we wish to develop an on-demand code loading scheme in order to reduce the static RAM footprint of an operating system kernel. The most important design criteria are: – Reliability: the correct working of the kernel must in no way be compromised by the code loading scheme. – Performance: the OS kernel is a performance-critical part of the system, especially as it communicates directly with the hardware devices. Loading code must not slow down the system too much. – Guaranteed size reduction: unlike the approach proposed in [8], our approach should guarantee a hard upper bound on the kernel code memory usage, and it should be considerably lower than that of the original kernel. Note that this does not mean that the user can set a limit up front (e.g. “the kernel code should only use x bytes of memory”) but rather that, after our technique is applied, the user knows exactly how much memory will be used by the kernel code. – Transparency: there should be no major rewriting or manual annotation of the kernel code necessary. – Automation: the code to be loaded on demand must be selected and partitioned into loadable fragments automatically. Users of the technique should not need to have an intimate knowledge of the kernel code. The requirement of a hard upper bound on the kernel code’s memory usage implies that any suitable scheme not only implements on-demand code loading, but

176

D. Chanet et al.

also code eviction whenever the memory allocated for the kernel code is full and new code needs to be loaded. However, due to the inherent multithreaded nature of most OS kernels, this raises concurrency issues. If a code fragment has to be evicted, the scheme must ensure the reliable execution of the kernel threads that may currently be executing the evicted code fragment. Automatically inserting locks in the kernel code to protect the loadable code fragments from untimely eviction creates a very real risk of introducing deadlocks. Therefore, an alternative to the introduction of locking must be found to guarantee the kernel’s reliability. In the remainder of this section, we first investigate the Linux kernel’s module loading scheme, which can be considered a form of on-demand code loading, and then propose our own solution to the problem. 2.1

Linux Kernel Modules

The Linux build system allows the developer to compile parts of the kernel (e.g., certain hardware drivers) as loadable modules. These modules can then later be loaded on demand when their functionality is needed. This facility is mostly intended for the distribution of kernels for generic machines. Drivers for a wide range of peripherals are compiled as modules, and upon booting the kernel then loads only those drivers that are necessary for the specific hardware it is running on. We feel this scheme is less suited to embedded systems, where the hardware is known in advance and the necessary drivers can easily be compiled into the kernel, obviating the need for the module loading code altogether and thus reducing the kernel’s code size. Furthermore, the module granularity is determined by the kernel’s configuration system, and this is not fine-grained enough for our purposes. We wish to remove individual, infrequently executed, code paths from a driver or kernel subsystem, and not a driver or subsystem as a whole. 2.2

Our Solution

With our technique, which is only applicable to platforms that support virtual memory, the infrequently executed (henceforth called cold) code in the kernel is removed from physical memory, even though it is still present in the kernel’s virtual memory image, and stored on a fast secondary storage medium. This storage is preferably Flash memory as this is already available in most embedded systems for storing the device’s firmware. However, there are other possibilities, like storing the cold code in a compressed form in main memory (thus improving on [8]). When the kernel tries to execute the cold code, a page fault occurs. The (modified) page fault handler then locates the needed page in the secondary memory, loads it in one of a set of pre-allocated physical page frames and adjusts the page tables to map it to the correct virtual address, after which execution can continue. This basically means that the page fault trap is used as the trigger to load code.

Linux Kernel Compaction through Cold Code Swapping

177

Fig. 1. An example of the concurrency issues involved in evicting code from memory. The left side of the figure shows a time line of the execution of two different threads, the right side shows the contents of the code cache at times T0 , T1 and T2 . The downwardpointing arrow indicates the next page to be replaced according to the replacement policy.

The only modification necessary to the kernel source code is the extension of the page fault handler to insert the code loading mechanism. As such, we feel the transparency requirement has been fulfilled. In order to fulfill the “hard upper bound on memory usage” requirement, we use a fixed-size code cache of physical page frames to map the cold code. The cache is managed through some replacement policy (for instance, round robin) that does not explicitly check whether code on a page selected for eviction is being executed in another thread. Nevertheless, the reliability of the system is not compromised. This is illustrated in Figure 1. Assume there are two threads in the kernel, the cache can hold two pages, and we use a round-robin replacement policy. Thread 1 is executing cold code from virtual page V1 in the physical cache frame P1 , while thread 2 is executing hot, non-swappable code. The second cache frame contains a previously loaded cold page V9 from which no code is being executed any more. At some point, the second thread has to execute some cold code from virtual page V2 , which is currently not in the cache, causing a page fault. The page fault handler runs in the execution context of thread 2, locates the necessary code in secondary memory and because of the round-robin replacement policy decides to put V2 in P1 . Once V1 is unmapped from memory, a page fault occurs in thread 1 when the next instruction in this thread is loaded. The page fault handler runs in the context of thread 1, find V1 in secondary storage and map it in P2 because of the round-robin policy, after which execution in thread 1 can continue as before. While this scenario means that thread 1 has been temporarily interrupted, the integrity of the execution has not been compromised. In the worst case, this scenario could cause a cascade of code cache refills for all kernel threads, but it is easily shown that the system

178

D. Chanet et al.

will not deadlock as long as there are at least as many code cache frames as there are kernel threads. There is only one manual step involved in selecting which code is loaded on demand: the profiling step. The user has to run an instrumented kernel on the target system in order to collect a basic block profile. This profile is then used to identify the cold code in the kernel, and from then on the whole process runs without user intervention. This satisfies the automation design criterium. In order to fulfill the performance requirement, there are several issues we have to take into account. First, loading the page from secondary memory should be sufficiently fast. We believe this requirement to be fulfilled with the use of Flash memory as a secondary storage medium. For currently available Flash memory parts (Intel Embedded Strataflash P33), a 4 KiB page can be read in approximately 40 microseconds. As only code, which is read-only, is swapped in, there is no need to write back pages to Flash memory when they are evicted from memory, avoiding costly Flash write operations that would slow down the process. Secondly, only cold code should be swapped out in this way, to reduce the amount of needed code cache refills. Thirdly, an intelligent code placement algorithm should be used to avoid that related cold code fragments span page boundaries, because this would cause more code cache refills than necessary. Our code placement algorithms are detailed in Section 3.2. Our approach is essentially a variation on the well-known virtual memory swapping technique [20]. Generic swapping can store any virtual memory page from any process on a secondary storage medium (typically a hard disk), thus freeing up physical memory for other virtual memory pages. Because of the high latency involved in reading a page from disk, the OS usually puts the process causing a page fault to sleep and schedules another process to run instead. This makes this technique less suitable for use in the OS kernel itself, and indeed Linux does not implement swapping for kernel memory. While this has been repeatedly proposed in the past, the kernel developers reject the idea because of the amount of timing-critical code in the kernel that cannot sleep. Separating this timing-critical code and data from other code and data would be too involved and error-prone to be practical [5]. We believe that our approach is not susceptible to these objections. As Flash memory is an order of magnitude faster than hard disks, there is no need to put the faulting execution thread to sleep, thus avoiding the problems usually associated with swapping out kernel memory. Note that it would also be possible to extend this technique to the kernel’s read-only data (i.e., strings for error description). However, we have left this problem for future work. Extending the technique to incorporate writable data is not advisable, as Flash memory wears down after too many write cycles. Consequently, the repeated write-back operations that swapping out writable data would entail, would severely limit the device’s lifetime. Furthermore, the write-back operations would significantly slow down the swapping process.

Linux Kernel Compaction through Cold Code Swapping

3

179

Swappable Code Selection and Placement

In this section, we discuss how the code to be loaded on demand is selected, and we present the code layout algorithm that maps the swappable code to individual virtual memory pages in such a way that the need for code loading operations is minimized. 3.1

Code Selection

As mentioned in Section 2.2, only infrequently executed code is considered for on-demand code loading. Based on basic block profile information gathered for the kernel (the instrumentation technique is discussed in Section 4.2), the kernel code is divided into three categories: 1. The core code: this is the code that always has to be present in memory for the system to work correctly. Basically this portion of the code consists of all code that can be executed before our code loading mechanism is initialized, the page fault handling mechanism and the code needed to read the secondary storage medium. 2. The base code: this is the frequently executed (hot) kernel code, which we want to keep permanently resident for performance reasons, even though there are no technical difficulties in swapping it out. 3. The swappable code: this is the remaining code, which is either infrequently (cold) or never (frozen) executed. This is the code that is removed from the kernel image and stored on the secondary storage medium for on-demand loading. It is important to note that by design the Linux kernel code is split into two categories: initialization code and non-initialization code, henceforth called init code and non-init code respectively. Because the kernel’s first task at boot time is to initialize the system and create an environment in which application programs can run, it contains a lot of code and data that is only used once at boot time. As soon as this initialization code and data are no longer needed, the kernel removes them from memory. As such it is not very useful to apply the on-demand code loading technique to the init code: by the time the user space processes start executing, and the device’s full memory capacity is needed, it is already removed from memory. Consequently, we consider all init code to be part of the core code. As mentioned before, all code that is executed before our code loading mechanism is initialized has to be considered core code. While most of this code is init code, it also includes a number of non-init utility procedures that are called from the init code. We can reduce the amount of non-init core code by duplicating all non-init procedures that are only called from init code prior to the initialization of our code loading mechanism. All calls from init code are moved to the duplicate procedures, which can then be considered init code as well. The original procedures are then no longer called prior to the initialization of our mechanism, and can be considered swappable. As the init code is released from

180

D. Chanet et al.

Fig. 2. A slice of the call graph (a) before and (b) after procedure duplication. Gray blocks represent init code, white blocks non-init code. Nodes with heavy borders are considered core code, those with a dashed border are infrequently executed, those with a solid border are frequently executed.

memory during the boot process, the duplicated procedures incur no memory overhead during the system’s steady state operation. Figure 2 illustrates this process. In part (a) we see a slice of the kernel’s call graph before duplication. Non-init procedure D is called by init procedures A and B before the code loading mechanism is initialized. The call from non-init procedure C can only occur after the mechanism is initialized. Because of the calls from A and B, D and its descendants in the call graph E and F must be considered core code. In part (b) the situation after duplication is shown. F is not duplicated as it is hot code, and is swapped out anyway. The duplicated procedures D’ and E’ are only reachable from init code, and can thus be considered init code themselves. The original procedures D and E can now only be called after the code loading mechanism is initialized and can hence be considered swappable instead of core code. 3.2

Code Placement

The idea of using code placement techniques as a means to minimize page faults has been studied before. An overview of the existing literature can be found in Section 6.3. All existing algorithms use some variation on run-time profile data as input, and of course they concentrate on achieving a good placement for the most-frequently executed code. As we only have to place the least-frequently executed code, for which there is much less profile information available, these algorithms are not guaranteed to achieve good results. This is especially true in the case where only frozen code is considered swappable, because for this code all execution counts are zero. Therefore, we have implemented two different code placement algorithms. The first makes use of whatever profile information is available to achieve a

Linux Kernel Compaction through Cold Code Swapping

181

good placement, whereas the second aims to minimize, for each entry point in the swappable code, the total number of pages needed to load all swappable code that is directly reachable from that entry point. The second algorithm makes no use of profile information and relies only on an analysis of the static structure of the code. Both algorithms assume that the swappable code can be placed independently from the hot and core code, i.e. there are no fall-through control flow paths connecting cold code to other code. How this is achieved in practice is explained in Section 4.3. The Profile-Based Algorithm. In this algorithm, which is similar to the one proposed by Pettis and Hansen [23], the code is placed with a chain granularity. A chain is a set of basic blocks that have to be placed in a predetermined order because of control flow dependencies (e.g. fall-through paths or a function call and its corresponding return site). Control flow between chains is always explicit, in the form of function calls, returns or jumps. Consequently, the order of the chains is not important for the correct working of the code. Indirect control flow (i.e. indirect jumps and function calls) is not taken into account by the algorithm. We use a graph representation of the problem as proposed by Ferrari [14]. The graph nodes represent chains. While hot chains will not be placed on the swappable pages, and their final layout will not be influenced by this algorithm, they are also represented in the graph. The size of a node representing a cold chain is equal to the size of the chain in bytes, whereas nodes representing hot chains have size 0. The (undirected) graph edges represent direct control flow between chains. The edge weights are computed by the following formula:  (1 + execcount(e)) weight(eij ) = e∈(Ei→j ∪Ej→i )

where Ei→j is the set of direct control flow edges from chain i to chain j and execcount(e) is the traversal count of control flow edge e according to edge profile information. In our current implementation, the edge profiles are estimated from the basic block profile information we have available. It is also possible to obtain exact edge profiles by inserting the appropriate instrumentation into the kernel, just like we did for obtaining the basic block profiles, but, as shown in the evaluation section, the estimated edge profiles are accurate enough to derive a good code placement. The traversal counts are incremented by one to ensure that the substantial body of frozen code in the kernel, whose edge traversal counts are zero, is not ignored during placement. If each node is placed on a separate virtual memory page, the graph’s total edge weight is an estimate of the number of page faults that occur at run time. The hot chains are represented in the graph to make sure that related cold code fragments are not placed independently. For example, suppose a procedure has a cold prologue and epilogue, but the actual procedure body is hot. As the only way for the control to flow from the prologue to the epilogue is through hot code, the prologue and epilogue chains would not be connected in the graph

182

D. Chanet et al.

if only the cold chains are represented. As a consequence, there is a big chance that the prologue and epilogue are placed on different code pages, which would result in two page faults for an execution of the procedure, as opposed to only one when they are placed on the same page. The nodes in the graph are clustered in such a way that node sizes never exceed the virtual memory page size. This is done in three steps: 1. We try to minimize the total edge weight of the graph. This is done with a greedy heuristic by iteratively selecting the heaviest edge whose head and tail can still be merged without exceeding the page size. In case of a tie, we select the edge with the maximum commonweight, which is defined as:  (weight(eik ) + weight(ejk )) commonweight(eij ) = k∈succ(i)∩succ(j)

where size(i) + size(j) + size(k) ≤ PAGESIZE. In this way, we try to obtain a graph with less, but heavier edges instead of one with many light edges. If there still is a tie, we select the pair of nodes that exhibit the best locality, i.e., the pair of pages that contain code that was placed closest together in the original kernel. The intuition here is that code that was placed closely together is likely to be related. After this step, the total edge weight cannot be reduced any further. 2. We try to maximize the weight of individual edges by iteratively merging sibling nodes (nodes not connected to each other but connected to a common third node). In each iteration we select the nodes for whom the sum of the weights of the edges connecting them to their common parent is maximal. The idea behind this step is that, if more than one page is available in the code cache, the probability of page j already being in the cache upon a control transfer from page i is proportional to weight(eij ). 3. For each connected subgraph, nodes are merged with a best fit algorithm. This step minimizes the total number of pages needed for each connected subgraph. We do not yet merge nodes from different subgraphs, because we do not want to pollute the pages for one connected subgraph with code from another subgraph. After all, the likelihood that node j is needed in memory before node i is removed from the code cache is higher if i and j belong to the same connected subgraph. The Per-Entry Point Minimization Algorithm. This algorithm makes no use of profile information to guide the code placement. The swappable code is first partitioned into single-entry regions that do not span procedure boundaries. These single-entry regions (henceforth simply called regions) are the basic units of code placement. Regions that have incoming control flow edges from base or core code are called entry points. As a simplification, we assume that the entry points are independent of each other, i.e., that the fact that entry point i was entered at time T has no influence on the probability of any specific entry point j being entered at time T ′ > T . Under this assumption, it makes sense to place

Linux Kernel Compaction through Cold Code Swapping

183

the code in such a way that only a minimal amount of pages is reachable from each entry point. After all, in the absence of meaningful profile information we have to assume that all code paths through cold code are equally likely to be followed, so we cannot favor one code path over another for placement on a minimum number of pages. Initially, each region is placed on its own page. Let P be the set of pages, and E the set of entry points (E ⊆ P ). We define two functions: ∀p ∈ P : entries(p) = {e ∈ E | p is reachable from e} and ∀e ∈ E : pcount(e) = ♯{p ∈ P | e ∈ entries(p)} . entries(p) returns the set of entry points from which code on a page p is reachable, without passing through hot or core code. pcount(e) computes the number of pages that are reachable from entry point e. The code placement algorithm tries to minimize the pcount for each entry point by iteratively executing the following steps: 1. Build the set M containing the entry points with maximal pcount. 2. Select pages pi and pj such that size(pi ) + size(pj ) ≤ PAGESIZE and pi and pj have a maximum number of entry points in common with M and each other, i.e. ♯(M ∩ entries(pi ) ∩ entries(pj )) is maximal. In case there are multiple eligible pairs, select the pair that has the most entry points in common. Stop if no pair can be found. 3. Merge pages pi and pj . Reducing Fragmentation. Both described code placement algorithms terminate with a lot of small pages left that aren’t merged because there are no direct control flow edges between the code on the pages. In order to reduce fragmentation, a post-pass merges these small pages. This happens in two steps: 1. Small pages that contain code that was placed close together in the original kernel are merged. If the code fragments originally were placed close together, they probably originate from the same source code file, which greatly increases the likelihood of the code fragments being related to one another. 2. The remaining pages are merged using a best fit approach.

4

Implementation

In this section we discuss the actual implementation of our technique. The binary rewriting operations are implemented with a modified version of the Diablo linktime binary rewriting framework that is suitable for rewriting Linux kernels [8]. The rewriting intervenes in the kernel build process just after all object files are linked together to create the executable kernel image (the so-called vmlinux file). Note that this is not the last step in the build process, as this executable image is usually transformed into a self-extracting executable (the bzImage file) to save disk space and reduce the kernel’s load time.

184

4.1

D. Chanet et al.

Linux/i386 Virtual Memory Management

Before describing our implementation, we review, focusing on the i386 architecture and Linux, address spaces, address translation, the organization of the virtual address space and page fault handling. Address Spaces and Address Translation. The i386 architecture supports 32-bit virtual address spaces and a 32-bit physical address space. Although extensions to i386 support bigger spaces, these are not relevant to the embedded context and are henceforth ignored. Virtual to physical address translation is performed by a paging mechanism. Pages are either 4 KiB or 4 MiB in size. The page table is organized as a two-level structure: – The first-level page table (Page Directory) is recorded on a 4 KiB page, consisting of 1024 32-bit entries. Each entry contains a base physical address and some flag bits (present (valid), access rights, accessed, dirty, page size, . . . ). Each valid entry with the page size flag set represents a 4 MiB page; the remaining valid entries point to second-level page tables. – Each second-level page table is also comprised of a 4 KiB page divided into 32-bit entries. These entries are similar to those of the Page Directory, except for the absence of the page size flag. Each valid entry of the second-level page table represents a 4 KiB page. Each process has its own Page Directory. The cr3 control register contains the physical address of the Page Directory of the running process. On each context switch, it is updated by the operating system. Accessing either a Page Directory entry or a second-level page table entry whose present (valid) flag is not set triggers a page fault exception. When this happens, the virtual address that produced the fault is loaded into the cr2 control register and a code that reflects the exception cause is pushed on the stack. To speed up address translation, i386-family processors implement TLBs (Translation Lookaside Buffers) that keep some page translations in a cache memory. Storing a value in the cr3 control register flushes the TLBs. The TLBs can also be flushed with a special instruction. Organization of the Virtual Address Space. Linux divides the 32-bit virtual address space into the user space and the kernel space. While each active process has its own user space, the kernel space is shared among all the processes. The user space takes up the first 3 GiB of the virtual address space, up to address 0xbfffffff. The fourth gigabyte of the address space is designated the kernel space. The organization of the kernel space is dependent on whether the physical memory size exceeds 896 MiB. As this is not the case in the memoryconstrained embedded devices we target, we ignore this possibility. The kernel space is divided into three areas: – Physical Memory Mapping: This area provides a one-to-one mapping of physical to virtual addresses. Each physical address pa is mapped to virtual

Linux Kernel Compaction through Cold Code Swapping C000000 + phys.mem.

C0000000

Physical memory mapping

VMALLOC START

8MiB

call1 8KiB

VMALLOC END

4KiB

call2 4KiB 4KiB

FIXADDR START

8KiB

185

FFFFFFFF

Fix-mapped 4KiB addresses

Fig. 3. Organization of the kernel address space

address 0xc0000000 + pa. Contiguous virtual pages in this area are also contiguous in physical memory. The page table entries related to this area are initialized at boot time. – vmalloc: This area avoids external fragmentation when the kernel allocates a contiguous multiple-page virtual space; contiguous pages in this area of the virtual space are not necessarily contiguous in the physical space. The area starts at the virtual address VMALLOC START (typically 8 MiB after the end of the physical memory mapping) and ends at VMALLOC END. Each allocation in this area is separated by a 4 KiB safety gap from the previous allocation. That is, the virtual pages adjacent to each allocation in this area are tagged as invalid in the page table. The page table entries related to this area are dynamically initialized as the vmalloc routine is called. – Fix-mapped addresses: This area allows mapping a virtual page to an arbitrary physical frame. This area is placed almost at the end of the virtual address space. Fixed mappings are retrieved with the fix to virt routine. At compile time, the compiler is able to substitute all valid references to this routine with the corresponding virtual addresses. It is useful for subsystems that need to know the virtual addresses at compile time. The page table entries related to this area are dynamically initialized as the set fixmap routine is called. Figure 3 shows a diagram with the location of the three areas and the security gaps. We assume that vmalloc has been called twice, the first call has allocated 8 KiB and the second 4 KiB. Note that the Page Directory for each process maps both the user and kernel spaces. Moreover, the kernel maintains a Page Directory called the Master Page Table that is initialized, at boot time, with the physical memory mapping. During system operation, the kernel portion of a processes’ Page Directory is initialized from the Master Page Table, and the Master Page Table is updated as the kernel’s memory management routines are called. Page Fault Handling. The use of page fault exceptions on Linux depends on which portion of the virtual address space (user or kernel) the address belongs to. In the user portion of the address space, page fault exceptions are mainly related to triggering code cache page refill. However, other scenarios can also produce these exceptions: copy-on-write handling, dynamic loading of the binary file, stack growth and detecting invalid memory accesses. In the kernel portion of the address space, page fault exceptions are not related to code cache refills because the whole Linux kernel is permanently resident

186

D. Chanet et al.

in physical memory. However, the kernel portion of the virtual address space can be modified dynamically through several kernel routines. Although these modifications are reflected in the Master Page Table, they are not propagated into the kernel portion of the Page Directory of all user space processes. Consequently, if the processor is running in privileged mode using the Page Directory of the running process, an exception may arise when accessing a kernel area outside the physical-memory mapping. Then, the page fault handler is responsible for synchronizing the contents of the Master Page Table with the process’ Page Directory. Note that the second-level page tables related to the kernel space are shared by the kernel and by all processes, so those need not be synchronized. 4.2

Gathering Profile Information

As mentioned in Section 3, we need accurate basic block profile information to select the code to be loaded on demand. To collect this information, we generate an instrumented version of the kernel that is run on the system under typical workloads. The instrumentation added to the kernel is very straightforward: an extra zero-initialized data section is added to the kernel that contains a 32-bit counter for each basic block in the kernel. At the beginning of each basic block we insert an inc $counter instruction to increment the counter for that block. As the inc instruction affects the processor’s condition flags, we need to make sure that the original flags are restored if their value is still needed afterwards. This can be determined using interprocedural register liveness analysis, which is already provided by the link-time rewriting framework we use. If the flags need to be preserved, a pushf instruction is added before and a popf instruction after the inc. There are no special accommodations for reading out the counter values. The Linux kernel already offers the possibility to access the contents of the kernel’s memory through the /proc/kcore interface, so the counter values are read directly from this interface. In the next step, where the swappable code is separated from the alwaysresident (base and core) code, the basic block profile information is used to distinguish hot code from cold code based on a user-configurable threshold value T . For example, for T = 0.95, the most-executed basic blocks that together constitute (approximately) 95% of the kernel’s execution time is considered hot, hence going into the base code partition. The hot code is identified with the following algorithm: 1. Computethe control flow graph’s total weight. The total weight is defined n as W = i=1 weight(blocki ), where n is the number of basic blocks in the graph, weight(blocki ) is the execution count of the ith block multiplied by its number of instructions. 2. Sort the basic blocks on execution count in descending order. 3. Walk the sorted block list, summing the block weights until the accumulated weight is higher than or equal to T ∗ W .

Linux Kernel Compaction through Cold Code Swapping

187

4. The control flow graph’s hotness threshold H is then equal to the execution count of the last-visited block. All blocks whose execution count is higher than or equal to H are considered hot, all other blocks are cold. Note that this algorithm is not exact, though the approximation it provides is sufficiently accurate to be useful. 4.3

Rewriting the Kernel

Our binary rewriter builds a control flow graph of the complete kernel, as described in [8]. On this graph, some preliminary optimizations are performed to reduce the kernel’s memory footprint. First, all unreachable code and data are removed from the kernel. Next, using a technique described in [7], all non-init code that is only reachable from the init sections (and is thus unreachable after the init code is removed from memory) is identified and moved to the initialization code section. In this way, we minimize the amount of code that has to be considered for swapping. Based on the profile information, the remaining non-init code is divided into core, base and swappable code. The swappable code is first partitioned into single-entry regions, which are then made individually relocatable. This means there are no control flow dependencies (such as fall-through paths or function call and return pairs) between the different regions or between swappable and always-resident code, so we can freely move the regions to new virtual addresses. To make the regions individually relocatable, we just break up all fall-through paths in and out of the regions by inserting direct jump instructions. The swappable code is then partitioned into page-sized clusters according to the algorithms described in Section 3.2. Each cluster is then placed in a new code section, which is padded up to the 4KiB boundary. The rewriter’s code layout phase then places these code sections in the virtual address region reserved for the code cache and adjusts all jump offsets and addresses in the kernel accordingly. The Linux kernel code assumes that virtual addresses belonging to the Physical Memory Mapping area can be translated to physical addresses just by subtracting the constant 0xc0000000, that is, Linux assumes that this virtual address range is always present in physical memory. Consequently, it makes no sense to place the code cache in this virtual address range as we cannot free the corresponding physical memory. Instead, we decided to place the code cache outside the Physical Memory Mapping area, at the end of the vmalloc area. After the kernel image is emitted as an ELF executable, a simple GNU objcopy script extracts the swappable pages from the image and places them in a second file. The remaining kernel image, which now no longer includes the swappable pages, is then used to generate the bzImage file. The swappable pages are stored in a separate partition of the device’s Flash memory, and the generated bzImage is installed and booted just like a regular kernel.

188

4.4

D. Chanet et al.

The Modified Page Fault Handler

Our implementation is embedded into a driver statically linked in the Linux kernel. The driver’s init procedure is called by the init kernel thread. This procedure saves the address of the original page fault handler and replaces it with a new handler, whose functionality is described later in this section. Also, the initialization procedure reserves a virtual address range at the end of the vmalloc area for the code cache. Moreover, it initializes the page table entries corresponding to the swappable code pages in the Master Page Table. Although the Linux kernel makes use, when possible, of 4 MiB pages, we should split the 4 MiB pages related to the swappable code because our implementation works on a 4 KiB page granularity. We create second-level page tables and initialize their entries as not present. Splitting 4 MiB pages can affect the TLB hit rate. Assuming that the kernel code size is smaller than 4 MiB, a TLB entry related to a 4 MiB page table entry maps the entire kernel code space. Note that these modifications to the Master Page Table are performed before creating any user process, so it is not necessary to propagate them to any processes’ page tables. When a page fault occurs, our new page fault handler checks if the virtual address responsible for the fault belongs to the swappable code address range. If that is not the case, control is handed over to the original page fault handler. Otherwise, our handler deals with the fault: a page is allocated in the code cache, the corresponding page is copied from secondary storage to physical memory, the corresponding second-level kernel page table is updated accordingly, and the faulting instruction is re-executed. Note that swapping cold pages in and out of memory does not affect the first-level page tables. It only affects the second-level kernel space page tables, which are shared among the kernel and all processes. We have implemented three basic replacement algorithms: round robin, random and not recently used (NRU). The latter is implemented by periodically resetting the accessed flag of the page-table entries and flushing the TLB. When the cache is full, the page to be evicted is chosen randomly from those that have an unset accessed flag. 4.5

Portability

The proposed technique is easily portable to other operating systems. The only prerequisite for the OS kernel is that it supports virtual memory, and that it is possible to adapt the page fault handler so that it loads the cold code from the repository. We have studied the source code of FreeBSD (a Unix-like OS) and ReactOS (an open source Windows NT clone), and in both cases we were able to easily identify the page fault handler routine in which to insert our page loading code. Of course, to gather the profile information and to split the cold code from the unswappable code, one also needs to have a binary rewriter that is capable of rewriting the OS kernel. While our binary rewriter is only capable of rewriting Linux kernels, it does not rely on any specific Linux concepts to enable reliable binary rewriting. Consequently, we believe porting the binary rewriter to other OS kernels is only an implementation challenge, not a conceptual one.

Linux Kernel Compaction through Cold Code Swapping

5

189

Evaluation

In this section, we first describe the environment used in our evaluations. Second, we evaluate the partitioning algorithms proposed in this work. Finally, we explore several dimensions of the design space of the proposed mechanism (code cache size and page replacement algorithm). 5.1

Evaluation Environment

The evaluation has been carried out using the 2.4.25 Linux kernel on an i386 system, but our mechanism can easily be applied to other operating systems on other platforms with only minor changes. The benchmark suite used to stress the system is Mediabench II [1], which is suitable for testing embedded systems since it is composed of very specific multimedia applications. The full set of programs is: cjpeg, djpeg, h263dec, h263enc, h264dec, h264enc1 , jpg2000dec, jpg2000enc, mpeg2dec, mpeg2enc, mpeg4dec, and mpeg4enc. Each program has been executed 5 times in order to obtain more accurate results. The used input datasets are the ones bundled with the source code of the benchmark suite. Basic block profiles have been collected by running the full benchmark suite on the target machine. In order to reduce external activities in the system, all the user-space daemons have been stopped. This way, most of the kernel code used by the benchmark applications is considered hot, and is hence placed in the base partition. There could be some noise in the profile since it is collected from the boot of the system to the end of the benchmark execution; therefore some code only executed during the boot could be considered hot although it might not be used by the benchmark applications. The performance metrics considered in our evaluations are the kernel code memory footprint, the number of page faults caused by our mechanism, and the system-mode execution time observed during the sequential execution of all the applications in the benchmark. The used hardware platform is an i386-compatible VIA C3 processor clocked at 1200Mhz, 256 MiB of RAM (limited to 64 MiB as we explain later), a VT823x chipset, and an IDE UDMA2 hard disk. As we did not have access to an embedded system with integrated Flash memory at the time of writing, we cannot give accurate measurements for the slowdown incurred by the code loading operations. However, we believe that we can still provide very accurate estimates of the slowdown by inserting a realistic delay in the code loading mechanism that simulates the latency caused by reading a 4 KiB page from Flash memory. In our current implementation, the swappable code resides in RAM, but in a physical range that is not used by the kernel (the kernel’s physical memory usage was limited through the mem command line parameter at boot time). The swapped-out code is loaded into this physical range by a modified version of the GRUB boot loader. Loading code into the kernel-visible physical memory is then 1

H264 encoder has been excluded due to its excessive execution time.

190

D. Chanet et al.

original minimized non-init code modules core and base code + data + code cache core and base code + data core code + data data

1200

1000

Size (KiB)

800

600

400

200

0 100

99.99

99.98

99.95

99.9

99.8

99.5

99

98

95

90

80

50

Threshold (%)

Fig. 4. Static non-init kernel footprint per threshold with and without a 16-page code cache

simply implemented by copying the appropriate page from this memory range into the physical code cache frames. The latency that would be incurred by loading from a Flash device is simulated by a delay that is inserted in the copying code. In order to get a good estimate for this delay value, we have measured the time needed to read a 4 KiB block from Flash memory on a real embedded device. For this measurement, we have used an Intrynsic CerfCube 255, with a PXA255 XScale (ARM-based) processor clocked at 450 Mhz and 32 MiB of Intel StrataFlash J3 NOR-based Flash ROM. On this system, reading a contiguous 4096-byte block from Flash takes approximately 442 µs. Consequently, we have used a delay value of 442 µs for our measurements. 5.2

Results

We first investigate the impact of the hot code threshold value T (see Section 4.2) on the kernel code size. Next, we study the effect of the different code placement strategies described in Section 3 on the kernel performance. Finally, we study the impact of the code cache size and the page replacement policy on the performance, given a fixed code placement algorithm and a fixed value of the threshold T . Table 1. Original kernel characterization (sizes are in KiB and times in seconds) Name Code size Data size System ex. time User ex. time Original 657 312 7.00 395.76 Minimized non-init code 581 286 7.27 395.60

Linux Kernel Compaction through Cold Code Swapping 35000

191

per entry point (16) per entry point (32) per entry point (64) profile based (16) profile based (32) profile based (64)

30000

Kernel page faults

25000

20000

15000

10000

5000

0 100

99.99

99.98

99.95

99.9

99.8

99.5

99

98

95

90

80

50

Threshold (%)

Fig. 5. Kernel code page faults for the different code placement strategies

Table 1 summarizes the characteristics of both the original kernel and one in which the amount of non-init code has been minimized by removing all unreachable code and data and moving non-init code to the init sections where possible, as described in Section 4.3. The table shows the sizes of the non-init code and data and the system mode and user mode execution time for one run of the benchmark set. The system-mode execution time for the kernel with minimized non-init code is 4% higher than for the original kernel. This difference can be ascribed to the different code layouts in both kernels, which has an effect on I-cache utilization, and thus results in small variations in execution time. Influence of the Hot Code Threshold on Kernel Code Size. The graph in Figure 4 shows the kernel’s static memory footprint in function of the hot code threshold T , both with and without a 16-page code cache included. The sizes shown here do not include the initialization code and data, as these are removed from memory during the boot process and as such are not relevant for the kernel’s memory usage during the time the system is actually in use. For reference, the footprint of the original kernel, one with minimized non-init code and one with all optional functionality compiled as modules are also shown. Note that the modules line only refers to the size of the image of a kernel compiled with module support but it does not include the footprint of any of the modules, so it marks the lower bound of its code footprint. Some of the functionality provided by the modules is always needed (e.g. root filesystem, disk device driver) and they are almost permanently resident in memory. Other modules are loaded and unloaded as their functionality is used (e.g. ELF loading). Therefore, during normal operation, the actual code footprint is higher since some modules are resident in memory. While the static non-init footprint of the original kernel is 978 KiB, it is reduced to 867 KiB in the kernel with minimized non-init code, and to 718 KiB

192

D. Chanet et al. 40

per entry point (16) per entry point (32) per entry point (64) profile based (16) profile based (32) profile based (64) original

35

Execution time (seconds)

30

25

20

15

10

5

0 100

99.99

99.98

99.95

99.9

99.8

99.5

99

98

95

90

80

50

Threshold (%)

Fig. 6. System-mode execution time of the benchmark suite for the different code placement strategies

in a kernel compiled with module support. With the cold code removed from memory, the footprint ranges from 548 KiB (T = 100%) to 398 KiB (T = 50%). Taking into account a realistic code cache size of 16 pages, this becomes 612 KiB to 462 KiB, which amounts to a gain of 37.5% to 53.8% respectively when compared to the original kernel. Looking only at the non-init code sizes, the gains are 53% (code + code cache size 324 KiB) to 74.8% (code + code cache size 174 KiB) respectively. As Figure 4 clearly shows, the interesting range for T values lies between 100% and 98%. Swapping out code at lower threshold values has almost no impact on the footprint reduction, whereas that it does have a significant impact on the kernel’s performance, as we show later. Code Partitioning Evaluation. In order to evaluate the effectiveness of the code placement strategies discussed in Section 3.2, we have generated kernels for the same threshold values as used in Figure 4, for each combination of the placement strategies (profile-based and per-entry point) and three different code cache sizes (16, 32, and 64 pages). For each of these kernels, we recorded the total number of code loading events (i.e., kernel space page faults, Figure 5) and the total system-mode execution time (Figure 6) during a run of the benchmark suite. Pages in the code cache are replaced according to an NRU policy. The user-mode execution times for the benchmark runs remained largely the same as for the original kernel, and are not shown here. As can be expected, the profile-based code placement strategy is much more effective than the per-entry point strategy for lower values of T : the hotter the code that is swapped out, the more sense it makes to rely on profile information to guide the placement. For threshold values lower than 99.95 (using a 16-page or 32-page code cache), the number of page faults shown by the per-entry point algorithm

Linux Kernel Compaction through Cold Code Swapping 1200

193

Random Round Robin NRU

1000

Faults

800

600

400

200 48

48

48

0 16

32

64

Swap-in buffer size (pages)

Fig. 7. Influence of the code cache size and the cache eviction policy on the amount of kernel code page faults

grows exponentially. On the other hand, the profile-based strategy shows acceptable results even for a 16-page code cache. Using a 64-page code cache shows a reasonable number of page faults for the per-entry algorithm, although it remains much lower and almost constant for the profile-based algorithm. For reference, Figure 6 also includes the system-mode execution time of the benchmark suite on the original kernel (7 s). Interestingly, when the number of page faults is low enough (less than 1000) the kernel with swapping enabled outperforms the original kernel for all placement strategies. This can be attributed to a better I-cache utilization, as in the rewritten kernels the hot code is separated from the cold code, which is the basic concept behind code layout optimizations for improved cache utilization like the one proposed by Pettis and Hansen [23]. For lower values the per-entry point algorithm shows a large performance degradation caused by the huge number of page faults, while the profile-based algorithm obtains much better results, especially when using a 32page (maximum slowdown of 1.3 s or 19%) or 64-page code cache (speedups from 0.7 to 0.2 s – 10% to 3%). These results suggest that it is not very useful to use T values lower than 99.9%, as there is only little code size to be gained. With T = 99.9%, a 16-page cold code cache suffices to get adequate performance: a code footprint reduction of 68% is coupled with a slight speedup of 2.2% in the system-mode execution time. Design Space Exploration. Finally, we explore the impact of the in-kernel eviction policy on the performance of the code-loading scheme. For these experiments, we focused on a kernel rewritten using the profile-based code placement strategy and a hot code threshold value T = 99.9%.

194

D. Chanet et al. 12

Random Round Robin NRU original

Execution time (seconds)

10

8

6

4

2

0 16

32

64

Swap-in buffer size (pages)

Fig. 8. Influence of the code cache size and the cache eviction policy on the systemmode execution time

Like in the previous graphs, code cache sizes are varied between 16, 32 and 64 pages, the following cache eviction policies are tried: round robin, random replacement and not-recently-used (NRU) replacement. First, the influence of the code cache size and the cache eviction policy on the number of page faults is shown in Figure 7. We observe that a 64-page code cache holds the entire working set of swappable code pages. Decreasing the code cache size significantly increases the amount of kernel page faults. However this number of faults is not enough to significantly penalize the system level execution time, as shown in Figure 8. There is a slight speedup when compared to the original kernel on all our tests using different code cache sizes. We presume this to be caused by the aforementioned I-cache utilization effects. The difference in execution times for the different combinations of cold code cache size and replacement algorithm is proportional to the measured number of page transfers. Focusing on the cache eviction policy, we observe that NRU always performs best when looking at the number of page faults. The same goes for system-mode execution times, with the exception of the 64 page case. There, as the whole working set of the kernel fits into the code cache, the number of page faults remains constant for the 3 eviction policies. Impact on the Performance of I/O-Heavy Applications. For each kernel page fault, a 4 KiB page has to be read from Flash. One might wonder in which way this affects the performance of user space programs that make heavy use of the Flash memory for loading and storing data. Fortunately, the impact of the code loading on the total available Flash bandwidth is small. Assume that we have chosen a threshold of T = 99.9%, with a 16-page cold code cache and the NRU replacement policy. For our test environment, this boils down to a total of 818 page faults over the execution of the benchmarks. Each page fault takes

Linux Kernel Compaction through Cold Code Swapping

195

approximately 442 µs, so the total time spent reading cold pages from Flash is approximately 0.36 s. Given the total execution time of the benchmarks (user time + system time) of 403.2 s, this means that the code loading takes up only 0.09% of the total Flash bandwidth.

6

Related Work

In this section, we will discuss the related work with regards to OS kernel memory footprint reduction, on-demand code loading and minimization of page fault occurrences through code reordering. 6.1

Operating System Memory Footprint Reduction

The idea of specializing the Linux kernel for a specific application was first explored by Lee et al. [21]. Based on source code analysis, a system-wide call graph that spans the application, the libraries and the kernel is built. On this graph, a reachability analysis is performed, resulting in a compaction of a Linux 2.2 kernel of 17% in a simple case study. Chanet et al. [7] use link-time binary compaction techniques to reduce the memory footprint of the Linux kernel. For systems that have a known, fixed hardware and software configuration, several specialization techniques that reduce the memory footprint even further are introduced. Run-time static memory footprint reductions (that is, not counting the kernel’s dynamically allocated memory) of about 16% were achieved for Linux 2.4 kernels compiled for the ARM and i386 architectures. He et al. [19] use similar binary rewriting techniques to reduce the code size of the Linux kernel. A novelty in their approach is the use of approximate decompilation to generate C source code for hand-written assembly code in the kernel. This allows the use of a source code based pointer analysis (the FAanalysis [22]) for the identification of targets of indirect function calls. While the generated source code is not functionally equivalent to the original assembler code, it exhibits the same properties with regards to this FA-analysis. On a Linux 2.4 kernel without networking support, they report a code size reduction of 23.83%. For the same test system as used by Chanet et al. the results roughly correspond to those from [7], suggesting the two techniques are approximately equal in strength. Later work by Chanet et al. [8] extends on the previous techniques by means of code compression techniques. Through code coverage analysis, frozen code is identified. This code is then stored in compressed format and decompressed at run time only if it is actually needed. To avoid concurrency issues, once code is decompressed, it is never evicted from memory. This makes this technique useless for the more general cold code compression, as the cold code will be executed at least once, and thus still take up memory for the rest of the system’s run time. However, even after compaction and specialization of the kernel, there is still over 50% frozen code in the kernel, making this technique worthwhile. The combined compaction, specialization and compression techniques reduce the static

196

D. Chanet et al.

memory footprint of a Linux 2.4 kernel with 23.3% for the i386 architecture and 28% for the ARM architecture. The smaller kernels suffered from a performance degradation of 2.86% (i386) and 1.97% (ARM). The same evaluation systems were used as in [7], so the results can be compared directly. An alternative approach to customize an OS for use in embedded devices is proposed by Bhatia et al. [6]. The authors of this paper propose to remotely customize OS modules on demand. A customization server provides a highly optimized and specialized version of an OS function on demand of an application. The embedded device needs to send the customization context and the required function to the server and on receipt of the customized version, applications can start using it. The size of the customized code is reduced up to a factor of 20 for a TCP/IP stack implementation for ARM Linux, while the code runs 25% faster and throughput increases by up to 21%. While our approach to minimize the kernel’s memory footprint is top-down in that we start with a full-featured kernel and strip away as much unneeded functionality as possible, there is a number of projects that take a bottomup approach. The Flux OSKit [15], Think [13] and TinyOS [16] are operating system construction frameworks that offer a library of system components to the developer, allowing him to assemble an operating system kernel containing only the needed functionality for the system. 6.2

On-Demand Code Loading

The technique proposed in this paper is of course very much influenced by the virtual memory techniques used in most modern processors and operating systems [20]. The most important differences are that we selectively swap only parts of the kernel code in order to reduce the number of necessary code cache refills and that most VM systems use the hard disk, which is very slow compared to main memory, as the off-line storage medium. As such, swapping incurs much bigger latencies, that make the method less suitable for application in timingcritical programs like an OS kernel. Citron et al. [9] propose to remove frozen code and data from the memory image of a program. Control transfers to frozen code and memory accesses to frozen data are replaced by illegal instructions. The interrupt that occurs on execution of these illegal instructions is then intercepted and used as a trigger to load the needed code or data. A size reduction of 78% for the MediaBench suite of programs is reported. As only frozen code and data are loaded on demand, there are very little load events necessary, which makes the performance impact negligible. The approach is conceptually similar to ours, with the illegal opcode exception replacing the page fault exception as the trigger to load new code. Because this approach is not bound to the VM system, it is possible to load code and data at smaller granularity than the 4 KiB blocks that we use. However, while the paper mentions that once-loaded code may be evicted when memory is low, the authors do not discuss the concurrency issues this entails in multithreaded programs. As such, it is not clear how well their technique holds up for this kind of programs.

Linux Kernel Compaction through Cold Code Swapping

197

Debray and Evans [11] use software-controlled code compression to reduce the code size of programs for the Alpha architecture. With profile data infrequently executed code fragments are detected. These fragments are stored in memory in a compressed form, and replaced by stubs. Upon execution of a stub, the necessary code is decompressed in a fixed-size buffer. When applied to programs that were already optimized for code size using link-time compaction techniques [12], additional code size reductions of 13.7% to 18.8% were achieved. The performance impact ranges from a slight speedup to a 28% slowdown. Again, no attention is given to concurrency issues that may arise with code eviction in multithreaded programs. 6.3

Code Reordering for Page Fault Minimization

Hatfield and Gerald [18] describe a technique that aims to minimize the number of page faults for both code and data references. The code and data are divided into a set of relocatable blocks (e.g. an array or a procedure). Using profile data, a nearness matrix is constructed, with one row and column for each relocatable block. Entry cij of this matrix represents the count of references from block i to block j. Virtual memory pages correspond to square regions along the diagonal of the matrix. By reordering the rows and columns of the matrix, the largest entries are brought closest to the diagonal, which corresponds to placing the blocks that reference each other most on the same page. Ferrari [14] formulates the problem as a graph clustering problem. Nodes in the graph represent relocatable blocks. The weight of a node equals the size of the block it represents. Edges in the graph represent interblock references, and can be weighted according to various cost functions. An optimal ordering is then sought by clustering graph nodes in such a way that no node becomes larger than the page size and the total weight of the remaining edges is minimal. If the edges are weighted according to profile information, this method is equivalent to that of Hatfield and Gerald. The author proposes a better-performing edge weighting, however, that is based on a trace of the block references during execution instead of mere profile data. Pettis and Hansen [23] propose to reorder the procedures in a program in such a way that those procedures that call each other most frequently are placed closest together. Their main aim is to reduce the number of conflict misses in the instruction cache, but they note that this placement algorithm also reduces the number of page faults during program execution. Once again, the program is represented as a graph, with the nodes representing the procedures, and edges representing procedure calls. The edges are weighted according to profile information. In each step of the algorithm, the edge with the highest weight is selected and its head and tail nodes are merged. This method does not prevent procedures from spanning page boundaries. Gloy and Smith [17] also reorder procedures to improve a program’s instruction memory hierarchy behaviour. Their technique is similar to Pettis and Hansen’s, but instead of profile information they use temporal ordering information, which not only summarizes the number of calls from one procedure to

198

D. Chanet et al.

another, but also in which way these calls are interleaved. While their approach is in the first place directed towards optimization of the cache utilization, the authors also discuss an extension of the technique to minimize the amount of page faults.

7

Conclusions and Future Work

In this paper we introduced a novel on-demand code loading technique aimed at reducing the memory footprint of operating systems kernels for embedded systems with support for virtual memory. The page fault mechanism provided by the processor is used to trigger code loading and to avoid concurrency issues related to eviction of already-loaded code from memory. By using profile information, we can limit the code loading scheme to infrequently executed code and thus limit the performance impact. For a case study involving the Linux 2.4.25 kernel on an i386 platform, we were able to reduce the static kernel memory footprint (code + data) with up to 54.5%, with a slight speedup of the system-mode operations of 2.2% for our best-performing code placement strategy. When taking user-mode execution time into account, the speedup drops to 0.04%, which is negligible. Consequently, the proposed technique is a viable means of reducing the memory footprint of an OS kernel for use in an embedded system. We have investigated two different code placement strategies: one based on profile information and one that just takes the static structure of the code into account, and have shown that the profile-based strategy is always best, even when placing only cold code, for which there is little profile information available. In future work, we will focus on improving the code placement strategies to reduce the amount of page faults that still occur, and extend the technique to on-demand loading of read-only kernel data sections.

Acknowledgments The authors wish to acknowledge the HiPEAC European Network of Excellence for the support it has given to our research. The work of Dominique Chanet was funded in part by the Flanders Fund for Scientific Research (FWO-Vlaanderen). We would also like to thank Bruno De Bus for his excellent insights and suggestions.

References 1. Mediabench II benchmark, http://euler.slu.edu/~ fritts/mediabench/ 2. The Intel XScale microarchitecture technical summary, http://download.intel.com/design/intelxscale/XScaleDatasheet4.pdf 3. MIPS32 4Kc processor core data sheet, http://www.mips.com/content/Documentation/MIPSDocumentation/ ProcessorCores/4KFamily/MD00039-2B-4KC-DTS-01.07.pdf/getDownload

Linux Kernel Compaction through Cold Code Swapping

199

4. The Texas Instrument OMAP platform, http://www.ti.com/omap 5. Discussion on kernel paging on the linux kernel mailing list (April 2001), http://lkml.org/lkml/2001/4/17/115 6. Bhatia, S., Consel, C., Pu, C.: Remote customization of systems code for embedded devices. In: EMSOFT 2004: Proceedings of the 4th ACM international conference on Embedded software, pp. 7–15. ACM Press, New York (2004) 7. Chanet, D., De Sutter, B., De Bus, B., Van Put, L., De Bosschere, K.: Systemwide compaction and specialization of the Linux kernel. In: Proc. of the 2005 ACM SIGPLAN/SIGBED Conference on Languages, Compilers, and Tools for Embedded Systems (LCTES), pp. 95–104 (2005) 8. Chanet, D., De Sutter, B., De Bus, B., Van Put, L., De Bosschere, K.: Automated reduction of the memory footprint of the linux kernel. ACM Transactions on Embedded Computing Systems (TECS) 6(2) (to appear, 2007) 9. Citron, D., Haber, G., Levin, R.: Reducing program image size by extracting frozen code and data. In: EMSOFT 2004: Proceedings of the 4th ACM international conference on Embedded software, pp. 297–305. ACM Press, New York (2004) 10. De Sutter, B., De Bus, B., De Bosschere, K.: Link-time binary rewriting techniques for program compaction. ACM Transactions on Programming Languages and Systems 27(5), 882–945 (2005) 11. Debray, S., Evans, W.: Profile-guided code compression. In: PLDI 2002: Proceedings of the ACM SIGPLAN 2002 Conference on Programming language design and implementation, pp. 95–105. ACM Press, New York (2002) 12. Debray, S.K., Evans, W., Muth, R., De Sutter, B.: Compiler techniques for code compaction. ACM Transactions on Programming Languages and Systems 22(2), 378–415 (2002) 13. Fassino, J.-P., Stefani, J.-B., Lawall, J.L., Muller, G.: Think: A software framework for component-based operating system kernels. In: Proceedings of the General Track: 2002 USENIX Annual Technical Conference, Berkeley, CA, USA, pp. 73–86. USENIX Association (2002) 14. Ferrari, D.: Improving locality by critical working sets. Commun. ACM 17(11), 614–620 (1974) 15. Ford, B., Back, G., Benson, G., Lepreau, J., Lin, A., Shivers, O.: The Flux OSKit: a substrate for kernel and language research. In: SOSP 1997: Proceedings of the sixteenth ACM symposium on Operating systems principles, pp. 38–51. ACM Press, New York (1997) 16. Gay, D., Levis, P., von Behren, R., Welsh, M., Brewer, E., Culler, D.: The nesC language: A holistic approach to networked embedded systems. In: PLDI 2003: Proceedings of the ACM SIGPLAN 2003 conference on Programming language design and implementation, pp. 1–11. ACM Press, New York (2003) 17. Gloy, N., Smith, M.D.: Procedure placement using temporal-ordering information. ACM Trans. Program. Lang. Syst. 21(5), 977–1027 (1999) 18. Hatfield, D.J., Gerald, J.: Program restructuring for virtual memory. IBM Systems Journal 10(3), 168–192 (1971) 19. He, H., Trimble, J., Perianayagam, S., Debray, S., Andrews, G.: Code compaction of an operating system kernel. In: Proceedings of Code Generation and Optimization (CGO) (March 2007) (to appear) 20. Hennessy, J.L., Patterson, D.A.: Computer architecture: a quantitative approach, ch. 5. Morgan Kaufmann Publishers Inc., San Francisco (2002)

200

D. Chanet et al.

21. Lee, C.-T., Lin, J.-M., Hong, Z.-W., Lee, W.-T.: An application-oriented Linux kernel customization for embedded systems. Journal of Information Science and Engineering 20(6), 1093–1107 (2004) 22. Milanova, A., Rountev, A., Ryder, B.G.: Precise call graphs for C programs with function pointers. Automated Software Engg. 11(1), 7–26 (2004) 23. Pettis, K., Hansen, R.C.: Profile guided code positioning. In: PLDI 1990: Proceedings of the ACM SIGPLAN 1990 conference on Programming language design and implementation, pp. 16–27. ACM Press, New York (1990)

Complexity Effective Bypass Networks Aneesh Aggarwal Department of Electrical and Computer Engineering Binghamton University, Binghamton, NY 13902 [email protected]

Abstract. Superscalar processors depend heavily on broadcast-based bypass networks to improve performance by exploiting more instruction level parallelism. However, increasing clock speeds and shrinking technology make broadcasting slower and diffi cu lt to implement, especially for wide issue and deeply pipelined processors. High latency bypass networks delay the execution of dependent instructions, which could result in significant performance loss. In this paper, we first perform a detailed analysis of the performance impact due to delays in the execution of dependent instructions caused by high latency bypass networks. We found that the performance impact due to delayed data-dependent instruction execution varies based on the data dependence present in a program and on the type of instructions constituting the program code. We also found that the performance impact varies significantly with the hardware configuration, and that with a high latency bypass network, the processor hardware critical for nearmaximal performance reduces considerably. We then propose Single FU bypass networks to reduce the bypass network latency, where results from an FU are forwarded only to itself. The new bypass network design is based on the observations that an instruction’s result is mostly required by just one other instruction and that the operands of many instructions come from a single other instruction. The new bypass network results in significant reduction in the data forwarding latency, while incurring only a small impact (about 2% for most of the SPEC2K benchmarks) on the instructions per cycle (IPC) count. However, reduced bypass latency can potentially increase the clock speed. Single FU bypass networks are also much more scalable than the broadcast-based bypass networks, for more wide and more deeply pipelined future microprocessors.

1

Introduction

The bypass network lies in the most critical loop in pipelined processors that enables data dependent instructions to execute in consecutive cycles [6]. Prior studies [5][10][11] have shown that an increase of a single cycle in this critical loop reduces the instruction throughput dramatically. Most modern processors use a broadcast-based bypass network, where a result produced by a functional unit (FU) is made available at the inputs of all the other FUs. With a broadcastbased bypass network, bypassing can take significant amounts of wiring area on P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 201–221, 2009. c Springer-Verlag Berlin Heidelberg 2009 

202

A. Aggarwal

the chip [3]. Furthermore, the wire complexity grows proportional to the square of the issue width and the pipeline depth [3][7]. This increases the wire path delay from the source to the destination of bypass network [9]. The problem is further exacerbated with the growing importance of wire delays in the submicron technology era [1][12]. In fact, bypass network latency is expected to be one of the major bottlenecks in future micro-processors [7][9]. The overall impact of the broadcast-based bypass network complexity is that multiple cycles may be required to forward the values from the producer instruction to the consumer instructions [9]. A multi-cycle bypass network delays the execution of dependent instructions, decreasing the instruction throughput and the overall performance. Increased bypass widths not only impact performance, but also increase the power consumption (which has become a primary design issue [23][24]) due to the wide multiplexors at each destination and the increased number of long wires, and reduce reliability by increasing the cross-talk between the wires and by reducing the signal strength. In the first part of this paper, we study – both qualitatively and quantitatively – the performance impact due to delays in the execution of dependent instructions (because of multi-cycle data forwarding). The forwarding latencies are expected to increase in future microprocessors and they may not be able to execute the dependent instructions in consecutive cycles [8]. Hence, such a study becomes very important to develop efficient techniques (both hardware as well as software) to avoid the performance impact. In particular, we address the following questions: – What kind of programs incur more performance impact due to multi-cycle data forwarding? – How does hardware configurations of a processor affect the performance impact? – What is the minimum hardware configuration that is critical for getting near-maximal performance with delayed execution of dependent instructions? We found that the performance impact due to multi-cycle data forwarding depends on the data dependence present in a program and on the type of instructions constituting the program code. We also found that the performance impact varies significantly with the processor configuration in terms of the issue width, fetch width, issue queue size, and cache configuration, and that the hardware required for near-maximal performance reduces considerably if the dependent instructions are delayed in their execution. For instance, with a 2-cycle delay in the execution of dependent instructions, the performance obtained with a fetchwidth of 2 is almost the same as that obtained with a fetch width of 16, whereas with no delays, the difference in performance of the two cases is about 15%. In the second part of this paper, portions of which are also presented in [2], we propose a Single FU (Functional Unit) bypass network to reduce the data forwarding latency. In this bypass network, an FU’s output is forwarded only to its own inputs. Single FU bypass network facilitates low latency and energy-efficient data-forwarding by dramatically reducing the bypass network

Complexity E ff ective Bypass Networks

203

complexity. These benefits are obtained with an Instructions Per Cycle (IPC) impact of only about 2% as compared to a broadcast-based bypass network. We also discuss further reducing the number of bypass paths using a Single Input Single FU bypass network. We also show that a Single FU bypass network is much more scalable than its boradcast-based counterpart, as the width and the depth of the processor pipeline is increased.

2 2.1

Qualitative Analysis Program Characteristics

The two program characteristics that affect the performance impact due to multicycle data forwarding are the data-dependence present in a program and the type of instructions constituting the program code. Data Dependence Characteristics. The performance impact due to multicycle data forwarding can be reduced if independent instructions are available to hide the delays. If the average number of dependent instructions (for an instruction in a program) is larger, then once a result is produced, it is used by more instructions. This amortizes the delays due to data forwarding and increases the probability that independent instructions are available to hide the delays. Such programs are expected to have a lower performance impact due to multi-cycle data forwarding latencies. Even if the average number of dependants is low, but multiple independent strands of data-dependent instructions are present in a program, the performance impact may still reduce. Figure 1 illustrates an instance showing the effect of the number of dependants (for the same set of 9 instructions) on the performance impact due to multicycle data forwarding. In Figure 1, we assume an issue width of 2 instructions per cycle, and a one cycle delay in the execution of dependent instructions. In Figure 1(a), instructions have a higher average number of dependent instructions, and the overall delay is 1 cycle or 20%. Note that the delays in the execution of i5 and i6 are hidden by i4 and i9 , and that in the execution of i7 and i8 by i5 and i6 . In Figure 1(b), instructions have a smaller average number of dependent instructions, and the overall delay is 4 cycles or 80%, because none of the delays could be hidden. In Figure 1(c), even though the average number of data dependent instructions is small, multiple independent strands of instructions result in an overall delay of 0 cycles, because all the delays in the execution of dependent instructions are hidden. Another data-dependence characteristic that affects the impact due to multicycle data forwarding is the effective distance between the producer and the consumer instructions. The larger the distance, the higher the probability that the consumer instruction is not in the issue queue when the producer executes, thus hiding the delays (or at least part of it) due to the bypass network. An increase in dynamically dead instructions, i.e. instructions whose results are never used, also reduces the performance impact due to delays.

204

A. Aggarwal

(no delay) (1−cycle delay)

(no delay) cycle t

cycle t + 1

i2

cycle t + 3

i4

i5

i9

cycle t

i8

cycle t + 5

cycle t + 4

i3

i4

cycle t + 3

cycle t + 4

i7

i2

cycle t + 2

cycle t + 3

i6

cycle t + 4

i1

cycle t + 1

cycle t + 2

i3

cycle t + 2

(1−cycle delay)

cycle t

cycle t

i1

i5

cycle t + 4

i9

i6

i7

cycle t + 2

i8

cycle t + 6

cycle t + 8

(b)

(a)

(no delay) cycle t

(1−cycle delay) i1

i2

cycle t + 1

cycle t + 2

cycle t + 3

cycle t

i4

i5

cycle t + 1

i3

cycle t + 2

i6

i7

i8

i9

cycle t + 4

cycle t + 3

cycle t + 4

(c)

Fig. 1. Illustration (a) Broad Dependence Chain Tree; (b) Long Narrow Dependence Chain; (c) Multiple Long Narrow Dependence Chains

Program Code Constitution. We define program code constitution as the number and type of different instructions constituting the program code. Program code constitution affects the performance impact due to multi-cycle data forwarding latencies because delays in the execution of some instructions may impact performance more than others. For instance, delays in the execution of stores and correctly predicted branch instructions may not impact performance, whereas that in the execution of load instructions may. Note that, a delay in the execution of some store instructions may result in some load instructions getting delayed, either because of delays in address disambiguation, or because of delays in store-load forwarding. Similarly, delayed execution of a mispredicted branch instruction increases the branch misprediction penalty. Nevertheless, a larger percentage of branch and store instructions in a program reduces the percentage of result-producing instructions and increases the percentage of instructions whose results are only used by the branch and store instructions, reducing the overall performance impact. Furthermore, the performance impact is higher for long latency instructions, such as multiply, divide, and load instructions missed in the cache, because the effective distance between a such instructions and their dependants is less. However, our studies showed that variations in the number of long latency instructions had a negligible effect on performance impact due to multi-cycle data forwarding latencies, because the dependants are usually in

Complexity Effective Bypass Networks

205

the issue queue before the producer instruction is issued. A program with a high branch misprediction rate will have lower performance impact because many of the instructions (fetched after a branch misprediction) do not get their operands from the bypass network, instead they get their operands from the register file. 2.2

Hardware Parameters

In this section, we discuss the effect of different hardware parameters, on the performance impact due to high latency bypass networks. Issue Width. Reduction in issue width increases the delay hiding capability in the execution of dependent instructions. For instance, in Figure 1(a), for an issue width of 6, i2, i3, i4, and i9 will execute in the same cycle, and the delays in i5, i6, i7, and i8 are not hidden, and the overall delay is 2 cycles (with 3 cycles required for no delays) or almost 70%. This effect of the issue width variation also depends on the average number of dependants per instruction. For instance, comparing the examples in Figure 1(a) and 1(b), reduction of issue width from 6 to 2, reduces the overall delay for the example in Figure 1(a) from 70% to 20%, whereas it does not affect the overall delay for the example in Figure 1(b). Similarly, the effect of the issue width variation is expected to be higher for higher IPC benchmarks. Reduced issue widths may also slightly reduce the number of independent ready instructions because of fewer instructions being issued. However, we observed that the effects of hiding the delays overweigh the effects of reduced number of independent instructions per cycle. Fetch/Dispatch Width. A decrease in the fetch/dispatch width increases the probability of dispatched instructions being ready to execute. Reduced fetch width also increases the effective distance between the producer and the consumer instructions. Both these effects increase the delay hiding capability of the processor and will be more prominent for smaller fetch widths. However, reduced fetch width has the disadvantage of reducing the number of independent ready instructions per cycle. For reduced fetch width also, we observed that the ability to hide the delays overweighs the negative effects of reducing the fetch width. Issue Queue Size. The effect of reduction in issue queue size is very similar to that of reduction in fetch width, because with a reduced issue queue size, once the queue fills up, the number of instructions dispatched into the issue queue is equal to the ones that are issued out of the queue, which effectively reduces the fetch/dispatch width. Cache Performance. The effect of cache performance on the performance impact due to multi-cycle data forwarding is different for the data and the instruction caches. An increase in the data cache miss rate increases the number of instructions with larger latency, which can increase the performance impact due to the delays. However, an increase in the instruction cache miss rate has an effect very similar to reducing the effective fetch width because of the increased number of cycles required to fetch many of the instructions.

206

3 3.1

A. Aggarwal

Performance Impact Experimental Setup

The processor parameters used in our experiments are given in Table 1. We use a modified version of the Simplescalar simulator [25] for our experiments, using a 32bit PISA instruction set architecture. In our simulator, instead of a single RUU structure depicting the ROB, issue queue, and register file, we have a separate ROB, issue queue and integer and floating point register files. For benchmarks, we use 7 Integer (gzip, vpr, gcc, mcf, parser, bzip2, and twolf) and 7 Floatingpoint (wupwise, art, swim, ammp, equake, apsi, and mesa) benchmarks from the SPEC2K benchmark suite, compiled with the options provided with the suite, and using the ref inputs. We use this subset because we could only compile these benchmarks in our simulation environment. For the measurements, we skip the first 500M instructions and collect the statistics for the next 500M instructions. 3.2

Impact of Multi-cycle Forwarding

First, we measure the impact of increasing forwarding latency from zero to two cycles on IPC. Dependent instructions can execute in consecutive cycles only with a 0-cycle forwarding latency. Figure 2 shows the IPC along the Y-axis. As can be seen in Figure 2, there is a significant reduction in the IPC as the forwarding latency is increased. For instance, as compared to a 0-cycle forwarding latency, IPC reduces by about 15% for a 1-cycle forwarding latency for both integer and floating point benchmarks. The impact of increased forwarding latency is relatively higher for higher IPC benchmarks. More importantly, the IPC impact varies from program to program. For instance, gcc has an IPC impact of about 15% when going from a 0-cycle latency to a 2-cycle latency, whereas gzip (which has an IPC very close to that of gcc) has a performance impact of about 40%. 3.3

Analysis

We measure the average number of result-producing dependent instructions for each result-producing instruction in a program. These statistics are collected Table 1. Default Parameters for the Experimental Evaluation Parameter Fetch/Decode Width Phy. Register File

Value 8 instructions 128 Int/ 128 FP

Parameter Instr. Window Int. FUs

Value 128 instructions 3 ALU, 1 Mul/Div, 2 ld/2 st Issue/Commit Width 6 instructions FP FUs 3 ALU, 1 Mul/Div Branch Predictor bi-modal 4K entries BTB Size 2048 entries, 2-way assoc. L1 - I-cache 32K, direct-map, L1 - D-cache 32K, 4-way assoc., 2 cycle latency 2 cycle latency Memory Latency 40 cycles first chunk L2 - cache unified 512K, 1 cycles/inter-chunk 8-way assoc., 6 cycles

Complexity Effective Bypass Networks

207

Fig. 2. Impact of Multi-cycle Bypass Networks (a)Integer Benchmarks (b)Floating Point Benchmarks

at commit of instructions. Figure 3 presents these measurements in the form of a stacked bar graph. The top portion in each bar represents the non-result producing branch and store instructions. In Figure 3, the bar with 0 dependents also includes the instructions that have only branch and store instructions as dependent instructions. Figure 3 corroborates the qualitative analysis of Section 2.1 to explain the different reduction in IPC observed for different benchmarks in Figure 2. Benchmarks with a higher percentage of one result-producing dependant and a lower percentage of branch and store instructions (e.g. vpr, bzip2, twolf, wupwise, art) (and thus having long and narrow data-dependent chains of instructions) have a higher performance impact. On the other hand, benchmarks with a lower percentage of one result-producing dependant and a higher percentage of branch and store instructions (e.g. gcc, mcf. equake) have a lower performance impact. For instance, consider the benchmarks vpr and gcc. Both the benchmarks have almost the same IPC with a 0-cycle forwarding latency. However, with increased forwarding latency, gcc has significantly less performance impact than vpr. Ammp also has a high percentage of instructions with just one consumer, but its extremely low IPC results in a low IPC impact in its case.

Fig. 3. Instruction Distribution for SPEC2000 (a) Integer Benchmarks and (b) Floating Point Benchmarks

208

4 4.1

A. Aggarwal

Eff e ct of Hardware Configuration Variations in Issue Width

Figure 4 shows the IPCs with issue widths of 6 (first overlayed bar), 4 (second overlayed bar), and 2 (third overlayed bar) for 0, 1, and 2-cycles delay in dependent instruction execution. The rest of the parameters are kept at their default values. As discussed in Section 2.3, if the number of functional units are also reduced along with the issue width, then the bypass latency can be expected to reduce. In these experiments, we keep the number of functional units at its default value. As seen in Figure 4, for both the integer and FP benchmarks, the performance impact due to multi-cycle data forwarding reduces with a decrease in the issue width, conforming the analysis of Section 2. For instance, for gzip, for an issue width of 6, the IPC with a 0-cycle delay is about 60% more than that with a 2-cycle delay, and for an issue width of 2, the difference reduces to about 20%. Another important observation that can be made from Figure 4 is that higher the delays incurred by dependent instructions, lower is the critical issue width required for near-maximal performance. For instance, for a delay of 2 cycles, the IPC obtained with an issue width of 2 is almost equal to that obtained with an issue width of 6 for many of the benchmarks, such as gzip, vpr, gcc, mcf, twolf, wupwise, art, and ammp. From Figures 2, 3 and 4, it can also be observed that benchmarks with a higher IPC, and with a larger average number of dependent instructions have a higher effect of variations in the issue width. 4.2

Variations in Fetch/Dispatch Width

Figure 5 shows that, in general, the performance impact of delayed dependent instructions reduces with a decrease in the fetch width. The results in Figure 5 follow the qualitative analysis performed in Section 2. For instance, for gzip, for a fetch width of 16, the IPC obtained with a 0-cycle delay is about 65% more than that obtained with a 2-cycle delay, and for a fetch width of 2, the IPC

Fig. 4. IPCs for Variations in Issue Width (a) Integer Benchmarks and (b) FloatingPoint Benchmarks

Complexity Effective Bypass Networks

209

Fig. 5. IPCs for Variations in Fetch Width (a) Integer Benchmarks and (b) FloatingPoint Benchmarks

difference reduces to about 40%. Furthermore, Figure 5 suggests that with higher delays in the execution of dependent instructions, fetch width can be lowered, while still obtaining near-maximal performance. For instance, for almost all the benchmarks, with a delay of 2 cycles in the execution of dependent instructions, the IPC obtained with all the fetch widths (16, 8, 4, and 2) is almost the same. 4.3

Variations in Issue Queue Size

Figure 6 shows the IPC for the different issue queue sizes (keeping the other processor parameters at their default values) with delays of 0, 1, and 2 cycles in the execution of dependent instructions. As seen in Figure 6, with a reduction in the issue queue size, the performance impact due to multi-cycle data forwarding reduces. As discussed in Section 2, reduction in issue queue size has an effect similar to reduction in the fetch width. The effect of the variations in issue queue size is not much for issue queue sizes of greater than 16. Our studies found that with very small issue queue sizes of 2 and 4, delays in the execution of dependent instructions almost had no performance impact for many of the benchmarks.

Fig. 6. IPCs for Variations in Issue Queue Size (a) Integer Benchmarks and (b) Floating-Point Benchmarks

210

A. Aggarwal

Fig. 7. IPCs for Variations in Instruction Cache Size (a) Integer Benchmarks and (b) Floating-Point Benchmarks

4.4

Variations in Cache Configuration

We do not present the effect of variation in the data cache configuration on the performance impact due to delays, because the effect is negligible. This is because a variation in data cache configuration affects the performance impact only if an instruction dependent on a load instruction is a result producing instruction and the variation affects the presence of the dependant in the issue queue when the value is loaded. With the parameters used for the experiments, such situations were very rare and hence the effect of data cache configuration on the performance mpact due to the delays was negligible. However, the effect of the instruction cache configuration has a significant effect on the performance impact, as seen in Figure 7. Figure 7 presents the IPC for instruction caches of size 32K, 8K and 2K for delays of 0, 1, and 2 cycles in the execution of dependent instructions. It can be observed from Figure 7 that the performance impact almost disappears for an instruction cache size of 2KB. This is because, the cache miss rate becomes so high that the consumers of many instructions are delayed significantly. This gives enough time for the producers to write their results to the register file.

5

Single FU (SFU) Bypass Network

In this section, we propose Single FU (SFU) bypass networks, where an FU’s output is forwarded only to itself, thus reducing the bypass network complexity. 5.1

Basic Idea

Figure 8(a) shows a Pentium 4 [8] style broadcast-based bypass network for the integer units. A similar bypass network could be implemented for the floating point units. The multi-stage bypass network is responsible for forwarding the correct values from the other FUs and the latter stage of the pipeline. Figure 8(b) shows one configuration of the SFU bypass network for the same set of integer

Complexity Effective Bypass Networks

211

Fig. 8. (a) Conventional Bypass Network; (b) Limited SFU; (c) Extreme SFU

FUs. In this configuration, the output of an ALU is immediately forwarded to only its own inputs. However, the values loaded are typically required in IALU instructions. Hence, instead of forwarding the output of the load unit to itself, it is forwarded to one of the ALUs. In addition, load units typically read the base address from the register file or are, in some cases, forwarded the value from the ALUs. Hence, the output of one of the ALUs is also forwarded to the load unit. Without loss of generality, in Figure 8(b), the output of ALU2 is also forwarded to the load unit and that of load unit is forwarded to ALU2. In this configuration, the results from an FU are immediately available to the ones it is directly connected to, and are available to all the FUs after an additional cycle. We call this bypass network as Limited SFU (LSFU) bypass network. Figure 8(c) illustrates another configuration of the SFU bypass network. In this configuration, the multiplier and store units are completely isolated, i.e. their results are available only from the register file and they read their operands only from the register file. In addition, the results from an FU are only available to the one it is directly connected to, even for bypasses from the latter stages of the pipeline. We call this bypass network as Extreme SFU (ESFU) bypass network. For all the configurations, a similar bypass network can be assumed for the floating point units. 5.2

Motivation

In this section, we present the motivation behind the SFU bypass network. For this, we measure the typical data dependence characteristics in the programs, in terms of the type of instruction producing a value and the type and number of instructions using that value. The type of an instruction is defined by type of the functional unit it uses to execute. We define 6 types of instructions: IALU (simple integer instructions using an ALU), IMULT (complex integer instructions using a multiplier), LOAD (load instructions), STORE (store instructions), FPALU (simple floating point instructions using an ALU), and FPMULT (complex floating point instructions using a multiplier). Figure 9 presents these measure-

212

A. Aggarwal

Fig. 9. Result Usage Characteristics for SPEC2K (a) Integer Benchmarks and (b) Floating-Point Benchmarks

ments in the form of a stacked bar graph. For each integer benchmark, there are 2 sets of stacked bars, where each stacked bar represents the type of instruction producing the register value. Similarly, for each FP benchmark, there are 4 sets of stacked bars. The value on top of each stacked bar represents the percentage of results (out of the total) produced by instructions of that particular type. The total of all the stacked bars is less than 100% because some results are not used at all and some results are produced by instructions of other types. For instance, for gzip, about 35% of the values (out of the total produced) are produced by an IALU instruction and used by just one other IALU instruction. Equake has 0 FP instructions, because FP instructions in equake are encountered beyond the range of instructions simulated. As can be seen in Figure 9, most of the values (about 70%) produced are only used by just one instruction. Next we look at the data dependence characteristics of programs from the consumer’s perspective. Figure 10 presents these statistics1 in a manner similar to Figure 9, except that each stack now represents operand-producing instructions rather than result-consuming instructions. For instance, for gzip, about 68% of the instructions (out of the total executed) are IALU instructions, and about 30% of instructions (out of the total executed) are IALU instructions whose operands are produced by just one other IALU instruction. Figure 10 shows that a significant percentage (about 70%) of the total instructions executed have their operands produced either by just one other instruction or by no instructions. The load instructions that do not have any producer instructions for their operands are the ones which use the same register operand and this register operand is produced before the start of the collection of the statistics. The integer ALU instructions that do not have any producer instructions for their operands are mostly the ones which load an immediate value, or the ones that use register r0 to set a register to an immediate value. 1

We only show Integer instructions even for the FP benchmarks because of the low percentage of FP instructions in the benchmarks.

Complexity Effective Bypass Networks

213

Fig. 10. Operand Production Characteristics for SPEC2K (a) Integer Benchmarks and (b) Floating-Point Benchmarks

Overall, it is observed that for most of the instructions, their results are used by just one other instruction and their operands are produced by just one other instruction. This motivates us to investigate Single FU bypass networks, where the results produced in a FU are only forwarded to its own inputs. 5.3

FU Assignment

The performance of the new SFU bypass network design relies heavily on the ability to assign instructions to the FUs where their operands are available through the bypass network. We propose a post-schedule FU assignment scheme. In this scheme, the FUs assigned to the instructions by the select logic are selectively discarded2 . A pre-schedule FU assignment may require either clustering of the issue queue or additional scheduler logic to decode the assigned FUs. Furthermore, pre-schedule FU assignment may result in considerable performance drop because of unbalanced workload distribution between the different FUs. Once an instruction is scheduled for execution, it is assigned an FU based on where its operands will be available. Figure 11 shows the new pipeline for this FU assignment scheme. All the scheduled instructions access an FU table (in the FU assign/arbiter stage) for each valid operand, to get an FU assigned to them. The number of entries in the FU table is equal to the number of registers. Each FU table entry is only a few bits, equal to log2 (#of F U s), indicating the FU where that register is being produced. Once the FU for an instruction is decided, the FU table entry for its destination register is updated. When the register is written into the register file, its FU table entry is invalidated. From the FU table, the following information is obtained for each valid operand: (i) whether it is available from the bypass network, and if it is, then in which FU, or (ii) whether it is available from the register file. An FU is assigned for the instruction as follows: 2

However, the conventional select logic is still needed to select the right instructions (based on the priority scheme used, which could be the “oldest” first) to be scheduled.

214

A. Aggarwal

Fig. 11. Post-schedule FU Assignment Pipeline

– If an instruction has only one operand with a valid FU where the operand is available from the bypass network, and the other operand is either not present or is available from the register file or is available in any FU from the bypass network (for the LSFU network), then it is assigned the valid FU. – If an instruction has multiple operands with valid FUs or an operand cannot be obtained from the bypass network, then that instruction is marked “unscheduled” and it remains in the issue queue. – If an instruction does not have any register operands or all its operands are available from the register file or all its operands are available in all the FUs from the bypass network (for the LSFU network), then the FU assigned by the select logic is used. In this scheme, FU arbitration is used because multiple scheduled instructions may be assigned the same FU. The FU mappings read from the FU table for all scheduled instructions are decoded and requests are sent for the assigned FUs. FU arbiters grant these requests, based on priorities which could be the same as that used by the scheduler. If an instruction cannot acquire the assigned FU, it is “unscheduled” and is again scheduled in the following cycles. The “unscheduling” of instructions utilizes the same hardware as that used to “unschedule” and “reschedule” instructions dependent on load instructions that miss in the L1 cache, i.e. the scheduled instructions are not removed from the issue queue till they are free of replays. Figure 12 shows the operation of the FU assign/arbiter stage for one instruction. The register operands are used to index into the FU table. The FU table entry for an instruction’s destination register is speculatively updated with the FU assigned to the instruction in parallel to FU arbitration. If that instruction is re-scheduled, then its FU table entry is re-updated. This does not result in incorrect functioning of the dependants as seen later in the section. However, this approach removes the FU table update from the critical path. To perform FU assignment and arbitration in a single stage, these operations need to be fast. FU arbitration logic is very similar to the scheduler select logic, but of significantly lower complexity because of much fewer requests. FU arbitration can be simply implemented using position based priority [7]. Since all the arbiters operate in parallel, because each instruction only sends a request to just one arbiter, FU arbitration is very quick. The calculations in [7] suggest that the latency of FU arbitration is about 80% less than that of the select logic, for the parameters in Table 1. For a faster FU table access, we stack the FUs for multiple registers in a single FU table entry. The higher bits of a register tag are used to index into the table and the lower bits give the offset of the FU for a particular register tag. With this design, the access latency of a FU table is

Register Operands

215

FU−1 arbiter

FU Table

FU Maps

FU

Decode

Mux

FU−n arbiter

Valid Bits

Complexity Effective Bypass Networks

Fig. 12. Schematic FU assign/arbiter stage

about 90% less than that of a 128-entry physical register file used for a 6-way issue processor, based on the calculations in [7]. The FU table is read during FU assignment and updated during FU arbitration. With such small FU assignment and arbitration latencies, we assume a single FU assign/arbiter stage. We expect that the additional stage will not lie on the critical path because of its lower complexity as compared to other stages in the pipeline. If a scheduled instruction immediately wakes up the dependent instructions, “unscheduling” an instruction may lead to the consumer instruction getting executed before the producer instruction. This situation is avoided by keeping a bit-vector. The bit-vector has one bit per physical register and the number of read and write ports into the bit-vector is equal to the issue width of the processor. The bit for a register indicates whether the instruction producing that register has been dispatched to the FUs. The instructions check this bit-vector in parallel to FU arbitration, and hence its access does not incur any additional latency. When an instruction is dispatched to a FU (in the issue stage in Figure 11), the bit for its destination physical register is set. The bit for a physical register is reset when that register is allocated to an instruction. Hence, the bit-vector is not corrupted by any misspeculation rollbacks. If the producer of an instruction’s operand has not been dispatched to the FU, it is also “unscheduled”. It is important to note that the processor architecture remains exactly the same for simultaneous multi-threading (SMT) processors that share the registers between the different threads. This is because, the new FU assign/arbiter stage in Figure 11 only requires the physical register identifiers, which are distinct in the different threads in any given cycle.

6

Results

We use the processor parameters given in Table 1 for obtaining the results. First we show the benefit of SFU in reducing the bypass network latency using the calculations in [7]. We measure the forwarding latencies of the fully connected bypass network of Figure 8(a) and the SFU bypass network of Figure 8(b) for a 0.18µm technology. Note that we assume that only the integer ALUs and the

216

A. Aggarwal

Fig. 13. IPCs for SPEC 2000 (a) Integer Benchmarks; LSFU is about 5% worse than FUL0 and about 9% better than FUL1; ESFU is about 15% and 3% worse than FUL0 and FUL1; and (b) FP Benchmarks; LSFU is about 3% worse than FUL0 and about 13% better than FUL1; ESFU is about 16% and 3% worse than FUL0 and FUL1

load units are fully connected in Figure 8(a). The rest of data forwarding uses the multi-stage bypass network. We found that the forwarding latency of the SFU bypass network is about 70% less than the FUL0 bypass network. Thus, SFU can be used to reduce the number of cycles taken for data forwarding. For instance, a 0-cycle latency SFU bypass network can be used instead of a fully connected bypass network with 1-cycle latency (FUL1), which may result in overall IPC improvement. If reducing the bypass network latency is not the goal, then SFU can be used to reduce the overall complexity of the bypass network, while incurring a small IPC impact. Hence, we measure the performance of a 0-cycle latency LSFU and ESFU in terms of IPC and compare it against that of a 0-cycle latency fully connected bypass network (FUL0) and FUL1. 6.1

IPC Results and Analysis

Figure 13 gives the IPCs for the SPEC 2000 INT and FP benchmarks. As seen in Figure 13(a), for integer benchmarks, the IPC of LSFU is only about 5% worse than that obtained from FUL0, and about 9% better than that from FUL1. For floating-point benchmarks, LSFU performs 3% worse and 13% better than FUL0 and FUL1, respectively. ESFU with minimal bypass hardware incurs about 15% and 3% IPC impact compared to FUL0 and FUL1, respectively. LSFU performs better than FUL1, because in FUL1, all the instructions incur a 1 cycle forwarding latency. However, in LSFU, most instructions do not incur any delays due to forwarding, a few instructions suffer a 2-cycle delay because of getting re-scheduled and a negligible number of instructions incur a delay of more than 2 cycles (if they do not get any FU during the second try as well). The fact that many instructions do not incur any forwarding latency delays is also the reason that the IPC of ESFU does not reduce significantly. However, in case of ESFU, if an instruction is not able to get its operands from the bypass network, then it has to wait at least 4 cycles so that it can read the operand values from the register file (it takes 2 cycles to write the values into the register file).

Complexity Effective Bypass Networks

217

Fig. 14. IPCs for SPEC 2000 (a) Integer Benchmarks and (b) FP Benchmarks

6.2

Lower Priority to Branch Instructions

One of the main reasons for the performance impact when using single FU bypass network is the delayed execution of some of the instructions. To recover the performance loss, we investigate a technique that gives lower priority to branch instructions. Not all the branch instructions affect performance. Only the mispredicted branches affect performance, and delayed execution of correctly predicted branches does not impact performance. However, branches with low prediction accuracy need to be executed at the earliest to know their outcome. Branch instructions, on the other hand, compete with other instructions for the valuable forwarding paths. For instance, if a branch instruction and a resultproducing instruction are both dependent on the same instruction, then both the instructions will be assigned the FU used by the producer instruction. If the branch gets the FU, the result-producing instruction is delayed, and the performance is hit. Hence, to improve performance with a SFU bypass network, branch instructions are given a lower priority during FU arbitration. For this, each instruction is assigned a bit (called the “type bit”), which indicates whether the instruction is a branch instruction or not, and in case of a collision for the same FU, lower priority branch instruction gets “un-scheduled”. Figure 14 shows the IPC results when this technique is employed for the SFU bypass network. As can be seen in Figure 14, a significant performance improvement is observed for many benchmarks, bringing the IPC of the LSFU bypass network almost equal to that of the FUL0 network. 6.3

Single Input Single FU Bypass Network

The bypass network complexity can be further reduced by half by forwarding the values to only one of the FU inputs [3]. In the SFU bypass network design, only the instructions that have at most one operand forwarded from the bypass network are scheduled, and having a single input forwarding is a natural extension of this design. This would require a switch in the operand locations in the instructions, so that the correct operand is bypassed the correct value. Since the values are forwarded to the inputs of the same FU in the SFU design, the

218

A. Aggarwal

switch can be performed once the operand that will be forwarded is known. No additional performance loss is observed for a Single Input SFU bypass network. However, with single input SFU bypass network, the forwarding latency further reduces to about 85% less than the broadcast-based bypass network. This kind of data forwarding has one of the minimum bypass network hardware, apart from the case when there is no data forwarding.

7

Scalability of SFU

In this section, we measure the scalability of the SFU bypass network as compared to the broadcast-based bypass network, as the issue width and the issue queue size are increased. With an increase in the issue width, the bypass network complexity increases dramatically for a broadcast-based bypass network. However, for the SFU bypass network, the complexity and hence the latency of the bypass network remains unchanged. The complexity of the additional hardware used for FU assignment (Section 5.3) also remains small relative to the complexity of the other hardware structures (the select logic and the register file) used in the processor. However, with an increase in the issue width and issue queue size, the performance impact of SFU is expected to increase because of an increase in the IPC. Hence, as a pathological case for the scalability of SFU, we measure the IPC obtained with FUL0, FUL1, and LSFU with a large issue width of 16 and a large issue queue of 1024 entries. We also increase the number of functional units and register file size accordingly and use a perfect branch predictor to increase the average number of instructions in the issue queue. All the other parameters are maintained at their default values given in Table 1. Figure 15 presents the IPC results for the FUL0, FUL1, and LSFU bypass networks for the new hardware parameters. Figures 15 and 14 show that, as the issue width and the issue queue size are increased, even though the percentage reduction in IPC (compared to FUL0) increases when using LSFU, it is still quite close to that of FUL0 and significantly higher than FUL1. Figure 15 also shows that the IPC impact of SFU

Fig. 15. IPCs for SPEC 2000 (a) Integer Benchmarks and (b) Floating-Point Benchmarks; Issue Width = 16, Issue Queue = 1024, Perfect Branch Prediction

Complexity Effective Bypass Networks

219

bypass network increases more for the integer benchmarks than the floatingpoint benchmarks. This is exlained by Figure 9, where it can be seen that in general, more generated results are used by a single other instruction in the floating-point benchmarks as compared to the integer benchmarks. Figures 15 and 13 also show that the IPC impact increases for FUL1 as the issue width and the issue queue size are increased, indicating that the broadcast-based bypass network not only incurs higher latency as the processor width is increased, but also the IPC IPC loss incurred due to delays introduced in the execution of dependent instructions increases. This suggests that the SFU bypass network can potentially be much more useful for wider machines used to exploit more instruction level parallelism to improve performance.

8

Related Work

Bypassing is an old idea and was first described in 1959 by Bloch [4]. Since then, the issue width of the processors and the number of functional units in the processors have increased considerably. Unfortunately, not enough work has been done for efficient data forwarding. In addition, a study of the performance impact due to delayed execution of dependent instructions has also never been performed. Here, we classify the proposed efficient bypass networks into 2 broad categories: limited bypassing, and partitioned bypassing. In limited bypassing, certain paths are missing from the bypass network. Ahuja et al [3] study bypass networks where the results from the FUs are forwarded to only one of the inputs of all the FUs. They propose simple code transformations such as interchange of operands and instruction scheduling to avoid the stalls generated due to missing bypasses. For efficient bypassing, the Pentium 4 processor [8], limits the number of bypass inputs into each FU as well as the number of bypass outputs from each FU. For this, [8] uses a complex multi-stage bypass network that stores and selectively forwards the results to be bypassed from the latter stages of the pipeline. Partitioned bypassing is used in clustered processor architectures, where the entire issue width and the functional units in a processor are partitioned into smaller independent clsuters. In these architectures, there is typically a broadcast-based bypassing within a cluster, and either broadcast-based inter-cluster bypassing [17][16][14][7], or point-to-point inter-cluster bypassing [15][18][19][20][21]. Since the issue width within a cluster is small, intra-cluster bypassing is fast because of reduction in the number of the bypass paths, whereas inter-cluster bypassing may take additional cycles because of longer wires and/or multiple hops required. Clustered architectures do not achieve the single-threaded IPCs obtained with a centralized superscalar [22].

9

Conclusions

For a wide-issue and deeply pipelined processor, broadcast-based bypass networks can take multiple cycles, especially with the relative increase in wire

220

A. Aggarwal

delays in the sub-micron technology, delaying the execution of dependent instructions and reducing performance significantly. In this paper, we perform a detailed analysis of the performance impact due to multi-cycle data forwarding. We found that this performance impact differs from program to program based on the data dependence present in the programs and on the instructions constituting the program code. In particular, we found that programs with a higher percentage of branch and store instructions and programs with a higher number of dependants per instruction tend to incur less performance impact. We also found that the performance impact due to delays in data-dependent instruction execution varies with the processor parameters such as issue width, fetch width, issue queue size, and cache configuration. The analysis performed in this paper can be used by the designer to determine the minimum amount of hardware resources required to obtain the maximal performance in a processor with multi-cycle forwarding latency. Furthermore, we observed that the results of most instructions are consumed by just one other instruction and the operands of many instructions come from a single other instruction. Based on this observation, we proposed a Single FU bypass network, where the results for an FU are only bypassed to its own inputs, thus reducing the bypass network complexity significantly, and facilitating fast forwarding. Our studies showed that the forwarding latency can be reduced by more than 70%, while incurring a small IPC impact of about 2% for most of the benchmarks. Single FU bypass networks are also much more scalable than the broadcast-based bypass networks, as the future microprocessors become more wide and more deeply pipelined.

References 1. Agarwal, V., Hrishikesh, M.S., Keckler, S.W., Burger, D.: Clock rate versus IPC: the end of the road for conventional microarchitectures. In: Proceedings of International Symposium on Computer Architecture (ISCA-27) (2000) 2. Aggarwal, A.: Single FU bypass networks for high clock rate superscalar processors. In: Boug´e, L., Prasanna, V.K. (eds.) HiPC 2004. LNCS, vol. 3296, pp. 319–332. Springer, Heidelberg (2004) 3. Ahuja, P., Clark, D., Rogers, A.: The performance impact of incomplete bypassing in processor pipelines. In: Proc. of Intl. Symp. on Microarchitecture (1995) 4. Bloch, E.: The Engineering Design of the Stretch Computer. In: Proceedings of Eastern Joint Computer Conference (1959) 5. Brown, M., Stark, J., Patt, Y.: Select-free Instruction Scheduling Logic. In: Proceedings of International Symposium on Microarchitecture (Micro-34) (2001) 6. Hennessy, J., Patterson, D.: Computer Architecture: A Quantitative Approach. Morgan Kaufmann Publishers, San Francisco (2002) 7. Palacharla, S., Jouppi, N.P., Smith, J.E.: Complexity-Effective Superscalar Processors. In: Proc. of Int’l. Symp. on Computer Architecture (1997) 8. Hinton, G., et al.: A 0.18-um CMOS IA-32 Processor With a 4-GHz Integer Execution Unit. IEEE Journal of Solid-State Circuits 36(11) (November 2001) 9. Sankaralingam, K., Singh, V., Keckler, S., Burger, D.: Routed Inter-ALU Networks for ILP Scalability and Performance. In: Proceedings of International Conference on Computer Design (ICCD) (2003)

Complexity Effective Bypass Networks

221

10. Sprangle, E., Carmean, D.: Increasing Processor Performance by Implementing Deeper Pipelines. In: Proc. of Int’l. Symp. on Computer Architecture (2002) 11. Stark, J., Brown, M., Patt, Y.: On Pipelining Dynamic Instruction Scheduling Logic. In: Proc. of International Symp. on Microarchitecture (2000) 12. The National Technology Roadmap for Semiconductors, Semiconductor Industry Association (2001) 13. Rotenberg, E., et al.: Trace Processors. In: Proc. of Int’l. Symp. on Microarchitecture (1997) 14. Leibholz, D., Razdan, R.: The Alpha 21264: A 500 MHz Out-of-Order Execution Microprocessor. In: Proceedings of Compcon., pp. 28–36 (1997) 15. Farkas, K., et al.: The Multicluster Architecture: Reducing Cycle Time Through Partitioning. In: Proc. of Int’l. Sym. on Microarchitecture (1997) 16. Canal, R., Parcerisa, J.M., Gonzalez, A.: Dynamic Cluster Assignment Mechanisms. In: Proc. of Int’l. Symp. on High-Performance Computer Architecture (2000) 17. Baniasadi, A., Moshovos, A.: Instruction Distribution Heuristics for Quad-Cluster, Dynamically-Scheduled, Superscalar Processors. In: Proceedings of International Symposium on Microarchitecture (MICRO-33) (2000) 18. Parcerisa, J.M., Sahuquillo, J., Gonzalez, A., Duato, J.: Efficient Interconnects for Clustered Microarchitectures. In: Proceedings of International Symposium on Parallel Architectures and Compiler Techniques (PACT-11) (2002) 19. Nagarajan, R., et al.: A design space evaluation of grid processor architectures. In: Proceedings of International Symposium on Microarchitecture (Micro-34) (2001) 20. Waingold, E., et al.: Baring it all to software: RAW machines. IEEE Computer 30(9), 86–93 (1997) 21. Fillo, M., et al.: The M-Machine Multicomputer. In: Proceedings of International Symposium on Microarchitecture (Micro-28) (1995) 22. Aggarwal, A., Franklin, M.: Instruction Replication: Reducing Delays due to InterCommunication Latency. In: Proceedings of International Symposium on Parallel Architectures and Compiler Techniques (PACT) (2003) 23. Gowan, M.K., et al.: Power Considerations in the Design of the Alpha 21264 Microprocessor. In: Proceedings of Design Automation Conference (DAC) (1998) 24. Tiwari, V., et al.: Reducing Power in High-performance Microprocessors. In: Proceedings of Design Automation Conference (DAC) (1998) 25. Burger, D., Austin, T.: The Simplescalar Tool Set. Technical Report, Computer Sciences Department, University of Wisconsin (June 1997)

A Context-Parameterized Model for Static Analysis of Execution Times Christine Rochange and Pascal Sainrat Institut de Recherche en Informatique de Toulouse, Universit´e de Toulouse - CNRS, France {rochange,sainrat}@irit.fr http://www.irit.fr/TRACES

Abstract. The static analysis of the execution time of a program (i.e. the evaluation of this time for any input data set) can be useful for the purpose of optimizing the code or verifying that strict real-time deadlines can be met. This analysis generally goes through determining the execution times of partial execution paths, typically basic blocks. Now, as soon as the target processor architecture features a superscalar pipeline, possibly with dynamic instruction scheduling, the execution time of a basic block highly depends on the pipeline state, that is on the instructions executed before it. In this paper, we propose a model to specify the local execution context of a basic block as a set of parameters. The execution time of the block can then be computed as a function of these parameters. We show how this model can be used to determine an upper bound of the execution time of a basic block, that can be used for computing the Worst-Case Execution Time of the program. Experimental results give an insight into the tightness of the estimations. Keywords: timing analysis, Worst-Case Execution Time.

1

Introduction

A static analysis of the execution time of a program consists in evaluating the execution time without considering any particular input data set, or more precisely considering any possible input data set. Since exploring all the possible paths one by one is generally inconceivable, static analysis is based on the execution times of partial execution paths, e.g. basic blocks. In this paper, we are interested in how these partial execution times can be determined. Being able to get information about the execution time of a basic block for every possible execution context might be useful for a back-end code optimizer. It is also needed for the estimation of the Worst-Case Execution Time (WCET) of realtime applications that must meet strict deadlines. For example, the Implicit Path Enumeration Technique [8] expresses the execution time of a program as the sum of the execution times of its basic blocks weighted by their respective execution counts. The WCET is obtained by determining the block execution P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 222–241, 2009. c Springer-Verlag Berlin Heidelberg 2009 

A Context-Parameterized Model for Static Analysis

223

counts that maximize this expression under a set of linear constraints that convey the structure of the Control Flow Graph as well as flow facts like loop bounds and infeasible paths. When the program executes on a high-performance processor, the execution time of a basic block naturally depends on the state of the pipeline when the block is fetched: previous instructions still present in the pipeline might stall some instructions of the block due to resource conflicts; they also might have an impact on the groups of instructions that are processed in parallel in a superscalar pipeline; they can alter the scheduling of the block instructions particularly if the processor features out-of-order execution capabilities. In this paper, our goal is to define a model that can help in estimating the execution time of a basic block as a function of the execution context. We propose to specify the context as a set of parameters that stand for the release times of all the resources required to execute the block (pipeline stages, functional units, register values, etc.). Then, we show how the execution pattern of the block can be expressed as a function of these parameters. This pattern can then be analyzed to compute an upper bound of the block execution time. The paper is organized as follows. In Section 2 we define the execution cost of a basic block and we discuss the concept of execution context. We also give an overview of related work. Section 3 develops our model to express the context and to estimate the block execution pattern and cost as a function of the context. How this model can be used to determine the worst-case cost of a basic block when estimating the WCET of a real-time task is shown in Section 4. Experimental results are provided and discussed in Section 5. Section 6 concludes the paper.

2 2.1

Computing Basic Block Execution Times Cost of a Basic Block

Most modern processors include a pipeline and the executions of successive instructions, and thus of successive basic blocks, overlap. Then the execution time of a n-block sequence {B1 −B2 −. . . −Bn }is usually shorter than the sum of the block execution times, as illustrated on the left part of Figure 1. The execution cost of a basic block is defined as the time between the completion of the last preceding instruction and the completion of its last instruction (this definition makes sense only if the instructions terminate in the program order, which is the case for most of the processors). The execution cost of the first basic block of the program is equal to its execution time. The execution time of a sequence can be computed as the sum of its block costs:  CBi tB1 −B2 −...−Bn = 1≤i≤n

As illustrated in the right part of Figure 1, the execution time of a basic block often depends on the sequence of blocks executed before it (which will be referred to as the prefix ). Because of data dependencies and resource conflicts, it might

224

C. Rochange and P. Sainrat

(a)

A

(b)

C (c)

Fig. 1. Execution cost of a basic block

be longer (case b) than when the block is executed in an empty pipeline (case a). In the case where the processor features dynamic instruction scheduling, the execution time of a basic block also depends on what is executed after it, referred to as the suffix. After a given prefix and before a given suffix, the execution time of a basic block might be longer or shorter (case c) than when the block is executed alone. The cost of the basic block, i.e. the time difference between the end of the block and the end of the prefix, is noted CB/pref ix\suf f ix . A different cost might be observed for each {prefix, suffix } pair which can be thought as the execution context of the block. Note that considering partial prefixes and suffixes might not be acceptable to estimate accurate block execution times: as shown in [7], complete paths must be analyzed. 2.2

Local Execution Context

A basic block is likely to be impacted by its execution context if not all resources are available when it enters the pipeline, or more precisely at the time it needs them. By the generic term of resource, we encompass: – structural resources: pipeline stages, functional units, slots in instructions queues, etc. An instruction of the block might be delayed when it competes with a previous instruction (or a later one, in case of out-of-order execution) for the use of a structural resource. – functional resources, like register values. To start its execution in a functional unit, an instruction must wait for its operands being ready (which can be seen as the release of resources by the producing instructions). The latency of instruction and data fetches also depends on the caches contents at the start of the block execution. Also, the cost of the block might be augmented by a branch misprediction penalty depending on the contents of the branch predictor tables. The state of the instruction and data caches and of the branch predictor is generally analyzed in a preliminary step and several techniques have

A Context-Parameterized Model for Static Analysis

225

been proposed for this purpose in the literature [1][2][5]. In this paper, we do not address this question and we thus consider perfect (always-hit) caches and an oracle branch predictor. 2.3

Related Work

Approaches to analyze basic block execution times have been widely investigated with the objective of estimating the Worst-Case Execution Time (WCET) of strict real-time tasks [6]. Three main ways of taking contexts into account have been proposed in the literature. Enumeration of Possible Contexts. The most trivial solution — at least in the concept — consists in enumerating all the possible execution contexts for the block, i.e. in analyzing all the possible paths in the code. It is commonly admitted that the definition of data sets that would cover all the possible paths is complex and even impossible in most of the cases (in particular with floating-point input data). However, symbolic simulation used by Lundqvist and Stenstr¨ om [11] makes it possible to avoid the specification of data sets by marking all input data as unknown and by propagating unknown values through the computations. After each conditional branch with an unknown condition, both paths are explored. This way, all the possible paths are considered and, each time every basic block is executed (or more exactly simulated) in a precise context. The cost (in terms of CPU time) of the evaluation of the block execution times can be reduced by stopping at a fixed point when the analyzed code includes loops. However it generally remains prohibitive when the algorithmic structure of the code is complex. Abstract Local Context. The second approach consists in building a representation of what a context can be according to the various possible paths in the Control Flow Graph (CFG). It is the strategy implemented in the aiT software commercialized by the AbsInt company (www.absint.com). The abstract execution context of a basic block is expressed as a set of concrete contexts derived by an abstract interpretation of the application. The effect of the execution of the block on the abstract input context is analyzed and the abstract output context is propagated to the successor blocks until a fixed point is reached [12]. The main drawback of this approach is its cost both in CPU time and memory usage. Worst-Case Local Context. The third approach, proposed by Li et al. [10] is aimed at determining an upper bound of block execution times. It expresses the execution pattern of a basic block by an execution graph that conveys the precedence constraints between instructions (the concept of execution graph will be detailed in Section 3). The execution graph is analyzed to derive the start and finish times of each instruction considering pessimistic assumptions about the execution context. We have found that this method significantly overestimates WCETs, especially for architectures that feature a wide instruction window. This led us to propose the new technique, also based on execution graphs, that we present in this paper. Further details on Li’s algorithm will be provided in Section 5 when discussing our respective results.

226

3

C. Rochange and P. Sainrat

Analyzing the Execution Cost of a Basic Block

In this section, our objective is to set up a framework to analyze the execution time of a basic block. It first consists in expressing the constraints that drive the execution of the block in the pipeline: as detailed in Section 3.1, we use the execution graph model that was first proposed by Li et al. [9] and that we extended in [3]. 3.1

Modeling the Execution of a Basic Block

Scalar Pipelined Execution. As proposed by Li et al. [9], the execution of a basic block can be represented by an execution graph where each node stands for the processing of an instruction by a pipeline stage or a functional unit. Directed edges express precedence constraints between nodes that can be related to the program order (e.g. instructions are fetched in order), to the structure of the pipeline (e.g. an instruction must be fetched before being decoded), to capacitylimited instruction queues or to data dependencies. For the sake of clarity, we will illustrate our discussion with a simple processor including a 3-stage scalar pipeline (fetch, execute, commit ) with in-order execution, two functional units with a fixed latency of one cycle and a 4-entry instruction window (buffer that stores all the instructions simultaneously active in the processor). This processor is illustrated in Figure 2. Figure 3 shows the execution graph of a 5-instruction basic block executed in the processor shown in Figure 2. The horizontal edges express the processing flow through the pipeline, while the vertical edges express the program order or the data dependencies (e.g. edge MEM(i0 ) → ALU(i1 )). The 4-instruction capacity of the instruction window is denoted by edge CM(i0 ) → IF(i4 ). Analyzing the execution pattern of a basic block from its execution graph comes to estimating the time at which each node N is ready (ρN ). Algorithm 1 shows how ready times can be determined (ℓP is the latency of node P). Parallel (superscalar) Execution. In [3] we extended the execution graph model to express superscalar execution: two nodes that might be processed in parallel but in the program order are linked by a slashed edge (e.g. two adjacent instructions can be decoded at the same cycle, or the second one can be decoded one cycle after the first one, but the second one cannot be decoded before the first one). This means that the second node (N) can start at the same time at the first one (P), which would change line 3 of Algorithm 1 into: delay = ρP .

Fig. 2. Example processor

A Context-Parameterized Model for Static Analysis

i0: i1: i2: i3: i4:

r0 r1 r2 r3 r4

MEM[@x] r0 + 8 MEM[@y] r10 + 12 r2 + r3

IF(i0)

MEM(i0)

CM(i0)

IF(i1)

ALU(i1)

CM(i1)

IF(i2)

MEM(i2)

CM(i2)

IF(i3)

ALU(i3)

CM(i3)

IF(i4)

ALU(i4)

CM(i4)

227

Fig. 3. Example execution graph (scalar core with in-order execution)

Algorithm 1. Computing the ready time of a node 1 2 3 4 5

ρN = 0 ; foreach predecessor P of node N do delay = ρP + ℓP ; if delay > ρN then ρN = delay ; end

This is illustrated in Figure 4 that shows the execution graph of the basic block given in Figure 3 in a 2-way superscalar pipeline: we assume that four instructions can be fetched at each cycle, and that instruction i0 is aligned on a cache line boundary, so that instructions i0 to i3 can be fetched in parallel (slashed edges). Slashed and plain edges between CM nodes mean that two adjacent instructions can be committed at the same cycle. Out-of-Order Execution. When instructions are dynamically-scheduled to a given resource (typically a functional unit), the nodes related to this resource are linked by dashed edges that can be labeled to indicate the width (capacity) of the resource. Slightly different approaches to handle possible contentions for the use of this kind of resource have been proposed in [9] and [3]. Since the work we present here uses a radically different algorithm, we do not detail them here. In Figure 4, we assume that the processor features three dynamically-scheduled functional units (two ALUs and one MEM unit). Labeled dashed edges express that instructions i1 , i3 and i4 can use the ALU in any order and that at most two them can be executed at the same cycle (however, the data dependency between i3 and i4 limits the scheduling possibilities). 3.2

Modeling the Execution Context

Our goal here is to define a parametric model of the execution context of an instruction sequence (e.g. a basic block). This model should be able to express any information about the context that is relevant for evaluating the execution

228

C. Rochange and P. Sainrat

IF(i0)

MEM(i0)

IF(i1)

ALU(i1)

CM(i0)

CM(i1) 1

IF(i2)

MEM(i2)

CM(i2)

IF(i3)

ALU(i3)

CM(i3)

IF(i4)

ALU(i4)

CM(i4)

2

Fig. 4. Example execution graph (superscalar core with out-of-order execution)

time of the sequence. The main goal is to determine the finish time of each node in the execution graph with respect to the context parameters. The execution context of an instruction sequence is defined by the release times of the resources used by the sequence. As said before, resources include pipeline stages, functional units, slots in instructions queues, but also register values. For an n-way superscalar processor, a pipeline stage is seen as n different resources. Let Rbe the set of the resources involved in the context specification. This set is extended with a virtual resource that stands for the time at which the sequence enters the pipeline. The context can be specified by a vector A of parameters, each standing for the release time of a resource that belongs to R: A= {ar |r ∈ R} The constraints on the ready time of any node N in the execution graph are expressed by: – a vector E¯N of boolean flags that indicate the dependence of the node ready time to each of the context parameters: E¯N = {¯ er ∈ {0, 1} |r ∈ R} N

¯ N of delays that express the minimal distance between the resource – a vector D release times and the node ready time:   ¯ N = d¯r |r ∈ R D N The ready time of the node can then be computed as:    ρ¯N = max e¯rN . d¯rN + ar r∈R

3.3

(1)

Computing Node Ready Times

¯N for each Initialization. The first step consists in initializing vectors E¯N and D node N in the execution graph : if N is the first node in the sequence to use the

A Context-Parameterized Model for Static Analysis

229

resource r ∈ R, then e¯rN ←1 and d¯rN ←0. This means that N requires r and can start only when r has been released. Propagation. The graph is processed in a topological order that ensures that each node is evaluated after all of its predecessors, as shown in Algorithm 2. The dependence flags and the delays are propagated from node to node: a node depends on a given resource as soon as one of its predecessors depends on it [line 6 ], and its delay relating to this resource release time is computed from the maximum delay among its predecessors (incremented by the predecessor latency if the edge is plain) [line 7 ].

Algorithm 2. Computing node delays 1 2 3 4 5 6 7 8 9 10

foreach predecessor P of node N do if edge P →N is slashed (superscalarity) then lat = 0; else lat = ℓP ; foreach resource r in Rdo if e¯rP == 1 then if e¯rN == 0 then e¯rN = 1; d¯rN = d¯rP + lat; else if (d¯rP + lat) > d¯rN then d¯rN = d¯rP + lat; end end end

Example. For the sake of simplicity, we will illustrate the algorithm with the example of Figure 3 where the processor is scalar and executes the instructions in order. Considering a superscalar core would not change the complexity of computation (see the algorithm). Figure 5 shows the node start times for the example execution graph extended with a prefix instruction (i−1 ). The legend on top of the Figure specifies the different context parameters (iws stands for slot s of the instruction window). For each resource (parameter), a thick-line frame indicates the first node that requires it. For example, the delays for node ALU(i4 ) mean that this node can start at the earliest : – six cycles after IF(i−1 ) – four cycles after slot iw2 of the instruction window has been released (this comes from an indirect dependency to IF(i1 )) – four cycles after the ALU has been released (this comes from path ALU(i−1 )CM(i−1 )-IF(i3 )-IF(i4 )-ALU(i4 )) and three cycles after the MEM unit has been released (path MEM(i0 )-ALU(i1 )-ALU(i3 )-ALU(i4 )) – three cycles after the release of the commit stage due to the limited capacity of the instruction window – one cycle after r10 has been produced due to a data dependency to ALU(i3) that itself uses register r10

230

C. Rochange and P. Sainrat iw2

iw0

from first node

iw1

iw3

ALU CM MEM r10

e0 e1 e2 e3 e4 e5 e6 e7 e8 d0 d1 d2 d3 d4 d5 d6 d7 d8 ALU(i-1)

IF(i-1)

CM(i-1)

1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 1 0 0 0 1 0 0 0 1 1 0 0 0 0 0 0 0

1 1 0 0 0 1 0 1 0 L P 2 2 0 0 0 1 0 0 0

IF(i0)

MEM(i0)

CM(i0)

1 1 1 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0

1 1 1 0 0 0 1 0 0 2 2 1 0 0 0 0 0 0

1 1 1 0 0 1 1 1 0 3 3 2 0 0 2 1 1 0

IF(i1)

ALU(i1)

1 1 1 1 0 0 0 0 0 2 2 1 0 0 0 0 0 0

1 1 1 1 0 1 1 0 0 3 3 2 1 0 1 1 0 0 MEM(i2)

IF(i2)

CM(i1)

1 1 1 1 0 1 1 1 0 4 4 3 2 0 3 2 2 0 CM(i2)

1 1 1 1 1 0 0 0 0 3 3 2 1 0 0 0 0 0

1 1 1 1 1 0 1 0 0 4 4 3 2 1 0 1 0 0

1 1 1 1 1 1 1 1 0 5 5 4 3 2 4 3 3 0

IF(i3)

ALU(i3)

CM(i3)

1 1 1 1 1 1 0 1 0 4 4 3 2 1 2 0 1 0

1 1 1 1 1 1 1 1 1 5 5 4 3 2 2 2 2 0

1 1 1 1 1 1 1 1 1 6 6 5 4 3 5 4 4 1

IF(i4)

ALU(i4)

CM(i4)

1 1 1 1 1 1 1 1 0 5 5 4 3 2 3 2 2 0

1 1 1 1 1 1 1 1 1 6 6 5 4 3 4 3 3 1

1 1 1 1 1 1 1 1 1 L 7 7 6 5 4 6 5 5 2 B

Fig. 5. Node start times (example)

3.4

Computing the Cost of a Basic Block

Once the node times have been computed, the execution time of the block can be computed as a function of the context, which is expressed by the vector of resource release times A. If LB is the last node of the block, the execution time of the block is given by: tB = ρLB + ℓLB . If the execution graph has been extended to a prefix of at least one instruction (as in Figure 5), it is also possible to evaluate the execution cost of the block. Let LP be the last node of the prefix. Then: (2) CB = ρLB + ℓLB −ρLP −ℓLP Whenever the execution context is known, i.e. the values of the resource release times (ar ) are known, it is possible determine the exact execution pattern of the block. Then, its exact execution cost can be computed using equation (2). Otherwise, this equation can be used to get an insight into how any possible context might impact the execution of the block. In the next section, we will also show in what extent this model can help to estimate the worst-case cost of a basic block. This value can be used to analyze the Worst-Case Execution Time of a real-time application submitted to strict deadlines. 3.5

Modeling Contentions for Dynamically-Scheduled Resources

Dynamically-scheduled functional units cannot be considered as having a unique release time since it might not be true that all the instructions of the sequence use the resource after all the instructions of the context. They might instead be interleaved. Then a different release time for the functional unit should be considered for each instruction in the sequence. As a result, a dynamically-scheduled

A Context-Parameterized Model for Static Analysis

ε

231

ε r

S1 S0 e 0 e1 e2 e3 e4 e 5 e6 e7 e8 d 0 d1 d2 d3 d4 d 5 d6 d7 d8 ALU(i -1) = S0

. . . . . 1 . . . . . . 3 .

CM(i -1)

1 0 0 0

MEM(i 0) = S1

. . . . . 1 . . . . . . 4 .

. . . . . 1 . 1 0 . . . . . 4 . 1 0 CM(i0)

1 1 1 0

. . . . . 1 . 1 1 . . . . . 5 . 2 1

Fig. 6. Node start times including contention delays (example)

functional unit used by n instructions in the sequence is represented by n release times, i.e. as n different resources. Now, one instruction in the block can still compete for the unit with other instructions in the block. This is what we call internal contentions. Let S be the set of the nodes in the execution graph that use a dynamically-scheduled functional unit and might be delayed by contentions. Due to internal contentions, the time at which a node N really starts its execution (σN ) might be later than its ready time (ρN ). We distinguish the contentions sustained by node N from the contentions sustained by another node and propagated to node N. The delay induced by the contentions that directly affect node N is noted ωN (if N does not use a dynamically-scheduled resource, ωN is null). Then the start time of node N is given by: σN = ρN + ωN . To analyze the propagated effects of internal contentions on the ready time of node N, we introduce: ˘ of boolean flags that indicate the dependence of the node ready – a vector E N time to each node S in S:   ˘N = e˘S ∈ {0, 1} |S ∈ S E N ˘ N of delays that stand for the minimal distance between the – a vector D effective start times of the nodes in S and the ready time of node N:   ˘ N = d˘S |S ∈ S D N This is illustrated by the example graph given in Figure 6. Node CM(i0 ) depends on resource r via nodes ALU(i−1 ) (also referred to as S0 ) and MEM(i0 ) (or S1 ) that both use dynamically-scheduled units. This means that any contention delay that would affect the start time of S0 or S1 might increase the distance between the release time of r and the ready time of node CM(i0 ). This is expressed by vectors

232

C. Rochange and P. Sainrat

˘ and D. ˘ For example, if node S1 is delayed by three cycles due to contentions E with others nodes using the MEM unit, this delay will propagate to node CM(i0 ) which will be ready only eight cycles after resource r has been released. ˘N are estimated ˘N and D Computing Contention-Related Delays. Vectors E ¯ ¯ in a similar way as EN and DN : – initialization: ∀S ∈ S, e˘SS ← 1 and d˘SS ← 0. – propagation: in Algorithm 2, the internal loop [lines 4-11 ] that computes R-related delays is duplicated to compute S-related delays in the same way. Node Ready Times. Let ρ˘N be the ready time of node N with respect to its dependencies to the nodes in S:

ρ˘N = max e˘SN d˘SN + σS S∈S

Putting it altogether, the ready time of node N is given by: ρN = max (¯ ρN , ρ˘N ) or: 

   ρN = max max e¯rN · d¯rN + ar , max e˘SN d˘SN + σS r∈R

S∈S

Proposition 1. The ready time of node N can be expressed as: ρN = max (erN (drN + ar ))

(3)



drN = max d¯rN , max e˘SN · erS d˘SN + drS + ωS

(4)

r∈R

with erN = e¯rN and: S∈S

Proof is given in the Appendix.

4 4.1

Evaluating the Worst-Case Cost of a Basic Block Bounding the Cost of a Basic Block

If LB is the last node of the block and LP the last node before the block, then ωLB = ωLP = 0 because the instructions leave the pipeline in the program order. Equation 2 can be written as: CB = ∆(LB , LP ) + ℓLB − ℓLP with ∆(LB , LP ) = ρLB − ρLP . Let λ be the resource related to the last pipeline stage (in our example, aλ is the release time of the CM stage). Every node in the graph related to the last pipeline stage (e.g. LB and LP ) has a dependency to resource λ (eλLB = eλLP = 1)

A Context-Parameterized Model for Static Analysis

233

because we assume that instructions are committed in the program order. An important point is that every resource in Ris released before λ: ∀r ∈ R, ar < aλ . It can even be determined how many cycles before λ each resource r must have been released, depending on the pipeline stage where it is used (this delay will be noted αr ). For example, in our 3-stage pipeline, a resource that stands for a functional unit must have been released one cycle before the CM stage (λ). Then: ∀r ∈ R, ar + αr ≤ aλ . Proposition 2. An upper bound of ∆(LB , LP ) is given by: ∆(LB , LP ) ≤ max (δ r (LB , LP ))

(5)

     δ r (LB , LP ) = erLB · drLB − erLP · drLP − 1 − erLP · dλLP + αr

(6)

r∈R

with:

Proof is given in the Appendix.

Implementation. Algorithm 3 can be used to compute the cost of block B. When an in-order processor core is considered, the computation of the δ r (LB , LP ) ¯ LP is straightforward from Equation 6. Things ¯ LB , E¯LP and D from vectors E¯LB , D are more complex when the pipeline includes dynamically-scheduled functional units. How contention delays can be taken into account is explained in the next section.

Algorithm 3. Computing the execution cost of a block 1 2 3 4 5

CB = 0; foreach r ∈ R do if δ r (LB , LP ) > CB then CB = δ r (LB , LP ); end CB = CB + ℓLB − ℓLP ;

4.2

Analyzing Contention Delays

Equation 6 refers to the values in vectors DLB and DLP . As stated by Equation 4, these values depend on the contention delays of dynamically-scheduled nodes. Considering an unknown context, it is generally impossible to compute the exact scheduling and then exact contention delays. However, they can be bounded. The maximum value of ωS could be conservatively estimated by considering that every node in the execution graph that uses the same functional unit is a possible contender and might delay node S. If an oldest first scheduling policy is assumed, at most one later instruction can also delay node S. The maximum number of contenders can be refined by ignoring nodes that can be proved as finishing before or being ready after node S. These nodes cannot request the unit at the same time as S. Algorithm 4 shows how the maximum number of

234

C. Rochange and P. Sainrat

Algorithm 4. Computing the maximum number of contenders for node S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

numContenders = 1; foreach node C using the same unit as S and in the same inst. window do finishedBefore = true; readyAfter = true; foreach resource r ∈ R do if e¯rC == 1 then if e¯rS == 0 then finishedBefore = false; else if d¯rS < d¯rC + ℓC then finishedBefore = false; end if e¯rS == 1 then if e¯rC == 0 then readyAfter = false; else if d¯rC dN then dN = tmp; end end end

r M AX 1 dN

2 3 4 5 6 7 8 9

Algorithm 6. Computing an upper bound of δ r (a, b) 1 2 3 4 5 6 7 8 9 10 11 12 13 14

if era == 0 then δ r (a, b)M AX = 0; else if erb == 0 then δ r (a, b)M AX = dra M AX − d¯λb − αr ; else δ r (a, b)M AX = d¯ra − d¯rb ; foreach S ∈ S do if e˘S a == 1 then S S if e˘S b == 1 then tmp = da − db ; S M AX r else tmp = da + ωS + δ (S, b)M AX ; //recursive call r M AX if tmp > δ (a, b) then δ r (a, b)M AX = tmp; end end end end

depends on resource r but b does not, we have: δ r (a, b) ≤ dra − dλb − αr . Since d¯λb ≤dλb (see Equation 4), we can write [line 3 ]: MAX

≤dra MAX − d¯λb − αr

δ r (a, b)

When both a and b depend on resource r, we have: δ r (a, b) = dra − drb . If a is not delayed by S-nodes, we have: dra == d¯ra . We also have d¯rb ≤ drb , then: MAX

δ r (a, b)

≤ d¯ra − d¯rb

Otherwise, if ∃S ∈ S|dra == d˘Sa + drS + ωS , we get: MAX

δ r (a, b)

MAX

= d˘Sa + drS MAX + ωSMAX − drb = d˘Sa + ωSMAX + δ r (S, b)

Finally, we write [lines 5-12 ]: 

MAX MAX r r r S MAX r ¯ ¯ ˘ δ (a, b) ≤ max da − db , max da + ωS + δ (S, b) S∈S

236

5 5.1

C. Rochange and P. Sainrat

Experimental Results Methodology

In this section, we give some experimental results that were obtained using the OTAWA framework [4]. OTAWA implements an infrastructure to support several kinds of analyses that can be combined to estimate the WCET of an application. It includes: – a basic flow analyzer that builds the CFG from the object code and retrieves user-specified flow facts (e.g. loop bounds) – a cycle-level simulator that computes the execution times of sequences of blocks: by simulating all the possible 8-block sequences ending with each block B, we were able to get an estimate of the worst-case cost of B (we are aware that these estimates are not safe because of possible long timing effects [7] but they can be considered as probably close to the real worst-case costs). The simulator accepts PowerPC object code as input. – a module that builds execution graphs and analyzes them either with Li’s approach [10] or with the approach presented in this paper. – a module that generates the constraints for WCET estimation with IPET and gets a WCET estimate from the lp solve ILP solver. We considered a 2-way superscalar processor with dynamic instruction scheduling. The instruction and data caches and the branch predictor are assumed to be perfect (this work focuses on pipeline modeling). The processor includes an 8entry fetch queue and a 32-entry reorder buffer. It has 6 functional units: 2 ALUs (1-cycle latency), one memory unit (2-cycle), one floating-point ALU (3-cycle), one multiplier (6-cycle) and one divider (15-cycle, non pipelined). The benchmarks come from the SNU-suite [13] and are listed in Table 4. They were compiled to PowerPC code using gcc with the -O0 optimization level option. They are rather small benchmarks but this is of little importance since we analyze local effects. They have the advantage of making the ”manual” analysis of their flow facts (loop bounds and infeasible paths) tractable. 5.2

Tightness of Cost Estimates

We have estimated the worst-case cost of basic blocks using the method presented in this paper. While the method only requires considering single-instruction prefixes, we have considered 1- to 4-block prefixes to improve the tightness of the results (with a longer prefix, there is more chance that the node before the block (LP ) depend on the same resources as the last node of the block (LB ) because these resources are likely to be used by earlier instructions; then the estimated cost might be tighter). The worst-case cost considering n-block prefixes is the maximum of the costs obtained for every possible n-block prefix. The worst-case costs were used to compute the WCETs of the benchmarks using the IPET approach: a WCET is derived as the sum of the worst-case costs of the blocks weighted by their (worst-case) execution counts. Table 2 shows the WCET overestimation as a function of the prefix length.

A Context-Parameterized Model for Static Analysis

237

Table 1. Benchmarks function bs fft1 fft1k fibcall fir insertsort lms ludcmp matmul minver qurt select

CRC (Binary search) FFT (Fast Fourier Transform) using Cooly-Turkey algorithm FFT (Fast Fourier Transform) for 1K array of complex numbers Fibonacci series FIR filter with Gaussian number generation Insertion sort LMS adaptive signal enhancement LU decomposition Matrix product Matrix inversion Root computation of quadratic equations N-th largest number selection

When 1-block prefixes are considered, the WCET is overestimated by 31% on the average. However, the overestimation is noticeably small (less than 8%) for four of the benchmarks (fft1k, ludcmp, fibcall and insertsort). The two first ones contain some particularly long basic blocks which makes the cost estimates accurate. The overestimation is particularly high for crc (102%): a look at the per-block results showed that the overestimation did concern a small number of very short basic blocks that come to be executed many times on the worst-case path. As expected, the tightness of the worst-cost estimates is improved when longer prefixes are considered: it is doubled with 2-block prefixes and falls below 7% on a mean as soon as 3-block sequences are analyzed. The exception is lms that still exhibits a sensible overestimation even with 4-block prefixes (14%). As mentioned in Section 2.3, the method proposed in this paper has been inspired by the work by Li et al. [10] and intends to tackle its problem of overestimation. We have estimated the benchmarks WCETs using Li’s algorithm which we implemented in the OTAWA framework. We found that the WCETs were overestimated by 249% on a mean and not less than 100% for every benchmark. These results are higher than but in the same range as those published as part of the 2006 WCET Challenge [14] even if the benchmarks are not the same. The overestimation induced by Li’s model comes from the fact that the start times of the last node of a block and of the last node of its prefix are not estimated using the same assumptions. The prefix has a length equal to the size of the instruction window. On one hand, the last node of the block is analyzed considering a worst-case context: it is assumed that the last instruction of the context leaves the pipeline only when the block enters it (i.e. the instruction window is considered as full) and that all resources are released only at this time. These are clearly very pessimistic assumptions. On the other hand, the start time of the last node of the prefix is computed as if the context had no impact on the prefix. Then the overestimation is inevitable and is all the larger as the instruction window is wider.

238

C. Rochange and P. Sainrat Table 2. Tightness of WCET estimates

bs crc fft1 fft1k fibcall fir insertsort lms ludcmp matmul minver qurt select MEAN

Prefix length (#blocks) 1 2 3 4 69% 39% 11% 1% 102% 66% 3% 1% 19% 4% 0% 0% 8% 1% 0% 0% 3% 1% 0% 0% 24% 14% 6% 4% 2% 1% 1% 1% 37% 22% 15% 14% 5% 2% 1% 1% 23% 8% 4% 3% 29% 12% 2% 1% 27% 16% 5% 3% 51% 15% 7% 4% 31% 15% 4% 3%

Our approach instead confronts node start times obtained with identical assumptions. Moreover, it provides a means for analyzing the impact of each element of the context (i.e. each resource release time) on the cost of a block: the worst-case execution cost of a block is computed as the maximum of components related to the release times of different resources used by the block. The resource that gives the maximum value might be seen as critical for the block. Any processor for which the execution of an instruction sequence can be expressed as an execution graph can be analyzed using our model (we feel that this is true for most of the commercialized cores). Our algorithm to estimate contention delays assumes an oldest-first scheduling policy and should be modified to fit a different policy. Note that our model could easily be extended to take into account variable latencies for the functional units (this would be particularly useful to integrate the results of a separate cache analysis). We could implement the same solution as Li and consider minimum and maximum values ¯ and D. ˘ for the delays in vectors D

6

Conclusion

The goal of this paper was to propose a new method to analyze the execution time of basic blocks considering any possible prefix path. We have presented an approach that relies on the specification of local execution contexts as a set of parameters that stand for the release time of the structural and functional resources required by the basic blocks. Then the execution pattern of each block can be analyzed as a function of the context parameters. Our method handles superscalar pipelined processors with dynamically-scheduled functional units.

A Context-Parameterized Model for Static Analysis

239

We considered perfect instruction and data caches and perfect branch prediction: these components would generally be taken into account in a separate analysis. We have shown that it is possible to extract an upper bound of the cost of a basic block (i.e. its worst-case cost) from its parameterized execution pattern. The knowledge of worst-case costs is useful to estimate the Worst-Case Execution Time of a real-time task with strict deadlines. Reported experimental results show that the overestimation of Worst-Case Execution Times is moderate (31% on the average). It can be reduced down to 3% by considering longer prefixes. This increases the number of sequences and thus the estimation time but this time remains moderate since the algorithms only require a single graph traversal. As future work, we intend to investigate the possibilities of exploiting parameterized block execution patterns to improve the WCET predictability through code transformations.

References 1. Alt, M., Ferdinand, C., Martin, F., Wilhelm, R.: Cache Behavior Prediction by Abstract Interpretation. In: Static Analysis Symposium (1996) 2. Bate, I., Reutemann, R.: Efficient Integration of Bimodal Branch Prediction and Pipeline Analysis. In: 11th IEEE Int’l. Conf. on Embedded and Real-Time Computing Systems and Applications (2005) 3. Barre, J., Landet, C., Rochange, C., Sainrat, P.: Modeling Instruction-Level Parallelism for WCET Evaluation. In: 12th IEEE Int’l. Conf. on Embedded and RealTime Systems and Applications (2006) 4. Cass´e, H., Sainrat, P.: OTAWA, a framework for experimenting WCET computations. In: 3rd European Congress on Embedded Real-Time Software (2006) 5. Colin, A., Puaut, I.: Worst Case Execution Time Analysis for a Processor with Branch Prediction. In: Real-Time Systems, vol. 18(2). Kluwer, Dordrecht (2000) 6. Engblom, J., Ermedahl, A., Sj¨ odin, M., Gustafsson, J., Hansson, H.: Towards Industry-Strength Worst Case Execution Time Analysis. ASTEC 99/02 Report (1999) 7. Engblom, J.: Processor Pipelines and and Static Worst-Case Execution Time Analysis. Ph.D. thesis, University of Uppsala (2002) 8. Li, Y.-T.S., Malik, S.: Performance Analysis of Embedded Software using Implicit Path Enumeration. In: Workshop on Languages, Compilers, and Tools for Realtime Systems (1995) 9. Li, X., Roychoudhury, A., Mitra, T.: Modeling Out-of-Order Processors for Software Timing Analysis. In: IEEE Real-Time Systems Symposium (2004) 10. Li, X., Roychoudhury, A., Mitra, T.: Modeling out-of-order processors for WCET analysis. In: Real-Time Systems, vol. 34(3). Kluwer, Dordrecht (2006) 11. Lundqvist, T., Stenstr¨ om, P.: An Integrated Path and Timing Analysis Method based on Cycle-Level Symbolic Execution. In: Real-Time Systems, vol. 17(2-3). Kluwer, Dordrecht (1999) 12. Thesing, S.: Safe and Precise WCET Determination by Abstract Interpretation of Pipeline Models. PhD thesis, Universit¨at des Saarlandes (2004) 13. http://archi.snu.ac.kr/realtime/benchmark/ 14. Gustafsson, J.: WCET Challenge 2006. Technical Report of M¨ arladalen University (2007)

240

C. Rochange and P. Sainrat

Appendix Proof of Proposition 1 The statement holds for every node I that does not depend on any node in S: ρI = max (erI (drI + ar )) r∈R

with erI = e¯rI and drI = d¯rI . Now let us assume that the statement holds for every node P earlier than node N. Then ρ˘N can be written as:  

= max e˘S d˘S + max (er (dr + ar )) + ωS ρ˘N = max e˘S d˘S + σS N

S∈S

N

N

S∈S

N

r∈R

S

S

Since every node in the graph depends at least on one resource in R (e.g. the resource that stands for the block start time), we can write: ∀S ∈ S, ∃r ∈ R|erS = 1 Hence, ρ˘N can also be written as: 

ρ˘N = max max e˘SN · erS drS + ar + d˘SN + ωS S∈S

r∈R

or 

ρ˘N = max max e˘SN · erS drS + ar + d˘SN + ωS r∈R

S∈S

As a result:  

   ρN = max max e¯rN d¯rN +ar , max max e˘SN · erS drS +ar + d˘SN +ωS r∈R

r∈R

S∈S

thus  

  ρN = max max e¯rN d¯rN + ar , max e˘SN · erS drS + ar + d˘SN + ωS r∈R

S∈S

Now, by construction, if node N depends on node S ∈ S which in turn depends on resource r ∈ R, it must be that node N also depends on resource r: e¯rN ≥ e˘SN · e¯rS Then:   

ρN = max e¯rN · max d¯rN , max e˘SN · erS drS + d˘SN + ωS + ar r∈R

S∈S

This proves the proposition by induction. ⊓ ⊔

A Context-Parameterized Model for Static Analysis

241

Proof of Proposition 2 Let β ∈ R and π ∈ R be the most critical resources for nodes LB and LP respectively:

ρLB = dβLB + aβ ρLP = dπLP + aπ Then ∆(LB , LP ) = dβLB + aβ − dπLP − aπ . If node LP also depends on resource β (eβLP = 1), we can write: dβLP + aβ ≤ dπLP + aπ since β is not the most critical resource for LP (or β = π). Then: aβ ≤ dπLP + aπ − dβLP and:

∆(LB , LP ) ≤ dβLB − dβLP

(7)

Otherwise (eβLP = 0), we remark that ∀β ∈ R, aβ ≤ aλ − αβ . Thus: ∆(LB , LP ) ≤ dβLB + aλ − αβ − dπLP − aπ Moreover, λ is not the most critical resource for node LP (or λ = π). Then dλLP + aλ ≤ dπLP + aπ and aλ ≤ dπLP + aπ − dλLP . Hence: ∆(LB , LP ) ≤ dβLB − dλLP − αβ

(8)

Finally, equations (7) to (8) can be compiled into:       ∆(LB , LP ) ≤ max erLB · drLB − erLP · drLP − 1 − erLP · dλLP + αr r∈R

⊓ ⊔

Reexecution and Selective Reuse in Checkpoint Processors Amit Golander and Shlomo Weiss Tel Aviv University, Tel Aviv, 69978, Israel { amigos,weiss} @eng.tau.ac.il

Abstract. Resource-efficient checkpoint processors have been shown to recover to an earlier safe state very fast. Yet in order to complete the misprediction recovery they also need to reexecute the code segment between the recovered checkpoint and the mispredicted instruction. This paper evaluates two novel reuse methods which accelerate reexecution paths by reusing the results of instructions and the outcome of branches obtained during the first run. The paper also evaluates, in the context of checkpoint processors, two other reuse methods targeting trivial and repetitive arithmetic operations. A reuse approach combining all four methods requires an area of 0.87[mm2 ], consumes 51.6[mW], and improves the energy-delay product by 4.8% and 11.85% for the integer and floating point benchmarks respectively.

1

Introduction

Speculative processors may have a variety of misspeculation sources: branches, load values [21], and memory dependences [24] are just a few examples. A misspeculation initiates a recovery process, which in checkpoint microarchitectures has two phases: rollback, in which the latest safe checkpoint (a stored processor state) preceding the point of misprediction is recovered, and reexecution, in which the entire code segment between the recovered checkpoint and the mispredicting instruction is executed again. Since the same instruction instance is processed in both the first and the reexecuted run, nearly always carrying the same control and data dependences, checkpoint processors are characterized by a higher repetitive nature than processors that use a reorder buffer. Repetitiveness may be exploited by reuse methods that waive instruction execution by reusing stored results from past execution in order to save resources, resolve data dependences early, and improve processor performance. In this paper we study the unique characteristics of reexecution in checkpoint processors. To exploit reexecution we develop a scheme for monitoring instruction execution after taking a checkpoint. This scheme is the enabler for linking a reexecuted instruction to its stored result calculated during the first run. We present two complementary reuse methods, one handling flow control instructions (RbckBr ) and the other handling all other instruction types (RbckReuse). Searching for low-cost solutions, we reconsider the undiscriminating reuse approach represented by the RbckReuse method. Selective reuse is an alternative P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 242–268, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Reexecution and Selective Reuse in Checkpoint Processors

243

approach, it reduces the implementation cost by focusing on instruction types that have a significant impact on the speedup. We identify three such instruction types: branches, memory loads, and long latency arithmetic operations. Reexecuted branches are handled by the RbckBr method; reexecuted loads access trained caches and we do not attempt any further acceleration; the third and final type is arithmetic operations. To reuse an arithmetic operation it is sufficient to detect the reexecution of the operation with the same operands, although the instruction instance is different. The result of an arithmetic operation can be available without execution when the instruction degenerates to a trivial operation or the result was calculated in the past and saved. We refer to these two methods as Trivial and SelReuse respectively. Although reexecution is a natural candidate for reuse, reusing results during reexecution in checkpoint processors was never investigated. The contribution of this paper is as follows. 1. We introduce and investigate a method (RbckReuse) for accelerating reexecuted paths. 2. We introduce and investigate a method (RbckBr) for reusing the outcome of branches to achieve nearly perfect branch prediction during reexecution. 3. The above two methods are both related to rollback events in checkpoint processors. We also consider two other methods (Trivial and SelReuse) that accelerate arithmetic operations during normal instruction execution. Although both are well-known methods, this is the first paper that presents results in the context of checkpoint processors. 4. We place a price tag (power consumption, area, and access time) on each of the above four methods and based on it and on the performance results we recommend a combined approach. The remainder of the paper is organized as follows. Section 2 describes the experimental methodology. Section 3 analyzes reexecution characteristics. Section 4 is a classification of reuse methods, four of which are presented in sections 5 (Trivial), 6 (SelReuse), 7 (RbckReuse) and 8 (RbckBr). These methods and recommended combinations of them are compared in Section 9. Related work is surveyed in Section 10 and finally, conclusions are drawn in Section 11.

2

Experimental Methodology

We used SimpleScalar for performance simulation and Cacti [37] for power, area and access time estimates. We modified the sim-outorder model of SimpleScalar to implement checkpoints, variable length pipelines, variable size register files, instruction queues (IQ), a state-of-the-art branch predictor (TAGE [32]), and a mechanism that completely hides the rollback penalty, regardless of the state of the pipeline (CRB [14]). The simulation parameters of the baseline microarchitecture (refer to Table 1) mostly follow the parameters of the IBM Power4 and Power5. The latency of the floating-point multiply and add instructions are the same as in the Power4

244

A. Golander and S. Weiss Table 1. Baseline processor parameters Branch A five component, 70Kbit TAGE [32] predictor. predictor BTB: 128 sets, 4-way. RAS: 8-entry. Confidence estimator: 4K-entry, 4bit JRS [17]. Front end Decode: 2 cycles. Rename and dispatch: 3 cycles. Resources 4-deep in-order checkpoint buffer. M axDist threshold: 256 IQ: 96. LSQ: 96. Processor width: 4. INT/FP: 128/128 register file (a single cycle structure) Execution FP multiplication/division: 6/30 cycles, one unit. units FP add and subtract: 6 cycles, two execution units. FP rounding, conversion and move: 2 cycles. INT multiplication/division: 7/30 cycles, one unit. Caches and Inst.-L1: 64KB, 256 sets, 2-way, 2 cycles. memory Data-L1: 32KB, 64 sets, 4-way, 2 cycles. Level one caches use 128B blocks, broken into four 32B sectors. Unified-L2: 1MB, 8-way, 8 cycles, 128B blocks. Memory access time is 200 cycles.

and Power5. (The pipeline latencies of the Power4 are shown in Table 1 in [9]. An IEEE Micro article [18] on the Power5 explains that the Power4 and Power5 microprocessors have identical pipeline structures and the same latencies). We also use Power5 data for estimating the area budget of the proposed hardware enhancements (Section 9.3). A relatively large load-store queue (LSQ) is needed for checkpoint processors, which commit stores only when the relevant checkpoint is committed [11]. A reduced-cost implementation (SRL – Store Redo Log), using secondary buffers without CAM and search functions, is proposed in [12]. Currently our simulator does not support SRL and we do not evaluate its effect on the results. An indication of the effect that can be expected is provided by Gandhi et al. [12], who report that SRL is within 6% of the performance of an ideal store queue. All 26 SPEC CPU2000 benchmarks were used. Results were measured on a 300 million instruction interval, starting after half a billion instructions. Power, area, and latency costs of the hardware structures were estimated assuming a 64-bit machine, using version 4.2 of Cacti, and a 70[nm] technology, the most advanced technology the tool was verified on. Dynamic energy was translated to power consumption using a 2GHz clock frequency and switching information from the performance simulation. Leakage power was also taken into account using CACTI.

3

Reexecution

Assuming a free checkpoint exists, the processor will take a checkpoint at the following points in the program: at a branch that is predicted with a low confidence estimation level, at the first branch after rollback, when the number of instructions after the last checkpoint exceeds a certain threshold M axDist ,

Reexecution and Selective Reuse in Checkpoint Processors

INT

FP

18% 15% 12% 9% 6% 3% Ammp Applu Apsi Art Equake Facerec Fma3d Galgel Lucas Mesa Mgrid Sixtrack Swim Wupwis Mean

0% Bzip Crafty Eon Gap Gcc Gzip Mcf Parser Perlbmk Twolf Vortex Vpr Mean

Fraction of Reexecution

21%

245

Fig. 1. Fraction of reexecuted instructions

and when the number of store instructions after the last checkpoint exceeds another threshold. It is evident that every reexecution path will contain less than M axDist instructions. 3.1

Potential of Reexecution

A reexecuted path is faster for several reasons: (1) data dependences on instructions preceding the checkpoint have been resolved, (2) resources such as execution units, IQ entries, registers and checkpoints are freed during rollback, so they are likely to be available, (3) caches have been trained (for example the first level data cache miss rate was measured to decrease from 3.4% to 0.8%), and finally (4) branch predictor unit components, such as the BTB were also trained. Although reexecuted instructions inherently run faster they could still be further accelerated. Reusing ready results from the reexecution path is expected to have an impact on the IPC because of the following reasons: 1. The fraction of instructions that are reexecuted is substantial, especially in the integer benchmarks, as shown in Figure 1. These numbers are higher than the averages presented by Akkary et al. [1], primarily due to the different instruction set architecture, and the fewer resources used in this study, for instance in the checkpoint buffer. 2. Most reexecuted instructions have their results ready before the misspeculation is detected. Our study indicates that nearly all (92.5% for the integer benchmarks) results of instructions about to reexecute are already available when rollback is initiated. 3. Results are reused early enough in the pipeline stage and their results quickly propagate back into the register file. To avoid adding complexity to the pipeline, we use a dedicated RbckReuse structure. At reexecution, reused results (with the exception of branches, which we discuss later) are merged during the decode stage.

A.

Length [Numer of Instructions]

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Mean

INT FP

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140

INT FP

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140

100% Mean 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Cumulative Percentage of Code Segments between Adjacent Checkpoints

A. Golander and S. Weiss

Cumulative Percentage of Reexecuted Code Segments

246

B.

Length [Numer of Instructions]

Fig. 2. Cumulative distribution of: A. the length of reexecuted code segments, and B. the distance between adjacent checkpoints

Contribution to IPC is not the sole factor when considering the microarchitecture of a processor, ease and cost of the reuse are also important factors. Reexecution is a natural candidate for efficient reuse structures, mainly owing to three characteristics: reexecuted instructions almost always maintain the control and data dependences they had during the first run, each reexecuted path has a known start point (one of several checkpoints), and reexecution is limited to M axDist instructions. From the cumulative distribution of the length of reexecuted code segments in Figure 2A it is clear that reuse structures do not have to be M axDist deep. 3.2

Keeping Track of Reexecution

Akkary et al. [1] suggested counting branches in order to identify the mispredicted branch on reexecution to avoid repeating the same misprediction. We generalize this approach by counting all instructions, which allows a unique identifier per reexecuted instruction regardless of its type. Given that a reexecution path always begins at a checkpoint, we maintain a DistCP counter that is reset when a checkpoint is taken or a rollback event occurs, and incremented for each instruction that is decoded. The instruction carries the DistCP value with it, as it does with the checkpoint tag (CP ). For instructions preceding the mispredicted branch, the CP and DistCP fields constitute a unique identifier. We use this identifier for storing results in the reuse structures. A processor recovers to the latest checkpoint preceding the misspeculation, so all the instructions in the reexecution path originally belonged to a single checkpoint. During reexecution new checkpoints may be taken and all reexecuted instructions following the new checkpoint have new CP and DistCP parameters. This new instruction identifier, which will be used for searching results, is different from the identifier used to save them. To translate the new identifier to the old one, we add a second counter, DistCP Rbck , that is only used during reexecution. DistCP Rbck increments like the DistCP counter, but is reset only at

Reexecution and Selective Reuse in Checkpoint Processors

Reexecution path Wrong-path

Folded code segment

First run

CP1

CP = 1 DistCP = 9

Reexecution

Insta Instb Instc Instd Inste Instf BEQg Insth Insti Instj BEQk

CP2’

Misprediction occurred CP2

247

CP = 2 DistCP = 3 CPRbck = 1 DistCPRbck = 9

Instl Instm Instn

Misprediction detected

Fig. 3. Keeping track of reexecution

rollback events. In addition, we also save the checkpoint tag we have rolled back to and denote it as CP Rbck. Using CP Rbck and DistCP Rbck the reexecuted instruction can search the reuse structure using its original identifier, regardless of the number of new checkpoints taken during reexecution. Example 1: Keeping Track of Reexecution This process is illustrated in the example in Figure 3. The instruction BEQk of checkpoint one (CP1 ) is a mispredicted branch that is later detected, after another checkpoint CP2 is taken. When the rollback occurs, CP2 is folded (flushed) and the processor recovers its state from CP1 . Just before reexecution begins, the relevant parameters have the following values: CP Rbck==1, DistCP ==0 and DistCP Rbck ==0. The first six reexecuted instruction (Insta to Instf ) carry the same DistCP and CP as they did during the first run. The seventh reexecuted instruction BEQg is a branch, and a new checkpoint CP2′ is taken. (Note that CP2′ and CP2 are different, even though both follow CP1 and carry the same CP tag). Now Insti , for example, has a DistCP value of three, but we can lookup the result that the instruction produced during the first run using CP Rbck and DistCP Rbck , which were not affected by the new checkpoint CP2′ . The 11th instruction after CP1 in Figure 3 is the mispredicted branch. The outcome of that branch is known at rollback and is used during reexecution to prevent making the same misprediction twice. We add storage to save the DistCP field carried by the mispredicted branch, and another bit to save the correct branch direction. We further use this event to reset a ReexecutionFlag we later use for filtering.

248

4

A. Golander and S. Weiss

Reuse Methods

We define six reuse methods: two methods handling arithmetic operations, Trivial and SelReuse, that accelerate trivial and repeating long latency arithmetic computations, respectively; two reuse methods, RbckReuse and RbckBr, that improve the performance of the reexecution path following a rollback event; and two wrong-path reuse methods, WrPathReuse and WrPathBr that improve the performance of instructions following the point in which the correct and the wrong paths converge. These methods differ in several aspects as specified in Table 2, and though wrong-path reuse aspects are later analyzed in the context of the SelReuse method, only the first four methods are covered by this paper. We have included Trivial in this table even though it does not dynamically reuse any information. Many trivial calculations however would also be accelerated by the SelReuse method, therefore in a sense these two methods are partially substitutional. We later show that filtering trivial computations from SelReuse helps better utilize the SelReuse hardware. Table 2. Properties of reuse methods Method

Which When during instructions? the program run? Trivial Long latency Normal run calculations SelReuse Long latency Normal run calculations RbckReuse All Reexecution path RbckBr Flow control Reexecution path WrPathReuse All Following reexecution WrPathBr Flow control Following reexecution

5

Where What is Is the in the reused? instruction pipeline? executed? Execute stage N/A No Execute stage Results

No

Decode stage Results

Only loads Yes

Fetch stage

Branch outcomes Decode stage Results Fetch stage

Branch outcomes

Varies Yes

“Trivial” Arithmetic Operations

Arithmetic operations become trivial for certain input operands. This could happen when one of the inputs is zero or the neutral element (α±0, α×0, α×1, 0 α α α, 0 and 1 ), or both operands have the same magnitude (α −α, α + (−α) α and α). Compilers remove static trivial computations but cannot remove trivial computations that depend on runtime information. Results indicate that many computations are trivial. Table 3 shows the frequency of usage of long latency instructions and the percentage of trivial computations. Since the Trivial method is simple, we detect and accelerate all these

Reexecution and Selective Reuse in Checkpoint Processors

249

Table 3. Long latency arithmetic operations: the frequency of usage, fraction of trivial computations and breakdown according to the type of trivial computation. Note that division by zero occurs only when executing wrong-path instructions.

Instruction type

Instruction type Instruction type T rivial T rivial All instructions Instruction type All instructions Instruction type (FP benchmarks) (FP benchmarks) (INT benchmarks) (INT benchmarks)

11.95%

FP Add/Subtract

α±

0, 0 ± α α− α 8.83%

FP Multiplication

α× α×

0, 0 × 1, 1 ×

α α 0.61%

INT Multiplication

α× α×

0, 0 × 1, 1 ×

α α 0.18%

INT&FP Division

α/1 0/α α/α α/0

33.76% 28.86% 4.90% 38.76% 35.77% 2.99% 41.44% 11.94% 29.50% 25.96% 25.37% 0.31% 0.23% 0.04%

0.48%

25.01% 18.86% 6.15% 31.87% 26.71% 5.15% 20.32% 18.10% 2.22% 7.37% 3.56% 2.87% 0.02% 0.93%

0.26%

0.53%

0.07%

sources for trivial computations, including the infrequent ones. Certain trivial computations can further improve the performance by canceling true data dependences. A trivial instruction such as α×0 will have the same result regardless

Integer Trivial Detection:

Multiplication Result Selection:

MSbit

A LSbit

A=0 ? A=1 ?

A B

1

0

0

1

Result 0

A=B ?

A=0 A=1

MSbit

B LSbit

B=1 A=1

Hit B=0 ?

B=0 B=1

B=1 ?

Fig. 4. Implementation of a Trivial structure for integer multiplication. The left side of the figure includes logic to detect an operand value of zero, one, and identical operands (equality is only required for subtraction and division). The right side of the figure illustrates how to select results. A result is only valid when the hit indication is set.

250

A. Golander and S. Weiss B.

A.

Folded checkpoint

Operands and opcode

SelReuse Execution unit

Cancellation notification

Result writeback bus

CP

Replace

Opcode Operand A Operand B

= = = = Replacement policy

Result Number of entries

Trivial

Fully associative cache search criteria (tag)

Fig. 5. Handling long latency computations. A. Block diagram of a unit that executes long latency instruction. B. A closer look at a SelReuse cache. A hit occurs when both operands A, B and the opcode are identical.

of the value of α and instructions using this result are not truly dependent. In this work we do not cancel such data dependences. The hardware for detecting trivial computations and selecting the result consists primarily of comparators for the input operands and muxes for writeback. An integer multiplication Trivial structure is illustrated in Figure 4. Its location at the execution stage is shown in Figure 5A. The Trivial structures are estimated to take an area of 0.015[mm2 ] and consume 9.14[mW]. Access time is estimated to be within a single cycle operation.

6

“SelReuse” – Reusing Results of Arithmetic Operations

In this section we consider reusing the results of repetitive arithmetic operations. SelReuse and Trivial are complementary methods. SelReuse can accelerate both trivial and non-trivial computations that occur repetitively, dynamically adjusting per program and code characteristics. SelReuse however cannot accelerate the first occurrence of a trivial computation and will waste multiple entries on the same trivial computation type, if for example presented with X1 + 0 and X2 + 0 where X1 is different than X2 . Filtering trivial calculations from entering the SelReuse cache structure increases its effective size. Figure 5A illustrates the block diagram of a unit that handles long latency computations. At the execution stage, the operands and the opcode enter three units in parallel. The first unit responds to trivial computations within a cycle. The second unit (a SelReuse cache) attempts to reuse values from recent nontrivial computations and can supply the result within two cycles. Finally, the third entity is a regular long latency execution unit that calculates the result from scratch. When one of the first two structures succeeds, it provides the result and aborts the operation in any of the remaining slower structures. Calculated results also update the SelReuse cache. With the addition of the trivial and SelReuse structures in Figure 5A arithmetic operations have variable latencies. Variable latencies have been used in microprocessors generally in division operations [16,38,19] but also in integer multiplication (PowerPC 604 [35] and 620 [20], Alpha 21164 [5]). For example, in the Alpha 21164 [5] the multiplication latency depends on the size and type of

Reexecution and Selective Reuse in Checkpoint Processors

7%

251

INT Mult.

6%

FP Mult.

5%

Division FP Add

4% 3% 2% 1%

5+

4

3

2

1

0%

Number of Times SelReuse Results were Reused

Fig. 6. The number of times a result stored in the SelReuse structure is reused prior to being replaced

operands: eight cycles for 32-bit operands, 12 cycles for 64-bit signed operands, and 14 cycles for the 64-bit multiply unsigned high instruction. Variable latency is not a problem in functional units that have a dedicated result bus. This is indeed the case in the PowerPC 604 [35] for example, but not in the P6 [16]; in the latter the floating point unit and the integer unit (which handles integer multiplication and division) share the same result bus. Scheduling operations on a shared bus can be done by a simple shift register mechanism described in [33]. Figure 5B takes a closer look at the SelReuse cache. It is a small (4-entry) fully associative cache that uses a tag composed of the values of the operands and the opcode. Entries are marked as candidates for replacement when the checkpoint they belong to is folded. This replacement policy favors instructions that are yet to be committed. Figure 6 presents the number of times a result stored in the SelReuse structure is reused prior to being replaced. It reveals that most stored results are never reused and that even if a result is reused it will usually be reused only once or twice. Performing the same measurement on a processor using very large (4K-entry) SelReuse structures shows even lower efficiency. These averages however are heavily biased towards reusing results during the normal run of the program. SelReuse is much more efficient during code reexecution after misspeculation. Folded code segments as demonstrated in Figure 3 are comprised of a reexecution path followed by a wrong path. Arithmetic operations in the reexecution path will nearly always run again with identical operand values, and the same is true for a subset of the wrong-path arithmetic operations. This subset includes instructions following the point in which the correct and the wrong paths converge and whose operand values are not affected by the misspeculation. Figure 7A proves that the efficiency of SelReuse, following a rollback, is an order of magnitude higher. It shows that small SelReuse caches holding a few entries alone can achieve high reuse percentages for reexecution paths and to a

A. Golander and S. Weiss

31%

SelReuse Percentage

30%

Reexecution WrPath First run

25%

23%

20% 15%

SelReuse Percentage

252

10%

50% 40% 30% 20% 10% 0% 0

5% 3.4%

0% 0

A.

60%

1

2

3

4

5

6

7

8

Number of SelReuse Entries

4K

B.

1

2 3 4 5 6 7 Number INT Mult.of SelReuse Entries INT Mult. (Reexecution) FP Mult. FP Mult. (Reexecution) Division Division (Reexecution) FP Add FP Add (Reexecution)

8

Fig. 7. Reuse percentage as a function of the number of entries in the SelReuse cache. A. SelReuse hit rates (all instruction types) on the first run, reexecution and wrong path (WrPath). B. SelReuse hit rates on the first run and reexecution per instruction type. The total (100%) excludes trivial arithmetic operations.

lesser extent for instructions that were executed as part of the wrong path. The figure also specifies the reuse percentage achieved when essentially unlimited (4K-entry) SelReuse caches are used. Clearly, an 8-entry cache is sufficient for reusing most of the available results. Figure 7B is a breakdown of the reuse rate on the first run and reexecution per instruction type. It shows that the reuse percentage gap between first run and reexecution is large for all instruction types. The reuse percentage is higher for less frequent instructions such as division, as the SelReuse cache experiences fewer capacity misses. The figure also reveals that the reuse percentage for reexecuted instructions is far from 100% (100% excludes trivial operations), indicating that most long latency instructions did not complete execution prior to rollback due to true data dependences. We also checked the impact of the register file size on the Trivial and SelReuse advantage. Doubling the register file size to 256 registers reduces the performance advantage by 7.5% and 10.8% for the integer and floating point benchmarks respectively. The relatively small degradation derives from the resource-efficient baseline microarchitecture that uses early register release mechanisms to attain satisfactory performance with a reduced-size register file. In the remainder of this paper we use a 4-entry SelReuse cache for each long latency instruction type. The Add/Subtract SelReuse cache is somewhat different than others. It has two read and two write ports, as it serves two execution units. The estimated area and power consumption of the SelReuse caches are 0.030[mm2 ] and 10.08[mW] respectively. Access time depends mainly on the large tag comparator, which takes 0.83[ns], hence two 2GHz clock cycles.

Reexecution and Selective Reuse in Checkpoint Processors CP1 CP, DistCP (Wr)

V Result

253

CPMax V Result

DistCPRbck (Rd)

CPRbck (Rd)

Fig. 8. The logical view of a RbckReuse structure. The lookup tables (LUT) contain results and valid bits.

7

“RbckReuse” – Reusing Results on Reexecution

Figure 8 illustrates the logical view of a RbckReuse structure. The RbckReuse structure saves results from all instruction types, with the exception of flow control instructions which are handled in the fetch stage as described in Section 8. The RbckReuse structure is accessed using the counters defined earlier in Subsection 3.2. Ready results are saved using the DistCP and CP counters, and results are searched for, during reexecution, using the DistCP Rbck counter and CP Rbck tag. A reexecuted instruction can be accelerated if the result is ready (V==1) and saved (DistCP Rbck < RbckReuse depth). An accelerated instruction does not rename or read its source operands and does not use resources such as an IQ entry or an execution unit. The percentage of results reused during reexecution is illustrated in Figure 9. The figure shows that the results of over 90% of the instructions of the integer benchmarks can be reused if the RbckReuse structure has 128 entries or more. The floating point reuse percentage is lower because floating point data dependences take longer to resolve and a lower percentage of the results are ready when rollback is invoked. We later present an alternative RbckReuse structure which is not LUT-based and has better performance for smaller RbckReuse structures. 7.1

Verifying the Result of Load Instructions

So far, we have assumed the result of a load instruction will be identical if reexecuted. A reexecuted load instruction can yield a different value than stored in the RbckReuse buffer if the stored value was incorrect in the first place, or if the memory content has been modified since. An incorrect load result may stem from wrong predictions that are designed to overcome data dependences, such as memory dependence [23], load address [4], and load value [21] predictors. We considered two solutions to this problem. The first one follows the concept of a load value predictor [21]. In this scheme, the processor speculatively reuses the results of all instructions, including loads and dependent instructions. In

A. Golander and S. Weiss

RbckReuse Percentage

254

100% 90% 80% 70% 60% 50% 40% 30% 20%

INT FP

0

64 128 192 RbckReuse Structure Depth

256

Fig. 9. The percentage of results reused during reexecution

order to verify the speculation, all reused load instructions are also reexecuted. If the result turns out to be different a recovery process begins, flushing all data dependent instructions in the process. A second approach [31] prevents full rollbacks by reexecuting only the speculative load and dependent instructions that used the incorrect load value. We implemented the first solution, in which all load instructions are reexecuted for the purpose of detecting any changes relative to the values loaded during normal execution. The added overhead of the reexecuted load instructions reduces the energy delay product by less than 0.1%. 7.2

RbckReuse Implementation Cost

The logical view of the RbckReuse hardware presented in Figure 8 can be optimized as follows. We rearrange the structure so that each line holds results from four consecutive instructions from the same checkpoint. This enables us to use a single read port because instructions are decoded in program order. The results arrive out-of-order though and four write ports, matching the writeback width of the processor, are still required. The access time, area, and power of the RbckReuse structure are described in Table 4. The access time does not present a problem as it is within a single cycle of the 2GHz processor. The figures in the table were obtained with Cacti and are based on an SRAM implementation The SRAM implementation is efficient as far as the memory cell is concerned, but has a nonlinear overhead for the control logic circuits. Small SRAMs such as the 32 and 64-deep RbckReuse structures are inefficient as indicated by the low RAM cell percentage. Suresh et al. [36] suggested alternative implementations of small SRAMs that require less area. Unfortunately, Cacti only implements the SRAM-based version. The increase in the dynamic power with RbckReuse depth comes from two sources. First a deeper structure requires more energy per a read/write operation. In addition small structures have fewer read/write operations, they keep track only of instructions that are close to the restart point. Comparing the dynamic power of a 256and 128-deep RbckReuse structure shows a relatively moderate increase (13%) because the second source of power increase is negligible at that size (Figure 9).

Reexecution and Selective Reuse in Checkpoint Processors

255

Table 4. Estimated access time, area and power consumption for LUT-based RbckReuse structures of several depths

Access time Area RAM cell percentage Leakage power Dynamic power Total power

7.3

[ns] [mm2 ] [mW] [mW] [mW]

32 0.383 0.361 15.6% 4.73 5.30 10.03

64 0.423 0.663 16.8% 8.92 8.84 17.76

128 0.448 0.825 27.1% 15.87 16.51 32.37

256 0.486 1.630 27.4% 31.96 18.66 50.62

Alternative Implementation for Small RbckReuse Structures

The RbckReuse structure we have presented thus far requires one LUT per checkpoint. Most rollback processes recover a recent checkpoint, a pattern seen in figures 10A and 10C, in which more than 50% of the recoveries are to the last checkpoint or the one previous to that. We now evaluate an alternative RbckReuse structure that is based on giving priority to results from recent checkpoints, instead of storing results of each checkpoint in a separate LUT. A cache-based RbckReuse structure uses set-associativity to share resources between checkpoints. Sets are accessed by the DistCP value, and the CP is used as a tag. Results from recent checkpoints replace results from older ones, and folded checkpoints invalidate their cache entries. Figures 10B and 10D show that a 2-way cache-based RbckReuse structure outperforms the LUT-based design presented earlier for small structures. This trend is reversed for deeper RbckReuse structures, as LUT-based RbckReuse structures do not replace results that may still be reused. To compare the potential of RbckReuse with other methods, in the rest of this paper we use a 128-deep LUT-based structure. 7.4

RbckReuse Sensitivity Analysis

The speedup potential of the RbckReuse method primarily depends on the fraction of reexecuted instructions (see Figure 1). We now explore several approaches, each requiring additional hardware, to reduce the fraction of reexecuted instructions and to measure the benefit of the RbckReuse method. The number of reexecuted instruction can be reduced by using a deeper checkpoint buffer (refer to Figure 11). A deeper buffer prevents scenarios in which a mispredicted branch was estimated as a low confidence branch but a checkpoint was not allocated due to lack of space in the checkpoint buffer. Figure 11 verifies our assumption about the RbckReuse advantage being proportional to the fraction of reexecution. It shows that increasing the checkpoint buffer depth from four to eight entries reduces the RbckReuse advantage by about 20%; for larger buffers the fraction of reexecuted instructions reaches a steady state. Increasing the checkpoint buffer depth up to eight entries is beneficial, but has a negative impact on the latency of accessing the mapping table [3] and on the rollback time [14]. For this reason, we have chosen a 4-deep checkpoint buffer (Table 1).

8-deep checkpoint buffer

A.

4 3 2 1

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

LUT-based 2-way cache-based 0

INT

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

C.

RbckReuse Percentage

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

FP

8 7 6 5 4 3 2 1

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

FP

D.

32

48

64

LUT-based 2-way cache-based 4-way cache-based 0

INT

16

RbckReuse Depth

B. RbckReuse Percentage

4-deep checkpoint buffer

Distribution of Folded Checkpoints

A. Golander and S. Weiss

Distribution of Folded Checkpoints

256

16

32

48

64

RbckReuse Depth

Fig. 10. Motivation (A,C) and results (B,D) of an alternative implementation of a RbckReuse structure. A. Distribution of folded checkpoints. The total (100%) is the total number of recovery events. The percentage are the number of recovery events in which i = 1, 2, 3, 4 checkpoints were folded (we maintain a 4-deep checkpoint buffer). B. Reuse percentage achieved by the LUT-based and the cache-based RbckReuse structures. The cache-based structure depth is normalized to present the same effective storage capacity, for example if storage for 128 results is available, we can either implement a 32-deep LUT-based or a 64-deep 2-way cache-based structure. This normalization is slightly in favor of the cache-based structure as it ignores the logic required for the tags (a 4% overhead). Figures C and D are the equivalents of figures A and B for an 8-deep checkpoint buffer microarchitecture.

We also checked the effect of the branch predictor and confidence estimator on the fraction of reexecution. Doubling the branch predictor size to 140 kbit reduces the number of reexecuted segments by 1.7% and 6.9% but increases the average length of the reexecuted segments by about 2.7% and 3.1% for the integer and floating point benchmarks respectively. Hence the fraction of reexecution is not significantly affected (+0.9% and -4.1% for the integer and floating point benchmarks respectively). The reason for this is that increasing the accuracy of the branch prediction reduces the quality of the confidence estimation [14]. Having checked how RbckReuse performance advantage depends on the fraction of reexecuted instructions, we now validate our earlier assumption (Subsection 3.1) that registers are freed during rollback and are usually available at reexecution. Repeating simulations using a double size register file showed that the RbckReuse speedup advantage decreased by only 3% and 17% for the integer and floating point benchmarks respectively.

Reexecution and Selective Reuse in Checkpoint Processors

1.04 9% 1.03 6% 1.02 3%

Fration of reexecution "RbckReuse speedup"

0%

INT

1.01

Fraction of Reexecution

1.05

12%

12%

Fraction of reexecution RbckReuse speedup

FP

1.05 1.04

9%

1.03 6% 1.02 3%

1.01

0%

1.00 2 4 6 8 Checkpoint Buffer Depth

1.06

15%

IPC Speedup

1.06

IPC Speedup

Fraction of Reexecution

15%

257

1.00 2 4 6 8 Checkpoint Buffer Depth

Fig. 11. Fraction of reexecuted instructions and the speedup advantage RbckReuse achieves as a function of the checkpoint buffer depth. The IPC speedup RbckReuse achieves using each checkpoint buffer depth is normalized to the baseline processor using the same checkpoint buffer.

8

“RbckBr” – Reusing Branches after Rollback

In this section we consider keeping track of the branch results during normal execution of the program and reusing them after a rollback. During reexecution we ignore the branch predictor and predict branches using known branch results that were saved in the process of the normal run of the code. The RbckBr structure basically resembles the LUT-based RbckReuse structure but is optimized to achieve a substantial cost reduction by not saving the branch target address. The reuse structure maintains a single bit of the branch outcome and relies on the BTB for the branch target address. The downside of the optimized RbckBr structure is that we are speculating that the target address was not replaced in the BTB. This is reasonable due to the limited number of flow control instructions processed since the address was stored. Flow control instructions are executed in order to verify RbckBr predictions. Figure 12A illustrates how the optimized RbckBr structure is integrated into the branch prediction unit. During reexecution, in parallel to accessing the branch predictor and the confidence estimator, the reuse structure is searched. For every valid RbckBr entry, the Br bit replaces the direction prediction and the confidence level is marked high. The branch target address is obtained from the BTB. Figure 12B takes a closer look at the logical view of the optimized RbckBr structure. As in RbckReuse, the structure is accessed using the DistCP , CP , CP Rbck and DistCP Rbck qualifiers, but the timing is different. RbckBr reuse results are needed at the fetch rather than decode stage, complicating the management of the DistCP Rbck counter, which should only increment for fetched instructions that are also going to be decoded. The proposed RbckBr structure has one entry for every instruction, although most instructions are not branches, and is accessed using a distance pointer as an index. The obvious alternative, of using a fully associative table, is more

A. Golander and S. Weiss CP1

B.

A. RbckBr structure V

Branch predictor

Br

Confidence estimator

CP, DistCP (Wr)

DistCPRbck (Rd)

Br High

ReexecutionFlag

1

0

1

V Br

CPMax V Br

LUT depth

258

0

CPRbck (Rd)

DistCPRlbck < LUT depth

Br

Conf. level

Fig. 12. The optimized reexecution flow control reuse (RbckBr) structure. A. The modifications required to integrate it. B. The logical view. The valid bit indicates the branch was resolved, and the Br bit is the branch direction.

expensive because of the need of storing in every entry, in addition to the branch outcome bit, a tag that consists of the concatenation of the DistCP and CP . In an implementation of the logical structure presented in Figure 12B, we rearrange the table so that each line holds entries for four consecutive instructions from the same checkpoint, which allows a single read port as was done in the RbckReuse structure. Considering that the frequency of branch instructions in the code is usually less than 25%, a single write port is sufficient in the 4-way superscalar processor studied in this paper. The area and power consumption of a RbckBr reuse structure is low, estimated at 0.007[mm2 ] and 0.57[mW] for a 128-deep structure. For the purpose of the following discussion, we define a notation that classifies flow control instructions according to the success or failure of the prediction in the first run and the prediction in the second run (reexecution). The prediction in the second run is done using the branch predictor, without any fixes that could be made by reusing the branch results. Accordingly a branch instruction may be in one of the four categories: CC, M M , M C and CM , in which the first letter represents a correct (C) prediction or misprediction (M ) during the first run, and the second letter represents a correct (C) prediction or misprediction (M ) during reexecution. We use the notation Rxx for the resolved subset of every category, for instance RMM is a subcategory of M M and includes all the M M branches resolved before the rollback was invoked. Figure 13 quantifies the improvement in the prediction of reexecuted branches. The figure contains four pairs of bars, the leftmost pair presents the branch predictor accuracy of the original (first) run. Obviously, the prediction accuracy in the vicinity of hard-to-predict branches is much lower than the general branch prediction accuracy, which is 95.6% and 98.9% for the integer and floating point benchmarks respectively. The second pair of bars to the left represents the accuracy achieved during reexecution. As expected, training the predictor during the first run improved its performance. The third pair represents a processor that saves the target address of the mispredicting branch which caused the recovery process, as our baseline model does. The forth pair of bars represents the

Reexecution and Selective Reuse in Checkpoint Processors

259

100% Accuracy

95% 90% 85% 80%

INT FP

75% 70%

First run 1

Reexecution Reexecution Naïve 2 3 former 4 fixing the reexecution using misprediction RbckBr

Fig. 13. Gradual improvement of the branch predictor unit on reexecution paths. The leftmost pair of bars represents the accuracy on the first run ( CC+MCC+CM C+M M +CM ). The

second pair is the accuracy on reexecution assuming no additional corrections The third pair shows the accuracy of the second pair in addiCC+M C+RM M tion to resolved M M branches, which are fixed ( CC+M C+M M +CM ). The last pair C+RM M +RCM represents RbckBr which also fixes resolved CM branches ( CC+M CC+M C+M M +CM ).

CC+M C ( CC+M C+M M +CM ).

performance when the proposed (128-deep) RbckBr structure is used. Clearly it achieves near perfect branch prediction performance. We now take a closer look at the RbckBr unique contribution — the CM category flow control instructions that were predicted correctly during the first run, but would be falsely predicted at reexecution. Example 2: Code That May Lead to a CM Event Figure 14 illustrates two scenarios that may lead to a CM event. The code in Figure 14A contains two branches, each depending on a load that precedes it. On the first run the first load gets a L1 miss, and the access of the second load is a L1 hit. As a result, the branches are resolved out-of-order. On reexecution the cache is trained, both loads hit and the branches resolve in-order. The example shows how two branches that resolve in an opposite order when reexecuted will lead to a different branch predictor state. Avoiding this problem by updating the history in-order at commit requires a fine-grained bookkeeping method, such as a reorder buffer, which our checkpoint microarchitecture does not have. As a second example, consider a code sequence that includes a procedure call, return, call, and another return (refer to Figure 14B). Assume a checkpoint was taken between CALL1 and RET N1 . A rollback that occurs between CALL2 and RET N2 finds on the RAS the address of the instruction following CALL2 , which, during reexecution, is incorrectly used by RET N1 . Such RAS misspeculations can be avoided if the entire RAS structure is made a part of the checkpoint. Our simulation model settles for recovering the location of the top of stack. Figure 15 demonstrates how CM reuse percentage increases as the RbckBr structure has greater depth. The behavior of the integer and floating point programs is similar. A 128-deep RbckBr structure succeeds in fixing 92.2% and

260

A. Golander and S. Weiss

A.

First run:

Reexecution:

B.

First run:

Reexecution:

Entry X has the correct value

Entry X has an incorrect value

CALL1

Not taken Taken

LD R1, 0 (R11) BEQ R1, target1 LD R2, 0 (R12) BEQ R2, target2

L1 miss

L1 hit Pop

L1 hit

RETN1

L1 hit Push

CALL2 Second branch First branch resolves first resolves first History=…10 History=…01

Write over entry X

RETN2

Fig. 14. Example scenarios that may lead to a CM event: A. Two branches that resolve in an opposite order when reexecuted due to a data cache miss, and B. a return address stack entry which was used and run over during the first run

Fixed CM Branches

100% 90% 80% 70% 60% 50%

INT

40%

FP

30% 0

64

128

192

256

RbckBr Structure Depth

Fig. 15. Percentage of CM branches fixed by the RbckBr structure as a function of its depth

91.9% of the CM branches, that otherwise would be mispredicted, in the integer and floating point benchmarks respectively. The speedup potential of the RbckBr method, like the RbckReuse before it, depends on the fraction of reexecuted instructions. Repeating the experiments that measure the sensitivity of RbckBr to the size of the checkpoint buffer, branch predictor, and confidence estimator yields results that closely follow the tendency of the RbckReuse method described in Section 7.4.

9

Combining Methods

So far we have presented four reuse methods and estimated the performance and cost of each of them individually. These methods are to some extent complementary and we evaluate the contribution of a combination of them. Combining all four reuse methods sums up to 0.87[mm2 ] and 51.6[mW], however since RbckReuse method has a substantially higher cost than the other three methods, we also explore an alternative lower-cost combination that we call Resource-Efficient

Reexecution and Selective Reuse in Checkpoint Processors

1.05 IPC Speedup

Three combinations INT FP Power

60 50

1.04

40

1.03

30

1.02

20

1.01

10

1.00

Power Consumption [mW]

Four reuse methods

1.06

261

0 Trivial

SelReuse

RbckBr, RbckReuse, RbckBr RbckReuse (Includes SelReuse, SelReuse, SelReuse, Trivial Trivial Trivial RbckBr)

(RESR)

(All four)

Fig. 16. Speedup and power consumption of the four reuse methods and three combinations of them

Selective Reuse (RESR). RESR processes normally most reexecuted instructions and only attempts to reuse results from “expensive” instructions, such as flow control instructions which can lead to mispredictions, and long latency arithmetic operations. RESR consisists of the Trivial, SelReuse, and RbckBr reuse methods. Its total cost sums up to 0.051[mm2 ] and 19.80[mW]. In the rest of this section we analyze the performance, power-efficiency, and die size of these methods. 9.1

Performance Analysis

Figure 16 illustrates microarchitectures implementing the four basic methods and three combinations, and compares the power consumption and IPC speedup for the integer and floating point benchmarks. Analyzing these results, the integer benchmarks have a large fraction of their code reexecuted (as shown in Figure 1), and they mainly benefit from the RbckReuse and RbckBr methods. Implementing a RbckReuse structure is a tradeoff decision, RbckBr achieves half of the speedup of RbckReuse for 1.77% and 0.82% of its power and area budget respectively. The floating point benchmarks on the other hand do not mispredict as often, but have many long latency arithmetic operations. Floating point benchmarks mainly benefit from the Trivial and SelReuse methods. Figure 17 presents the baseline and RESR microarchitectures sensitivity to the execution unit latencies. The graph displays additional points for pipelines 33% shorter and 33% longer than the baseline parameters in Table 1 and reveals the extent of the increasing advantage of RESR, for longer pipelines, over the baseline model.

262

A. Golander and S. Weiss

1.08

IPC Speedup

1.06

2.3%

1.04

5.7%

1.02 1.00

9%

0.98

Baseline

0.96

RESR

0.94 33% shorter

Baseline Parameters

33% longer

Latency of the Long Latency Execution Units

Fig. 17. Baseline and RESR IPC speedup as a function of the long latency execution unit latencies. Results are shown for the floating-point benchmarks and are normalized to the baseline microarchitecture with latencies as described in Table 1.

Throughout this paper we have chosen resource-efficient structure sizes (128entry RbckBr and RbckReuse structures, and 4-entry SelReuse caches). Table 5 compares the performance of processors using resource-efficient and maximal size reuse structures, defining maximal as 256-entry (M AXDist ) RbckBr and RbckReuse structures, and 4K-entry SelReuse caches (larger caches do not improve the hit ratio). The high efficiency percentages (all above 95%) justify the use of small reuse structures. 9.2

Energy-Delay Product

Until now we have considered performance and power consumption separately. In microprocessors however these two metrics are strongly correlated. It is easy to lower power consumption by lowering performance. In this section we consider the energy-delay product, a widely used metric that has been shown [15] to remain relatively stable over a wide range of microprocessors designed to operate at different points in the power-performance spectrum. For a fixed amount of work (number of instructions) the energy-delay product is equivalent to the processor’s total power divided by the rate-squared. To visually conform to IPC measurements (“the higher the better”), we invert the energy-delay metric and IP C 2 calculate T otal power . Table 5. The effectiveness of the reuse methods SelReuse +Trivial INT FP Resource-efficient size 1.0046 1.0524 Maximal size 1.0048 1.0526 Effectiveness 95.8% 99.6%

RESR INT 1.0148 1.0150 98.7%

All four FP 1.0566 1.0573 98.8%

INT 1.0248 1.0254 97.6%

FP 1.0587 1.0595 98.7%

Energy-Delay Improvement

Reexecution and Selective Reuse in Checkpoint Processors Four reuse methods

1.12 1.11 1.10 1.09 1.08 1.07 1.06 1.05 1.04 1.03 1.02 1.01 1.00

263

Three combinations INT FP

Trivial

SelReuse

RbckBr

RbckBr, RbckReuse, RbckReuse (Includes SelReuse, SelReuse, SelReuse, Trivial Trivial Trivial RbckBr) (RESR) (All four)

Fig. 18. Energy-delay product improvement of the four reuse methods and three combinations of them

The results for the various methods are shown in Figure 18. We assume the baseline processor uses a 70[nm] technology and consumes 25 watt when operating at 100 degrees. This is a conservative figure, for example the Intel core 2 Duo, operating at the same clock frequency and manufactured in a 65[nm] technology, is reported to consume 65 watt. As illustrated in Figure 18, combining all four reuse methods results in the best power-efficiency improvement, achieving an improvement of 4.80% and 11.85% for the integer and floating point benchmarks respectively. Figure 18 resembles Figure 16 in its general shape and the relative performance of the various methods, with one major difference: the energy-delay improvement relative to the baseline is substantially higher than the IPC improvement. This underscores the fact that the IPC speedup is achieved by adding minimal hardware structures with limited power consumption. 9.3

Die Size Analysis

Area budget is another important processor constraint. A dual core Power5 die size is 389[mm2 ] and 243[mm2 ] when manufactured using 130[nm] and 90[nm] technologies respectively. To estimate the area of a single core in a 70[nm] tech702 nology we multiply a single core area ( 243 2 ) by the perfect scaling ratio ( 902 ) and further multiply it by the ratio of practical to perfect scaling taken from an 2 actual technology scaling of the Power5 processor ( 243×130 389×902 ). This results in a die size of 86[mm2 ]. Enhancing an 86[mm2 ] processor with all four reuse methods increases its area by 1.01%. On the other hand, enhancing it with RESR hardly has an impact on the area, increasing it by only 0.06%. We conclude that combining all four

264

A. Golander and S. Weiss

reuse methods is better for all processor cores, except the low-end cores, in which RESR is a better alternative.

10

Related Work

Research on reusing results from past execution can be roughly divided into two categories. The first category includes methods that accelerate arithmetic operations that have identical opcode and operand values. The second category includes methods reusing the result of the same instruction instance if its execution repeats. Within the latter class the case that received most attention is reusing wrong-path instructions. 10.1

Reusing the Results of Arithmetic Operations

Richardson [28] observed that many multiplication and division operations were trivial. Yi and Lilja [39] extended the concept of accelerating trivial computations to other instruction types. Richardson [28] was also the first to observe that many arithmetic operations are repetitive and suggested reusing their results. Similar reuse structures were further investigated by Oberman and Flynn [26] and Molina et al. [22]. Other relevant methods include using software profiling to build static reuse tables [40] and partial reuse in similar rather than identical division operations [6]. The last two methods are less effective in the context of misprediction recovery because rollbacks dynamically increase the repetitive nature of identical computations. The closest work to the SelReuse method reported here is the work of Citron and Feitelson [9]. They suggest having separate reuse structures for different calculation types and reduce the cost by filtering out trivial computation. We accelerate only long latency computations using small reuse caches, taking advantage of the repetitive nature of the execution that follows a rollback event. 10.2

Reusing the Result of the Same Instruction Instance

Sodani and Sohi [34] were the first to suggest saving instruction results in a wrong-path reuse buffer (referred to as the WrPathReuse method in Table 2). If following a misprediction the same instruction instance is encountered, the stored result is passed to the reorder buffer. Alternatively, Roth and Sohi [30] suggested using the register file to implement the wrong-path reuse method. Rotenberg et al. [29] and Collins et al. [10] studied how to identify control independent instructions. Control independent detection was used for fetching independent instructions while a hard-to-predict branch is being resolved in Cher and Vijaykumar [7], for improving memory prefetch in Pajuelo et al. [27], and for constructing more efficient wrong-path reuse mechanisms in Chou et al. [8]. Gandhi et al. [13] also proposed a wrong-path reuse mechanism, but focused on a subset of control independence (exact convergence) in which the correct path starts at the first convergence point and no correct path instructions are skipped.

Reexecution and Selective Reuse in Checkpoint Processors

265

Akkary et al. [2] suggested a different reuse method (referred to as the WrPathBr method in Table 2). They use the outcome of branches that reside on the wrong path to speculatively replace the direction recommendations made by the branch predictor. The RbckBr method presented in this paper also reuses the outcome of branches, but it does so for branch instructions that reside on the reexecuted path rather than on the wrong path. Because reexecution is easy to track, the extra branch recycling predictor proposed in [2] is not needed. Furthermore, unlike the WrPathBr method, the RbckBr achieves near perfect accuracy when used. Mutlu et al. [25] studied reuse in the Runahead microarchitecture framework. A Runahead processor takes a checkpoint upon switching from regular to prefetch mode, an event that is triggered by a L2 cache miss. In prefetch mode the processor skips over data-dependent instructions and flushes all results upon returning to the regular mode (a rollback). The authors evaluated a scheme that saves results calculated during prefetch mode and reuses them in normal mode, and concluded that this reuse scheme is not cost-effective. We believe the reason is related to the prefetch characteristics. L2 cache misses are fairly long and processing long code segments in prefetch mode increases the reuse structure cost. Of course the probability of staying on the correct path decreases with the path length. On the other hand, speculative code segments that are reexecuted, which is the topic we study here, are relatively short. Moreover, when compared to RbckReuse, the percentage of valid results for reuse is lower because in prefetch mode data dependent instructions are not executed. Finally, in the Runahead microarchitecture the frequency of reuse events is the frequency of L2 misses, which for most benchmarks are not as common as branch mispredictions.

11

Conclusions

We have studied reuse in a checkpoint microarchitecture, conceived for highlyefficient speculative processing. Reexecution increases the repetitive nature of the program in a predictable manner. Unlike previously studied wrong-path reuse, reexecution characteristics enable simple and efficient reuse structures. We have presented two such methods, RbckReuse that undiscriminatingly reuses results from all instruction types and RbckBr that only reuses the outcome of branches. We have explored two additional methods, SelReuse and Trivial, for accelerating long latency arithmetic operations. SelReuse fits well in the framework of rollback, as it is based on repetitiveness, which is high in folded code segments. Accelerating trivial computations further helps to settle for a small, efficient SelReuse structure. These four methods, RbckReuse, RbckBr, SelReuse, and Trivial can be implemented and combined in several ways. We recommend two configurations, each representing a different tradeoff of speedup and cost. The first combination is constructed from all four methods. This method achieves a mean IPC speedup of 2.5% and 5.9%, and an improvement in the energy-delay product of 4.80% and 11.85% for the integer and floating point benchmarks respectively, at a cost

266

A. Golander and S. Weiss

of 0.87[mm2 ] and 51.6[mW]. The second combination, RESR, is based on results that indicate that some instruction types contribute to speedup more than others. RESR incorporates only selective reuse methods, handling long latency computations and flow control instructions to achieve near perfect branch prediction during reexecution. RESR achieves nearly 60% and 96% of the speedup for the integer and floating point benchmarks respectively, for 38.4% of the power consumption and 5.9% of the area.

References 1. Akkary, H., Rajwar, R., Srinivasan, S.T.: An analysis of a resource efficient checkpoint architecture. ACM Transactions on Architecture and Code Optimization 1(4), 418–444 (2004) 2. Akkary, H., Srinivasan, S.T., Lai, K.: Recycling waste: Exploiting wrong-path execution to improve branch prediction. In: Proc. of the 17th annual Int’l. Conf. on Supercomputing, pp. 12–21 (June 2003) 3. Akl, P., Moshovos, A.I.: Branchtap: improving performance with very few checkpoints through adaptive speculation control. In: Proc. of the 20th annual Int’l. Conf. on Supercomputing, pp. 36–45 (June 2006) 4. Austin, T.M., Sohi, G.S.: Zero-cycle loads: microarchitecture support for reducing load latency. In: Proc. of the 28th annual Int’l. Symp. on Microarchitecture, pp. 82–92 (November 1995) 5. Bannon, P., Keller, J.: Internal architecture of Alpha 21164 microprocessor. In: COMPCON 1995: Proceedings of the 40th IEEE Computer Society International Conference, pp. 79–87 (1995) 6. Benowitz, E., Ercegovac, M., Fallah, F.: Reducing the latency of division operations with partial caching. In: Proc. of the 36th Asilomar Conf. on Signals, Systems and Computers, pp. 1598–1602 (November 2002) 7. Cher, C.Y., Vijaykumar, T.N.: Skipper: a microarchitecture for exploiting controlflow independence. In: Proc. of the 34th annual Int’l. Symp. on Microarchitecture, pp. 4–15 (December 2001) 8. Chou, Y.C., Fung, J., Shen, J.P.: Reducing branch misprediction penalties via dynamic control independence detection. In: Proc. of the 13th annual Int’l. Conf. on Supercomputing, pp. 109–118 (June 1999) 9. Citron, D., Feitelson, D.G.: Look it up or Do the math: An energy, area, and timing analysis of instruction reuse and memoization. In: Third Int’l. Workshop on Power - Aware Computer Systems, pp. 101–116 (December 2003) 10. Collins, J.D., Tullsen, D.M., Wang, H.: Control flow optimization via dynamic reconvergence prediction. In: Proc. of the 37th annual Int’l. Symp. on Microarchitecture, pp. 129–140 (December 2004) 11. Cristal, A., Santana, O.J., Valero, M., Martinez, J.F.: Toward kilo-instruction processors. ACM Transactions on Architecture and Code Optimization 1(4), 389–417 (2004) 12. Gandhi, A., Akkary, H., Rajwar, R., Srinivasan, S.T., Lai, K.: Scalable load and store processing in latency tolerant processors. In: Proc. of the 32nd annual Int’l. Symp. on Computer Architecture, pp. 446–457 (June 2005) 13. Gandhi, A., Akkary, H., Srinivasan, S.T.: Reducing branch misprediction penalty via selective branch recovery. In: Proc. of the 10th IEEE Int’l. Symp. on HighPerformance Computer Architecture, pp. 254–264 (February 2004)

Reexecution and Selective Reuse in Checkpoint Processors

267

14. Golander, A., Weiss, S.: Hiding the misprediction penalty of a resource-efficient high-performance processor. ACM Transactions on Architecture and Code Optimization (accepted) (to appear) 15. Gonzalez, R., Horowitz, M.: Energy dissipation in general purpose microprocessors. IEEE Journal of Solid State Circuits 31(9), 1277–1284 (1996) 16. Gwennap, L.: Intel’s P6 uses decoupled superscalar design. Microprocessor Report 9(2) (1995) 17. Jacobsen, E., Rotenberg, E., Smith, J.E.: Assigning confidence to conditional branch predictions. In: Proc. of the 29th annual Int’l. Symp. on Microarchitecture, pp. 142–152 (December 1996) 18. Kalla, R., Sinharoy, B., Tendler, J.M.: IBM POWER5 chip: A dual-core multithreaded processor. IEEE Micro. 24(2), 40–47 (2004) 19. Kessler, R.E.: The Alpha 21264 microprocessor. IEEE micro. 19(2), 24–36 (1999) 20. Levitan, D., Thomas, T., Tu, P.: The PowerPC 620 microprocessor: a high performance superscalar RISC microprocessor. In: COMPCON 1995: Proceedings of the 40th IEEE Computer Society International Conference, p. 285 (1995) 21. Lipasti, M.H., Shen, J.P.: Exceeding the dataflow limit via value prediction. In: Proc. of the 29th annual Int’l. Symp. on Microarchitecture, pp. 226–237 (December 1996) 22. Molina, C., Gonzalez, A., Tubella, J.: Dynamic removal of redundant computations. In: Proc. of the 13th annual Int’l. Conf. on Supercomputing, pp. 474–481 (June 1999) 23. Moshovos, A.I., Breach, S.E., Vijaykumar, T.N., Sohi, G.S.: Dynamic speculation and synchronization of data dependences. In: Proc. of the 24th annual Int’l. Symp. on Computer Architecture, pp. 181–193 (June 1997) 24. Moshovos, A.I., Sohi, G.S.: Read-after-read memory dependence prediction. In: Proc. of the 32nd annual Int’l. Symp. on Microarchitecture, pp. 177–185 (November 1999) 25. Mutlu, O., Kim, H., Stark, J., Patt, Y.N.: On reusing the results of pre-executed instructions in a runahead execution processor. IEEE Computer Architecture Letters 4 (2005) 26. Oberman, S.F., Flynn, M.J.: Reducing division latency with reciprocal caches. Reliable Computing 2(2), 147–153 (1996) 27. Pajuelo, A., Gonzalez, A., Valero, M.: Control-flow independence reuse via dynamic vectorization. In: 19th IEEE Int’l. Parallel and Distributed Processing Symp., p. 21a (April 2005) 28. Richardson, S.E.: Exploiting trivial and redundant computation. In: Proc. of the 11th Symp. on Computer Arithmetic, pp. 220–227 (June 1993) 29. Rotenberg, E., Jacobson, Q., Smith, J.: A study of control independence in superscalar processors. In: Proc. of the Fifth IEEE Int’l. Symp. on High-Performance Computer Architecture, pp. 115–124 (January 1999) 30. Roth, A., Sohi, G.S.: Squash reuse via a simplified implementation of register integration. Journal of Instruction-Level Parallelism 3 (October 2001) 31. Sarangi, S.R., Torrellas, J., Liu, W., Zhou, Y.: Reslice: Selective re-execution of long-retired misspeculated instructions using forward slicing. In: Proc. of the 38th annual Int’l. Symp. on Microarchitecture, pp. 257–270 (November 2005) 32. Seznec, A., Michaud, P.: A case for (partially) TAgged GEometric history length branch prediction. Journal of Instruction-Level Parallelism 8 (February 2006) 33. Smith, J.E., Pleszkun, A.R.: Implementing precise interrupts in pipelined processors. IEEE Transactions on Computers 37(5), 562–573 (1988)

268

A. Golander and S. Weiss

34. Sodani, A., Sohi, G.S.: Dynamic instruction reuse. In: Proc. of the 24th annual Int’l. Symp. on Computer Architecture, pp. 194–205 (June 1997) 35. Song, S.P., Denman, M., Chang, J.: The PowerPC 604 RISC microprocessor. IEEE Micro. 14(5), 8–17 (1994) 36. Suresh, B., Chaterjee, B., Harinath, R.: Synthesizable RAM-alternative to low configuration compiler memory for die area reduction. In: Proc. of the 13th Int’l. Conf. on VLSI Design, pp. 512–517 (2000) 37. Tarjan, D., Thoziyoor, S., Jouppi, N.P.: Cacti 4.0. Technical Report HPL-2006-86, HP Laboratories Palo Alto (June 2006) 38. Yeager, K.C.: The MIPS R10000 superscalar microprocessor. IEEE micro. 16(2), 28–40 (1996) 39. Yi, J.J., Lilja, D.J.: Improving processor performance by simplifying and bypassing trivial computations. In: Proc. of the 20th Int’l. Conf. on Computer Design, pp. 462–465 (October 2002) 40. Yi, J.J., Sendag, R., Lilja, D.J.: Increasing instruction-level parallelism with instruction precomputation. In: Monien, B., Feldmann, R.L. (eds.) Euro-Par 2002. LNCS, vol. 2400, pp. 481–485. Springer, Heidelberg (2002)

Compiler Support for Code Size Reduction Using a Queue-Based Processor Arquimedes Canedo, Ben Abderazek, and Masahiro Sowa Graduate School of Information Systems, University of Electro-Communications, Chofugaoka 1-5-1 Chofu-Shi 182-8585, Japan

Abstract. Queue computing delivers an attractive alternative for embedded systems. The main features of a queue-based processor are a dense instruction set, high-parallelism capabilities, and low hardware complexity. This paper presents the design of a code generation algorithm implemented in the queue compiler infrastructure to achieve high code density by using a queue-based instruction set processor. We present the efficiency of our code generation technique by comparing the code size and extracted parallelism for a set of embedded applications against a set of conventional embedded processors. The compiled code is, in average, 12.03% more compact than MIPS16 code, and 45.1% more compact than ARM/Thumb code. In addition, we show that the queue compiler, without optimizations, can deliver about 1.16 times more parallelism than fully optimized code for a register machine. Keywords: Code Generation, Code Size Reduction, Reduced bit-width Instruction Set, Queue Computation Model.

1

Introduction

One of the major concerns in the design of an embedded processor is code density. Code size is directly related to the size of the memory and therefore the cost of the system [1,2]. Compact instructions improve memory bandwidth, fetching time, and power consumption [3]. 0-operand instruction set computers are the best alternative for achieving high code density. Stack machines have demonstrated their capacity to generate compact programs [4] but their sequential nature [5,6] excludes them from the high-performance arena. Queue machines use a FIFO data structure to perform computations [7,8,9]. Opposite to the stack machines, the queue machines offer a natural parallel computation model as two different locations are used for reading and writing, thus avoiding a bottleneck. All read/write accesses to the FIFO queue are done at the head and rear of the queue respectively. Thus, instructions implicitly reference their operands. This characteristic allows the instruction set to be short and free of false dependencies freeing the hardware from the complexity of register renaming mechanism. Compiling for queue machines presents serious challenges as the problem of laying out a directed acyclic graph in a queue is known to be NP-complete [10]. Additional hardware support alleviates this P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 269–285, 2009. c Springer-Verlag Berlin Heidelberg 2009 

270

A. Canedo, B. Abderazek, and M. Sowa

task to a certain extent [21]. In [7,8,9], we proposed a Parallel Queue Processor (PQP), an architecture based on the producer order queue computation model. The PQP allows the execution of any data flow graph by allowing instructions to read their operands from a location different than the head of the queue, specified as an explicit offset reference in the instruction. Although the place to read operands can be specified, the place to write operands remains fixed at the rear of the queue. As the rule of writing data (producing data) remains fixed, this model is known as the producer order model. However, in real applications, instructions that require both of their operands to be read from a place different than the head of the queue are scarce [11]. Therefore, we present the possibility of reducing the instruction set to encode at most one operand offset reference placing additional burden on the compiler. This modification has the purpose of reducing the bits in the instruction and thus improving the code size of compiled programs. The PQP is a 32-bit processor with 16-bit instructions capable of executing up to four instructions in parallel. Instructions allow one of its operands to be read from anywhere in the queue by adding the offset reference provided in the instruction to the current position of the head of the queue. A special unit called Queue Computation Unit is in charge of finding the physical location of source operands and destination within the queue register file enabling parallel execution. For cases when there are insufficient bits to express large constants and memory offsets, a covop instruction is inserted. This special instruction extends the operand field of the following instruction by concatenating it to its operand. We have developed the queue compiler infrastructure for the producer order model [11] as part of the design space exploration chain for the PQP processor that includes a functional simulator, cycle accurate simulator, and CPU described in RTL level. In this compiler, the code generation algorithm produces code for an abstract PQP architecture that is insufficient for our present goals. Another serious limitation of the queue compiler is its inability to compile programs with system header files restricting its input only to simple programs. In this paper, we propose a code size-aware optimizing compiler infrastructure that efficiently generates compact code using a queue-based instruction set. The compiler deals with the potential increase of instructions by inserting a special queue instruction that creates a duplicate of a datum in the queue. The presented code generation algorithm selectively inserts these special instructions to constrain all instructions in the program to have at most one explicit operand. We analyze the compiling results for a set of embedded applications to show the potential of our technique highlighting the code size and parallelism at the instruction level. In summary, the contributions of this paper are: – To demonstrate that a compiler for the PQP processor is able to produce compact and efficient code. – The methods and algorithms to build the code size-aware compiling infrastructure for a producer order queue based processor. The remainder of the paper is as follows: Section 2 gives a summary of the related work. In Section 3 we give an overview of the queue computation model, the problems related to compilation of programs, and we motivate our target of using

Compiler Support for Code Size Reduction

271

the queue computation model to reduce code size in embedded applications. In Section 4 we describe the code generation algorithm developed for restricting directed acyclic graphs into a suitable form for executing in a constrained queue instruction set. Section 5 reports our experimental results. Section 6 we open discussion, and Section 7 concludes.

2

Related Work

Improving code density in CISC and RISC architectures has been a thoroughly studied problem. A popular architecture enhancement for RISC is to have two different instruction sets [12,13] in the same processor. These dual instruction set architectures provide a 32-bit instruction set, and a reduced instruction set of 16-bit. The premise is to provide a reduced 16-bit instruction set for the operations most frequently executed in applications so two instructions are fetched instead of one. The improvement in code density comes with a performance degradation since more 16-bit instructions are required to execute the same task when compared to 32-bit instructions. The ARM/Thumb [12] and MIPS16 [14] are examples of dual instruction sets. In [12], a 30% of code size reduction with a 15% of performance degradation is reported for the ARM/Thumb processor. Compiler support for dual instruction set architectures is crucial to maintain a balance between code size reduction and performance degradation. Different approaches have been proposed to cope with this problem [15,16,17,18,19,20]. Queue Computation Model (QCM) refers to the evaluation of expressions using a first-in first-out queue, called operand queue. This model establishes two rules for the insertion and removal of elements from the operand queue. Operands are inserted, or enqueued, at the rear of the queue. And operands are removed, or dequeued, from the head of the queue. Two references are needed to track the location of the head and the rear of the queue. The Queue Head, or QH, points to the head of the queue. And Queue Tail, or QT, points to the rear of the queue. Only a handful of queue machine hardware designs have been proposed. In 1985, Bruno [21] established that a level-order scheduling of an expression’s parse tree generates the sequence of instructions for correct evaluation. A level-order scheduling of directed acyclic graphs (DAG) still delivers the correct sequence of instructions but requires additional hardware support. This hardware solution is called an Indexed Queue Machine. The basic idea is to specify, for each instruction, the location with respect of the head of the queue (an index) where the result of the instruction will be used. An instruction may include several indexes if it has multiple parent nodes. All these ideas were in the form of an abstract queue machine until the hardware mechanisms of a superscalar queue machine were proposed by Okamoto [22]. The operand queue has been used as a supporting hardware for two register based processors for the efficient execution of loops. The WM architecture [23] is a register machine that reserves one of its registers to access the queue and demonstrates high streaming processing capabilities. Although the compiler support for the WM Architecture has been reported in [23], details about code

272

A. Canedo, B. Abderazek, and M. Sowa

generation are not discussed. In [24,25] the use of Register Queues (RQs) is demonstrated to effectively reduce the register pressure on software pipelined loops. The compiler techniques described in this processor do not present a significant contribution for queue compilation since the RQs are treated as special registers. In our previous work [7,8], we have designed a parallel queue processor (PQP) capable of executing any data flow graph. The PQP breaks the rule of dequeueing by allowing operands to be read from a location different than the head of the queue. This location is specified as an offset in the instruction. The fundamental difference with the indexed queue machine is that our design specifies an offset reference in the instruction for reading operands instead of specifying an index to write operands. In the PQP’s instruction set, the writing location at the rear of the queue remains fixed for all instructions. Compiling for queue machines still is an undecided art. Only few efforts have been made to develop the code generation algorithms for the queue computation model. A linear time algorithm to recognize the covering of a DAG in one queue has been demonstrated in [10] together with the proof of NP-completeness for recognizing a 2-queue DAG. In [26], Schmit. et. al propose a heuristic algorithm to cover any DAG in one queue by adding special instructions to the data flow graph. From their experimental results a large amount of additional instructions is reported, making this technique insufficient for achieving small code size. Despite the large amount of extra instructions, the resulting size of their tested programs is smaller when compared to RISC code. In [27], we developed a queue compiler based on a conventional retargetable compiler for register machines. To provide a clean model of the queue, a very large amount of general purpose registers were defined in the machine definition files of the compiler to avoid the spillage of registers by the register allocator. Nevertheless, mapping register code into the queue computation model turned into low code quality with excess of instructions making this approach inappropriate for both, a native compiler for our queue machines, and the generation of compact code. In this article we present part of the initiative to deliver a compiler technology designed specifically for the queue computation model.

3

Queue Computing Overview

Correct queue programs are obtained from a level-order traversal of the parse trees of the expressions [21]. Figure 1(a) shows the parse tree for a simple expression and the obtained pseudo-program. The first four executed instructions (L3 ) place four operands in the queue and QH points at the first loaded operand and QT to an empty location after the last loaded operand. The contents of the queue are the following: {a, b, a, b, ǫ}. The next two binary instructions from level L2 consume the four operands leaving the queue in the following status: (a + b), (a − b), ǫ. The only instruction in level L1 consumes the previously computed subexpressions and generates a new intermediate value: {(a+b)/(a−b), ǫ}. The last instruction in level L0 consumes the only operand in the queue and

Compiler Support for Code Size Reduction

L0

L1

x

/

/

+

L2

L3

x

a

b ld ld ld ld add sub div st

a

+ b

a b a b

a

273

b

ld ld add sub div st

a b 0, 1 -2,-1 0, 1 x

x

a). Queue program obtained from expression's parse tree

b). Queue program obtained from expression's directed acyclic graph

Fig. 1. Evaluation of expressions in the queue computation model. (a) queue code from a parse tree. (b) hardware support for evaluation of any directed acyclic graph.

stores it into memory leaving an empty queue with QH and QT pointing at the same empty location: {ǫ}. Notice that in the previous example, the operands a, b are loaded twice in level L3 . Complications arise when the program is scheduled from its DAG. Figure 1(b) shows the DAG for the same expression. Notice that operands a, b are loaded once and are shared by two operations (+, −). The addition correctly obtains its operands from QH and writes back the result into QT. By the time the subtraction is executed, QH does not point to any of its operands. If the basic enqueueing/dequeueing is kept it leads to incorrect results. Our PQP solves this problem by allowing the operands to be dequeued from a location different than QH. The location from where to dequeue the operands is represented as an offset in the instruction as shown in Figure 1(b). The offset represent the relative distance with respect of QH from where the operand must be read. We classify all binary and unary instructions into three categories: 2-offset, 1-offset, and 0-offset instructions. 2-offset instructions read both operands from a place different than QH, e.g. “sub -2, -1”. 1-offset instructions read one operand from QH and the other from a different location (not shown in the Figure). And 0-offset instructions read both operands directly from QH, e.g. “add 0,1”. The PQP updates the QH position automatically every time an operand is read directly from QH [7]. For 0-offset binary instructions QH is moved two positions after its execution. For 1-offset instructions QH is moved only one position, and for 0-offset

274

A. Canedo, B. Abderazek, and M. Sowa

instructions QH is not updated. This mechanism guarantees correct evaluation of any data flow graph. To have an insight of the demands of applications on PQP instruction set we compiled several applications and obtained the distribution of offsetted instructions. Table 1 shows the distribution of offsetted PQP instructions for a set of embedded and numerical applications. Notice that the 2-offset instructions represent from 0.1% to 2.6%. 1-offset instructions represent from 2.9% to 18.2%. And 0-offset instructions represent the majority of instructions in the applications. Restricting PQP’s instructions to encode at most one offset makes instructions shorter and covers the great majority of instructions in programs. This has direct effect over the code size of the compiled programs since only a single operand is encoded in the instruction format. In the following section we discuss the compiler technique required to deal with the correct evaluation of 2-offset instructions in programs on a constrained 1-offset instruction set.

4

Code Generation Algorithm

Figure 2 shows a 4-point fourier transform DAG and its evaluation using the PQP instruction set [7]. 0-offset instructions are represented by the white nodes, 1-offset instructions by the gray nodes, and 2-offset instructions by the black nodes. It is responsibility of the compiler to transform the DAG into a suitable form to be executed by our reduced PQP instruction set. One approach to deal with this problem is to re-schedule the instructions to execute a subexpression while reading the other intermediate result with the available offset reference. While this approach generates correct programs, it has the overhead of extra instructions. In order to efficiently generate compact programs we propose the utilization of a special queue instruction called dup instruction. The purpose of this instruction is to create a copy, duplicate, a datum in the queue. The dup instruction has one operand which is an offset reference that indicates the location with respect of QH from where to copy the datum into QT. Figure 3 shows the transformed DAG with extra dup instructions. These two dup instructions Table 1. Distribution of PQP offsetted instructions for a set of embedded and numerical applications Application MPEG2 H263 Susan FFT8G Livermore Linpack Equake

0-offset 90.0% 86.7% 97.0% 93.9% 82.2% 80.8% 85.5%

1-offset 2-offset 9.3% 0.7% 11.4% 1.9% 2.9% 0.1% 5.9% 0.2% 15.6% 2.2% 18.2% 1.0% 11.9% 2.6%

Compiler Support for Code Size Reduction

L0

X0

X1

X2

X3

L1

+5

+6

+7

+8

L2

+1

+2

+3

+4

L3

x0

neg

x2

neg

L4

x1

x3

ld ld ld neg ld neg add add add add add add add add st st st st

275

x1 x3 x0 0 x2 0 0, 1 -2,-1 0, 1 -2,-1 0,+2 0,+2 -2, 0 -2, 0 X0 X1 X2 X3

Fig. 2. 4-point Fourier transform directed acyclic graph

place a copy of the left hand operand for nodes +2 , +3 transforming them into 1-offset instructions. 4.1

Queue Compiler Infrastructure

For a better comprehension of our algorithm we introduce the queue compiler infrastructure [11]. The queue compiler parses C files using GCC 4.0.2 frontend. Our back-end takes GCC’s GIMPLE trees [28,29] as input and generates assembly code for the PQP processor. The first task of the back-end is to expand three-address code GIMPLE representation into unrestricted trees of arbitrary depth and width called QTrees. QTrees help in the construction of our core data structures, the level DAGs (LDAG) [10]. A LDAG is a data structure that assigns all nodes in a DAG into levels and expresses well the interaction between operations and operands under the queue computation model. The code generation algorithm takes LDAGs to perform the offset calculation for every instruction, the most important feature of the queue compiler. After all offset references have been computed, the code generator schedules the program in level-order manner and crafts a linear low level intermediate representation called QIR. The QIR inherits the characteristics of the queue computation model making it a single operand intermediate representation. The operand is used by memory and control flow operations to represent memory locations, immediate values, target label, and function names. Offset references for binary and unary instructions are considered attributes of the instructions rather than operands. The last phase of the compiler takes QIR and generates the final assembly code for the PQP. Figure 4 shows the phases of the queue compiler.

276

A. Canedo, B. Abderazek, and M. Sowa

Fig. 3. Fourier transform’s directed acyclic graph with dup instructions

4.2

Code Generation Algorithm for Code Size Reduction

We implemented the algorithm into the queue compiler infrastructure. The main task of this algorithm is to determine the correct location of dup instructions in the programs’ data flow graph. The algorithm accomplishes its task in two stages during code generation. The first stage converts QTrees to LDAGs augmented with ghost nodes. A ghost node is a node without operation that serves as placeholder for dup instructions. This first stage gathers information about what instructions violate the 1-offset instruction restriction. The second stage decides which ghost nodes are turned into dup nodes or are eliminated from the flow graph. Finally, a level-order traversal of the augmented LDAGs computes the offset references for all instructions and generates QIR as output. Augmented LDAG Construction. Algorithm 1 presents the leveling function that transforms QTrees into ghost nodes augmented LDAGs. The algorithm makes a post-order depth-first recursive traversal over the QTree. All nodes are recorded in a lookup table when they first appear, and are created in the corresponding level of the LDAG together with its edge to the parent node. Two restrictions are imposed over the LDAGs for the 1-offset P-Code QCM. Definition 1. The sink of an edge must be always in a deeper or same level than its source. Definition 2. An edge to a ghost node spans only one level. When an operand is found in the lookup table the Definition 1 must be kept. Line 1 in Algorithm 1 is reached when the operand is found in the lookup table

Compiler Support for Code Size Reduction

277

C source file

Queue Compiler Back-End

GCC Front-End

GCC Middle-End

GIMPLE

1-offset Code Generation

QTrees

Offset Calculation

Leveled DAGs

Instruction Scheduling QIR Assembly Generation

QueueCore Assembly

Fig. 4. Queue Compiler Infrastructure

and it has a shallow level compared to the new level. The function dag ghost move node() moves the operand to the new level, updates the lookup table, converts the old node into a ghost node, and creates an edge from the ghost node to the new created node. The function insert ghost same level() in Line 1 is reached when the level of the operand in the lookup table is the same as the new level. This function creates a new ghost node in the new level, makes an edge from the parent node to the ghost node, and an edge from the ghost node to the element matched in the lookup table. These two functions build LDAGs augmented with ghost nodes that obey Definitions 1 and 2. Figure 5 illustrates the result of leveling the QTree for the expression x = (a ∗ a)/(−a + (b − a)). Figure 5.b shows the resulting LDAG augmented with ghost nodes. dup Instruction Assignment and Ghost Nodes Elimination. The second stage of the algorithm works in two passes as shown in Lines 2 and 2 in Algorithm 2. Function dup assignment() decides whether ghost nodes are substituted by dup nodes or eliminated from the LDAG. Once all the ghost nodes have been transformed or eliminated, the second pass performs a level order traversal of the LDAG, and for every instruction the offset references with respect of QH are computed in the same way as in [11]. The output of the code generation algorithm is QIR where all instructions use at most one offset reference.

278

A. Canedo, B. Abderazek, and M. Sowa

Algorithm 1. dag levelize ghost (tree t, level) 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32.

nextlevel ⇐ level + 1 match ⇐ lookup (t) if match  = null then if match.level < nextlevel then relink ⇐ dag ghost move node (nextlevel, t, match) return r elink else if match.level = lookup (t) then relink ⇐ insert ghost same level (nextlevel, match) return r elink else return m atch end if end if /* Insert the node to a new level or existing one */ if nextlevel > get Last Level() then new ⇐ make new level (t, nextlevel) record (new) else new ⇐ append to level (t, nextlevel) record (new) end if /* Post-Order Depth First Recursion */ if tis binary operation then lhs ⇐ dag levelize ghost (t.left, nextlevel) make edge (new, lhs) rhs ⇐ dag levelize ghost (t.right, nextlevel) make edge (new, rhs) else if tis unary operation then child ⇐ dag levelize ghost (t.child, nextlevel) make edge (new, child) end if return n ew

The only operations that need a dup instruction are those binary operations whose both operands are away from QH. The augmented LDAG with ghost nodes facilitate the task of identifying those instructions. All binary operations having ghost nodes as their left and right children need to be transformed as follows. The ghost node in the left children is transformed into a dup node, and the ghost node in the right children is eliminated from the LDAG. For those binary operations with only one ghost node as the left or right children, the ghost node is eliminated from the LDAG. Algorithm 3 describes the function dup assignment(). The effect of Algorithm 3 is illustrated in Figure 6. The algorithm takes as input the LDAG with ghost nodes shown in Figure 5.b and performs the steps described in Algorithm 3 to finally obtain the LDAG with dup instructions as shown in Figure 6.a. The last step in the code generation is to perform a level-order traversal of the LDAG with dup nodes and compute for every operation, the

Compiler Support for Code Size Reduction

/

a

Leveling Function

+ a

/

L1

*

-

neg

a

b

a

*

L2

L3

279

ghost

ghost

-

neg

a

L4

a). QTree

+

b

ghost

b). LDAG with ghost nodes

Fig. 5. Leveling of QTree into augmented LDAG for expression x=

a·a −a+(b−a)

Algorithm 2. codegen () 1. for all basic blocks BB do 2. for all expressions Wk in BB do 3. for all instructions Ij in TopBottom (Wk ) do 4. dup assignment (Ij ) 5. end for 6. for all instructions Ij in LevelOrder (Wk ) do 7. p qcm compute offsets (Wk , Ij ) 8. end for 9. end for 10. end for

Algorithm 3. dup assignment (i) 1. if isBinary (i) then 2. if isGhost (i.left) and isGhost (i.right) then 3. dup assign node (i.left) 4. dag remove node (i.right) 5. else if isGhost (i.left) then 6. dag remove node (i.left) 7. else if isGhost (i.right) then 8. dag remove node (i.right) 9. end if 10. return 11. end if

offset value with respect of QH. dup instructions are treated as unary instructions by the offset calculation algorithm. The final constrained 1-offset QIR for the expression x = (a ∗ a)/(−a + (b − a)) is given in Figure 6.b.

280

A. Canedo, B. Abderazek, and M. Sowa

/

L1

*

L2

L3

L4

dup

ld ld dup neg sub mul add div st

+

-

neg

a

b

a). LDAG with dup instructions

a b 0 0 -1 -2 0 0 x

b). 1-offset P-Code program with dup instruction

Fig. 6. 1-offset constrained code generation from a LDAG

5 5.1

Experimental Results Code Size Comparison

We selected ten applications commonly used in embedded systems from MiBench and MediaBench suites [30,31]. This selection includes video compression applications (H263, MPEG2), signal processing (FFT, Adpcm), image recognition (Susan), encryption (SHA, Blowfish, Rijndael), and graph processing (Dijkstra, Patricia). We compiled these applications using our queue compiler infrastructure with the presented code generation algorithm. The resulting code is 1-offset PQP assembly code where every instruction is 16-bit long. We compare our result with the code of two dual-instruction embedded RISC processors: MIPS16 [13], ARM/Thumb [12]. With two traditional RISC machines: MIPS I [14], ARM [32]. And with a traditional CISC architecture: Pentium processor [33]. We prepared GCC 4.0.2 compiler for the other five architectures and measured the code size from the text segment of the object files. All compilers, including our compiler, were configured without optimizations in order to compare the density of the baseline code. Figure 7 shows the normalized code size for all applications with respect of MIPS code. These results confirm the higher code density of the embedded RISC processors over their original 32-bit versions. Our PQP code is, in average, 12.03% denser than MIPS16 code, and 45.1% denser than ARM/Thumb code. Compared to a traditional variable length instruction set CISC machine, our PQP achieves 12.58% denser code. 5.2

Effect of dup Instructions on Code Size

Our algorithm inserts dup instructions in the program’s data flow graph to constrain all instructions to at most one offset reference. Table 2 shows the extra

Compiler Support for Code Size Reduction MIPS

MIPS16

ARM

ARM/Thumb

Pentium

281

PQP

110

Code Size

88

66

44

22

0 H.263

MPEG2

FFT

Susan

Rijndael

Sha

Blowfish Dijkstra Patricia

Adpcm

Fig. 7. Code size comparison

dup instructions inserted over the original 2-offset PQP code. The increase in number of instructions is below 1% for the chosen embedded applications. This confirms that for embedded applications the 2-offset instructions are rare and our technique can take advantage of this characteristic to improve code size by reducing the bits in the instruction to encode at most one offset reference. Table 2. Number of inserted dup instructions for the compiled embedded applications Application H263 MPEG2 FFT Susan Rijndael Sha Blowfish Dijkstra Patricia Adpcm

5.3

dup 2-offset PQP 751 39376 299 42016 18 9127 11 11177 12 821 5 711 16 5377 0 910 1 1260 1 1213

Instruction Level Parallelism Analysis

The queue compiler exposes natural parallelism found in the programs from the level-order scheduling. All instructions belonging to the same level in the LDAG are independent from each other and can be executed in parallel by the underlying queue machine. We compare the parallelism extracted by our compiler against the parallelism extracted by the MIPS I compiler. Our compiler was set with all optimizations turned off, and the MIPS-GCC compiler was configured

282

A. Canedo, B. Abderazek, and M. Sowa

Fig. 8. Compile-time extracted instruction level parallelism

with maximum optimization level (-O3). The extracted parallelism for the MIPS architecture was measured from the assembly output code using a conservative analysis by instruction inspection [34] to detect the data dependencies between registers and grouping those instructions that can be executed in parallel. For the PQP code, data dependences are given by the levels in the LDAG and are not expressed in the instructions. The only information available in the instructions is their offset and it cannot be used to determine dependences as it is relative to QH. To measure the parallelism of our code the compiler emits marks at the beginning of every level in the LDAGs grouping all parallel instructions. Figure 8 shows the extracted parallelism by the two compilers. Our compiler extracts about 1.16 times more parallelism than fully optimized RISC code.

6

Discussion

The above results show that embedded applications require a small amount of 2offset instructions. This motivates the idea of shortening the PQP instruction set to support at most one offset reference. The compiler is responsible for preserving the data flow graph to fit the program into the constrained instruction set by the addition of a single dup instruction. We believe that there is a chance to reduce even more the code size of PQP programs by using a variable length instruction set. Instructions that read their operands directly from QH can be reduced to 0-operand instructions without wasting the field to encode their dead offset. Another possibility is to extend the presented algorithm to constrain all instructions to 0-offset format with the penalty of a larger increase of extra dup instructions. With our modification and 16-bit instructions configuration, our compiler generates denser code than embedded RISC processors, and a CISC processor.

Compiler Support for Code Size Reduction

283

Also we demonstrated that the level-order scheduling naturally exposes more parallelism than fully optimized RISC code. We believe that the addition of classical and ILP optimizations, our compiler can generate higher quality code.

7

Conclusion

In this paper we presented an improved queue compiler infrastructure to reduce code size by using a reduced queue-based instruction set of the PQP processor. The algorithm handles the correct transformation of the data flow graph to evaluate the programs using a reduced queue instruction set. The algorithm was successfully implemented in the queue compiler infrastructure. We have presented the potential of our technique by compiling a set of embedded applications and measuring the code size against a variety of embedded RISC processors, and a CISC processor. The compiled code is about 12.03% and 45.1% denser than MIPS16 and ARM/Thumb architectures. The efficiency of our code generation technique is not only limited to code size but also the generation of parallel code. Without any optimization, our compiler achieves, in average, 1.16 times more parallelism than fully optimized code for a RISC machine. Queue architectures are a viable alternative for executing applications that require small code size footprint and high performance.

References 1. Liao, S.Y., Devadas, S., Keutzer, K.: Code density optimization for embedded DSP processors using data compression techniques. In: Proceedings of the 16th Conference on Advanced Research in VLSI (ARVLSI 1995), p. 272 (1995) 2. Wolfe, A., Chanin, A.: Executing compressed programs on an embedded RISC architecture. In: Proceedings of the 25th annual international symposium on Microarchitecture, pp. 81–91 (1992) 3. Gordon-Ross, A., Cotterell, S., Vahid, F.: Tiny instruction caches for low power embedded systems. ACM Transactions on Embedded Computing Systems (TECS) 2(4), 449–481 (2003) 4. Koopman, P.J.: Stack Computers: the new wave. Ellis Horwood (1989) 5. Vijaykrishnan, N.: Issues in the Design of a Java Processor Architecture. PhD thesis, University of South Florida (1998) 6. Shi, H., Bailey, C.: Investigating Available Instruction Level Parallelism for Stack Based Machine Architectures. In: Proceedings of the Digital System Design, EUROMICRO Systems on (DSD 2004), pp. 112–120 (2004) 7. Sowa, M., Abderazek, B., Yoshinaga, T.: Parallel Queue Processor Architecture Based on Produced Order Computation Model. Journal of Supercomputing 32(3), 217–229 (2005) 8. Abderazek, B., Yoshinaga, T., Sowa, M.: High-Level Modeling and FPGA Prototyping of Produced Order Parallel Queue Processor Core. Journal of Supercomputing 38(1), 3–15 (2006) 9. Abderazek, B., Kawata, S., Sowa, M.: Design and Architecture for an Embedded 32-bit QueueCore. Journal of Embedded Computing 2(2), 191–205 (2006)

284

A. Canedo, B. Abderazek, and M. Sowa

10. Heath, L.S., Pemmaraju, S.V.: Stack and Queue Layouts of Directed Acyclic Graphs: Part I. SIAM Journal on Computing 28(4), 1510–1539 (1999) 11. Canedo, A.: Code Generation Algorithms for Consumed and Produced Order Queue Machines. Master’s thesis, University of Electro-Communications, Tokyo, Japan (September 2006) 12. Goudge, L., Segars, S.: Thumb: Reducing the Cost of 32-bit RISC Performance in Portable and Consumer Applications. In: Proceedings of COMPCON 1996, pp. 176–181 (1996) 13. Kissel, K.: MIPS16: High-density MIPS for the embedded market. Technical report, Silicon Graphics MIPS Group (1997) 14. Kane, G., Heinrich, J.: MIPS RISC Architecture. Prentice-Hall, Englewood Cliffs (1992) 15. Krishnaswamy, A., Gupta, R.: Profile Guided Selection of ARM and Thumb Instructions. In: ACM SIGPLAN conference on Languages, Compilers, and Tools for Embedded Systems, pp. 56–64 (2002) 16. Halambi, A., Shrivastava, A., Biswas, P., Dutt, N., Nicolau, A.: An Efficient Compiler Technique for Code Size Reduction using Reduced Bit-width ISAs. In: Proceedings of the Conference on Design, Automation and Test in Europe, p. 402 (2002) 17. Sheayun, L., Jaejin, L., Min, S.: Code Generation for a Dual Instruction Processor Based on Selective Code Transformation. LNCS, pp. 33–48. Springer, Heidelberg (2003) 18. Kwon, Y., Ma, X., Lee, H.J.: Pare: instruction set architecture for efficient code size reduction. Electronics Letters, 2098–2099 (1999) 19. Krishnaswamy, A., Gupta, R.: Enhancing the Performance of 16-bit Code Using Augmenting Instructions. In: Proceedings of the 2003 SIGPLAN Conference on Language, Compiler, and Tools for Embedded Systems, pp. 254–264 (2003) 20. Krishnaswamy, A.: Microarchitecture and Compiler Techniques for Dual Width ISA Processors. PhD thesis, University of Arizona (September 2006) 21. Preiss, B., Hamacher, C.: Data Flow on Queue Machines. In: 12th Int. IEEE Symposium on computer Architecture, pp. 342–351 (1985) 22. Okamoto, S.: Design of a Superscalar Processor Based on Queue Machine Computation Model. In: IEEE Pacific Rim Conference on Communications, Computers and Signal Processing, pp. 151–154 (1999) 23. Wulf, W.: Evaluation of the WM Architecture. In: Proceedings of the 19th annual international symposium on Computer architecture, pp. 382–390 (1992) 24. Smelyanskiy, M.G., Tyson, S., Davidson, E.S.: Register queues: a new hardware/software approach to efficientsoftware pipelining. In: Proceedings of Parallel Architectures and Compilation Techniques, pp. 3–12 (2000) 25. Fernandes, M.: Using Queues for Register File Organization in VLIW Architectures. Technical Report ECS-CSG-29-97, University of Edinburgh (1997) 26. Schmit, H., Levine, B., Ylvisaker, B.: Queue Machines: Hardware Computation in Hardware. In: 10th Annual IEEE Symposium on Field-Programmable Custom Computing Machines, p. 152 (2002) 27. Canedo, A., Abderazek, B., Sowa, M.: A GCC-based Compiler for the Queue Register Processor. In: Proceedings of International Workshop on Modern Science and Technology, pp. 250–255 (May 2006) 28. Merrill, J.: GENERIC and GIMPLE: A New Tree Representation for Entire Functions. In: Proceedings of GCC Developers Summit, pp. 171–180 (2003) 29. Novillo, D.: Design and Implementation of Tree SSA. In: Proceedings of GCC Developers Summit, pp. 119–130 (2004)

Compiler Support for Code Size Reduction

285

30. Guthaus, M.R., Ringenberg, J.S., Ernst, D., Austin, T.M., Mudge, T., Brown, R.B.: MiBench: A free, commercially representative embedded benchmark suite. In: IEEE 4th Annual Workshop on Workload Characterization, pp. 3–14 (2001) 31. Lee, C., Potkonjak, M., Mangione-Smith, W.: MediaBench: a tool for evaluating and synthesizing multimedia and communications systems. In: 30th Annual International Symposium on Microarchitecture (Micro 1997), p. 330 (1997) 32. Patankar, V., Jain, A., Bryant, R.: Formal verification of an ARM processor. In: Twelfth International Conference On VLSI Design, pp. 282–287 (1999) 33. Alpert, D., Avnon, D.: Architecture of the Pentium microprocessor. Micro. 13(3), 11–21 (1993) 34. Debray, S., Muth, R., Weippert, M.: Alias Analysis of Executable Code. In: Proceedings of the 25th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pp. 12–24 (1998)

Power-Aware Bus Coscheduling for Periodic Realtime Applications Running on Multiprocessor SoC Khaled Z. Ibrahim1 and Smail Niar2 1

2

Suez Canal University, 42563 Port Said, Egypt University of Valenciennes, 59313 Valenciennes Cedex 9, France

Abstract. Execution time for realtime processes running on multiprocessor system-on-chip platform varies due to the contention on the bus. Considering the worst case execution cycles necessitates over-clocking the system to meet the realtime deadlines, which has a negative impact on the system power requirements. For periodic applications coscheduled on multiprocessor with shared bus, the cycles needed by a memory transaction fluctuate based on the execution overlap between processes’ activities on bus. In this work, we show the effect on execution cycles of different scheduling overlap of processes. Experiments’ results demonstrate that the execution cycles, and therefore the clock frequency, can be lowered by up to 24% on a 4 processor MPSoC. As the power consumption varies cubically with frequency, this reduction can lead to a significant power saving. Instead of exhaustively simulating all configurations to search for optimal scheduling overlap, we devise a scheme to predict the effect of scheduling. We propose the use of shift-variance of bus traffic profile of applications running individually on the system to predict the effect when scheduling these applications simultaneously. We show that the devised predictor of scheduling effect highly correlates to the behavior observed through simulations.

1 Introduction In bus-based multiprocessor system, running multiple contending processes on the shared bus increases the completion time of bus transactions and consequently the number of cycles needed to finish these processes. In realtime system, worst case execution time (WCET) is usually considered while scheduling these processes. The system clock frequency is adjusted to meet the process realtime constraints. Unfortunately, increasing the frequency (and possibly increasing the voltage, as well) to meet realtime deadlines negatively impacts the power consumption of the system. The dynamic power dissipation varies linearly with frequency and quadratically with supply voltage. With a linear relation between voltage and frequency, the increase in the number of cycles needed to execute a process can lead to a cubical increase in the dynamic power. The variation of application requirements and system state with time usually necessitates the dynamic adaptation of the system voltage and frequency. Dynamic voltage/frequency scaling (DVS) technique adapts the system frequency and voltage to the realtime constraints of the system [1,2], thus optimizing for the system power and energy. P. Stenström (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 286–306, 2009. c Springer-Verlag Berlin Heidelberg 2009 

Power-Aware Bus Coscheduling for Periodic Realtime Applications

C1

287

C1

Periodic standalone bus traffic for process 1 C2

C2

Periodic standalone bus traffic for process 2 C1’

C1’ C2’

C2’

Bus overlap C1" ( < C1’) C2" ( < C2’)

C1" ( < C1’) C2" ( < C2’)

Optimal bus overlap

Fig. 1. The effect of bus traffic overlap of two processes sharing a common bus

Scheduling processes in multiprocessor system aims at coordinating the utilization of shared resources between competing processes [3,4,5]. The stress of each process on the memory system varies with time. The number of cycles to execute these processes can decrease if we could find an optimal overlap of the bus demands from processes running on the system. By “optimal overlap”, we mean an overlap that minimize the average clock cycles that each process needs to complete a memory transaction. In this work, we show the impact of coscheduling of processes on a shared bus for multiprocessor embedded system. We illustrate the variation in execution cycles based on the overlap of processes coscheduled on shared bus. Brute force search for optimal overlap of coscheduled processes requires simulations of the processes with all possible overlaps. This can be a prohibitively expensive process. Instead, we devise a scheme to predict optimal coschedule. This scheme narrows the search space for optimal overlap. We propose a process that comprises the following steps: identifying the initial phase of each process, finding periodicity in the process behavior, determining a common period between the coscheduled processes, finding profile of performance variation with all possible overlaps, and finally finding an optimal bus coschedule of the processes. We also introduce the use of scheduling barrier to maintain the optimal overlap of coscheduled processes. The proposed scheme predicts the effect of coscheduling on the performance under all possible execution overlaps. This helps in identifying a schedule with minimum negative impact on performance (through minimizing bus contention) and helps in reducing the number of cycles needed to execute each process. For realtime application reducing the number of cycles to execute a task reduces the power requirement because the system frequency and voltage can be reduced accordingly. The proposed technique can be applied for coscheduling applications with periodic pattern of accessing the memory systems. For this class of applications, the same processing is usually applied on different frames of data and the processing is independent

288

K.Z. Ibrahim and S. Niar

of the values of data processed. Even though the applicability of the proposed scheme is limited to this class of applications, specialized design process is common in embedded systems to achieve the best power consumption especially that these systems are usually dedicated to run a fixed set of applications. The rest of this paper is organized as follows: Section 2 introduces the impact of contention on shared bus and its effect on the number of cycles a periodic task needs for execution. The simulation environment is described in Section 3. Our proposed technique to predict optimal bus scheduling is detailed in Section 4. We extend our formulation of the proposed scheme, in Section 5, to systems running arbitrary number of processes. Section 6 summarizes related work as well as future work. Section 7 concludes our work.

2 Impact of Bus Overlap on Performance In this work, we constrain our discussion to multiprocessor system with applications known a priori. For clarity, we will consider system with two processes running concurrently. We will generalize our formulation in Section 5 In Figure 1, two processes with different traffic patterns are shown. The upper part of the figure shows the bus traffic for each application running individually on the system. Two different execution overlaps are shown in the lower part of the figure. The number of cycles (Ci) for each process depends on the overlap with the other processes running on the system. The clock frequency of the system f needed to meet deadline constraint is defined as C/ T where C is number of cycles of a process with the realtime period T . Increasing the number of cycles C of a process necessitates increasing the frequency of the system f . Different proposals [2,6,7] describe how to adapt the frequency to the demand of a process with realtime constraints. Increasing the frequency severely impacts the power requirements of the system. In CMOS based system, the commonly used technology in embedded systems, the power 2 · f where consumption is mainly due to dynamic power [8] that is given by P = Ce f ·Vdd Ce f is the effective switched capacitance, Vdd is the supply voltage. The frequency is almost linearly related to the supply voltage [8]. Consequently, the dynamic power is cubically related with the frequency (P ∝ f 3 ). The energy (computed as E = P · T ) is quadratically related with the frequency, which is an important factor for battery powered devices. Static power usually has negligible contribution to total power for embedded systems with low frequency. Static power is proportional to supply voltage Vdd [9,10], and thus can benefit linearly from decreasing frequency. Figure 2 shows the effect of all execution overlaps for six pairs of embedded applications. Execution overlaps are generated by sliding the execution of one application with respect to the other application in the pair. These pairs of applications are taken from three categories of MiBench suite [11], namely the security, the telecommunication and the consumer categories. Each group is run on a dual processor system with shared bus. Each application exhibits a periodic behavior in accessing the bus. Figure 2 shows the percentage of change in execution cycles when we have different overlaps for each pair of applications. Depending on overlap (or shift), the execution cycles change

Power-Aware Bus Coscheduling for Periodic Realtime Applications

10

0 25

100

20

Bus-busy percentage

200

300

400

Time sample SHA

30

20

10

0 15

100

200

300

10

5

0

400

Time sample PCM-encode

10

15

5

0 100

200

Time sample

300

400

40

Bus-busy percentage

20

30

20

10

0 15

200

400

600

800

Time sample PCM decode

10

5

200

Time sample

300

400

30

20

10

0 50

200

400

600

800

300

400

Time sample MAD

40

30

20

10

0

0 100

GSM decode

50

40

Bus-busy percentage

Bus-Busy Percentage

30

GSM encode

50

40

Bus-busy percentage

Bus-busy percentage

40

Rijndael encrypt/decrypt

Bus-busy percentage

50

Bus-busy percentage

Blowfish

50

289

100

200

Time sample

300

400

100

200

Time sample

Fig. 2. Effect of all execution overlaps on the total execution cycles for six pairs of embedded applications. The effect is shown as a percentage of the execution cycles of an initial overlap.

by up to 5.5% for Blowfish, 12% for Rijndael, 4% for SHA, 6.5% for GSM, 1.5% for PCM, and 12.5% for MAD. These changes consider the difference between the maximum and the minimum execution cycles. The changes in the execution time are due to the memory latency changes that arise because of the different contention scenarios that these applications face on the bus. The details of the simulation environment are given in Section 3. During the run of these applications, the execution cycles alternate between the values shown in Figure 2. To guarantee meeting deadline for real-time application, system designer usually considers worst-case execution cycles thus necessitating overclocking the system. Based on the early discussion, if we can enforce a coscheduling that provide the minimum execution cycles then we can obtain a saving in the dynamic power consumption, relative to the power associated with worst-case execution cycles, from 4% for PCM up to 33% for MAD assuming a dual-core MPSoC. Even if the system bus and memory are not affected by the frequency scaling and only the processor core and caches are affected, the gain in reducing the processor power is very large. This is attributable to the large ratio of power consumed by the processor core and caches compared with the power consumed by the bus and memory subsystems. For PCM, this ratio is 50:1 and the ratio for MAD is 4:1. The saving due to frequency reduction is especially important for battery powered systems. The cubic scaling of power with frequency is one of the main motives for building MPSoC for realtime systems running concurrent jobs, in constrained power environment. This alternative is based on using multiple processor cores with a lower frequency. The other, less efficient, alternative is to run the concurrent jobs on one processor core with higher frequency. Replicating the number of cores almost linearly increases the power demands while increasing frequency increase power demands cubically. The above discussion shows the importance of coscheduling processes and its severe impact on the system power requirements. The main problem is to search for optimal configuration. This involves simulation of all possible overlaps which can be exces-

290

K.Z. Ibrahim and S. Niar Table 1. Embedded Benchmarks used in this study

benchmark Description Blowfish Rijndael SHA GSM ADPCM MAD

Symmetric block cipher with multiple lengths key. A Symmetric block cipher encryption/decryption algorithm that is chosen by the National Institute of Standards as Advanced Encryption Standard (AES). Secure Hash Algorithm (SHA) that produces a 160-bit message digest. Global Standard for Mobile communication. Adaptive Differential Pulse Code Modulation (ADPCM). MPEG Audio Decoder. It supports MPEG-1, MPEG-2, and MPEG-2.5.

sively expensive. We simulated 99 configurations for Blowfish, 90 for Rijndael, 79 for SHA, 368 for GSM, 321 for PCM, and 98 for MAD. The number of simulated configurations for each application is chosen to avoid repeated simulations and is determined by the common periodicity detected for each pair of applications as will be detailed in Section 4.3. Searching for a local minimum execution cycles using a technique such as steepest descent can face difficulty because of the existence of multiple minima and the existence of fast-varying change (ripples) as an additional component to the more important slowly-varying change. It will also require many simulation runs. The determination of the optimal scheduling is an additional dimension to the design space for MPSoC. Multiple hardware configurations are usually explored in designing such systems. It is also common to have systems that run different combinations of applications simultaneously. For every set of applications, we need to find an optimal coschedule that saves execution cycles as well as energy. In this work, our objective is to predict the effect of coscheduling of multiple processes in a simple and accurate way that enables fast and precise design process. The details of our proposed technique are presented in Section 4.

3 Simulation Environment The simulation environment is based on MPARM [12] environment. MPARM models a Multiprocessor System-on-Chip (MPSoC). The processor models ARM 7 processors based on a software implementation called SWARM [13]. The system is interconnected using an AMBA [14] communication standard (architecture for high performance embedded systems). Multiple bus arbitration schemes are implemented by the MPARM. We choose a simple round-robin scheme. The simulated system has data cache (4KB size, 4-way set-associative) and instruction cache (2KB size, 2-way set-associative). Both caches are with writeback policy. The clock frequencies of all CPUs are the same, i.e., homogeneous processor cores. Uncontended memory latency is 40 cycles. Benchmarks used in this study are taken from security, telecommunication, and consumer categories of MiBench suite of embedded benchmarks [11]. Except for SHA and MAD, all benchmarks have a decode functionality in addition to the encode functionality. In this study, we run these two functionalities in pairs. For MAD and SHA, we run similar copies of the application. These pairs of applications can normally run concurrently in a multiprocessor embedded system.

Power-Aware Bus Coscheduling for Periodic Realtime Applications

291

Rijndael and MAD represent memory intensive applications with large percentage of cache misses, while PCM is less memory intensive application with small percentage of cache misses. The average memory access time is affected by the miss penalty that increases with the contention on shared bus. These applications are sequential applications that are run in parallel with non-shared memory spaces.

4 Finding Optimal Bus Coschedule This section presents our technique to search for an optimal coschedule for two processes running on system with a shared bus. The approach is generalized to more than two processes in Section 5. We target finding an optimal static coschedule between two running processes that exhibit periodicity. Many embedded applications exhibit periodicity in their execution and consequently periodicity in the traffic sent to the memory system. This periodic behavior usually appears after an initialization period and may be trailed by a termination period. We are more interested in the periodic part of the application because it dominates the execution time of the application.

10

0 25

100

20

Bus-busy percentage

200

300

400

Time sample SHA

30

20

10

0 15

100

200

300

PCM-encode

10

15

10

5

0

400

Time sample

5

200

Time sample

300

400

30

20

10

0 15

200

400

600

800

Time sample PCM decode

10

5

100

200

Time sample

300

400

30

20

10

0 50

200

400

600

800

300

400

Time sample MAD

40

0

0 100

40

Bus-busy percentage

20

GSM decode

50

40

Bus-busy percentage

Bus-Busy Percentage

30

GSM encode

50

40

Bus-busy percentage

Bus-busy percentage

40

Rijndael encrypt/decrypt

Bus-busy percentage

50

Bus-busy percentage

Blowfish

50

30

20

10

0 100

200

Time sample

300

400

100

200

Time sample

Fig. 3. Percentage of bus-busy for applications running standalone on the system bus

We start the search for optimal coschedule by running each process individually on the system. We record the bus-busy percentages over fixed interval of cycles. These recordings constitute a time-series of measurements bk , k = 1, . . . , n, where bk is the bus-busy percentage at time interval k. Figure 3 shows the percentages of the bus-busy cycles for our benchmarks. The time samples are 0.5K cycles for Blowfish and Rijndael; 2K cycles for GSM and PCM; and 20K cycles for SHA and MAD. These choices for sampling sizes are empirically chosen to compromise between having enough bus traffic details and limiting the number of scheduling decisions (bus traffic periodicity) to facilitate verifications. Decreasing the sampling interval increases the number of samples per application periodicity. While the proposed scheme has no difficulty in predicting performance with any number of

292

K.Z. Ibrahim and S. Niar

px Bus busy

Bus busy

Application X

Time sample

Time sample

py

Bus busy

Bus busy

Application Y

Time sample

Time sample

1- Detecting initialization

2- Detecting periodicity

pc Bus busy

Bus busy

p c~=3*p x

Time sample

Time sample

Scheduling Shift

p c~=2*p y Bus busy

Bus busy

pc

Time sample

3- Finding common period

Time sample

4- Scheduling applications

Fig. 4. Four steps in scheduling processes in MPSoC

samples per periodicity; it will be very difficult to simulate all configurations to verify the correlation between the performance predicted by our model and the outcomes of the simulations. As shown in Figure 3, Blowfish and GSM bus traffics have burstiness in accessing the memory. Execution is divided into phases of large bus traffic followed by almost idle phases. Applications such as Rijndael, on the other hand, have slowly varying bus traffic. The encode and the decode functionalities produce different traffic profile for GSM and PCM. For Rijndael and Blowfish, the bus traffic for encrypt and decrypt follows the same profile. All these applications show periodic behavior in dealing with the system bus. To find an optimal bus coschedule, we propose the following steps: 1. Isolation of the initialization part of the application from the periodic part for each application, Section 4.1. 2. Identification of the periodicity of the bus traffic for each application individually, Section 4.2. 3. Creation of common coscheduling period for all applications designated for coexistence on the system, Section 4.3. 4. Analysis of the effect of different coscheduling overlaps/shifts, for the common coscheduling period, on the execution cycles, Section 4.4. Figure 4 summarizes the four steps proposed to find optimal scheduling decision for a pair of processes. With the outlined technique, using simulation to exhaustively search for optimal coschedule is not needed. Enforcement of coscheduling decision to guarantee repetitiveness using scheduling barriers is introduced in Section 4.5. We asses the goodness of our technique in the prediction of optimal bus coschedule in Section 4.6.

Power-Aware Bus Coscheduling for Periodic Realtime Applications

293

4.1 Identifying Initialization Period To identify the bus-traffic initialization phase of each process, we start by forming an initialization approximate vector (IAV). The IAV vector is formed by taking an initial subset of the vector bk . We arbitrary choose a divisor d for the number of samples n. The IAV is chosen as bk , k = 1, . . . , n/d. Then, we compute the difference between  n/d  the IAV and shifted versions of the original time series, as Ds = ∑ j=1 b j+s − b j  , s = 1, . . . , 2n/d. We compute an approximation of the second order difference as ∆D2s = (Ds+1 − 2Ds + Ds−1 ), s = 2, . . . , 2n/d − 1. The peaks of the second degree difference (∆D2s ) occur at the possible end of initialization points. It is notable that multiple local maxima may arise due do the inclusion of part of the periodic behavior within the IAV. We used multiple divisors d to get confidence of the result. The outcome of initialization identification is independent of d as long as the initialization is a subset of IAV. This approach is analogous to that used in finding application initialization based on basic block difference graph [15]. After identifying the m initial intervals, we form a new time series gk with the initialization phase stripped; such that gk = bk+m , k = 1, . . . , r where r = n − m. Figure 5 shows Ds for the GSM pair of applications and Rijndael pair of applications. We show a small fraction of the computation curve for clarity. We used large value for d, 32 and 64, because the initialization part is a very small part of the execution time. It is notable that the maximum ∆D2s occurs at sharp local minima of the graphs. We choose the end of initialization interval at any point after the first maximum of ∆D2s . Any point that follows the initialization can be considered a start for the periodic behavior. The end of the initialization interval can be taken as the first scheduling point for the application; that is why it is not advisable to excessively delay the choice for the end of initialization. For the following analysis, though, we need to make sure that 2.5

1.6

Blowfish encrypt/decrypt Divisor 32 Divisor 64

2.0

3.0

Rijndael encrypt/decrypt

1.4

Ds

Ds

3.5

GSM decode

3.0 2.5

1.2 1.5

GSM encode

2.5

2.0 2.0

1.0

Ds

Ds

1.5

1.5

0.8 1.0

1.0

1.0

0.6 0.5

0.5

0.5 0.4 0

50

100

150

200

250

300

Time sample

6

0

50

4.5

5

150

200

250

300

Time sample

5.0

SHA

100

100

200

300

400

Ds

2

200

300

400

500

Time sample MAD

10 2.0

3.5

3

100

2.5

PCM decode

8

3.0

Ds

0

500

Time sample

12

PCM encode

4.0

4

0

1.5

2.5

Ds

2.0

6

Ds 1.0

4

1.5 1.0

1

2

0.5

0.5 0

0 0

50

100

150

200

Time sample

250

300

0

50

100

150

200

Time sample

250

300

0

50

100

150

200

Time sample

250

300

0

50

100

150

200

Time sample

Fig. 5. Initialization phase detection based on difference graph of bus traffic

250

300

294

K.Z. Ibrahim and S. Niar

enough cache warming has occurred before identifying a period representative to all other execution periods. In summary, we use early initialization point for scheduling synchronization, to be introduced later in Section 4.5, while for the sake of analysis we consider a late initialization point. 4.2 Periodicity Detection Detection of periodicity in experimental data has been studied by many researchers [16,17,15]. Autocorrelation (self-correlation) R is one of the mathematical formulas used to detect periodicity in a time series. The autocorrelation R is computed based on the autocovariance Cd , where d represent the time lag between the time series and its shifted version. The computation proceeds as follows: r   ¯ g(k−d) − g¯ Let g¯ be the average of the time series gk , then Cd = 1r ∑ (gk − g) k=d

and Rd = CCd0 . In this work, we adopted a methodology based on special form of autocorrelation called the folded autocorrelation. First, we define folded covariance as r   ¯ g(k+d) − g¯ . We assume gk = gk−r for all k > r. The folded autoFCd = 1r ∑ (gk − g) k=1

d correlation is then defined as FRd = FC FC0 . Folded autocorrelation assumes virtual periodicity, thus simplify identifying periodicity. Figure 6 shows folded autocorrelation of the bus traffic after striping the initialization period. The periods between peaks of autocorrelation are candidates for defining periodicity. Although, the analysis introduced in Section 4.1 shows early prediction of possible periodicity, it does not precisely identify periodicity partly because of the inclusion of initialization period. The first few periods are usually affected by the cold start of the cache. Stripping the initialization from the bus traffic is needed to provide accurate estimate of the periodicity. Folded autocorrelation gives an accurate estimate for the periodicity. Precise identification of periodicity is needed to guarantee no drift in scheduling decision. Except for GSM encode, periodicity can be detected easily both by inspection and mathematically. GSM encode has a large period that comprises five smaller periods with some similarity. We have chosen the larger period because this choice makes all periods almost similar for GSM encode.

4.3 Finding Common Periodicity Based on the analysis introduced in the previous sections, we find the standalone periodicity for each process that needs coscheduling on the system with a shared bus. A process X is defined by the tuple [xk , ix , px ] where xk is the percentage of bus busy cycles at interval k (ranging from 1 to px ) after skipping ix initial intervals. Similarly, we define the tuple [yk , iy , py ] for process Y . The periodicity can vary from one process to another. Finding optimal coschedule of bus traffic requires analyzing a common period that repeats for all processes sharing the bus. Common period makes the scheduling decision repeatable for all processes. A common period is composed of multiple basic periods of

Power-Aware Bus Coscheduling for Periodic Realtime Applications 1.0

1.0

1.0

0

50

100

200

250

300

Scheduling Shift

-0.4 1.0

Folded Autocorrelation (FRd )

150

GSM encode

0.4 0.2 0.0 -0.2 0

50

100

0.8 0.6 0.4 0.2 0.0

200

PCM encode

1.0

0

100

200

300

400

500

0.8 0.6 0.4 0.2 0.0 0

100

200

600

700

Scheduling Shift

300

400

0.8 0.6 0.4 0.2 0.0 0

50

100

150

200

250

300

250

300

-0.2 -0.4

Scheduling Shift

1.0

MAD

0.8 0.6 0.4 0.2 0.0 -0.2 0

50

100

150

200

-0.4

-0.4

-0.4

300

PCM decode

-0.2

-0.2

250

Scheduling Shift

-0.4

GSM decode

150

Folded Autocorrelation (FRd )

0.0

0.6

Folded Autocorrelation (FRd )

0.2

Folded Autocorrelation (FRd )

0.4

0.8

Folded Autocorrelation (FRd )

Folded Autocorrelation (FRd )

0.6

-0.2

SHA

Rijndael encrypt/decrypt

Blowfish encrypt/decrypt 0.8

295

Scheduling Shift

Scheduling Shift

Fig. 6. Folded autocorrelation for periodicity detection

the coscheduled applications. Multiple criteria can be used in finding common period, as follows: – For same realtime requirements for all processes: The shorter process is either appended with inactivity period to make all processes with the same period, or stretched assuming that it will run on a slower processor (heterogeneous system) such that all processes have similar periods. – For different realtime requirements for considered processes: We need to define a common period pc using the least common multiple of the two period count px , py . To avoid having a common period, pc , that is as large as px · py , it is sufficient to have pc such that (pc mod px )/px < tol and (pc mod py )/py < tol, where tol can be arbitrarily chosen, for instance less than 0.05. Increasing the common period (pc ) for coscheduling may reduce the effectiveness of the scheduling mechanism that is described in Section 4.5. To compute pc based on a certain tol, we start with an initial value for pc of px · py ; we repeat decreasing pc as long as the condition (pc mod pi )/pi < tol is satisfied for all processes. The minimum value for pc while satisfying the condition is considered as a common period for coscheduling. For Blowfish and Rijndael pairs of applications, we used the same realtime constraint for encrypt and decrypt. Both encrypt and decrypt have the same periodicity, which facilitates choosing a common period (pc = px = py ). Similarly, we used the same periodicity for SHA and MAD because two identical copies are run for both applications. For GSM pair of applications, GSM encode periodicity is almost three times the periodicity for GSM decode, assuming a tol = 0.01. While for most mobile computing applications the realtime constraint is the same for GSM encode and decode, we assumed an application where decode is needed for more frames than for encode, for

296

K.Z. Ibrahim and S. Niar

instance in conference calls. Using tol = 0. 01 for PCM pair, we find a common periodicity that coincides with three basic periods of PCM encode and with five basic periods of PCM decode. In future work, we can consider a system with heterogeneous processor core to handle the same realtime constraint for different computational requirement. 4.4 Predicting Optimal Overlap of Coscheduled Processes In this section, we aim at finding an overlap between two periodic processes such that the number of cycles needed to finish both processes is minimal. Simulation of all possible overlaps between coscheduled processes is extremely time consuming process, especially with large common periodicity of coscheduled processes. The computational requirement for simulation increases if we need to repeat the coscheduling search for different hardware configuration. We like to narrow the search space for optimal coschedule based on the information we get from running each process as a standalone process. Formally, we need to find a scheduling shift l between the processes to be coscheduled given the bus-busy percentages xk , yk for process X and Y ; respectively, where k = 1, . . . , pc . To achieve this objective, we propose the use of one of the following two metrics: 1. Find the minimum shift-variance of the sum of xk and yk shifted by l = 1, . . . , pc −1. let zlk = xk + y(k+l) mod pc Var(l) =

1 pc l ∑ (zk − z¯)2 pc k=1

(1)

pc pc where z¯ = p1c ∑k=1 zlk = p1c ∑k=1 (xk + yk ). Note that z¯ is the same for all overlaps, which leads to the simple form on the right hand side of the equation for z¯. 2. Find the minimum convolution of xk , yk for all scheduling shifts l.

Conv(l) =

1 pc ∑ xk · y(k+l) mod pc pc k=1

(2)

These two metrics give profiles of the effect of overlapping the bus traffic of the two processes. While these profiles help in knowing approximate area where minimum negative impact of overlap occurs, they do not provide a quantitative measure of the effect that optimal coschedule may introduce. These measures will be highly accurate if the effect of overlapping a bus traffic from one process with the traffic from the other process is self-contained. The effect of overlap of two points is hopefully not biased toward extending the execution time for one process over the other. This requirement necessitates using fair arbitration policy on the bus. Coscheduling on bus with prioritization scheme may be less fruitful. A necessary condition is that the timing does not accidentally favor one process over the others, where one process always acquires the bus ahead of the other processes. Round-robin with preemption is one of those fair schemes but is unfortunately difficult to implement on multiprocessor bus. Simple round-robin, used in this study, provides a relatively fair arbitration mechanism.

Power-Aware Bus Coscheduling for Periodic Realtime Applications

297

Another inherent assumption is that criticality of cache misses are mostly the same for all cache misses, thus delaying a bus transactions impact performance the same. This is mostly true for simple cores with blocking cache misses that is modeled in this study. For systems with more complex cores, further investigation may be needed. 4.5 Coscheduling Enforcement Using Barriers We propose to use barriers to define the overlap between coscheduled processes and to maintain this coschedule. We cannot use timing information collected by the standalone runs as a basis for scheduling synchronization. These timings are stretched due to the increase in memory latency because of the bus contention. A robust technique would be to identify barrier locations on source code and to add barrier calls where necessary. Inserting software barrier has difficulties: First it requires changing the source code for each scheduling decision. Second, it cannot be inserted easily anywhere in the code; only specific locations are suitable for barrier insertion. We adopted a simple approach based on the number of graduated instructions. The initialization and periodicity of an application are translated into instruction counts. The scheduling barriers are applied based on the number of graduated instructions. This simple mechanism can be applied easily in embedded environment where simple OS is used and program execution is repeatable and deterministic. In this study, we considered hardware barrier [18,19] for synchronization, which is associated with little execution overheads. Hardware barrier can be implemented using a simple wired-and line(s). Triggering synchronization is system dependent. A possible implementation that is explored in this study involves additional registers to hold the initial synchronization and the periodicity in terms of graduated instructions counts. These registers are part of the process context. An additional register is

Fig. 7. Effect of coscheduling on bus wait time

298

K.Z. Ibrahim and S. Niar

Rijndael

Blowfish

1.0

SHA

1.0

1.0 0.9

0.9

Normalized Convolution Normalized Shift-variance

0.9

0.8

0.7

0.8

0.7

0.6

0.7

0.8

0.6 0.5

0.5

0.6 0.4

0.4 0.5

0.3

0.3 0.2

0.4 0

20

40

60

80

100

0

20

Scheduling shift 1.2

40

60

0

80

20

GSM

1.0

0.9

0.9

0.8

0.6

0.8

0.7

0.4

0.7

0.6

0.2

0.6

0.5

0.0

0.5 0

50

100

150

200

250

Scheduling shift

300

350

80

MAD

PCM

0.8

60

1.0

1.1

1.0

40

Scheduling shift

Scheduling shift

0.4 0

50

100

150

200

Scheduling shift

250

300

0

20

40

60

80

100

Scheduling shift

Fig. 8. Coscheduling effect prediction based on convolution and shift-variance

needed to hold the graduated instructions count. This register is initialized with the synchronization point. It is decremented each time an instruction is graduated. When the register value reach zero, the barrier synchronization is acquired. The barrier synchronization is released after all processes reach the barrier (the wired-and barrier line is asserted by all processors). During release, the register is reset to the periodicity instruction count. Scheduling barriers do not represent any data dependency. They are not required for correct execution of coscheduled programs. They are used to ensure that no drift in scheduling occurs after executing many periods, and thus guaranteeing the persistence of the scheduling decision. With appropriate scheduling, this does not only maintain less execution cycles but also reduce the variability of execution because the memory traffic overlap will continuously repeat. Using barriers usually causes period of idleness for the processors finishing work earlier, i.e. arriving earlier to the barrier. The waiting period on the barrier is quantified in Section 4.6. 4.6 Accuracy of Coscheduling Prediction for Two Processes Systems In this section, we introduce the effect of coscheduling on memory access time and prediction accuracy of the effect of overlap. The memory access time affected by contention during the arbitration phase, the time to be granted the bus to start memory transfer, and the time to transfer data, especially that burst of data can be split into multiple non-contingent transfers. Figure 7 shows the effect of scheduling shift on the average wait time on the bus for the six pairs of applications. The figure shows the percentage of change of wait time per memory transaction compared with the wait time for the initial scheduling decision (the reference). Every memory transaction faces the un-contended latency in addition

Power-Aware Bus Coscheduling for Periodic Realtime Applications

299

to the additional wait time due to contention. The profiles for the bus-access wait-time follow the profiles predicted by equations 1 and equation 2. Both equations correlate to the wait on bus, as will be quantified later. The reference average wait-time per memory transaction is small (in the range of 2 to 8 cycles) for applications with low bus contentions; specifically for PCM, SHA and GSM benchmarks. For Blowfish, Rijndael, and MAD benchmarks, the reference average wait time per memory transaction ranges from 12 to 15 cycles. Percentage-wise the coscheduling decision impacts the wait time of some benchmarks, for instance GSM, more than others, for instance Rijndael, while scheduling decision impacts the latter’s performance more. This is attributable to the higher miss rate and the higher wait time involved such that the overall performance is more sensitive to the memory system performance. The execution cycles for different scheduling decisions (Figure 2) follow the profile of the bus performance introduced in Figure 8. To quantify the prediction accuracy of our proposed scheme, we use the correlation coefficient; defined as follows: let x ¯ be  the average value of a random variable X. The variance is defined as E (X − x) ¯2 = √ ¯ 2 . We also define σX ≡ σXX . A good measure of dependence σXX ≡ 1n ∑ni=1 (xi − x) between the two random variables X and Y is the correlation coefficient [20], defined as ρXY ≡

E {(X − x) ¯ (Y − y)} ¯ σXY = σX · σY σX · σY

(3)

The numerator of the right hand side is called the covariance σXY of X and Y . If X and Y are linearly dependent, then |ρXY | = 1. If |ρXY | = 0, then observing the value X has no value in estimating Y . We use the correlation coefficient to study the relation between equation 1, equation 2, bus access wait-time, and execution cycles. Table 2 shows the correlation coefficient between these measurements. Both shift-variance (defined by equation 1) and convolution (defined by equation 2) are very strongly related for two processors system. One can replace the other, and preference is given to computational simplicity which favors convolution. The correlation between the bus waits and the execution cycles ranges between a lowest of 0.72 for PCM and highest of 0.97 for GSM. These values can be classified as high to very high correlation according to Williams [21]. The bus performance does not perfectly correlate to the execution cycles (correlation of 1) because the effects of cache misses on the performance are not similar. Some bus transactions are more critical to performance than others, while all transactions contribute to the bus contention similarly. Additionally, the bus arbitration policy is not perfectly fair. The bus wait is more correlated to convolution (and to shift-variance), compared with correlation to execution cycles, because we used the bus traffic only in the convolution computation. These correlations, between bus wait and convolution, range from a lowest of 0.71 for Rijndael to a highest of 0.93 for GSM. The total execution cycles depends on the interaction with other components on the system. It is apparent that correlation coefficients between execution cycles and convolution are lowered if the correlation between bus wait and execution cycles is low, or if the correlation between bus wait and shift variance is low.

300

K.Z. Ibrahim and S. Niar

Table 2. Correlation coefficient (ρXY ) between bus wait time(Wait), execution cycles (Cycles), convolution(Conv), and shift-variance (SVar) Bench Conv/SVar Cycles/SVar Wait/SVar Wait/Cycles ∼ 0.7675 0.8043 0.8610 = 1.000 0.9974 0.7238 0.7085 0.7956 ∼ 0.7087 0.8707 0.8827 = 1.000 ∼ 0.8580 0.9266 0.9764 = 1.000 ∼ 0.5537 0.8538 0.7243 = 1.000 ∼ 0.8610 0.8858 0.9380 = 1.000

Blowfish Rijndael SHA GSM PCM MAD

These high correlation coefficients show that we can predict the effect of scheduling multiple processes sharing a bus. This prediction helps in identifying the best scheduling region. The exact performance difference, due to the scheduling, can be obtained through simulation of only few points of interest in the regions identified by the proposed scheme. Simulating the system without scheduling barriers, we found a drift in the execution overlap that leads to performance change from one scheduling period to the other. As discussed earlier, we propose using scheduling barrier to circumvent this problem. We show the effect on performance of scheduling barrier in Figure 9, which shows the percentage of barrier wait for different scheduling decision. The barrier overhead is reported as average and the variation in the wait time is reported as the 99th percentile around the average. Using scheduling barriers incurs a small overhead for all applications studied. The barrier synchronization time relative to the total execution time does not exceed 1.3% for Rijndael and Blowfish. We noticed that the variation of barrier wait time is largest for Rijndael and Blowfish, although almost two identical processes are overlapping. This shows that the variation is mostly caused by the changes on overlap of bus transactions and not the difference in the amount of work, which is adjusted by the choice of common periodicity. 4.7 Proposed Scheme Applicability Dynamic Voltage Scaling (DVS) and the proposed technique tackle the same problem of adjusting driving frequency/voltage for the sake of reducing the power of applications with varying execution time. For hard realtime applications, DVS increase clock/voltage of the system to meet worst case execution time and then tries to reclaim slack time by reducing voltage/frequency if it arises during execution [22,23]. The advantage of DVS is that this technique can be applied to all kind of tasks, but faces the following challenges: a) estimating the worst case execution time, needed for hard realtime applications, is not always feasible for multiprocessor machines; b) changing voltage/frequency usually involves complexity in design and delayed response that reduces the amount of saving in power. The dynamic adjustment may overclock frequency thus wasting power, or under-clock the system frequency thus not meeting real time deadlines.

Power-Aware Bus Coscheduling for Periodic Realtime Applications

301

Percentage of total execution time

1.4

Average barrier wait time

1.2 1.0 0.8 0.6 0.4 0.2 0.0 -0.2 Blowfish

Rijndael

SHA

GSM

PCM

MAD

Fig. 9. Scheduling barrier wait-time percentage with the 99th percentile around the average

The proposed technique in this paper is applicable for special class of applications with repeating pattern of accessing the memory. For these applications, a static schedule is selected that minimize the execution cycles and thus the frequency/voltage needed to drive the system. This schedule is maintained by low-overhead hardware barrier, thus reducing the variability in execution time. The proposed scheme exploits repetitiveness of bus traffic. We show that the design space exploration can be surrogated by static analysis that alleviates the need of seemingly infeasible simulations. We studied different embedded applications from three different categories of benchmarks included in MiBench suite. The sources of bus traffic repetitiveness are as follows: a) algorithmic repetition of processing, for instance, applying the same processing to multiple frames of data; b) control flow of the application that is not dependent on the data processed. Certainly these conditions do not apply for all embedded application. We found that some applications, from MiBench suite, may not benefit from the proposed scheme as summarized below: a) Applications that have amount and type of processing dependent on the data in the frame processed. The traffic generated on the bus is thus not cleanly periodic, although the processing of some frames can be similar. Lame application, GPL’d MP3 encoder, and JPEG application, a standard lossy compression standard, are example applications that exhibit this behavior. Fortunately, many of these applications are usually not hard realtime applications; b) Applications with constant bus traffic on the bus, for instance CRC32, 32-bit Cyclic Redundancy Check. Applications with constant behavior does not benefit from DVS, as well as our technique, because there is no variability in execution time; c) Applications with no periodic behavior, for instance FFT (Fast Fourier Transform). Some of these benchmarks are kernel codes that are called by higher level codes, and may be called in a repetitive way. In this case they can benefit from the proposed scheme. We do not view these as limitations of applicability because the design of embedded systems does not involve generalized design rules. Special techniques are needed for different classes of systems to achieve certain design objectives, for instance ultra low-power systems. The proposed scheme can be thought of as Static Voltage Scaling (SVS) technique that suites special applications that shows periodicity. For this class of applications, SVS does not require complex mechanism for detecting and changing the driving frequency and voltage. Additionally, SVS reduce the variability

302

K.Z. Ibrahim and S. Niar

in execution time by forcing repetitive overlap of contending processes through barrier synchronization.

5 Multidimensional Coscheduling Applying the proposed coscheduling techniques gain importance if the number of scheduled processes increases because the search space for scheduling increases exponentially. For instance, if we would like to explore one hundred scheduling decision for a certain application, then scheduling two processes of this application will require one hundred simulation runs, while scheduling four processes will require 106 simulation runs. For the applications considered in this study, GSM would have 49,836,032 scheduling decisions for four processes. Certainly, exploring such design spaces is not feasible through simulations; and a tool to predict points of interest in the design space is critically needed. For multiprocessor system, estimating WCET is very challenging because of the numerous contention scenarios that a process may encounter. Conventionally, a simple approach involves overestimating the clock frequency to guarantee probabilistic meet of deadlines. In this section, we introduce the following generalization of the formulation introduced earlier in Section 4.4 to predict the effect of coscheduling, as follows: Let the periodic time series for bus-busy percentages for the m applications that need to be coscheduled in the system x1k , x2k , · · ·,xm k ; The common periodicity pc is then computed such that (pc mod pxi )/pxi < tol. Extending the definition for the shift-variance will be as follows: We define scheduling-shift vector as L : (l1 , l2 , ..., lm−1 ), where li = 1, . . . , pc − 1. zLk  x1k + i ∑m i=2 x(k+l ) mod pc i−1

Var(L) = where z¯ =

1 pc

1 pc L ∑ (zk − z¯)2 pc k=1

(4)

p

c i ∑k=1 ∑m i=1 xk . The convolution definition can be formulated as follows:

Conv(L) =

1 pc 1 m i ∑ xk ∏ x(k+li−1 ) mod pc pc k=1 i=2

(5)

Traditionally, shared bus systems are limited to few processors on the system bus. We limited system exploration to a system of four processors. Unfortunately, we cannot perform full verification of the prediction of coscheduling effect on performance, because we cannot simulate all the points of the scheduling space. Instead, we simulated three hundreds randomly selected scheduling points of the design space for each pair of applications. We also conducted simulation of the best and worst scheduling points that were predicted by the multidimensional shift-variance defined by equation 4. To evaluate the correlation between the prediction function and the simulated point; the multidimensional space of scheduling is projected into a single dimensional space, and then the formulation defined by equation 3 is used. Table 3 summarizes the results for the same set of applications studied in Section 4.6. We explored systems of four processes. We doubled the processes by repeating the pairs

Power-Aware Bus Coscheduling for Periodic Realtime Applications

303

Table 3. Results summary for four processors system; Maximum difference of execution cycles (Cycles), and correlation with Convolution (Conv) and shift-variance (SVar) benchmark Conv/SVar Cycles/SVar Cycles/Conv %Cycles Diff. Blowfish Rijndael SHA GSM PCM MAD

0.9001 0.9675 0.9049 0.6427 0.9070 0.8691

0.6900 0.4705 0.7074 0.8241 0.6592 0.9002

0.6334 0.3875 0.4863 0.5015 0.4696 0.7368

9.35 23.84 5.02 13.74 4.17 20.96

introduced in Section 4.6. Table 3 shows percentage of change in the execution cycles. The percentage of change increases with increasing of the number of processes contending on the bus. The main reason is that the difference between coincidence of high traffic burst of four processes and the optimal distribution of bus traffic gets larger. Certainly, this trend happens because of the ability of bus to absorb the bandwidth required by all running processes. If the applications bandwidth demands exceeded the bus ability, we expect that all processes will be slowed down and the difference in execution cycles will become smaller. The execution cycles difference is computed for the randomly selected scheduling points that were simulated, including points predicted by the shift variance as candidate for minimum and maximum cycles. The difference reaches a peak of 24% for Rijndael. Even if these differences in cycles are not proved global minimum and global maximum, they show the impact of scheduling on performance, and the need to enforce a scheduling to avoid missing deadlines for realtime systems. Table 3 shows that the correlation between the convolution and shift variance is not perfect for multidimensional scheduling, as we have seen earlier for one-dimensional scheduling. The shift variance gives better correlation with the performance observed through simulations. This behavior is observed for all applications because the shift variance predicts performance based on the effect of the sum of all traffic instead of multiplying them. Another observation is that the correlation of cycles with shift-variance decreased with four processors systems compared with two processors system because the fairness of the bus gets stressed. Although there is a need for further improvements in this direction, simulations can not be seen as a feasible alternative.

6 Related and Future Works Recently numerous research proposals targeted optimizing communication architecture for efficient use of energy [24,25]. These proposals adapt communication architecture to the application need or more specifically to the traffic generated by the application. Adapting the system frequency/voltage to the application need is intensively studied for uniprocessor and multiprocessor systems. For uniprocessor machines, different proposals [6,26] address the problem of adapting frequency-voltage to meet realtime constraint and optimize for the energy consumption.

304

K.Z. Ibrahim and S. Niar

For multiprocessor, Yang et al. [27] propose dividing power-aware scheduling of processes into two phases: The first phase, done offline, summarizes the possible schedule decisions. During runtime, the second phase, the actual scheduling decision is taken based on the predetermined scheduling options. Static Power Management is utilized by Gruian [1] to adjust the best supply voltage/frequency for each processor. Zhu et al. [8] adjust the voltage/frequency level of the system to reclaim the slack time [28] in executing tasks by reducing the frequency of future tasks. In contrast, our work addresses one of the main causes of variability in shared memory multiprocessor which is the contention for memory on a shared bus. Our proposed technique finds a scheduling decision that reduces the number of cycles needed to execute a task by reducing the effect of bus contention. We predict a good offline static schedule for the applications. We verified our technique for six pairs of applications. We explored extending the coschedule to any number of coscheduled applications. One constraint to this work is that it is applicable for applications that exhibit periodic bus behavior. Future work includes exploring different bus arbitration policies and studying the best scheduling decision under these policies. Differentiating bus traffic into critical and less critical to performance need to be augmented to our formulation to reach a better estimate of performance and serve the needs for more complex processors systems. We believe that studying the effect of contention on shared resources should gain more attention from system designers.

7 Conclusions Minimization of execution cycles for a given periodic task is essential for power saving. For a given realtime deadline, the frequency increases linearly with the execution cycles. The power consumption varies cubically with the frequency. One source of increasing the execution cycles is the contention on the shared bus in multiprocessor system. Through cycle simulation, we show that the execution cycles can vary by up to 13% for benchmarks running on two processors system, and 24% for four processors systems. These executions cycles varies the system power requirements greatly, because of the need to adjust the clock frequency to meet the system realtime constraints. This dynamic power saving can reach 57% for Rijndael executed on a quad-core MPSoC. To alleviate the high cost of searching best scheduling overlap using simulation, we propose a scheme based on shift variance of the bus traffic profiles obtained while running these applications individually. We outlined the steps needed to strip application’s initialization period and to detect the application periodicity based on bus traffic. Using shift-variance, we show that we can predict the effect of coscheduling under multiple scheduling overlaps. We also propose the use of scheduling barrier to maintain scheduling decision, which incurs very little overhead. We show that the prediction of scheduling effect using shift variance is highly correlated to the results obtained using simulations. We also extended our performance prediction mechanism to systems of larger number of processors with acceptable prediction accuracy.

Power-Aware Bus Coscheduling for Periodic Realtime Applications

305

References 1. Gruian, F.: System-Level Design Methods for Low-Energy Architectures Containing Variable Voltage Processors. In: Falsafi, B., VijayKumar, T.N. (eds.) PACS 2000. LNCS, vol. 2008, pp. 1–12. Springer, Heidelberg (2001) 2. Shin, Y., Choi, K., Sakurai, T.: Power Optimization of Real-time Embedded Systems on Variable Speed Processors. In: The 2000 IEEE/ACM Int’l. Conf. on Computer-Aided Design (ICCAD 2000), pp. 365–368 (2000) 3. Weiser, M., Welch, B., Demers, A., Shenker, S.: Scheduling for Reduced CPU Energy. In: The First USENIX Symp. on Operating Systems Design and Implementation (OSDI 1994), pp. 13–23 (1994) 4. Nguyen, T.D., Vaswani, R., Zahorjan, J.: Using Runtime Measured Workload Characteristics in Parallel Processor Scheduling. In: Workshop on Job Scheduling Strategies for Parallel Processing (IPPS 1996), pp. 93–104 (1996) 5. Snavely, A., Tullsen, D.M.: Symbiotic Jobscheduling for a Simultaneous Multithreaded Processor. In: The 9th Int’l. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX), pp. 234–244 (2000) 6. Rotenberg, E.: Using Variable-MHz Microprocessors to Efficiently Handle Uncertainty in Real-time Systems. In: The 34th annual ACM/IEEE Int’l. Symp. on Microarchitecture (MICRO 34), pp. 28–39 (2001) 7. Seth, K., Anantaraman, A., Mueller, F., Rotenberg, E.: FAST: Frequency-Aware Static Timing Analysis. In: The 24th IEEE Real-Time Systems Symp. (RTSS 2003), pp. 40–51 (2003) 8. Zhu, D., Melhem, R., Childers, B.R.: Scheduling with Dynamic Voltage/Speed Adjustment Using Slack Reclamation in Multiprocessor Real-Time Systems. IEEE Trans. on Parallel and Distributed Systems 14(7), 686–700 (2003) 9. Butts, J.A., Sohi, G.S.: A Static Power Model for Architects, pp. 191–201 (2000) 10. Brandolese, C., Salice, F., Fornaciari, W., Sciuto, D.: Static Power Modeling of 32-bit Microprocessors. IEEE Trans. on Computer-Aided Design of Integrated Circuits and Systems 21(11), 1306–1316 (2002) 11. Guthaus, M.R., Ringenberg, J.S., Ernst, D., Austin, T.M., Mudge, T., Brown, R.B.: MiBench: A Free, Commercially Representative Embedded Benchmark Suite. In: The IEEE 4th Annual Workshop on Workload Characterization (2001) 12. Benini, L., Bertozzi, D., Bogliolo, A., Menichelli, F., Olivieri, M.: MPARM: Exploring the Multi-Processor SoC Design Space with SystemC. Journal of VLSI Signal Processing 41, 169–182 (2005) 13. Dales, M.: SWARM – Software ARM, http://www.cl.cam.ac.uk/mwd24/phd/swarm.html 14. ARM, AMBA Bus, http://www.arm.com/products/solutions/AMBA_Spec.html 15. Sherwood, T., Perelman, E., Calder, B.: Basic Block Distribution Analysis to Find Periodic Behavior and Simulation Points in Applications. In: Int’l. Conf. on Parallel Architectures and Compilation Techniques (2001) 16. Small, M., Judd, K.: Detecting Periodicity in Experimental Data Using Linear Modeling Techniques. Physical Review E 59(2), 1379–1385 (1999) 17. Sherwood, T., Perelman, E., Hamerly, G., Calder, B.: Automatically Characterizing Large Scale Program Behavior. In: The 10th Int’l. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-X), pp. 45–57 (2002) 18. Beckmann, C.J., Polychronopoulos, C.D.: Fast Barrier Synchronization Hardware. In: The 1990 ACM/IEEE conference on Supercomputing (Supercomputing 1990), pp. 180–189 (1990)

306

K.Z. Ibrahim and S. Niar

19. Sivaram, R., Stunkel, C.B., Panda, D.K.: A Reliable Hardware Barrier Synchronization Scheme. In: The 11th Int’l. Symp. on Parallel Processing (IPPS 1997), pp. 274–280 (1997) 20. Shanmugan, K.S., Breipohl, A.M.: Random Signals: Detection, Estimation and Data Analysis. Wiley, Chichester (1988) 21. Williams, F., Monge, P.: Reasoning with statistics: How to read quantitative research, 5th edn. Harcourt College Publishers, London (2001) 22. Lee, Y.H., Krishna, C.M.: Voltage-Clock Scaling for Low Energy Consumption in FixedPriority Real-Time Systems. Real-Time Systems 24(3), 303–317 (2003) 23. Krishna, C.M., Lee, Y.H.: Voltage-clock-scaling adaptive scheduling techniques for low power in hard real-time systems. IEEE Trans. on Computers 52(12), 1586–1593 (2003) 24. Nicoud, J.D., Skala, K.: REYSM, a High Performance, Low Power Multi-processor Bus. In: The 13th Int’l. Symp. on Computer Architecture, pp. 169–174 (1986) 25. Lahiri, K., Dey, S., Raghunathan, A.: Communication Architecture Based Power Management for Battery Efficient System Design. In: The 39th Conf. on Design Automation (DAC 2002), pp. 691–696 (2002) 26. Anantaraman, A., Seth, K., Patil, K., Rotenberg, E., Mueller, F.: Virtual Simple Architecture (VISA): Exceeding the Complexity Limit in Safe Real-time Systems. In: The 30th annual int’l. Symp. on Computer Architecture (ISCA 2003), pp. 350–361 (2003) 27. Yang, P., Wong, C., Marchal, P., Catthoor, F., Desmet, D., Verkest, D., Lauwereins, R.: Energy-Aware Runtime Scheduling for Embedded-Multiprocessor SOCs. IEEE Design and Test of Computers 18(5), 46–58 (2001) 28. Hua, S., Qu, G.: Power Minimization Techniques on Distributed Real-time Systems by Global and Local Slack Management. In: The 2005 Conf. on Asia South Pacific Design Automation (ASP-DAC 2005), pp. 830–835 (2005)

Performance Characterization for the Implementation of Content Addressable Memories Based on Parallel Hashing Memories Patrick Mahoney, Yvon Savaria, Guy Bois, and Patrice Plante ´ Ecole Polytechnique de Montr´eal, Groupe de Recherche en Micro´electronique, C.P. 6079, succ. Centre-Ville, Montr´eal Qu´ebec, Canada, H3C 3A7 {patrick.mahoney,yvon.savaria,guy.bois}@polymtl.ca, [email protected] http://www.grm.polymtl.ca/

Abstract. Content addressable memories (CAMs) are commonly used in applications requiring high speed access to some data set. This technology allows data items to be accessed in constant time based on content rather than on address. Unfortunately this technology has several drawbacks: it occupies more die area per bit, dissipates more power, and has higher latency. Recently, an efficient architecture based on a parallel hashing has been proposed as an alternative to CAM technology. In the present paper, we go a step further by backing preliminary simulation results of this proposed architecture by a complete analytical model. The insertion operations applied on the proposed architecture can be modeled with the balls and urns problem. We also propose a method to identify optimal configuration parameters in order to start designing efficiently. Finally, a VLSI implementation and optimizations of the proposed architecture are presented in order to obtain a more thorough understanding of how it could compare to commercial CAMs. Because of its simple design and of the widely spread use of the required tools, this new architecture offers a very appealing alternative to CAM technology.

1

Introduction

Content Addressable Memories (CAMs) allow lookup operations to be executed in a single cycle. This feature enables many applications to sustain high data throughput in a deterministic manner. Typical applications that leverage the CAM behaviour include “Ethernet address lookup, data compression, pattern-recognition, cache tags, high-bandwidth address filtering, and fast lookup of routing, user privilege, security or encryption information on a packet-by-packet basis for high-performance switches, firewalls, bridges and routers.”[1] Random Access Memory (RAM) technology does not offer single cycle lookup operations, yet it bears several key advantages over CAM technology. Typically, for a given storage capacity, it occupies smaller die area, it consumes less power, it possesses shorter access latency, it is offered in a wider variety of sizes and P. Stenstr¨ om (Ed.): Transactions on HiPEAC II, LNCS 5470, pp. 307–325, 2009. c Springer-Verlag Berlin Heidelberg 2009 

308

P. Mahoney et al.

Table 1. Static RAM and binary CAM technologies comparison. CMOS 0.18µm technology. 100 MHz operating frequency. CAM RAM CAM/RAM Factor Cost 19.2 6.75 2.8 (mm2 /M bit) Power Consumption Read operations 410 80.9 5.1 (mW/M bit) Power Consumption Write Operations 735 89.3 8.23 (mW/M bit)

flavours. Also, it is more generic and widely available and enables to avoid the heavy licensing and royalty costs charged by some CAM vendors. Table 1 illustrates some of the previously mentioned advantages of RAM technology over CAM technology. This comparison is based on data taken from specification sheets generated by a memory compiler of a common but undisclosed proprietary source. Considering the cost per bit metric, CAM is more expensive by a factor larger than 4. Note that tools and licensing fees may also represent a major part of the total cost of using CAMs in an application specific integrated circuit. Comparing power consumption is very tricky, as CAM’s consumption greatly varies depending on the operating frequency, the key content, and the match rate. Based on what could be defined as standard use for most applications, for a given memory size, the CAM consumes more power by a factor larger than 3 for read operations and by a factor larger than 4 for write operations. Because of all these reasons, many applications could benefit from an alternative solution to CAM technology, which could offer improved price over performance or power consumption over performance ratios. This paper aims at offering such an alternative solution. It strives at emulating CAM behavior by using parallel static RAM units, hence leveraging the benefits of using RAM technology so as to offer improved die area, power and energy consumption, as well as latency metrics. Recently, such an architecture based on parallel static RAM units has been proposed [2]. In the current paper we go a step further by backing preliminary simulation results [2] of this proposed architecture by a complete analytical model. The insertion operations applied on the proposed architecture can be modeled with the balls and urns problem. We also propose a method to identify optimal configuration parameters in order to facilitate system design. Finally, a VLSI implementation and optimizations of the proposed architecture are presented in order to obtain a more thorough understanding of how it could compare to commercial CAMs. While section 2 presents relevant work which has previously been accomplished in the field, section 3 describes the inner workings of the proposed architecture. Section 4 presents characterization of the proposed architecture done by simulations, while section 5 extends the characterization domain by means of

Performance Characterization for the Implementation of CAMs

309

an analytical model. Sections 6 and 7 provide a methodology for the implementation of the proposed architecture, while sections 8 presents design variations in order to optimize classical metrics. Finally Section 9 concludes the paper.

2

Previous Work

Many research projects aimed at reducing the power consumption of CAM cells. Most of them ( [3], [4], [5], [6], [7], [8], [9], [10] ) tried to achieve this goal by modifying the CAM unit cell that stores a single CAM bit. Several paths were explored, including the reorganization of the transistors so as to have as few transistors as possible discharging preloaded values to ground [8]. Such novel CAM structures are costly to use in very large scale integrated circuit (VLSI) designs because typically, they must be supported by some CAM compiler. An approach that may prove easier and less expensive is to use RAM technology combined with some hashing scheme to emulate a CAM behaviour. Broder [11] introduced a dynamic data structure that makes use of several tables, each possessing a unique hash function. Insertion operations consist in trying to sequentially insert an element into every table until a successful attempt is made. In the event where all insertion attempts produce a collision, Broder proposes to rehash a subset of the previously stored elements in a way that would allow all of them to be contained. Lim [12,13] extended this idea and proposed to implement it in hardware, using two RAM-based hash tables. Instead of rehashing, Lim added a small CAM unit which is used as an overflow table storing elements that produced collisions in other tables. In a system composed of two RAM-based tables and one small CAM, when a lookup operation is made, all three units are accessed simultaneously. A priority encoder then takes care of routing the right information to the output. Seznec [14] proposed a multi-bank cache configuration that could use more than one hash function. They were used to simultaneously map physical addresses to a distinct line in each bank. The performance gain is based on the high probability that addresses colliding in one bank would not collide in another. The current paper extends these concepts in order to offer an improved alternative to CAM technology.

3

Proposed Architecture

The proposed architecture is structured as sets of modules implementing multiple processing layers, each consisting of a hashing function, a static RAM unit and some combinational logic. The hashing functions translate the tag portion of the incoming data into an address of the RAM cell, as shown in Fig. 1. It is assumed, without loss of generality, that rows in memory contain at least 3 fields: the present field, the tag field and payload field. The present field occupies a single bit which is set if the row possesses valid data. The tag field contains the search key which will be compared to the supplied search key to

310

P. Mahoney et al.

Search Key

HF0

RAM unit

HF1

RAM unit

Control Logic

... HFn

RAM unit

Fig. 1. Proposed hardware structure

signal a match. The payload field contains the sought-after data associated with the search key. For example, an 8 port Ethernet bridge could have a 48 bit wide tag field representing MAC addresses and a 3 bit wide payload field representing the port through which the associated MAC address can be reached. Considering the one bit present field, every row in memory would thus span 52 bits. During a lookup operation, all layers execute a memory access to their RAM unit at the address specified by their respective hash key. The accessed data is then compared to the supplied search key. Every layer passes on the comparison results along with the payload of the accessed row to the output logic unit. This unit gathers the information and signals a successful lookup operation if one of the layers signals a match. In this case, it also gates the appropriate payload on the output pins. If no match occurs, the output logic unit signals a failed lookup operation. Insertion operations are analogous to their lookup counterpart. They can be split into 2 cases depending on whether or not the search key being inserted resides already in memory. If it is the case, the insertion operation consists in a payload update. If it is not, the insertion operation requires that a new row in memory be allocated to the data being inserted. For example, in an Ethernet bridge, executing an insertion operation with a non-resident search key might happen after receiving the initial frame from a given host. On the other hand, if this host has previously issued several packets, the inserted MAC address might already be present in memory. The inserted payload might be identical to the one being inserted, depending on whether or not the port through which the host can now be reached has changed since the last packet issued from the host was seen. In either case, it will be overwritten, and thus updated. The insertion operation itself can be seen as including a lookup operation. The procedures are identical up to the point where the output logic unit gathers the match related information from the layers. This time, it sends write authorizations back to the layers. A positive authorization will be given to the

Performance Characterization for the Implementation of CAMs

311

layer signaling a match if any, thus causing a payload update. If no layer signals a match, the positive authorization will be given to the highest priority layer among those signaling an empty row. If no such layer exists, the insertion operation will be considered as having failed. Because of the possibility of insertion failures, the designer will have to oversize the total architecture memory capacity in order to emulate a CAM unit of a given size. RAM’s low area cost per bit compared to CAM’s gives the designer the luxury of using only a fraction of the total memory capacity, while keeping the transistor count below the one of the emulated CAM unit. In this paper, we define the load factor as being the inverse of the oversizing factor. In addition to the load factor, directly linked to the total memory capacity, the number of layers is an application specific parameter chosen to obtain an acceptably small insertion failure rate. Another configuration parameter resides in the choice of the hashing functions used. A good hashing function will uniformly spread the probability of associating any given search key between all possible hash key values, hence decorrelating any similitude in search key traffic. If there was a known ideal hashing function for all applications, such a function would be selected. As none is known, the choice of the hashing function can be seen as being application dependent. For instance, Jain [15] has shown that bits generated by cyclic redundancy check (CRC) algorithms offer a performance close to those produced with ideal hashing functions. Thus such a hashing function would be a logical choice, yet CRC algorithms possess several parameters that can yield to very inadequate results if poorly chosen. This topic is complex, and even though further research would be justified, the issue is left for future work.

4

Performance Characterization Based on Simulations

The main behavioral difference between a CAM unit and the proposed architecture is that the former will successfully keep inserting new entries until it reaches full capacity, upon which time any new insertion attempt will fail. On the other hand, as will be shown in the following, our proposed architecture will slowly start failing before reaching full capacity. Because most applications do not possess hard upper limits on the number of entries needing to be supported, this gracefully degradable behaviour, combined with the possibility of oversizing the capacity, while keeping complexity below that of a competitive CAM, can best fit the needs of many applications where designers currently see CAM technology as being mandatory. Two characteristics need to be analyzed in order to appropriately choose the target load factor and the number of layers for a given application: the insertion failure rate and its logical derivative, the probability of failure of any given insertion attempt. A simulator was set up in order to characterize the failure probability as a function of the relevant design parameters. It is important to note that the search keys were generated by using the rand routine from Gnu’s GLIBC. This situation is equivalent to having either uncorrelated traffic or using ideal hashing

312

P. Mahoney et al.

functions, both yielding perfectly distributed search keys. Also, the performance and behaviour of the proposed architecture depend on several key variables. We thus define L to be the number of layers, α, the load factor, C, the total architecture memory capacity and τ , the insertion failure rate. The probability of failure of any given insertion attempt is defined as the probability that on a given insertion, in every layer, the row located at the address associated with the search key being inserted is occupied. This probability is bound to grow as the number of stored elements increases. It also depends on the number of layers, L, and on the total architecture capacity, C. Fig. 2 illustrates the relationships between performance and these variables and parameters. The graph also characterizes the system behavior when the number of insertion attempts is larger than the capacity. Indeed, as insertion attempts may fail before the capacity is reached, some entries are empty when the number of insertion attempts reaches that capacity. Then, further insertion attempts may be successful, even if the number of insertion attempts is greater than capacity.

1

Probability of failure

0.8

0.6

0.4

0.2

L=1 L=2 L=4 L=8 L=16

0 0

0.5

1 1.5 Insertion attempt index / Architecture capacity

2

2.5

Fig. 2. Sigmoid behaviour of the probability of failure as the number of insertion attempts varies

Fig. 2 shows that as L increases, the sigmoid relation allowing to identify the probability of failure for any given number of insertion gets sharper. In other words, as the number of layers increases, the architecture behaves more and more like a CAM unit. It is also worthwhile to note that this statement holds true even as L approaches C, where each RAM unit size gets closer to unity. In fact, a completely degenerated configuration where each RAM unit is of unity size, and where L equals C, can actually be seen as a non-optimized CAM unit implementation. With this information, designers can fine tune the architecture configuration so as to obtain the sigmoid relation that best fits the application needs. For example, increasing the number of layers allows the application to execute a

Performance Characterization for the Implementation of CAMs

313

1

Probability of Failure (τ)

0.8

0.6

0.4

0.2

emulated CAM α = 1/2.8 α = 1/2

0 0.5

1

1.5 2 2.5 3 Insertion attempt index / Capacity of emulated CAM

3.5

4

Fig. 3. Performance comparison with CAM at L=8 and α = 1/2.8 (cost per bit equivalent) and α = 1/2

0.25 Displayed valued of α: α = 1.00 α = 0.90 α = 0.85 α = 0.80

Failure Rate (τ)

0.2

0.15

0.1

0.05

0 2

4

6

8

10

12

14

16

18

20

Number of Layers (L)

Fig. 4. τ vs L for several α values with constant total architecture capacity

greater number of insertion attempts before reaching a given failure probability. As for the load factor, if the application possesses a soft upper-limit on the number of entries that it needs to store, decreasing the ratio between the number of insertion attempts and the capacity can be seen as shifting the sigmoid relation to the right. Fig. 3 compares behaviours of a CAM unit and of the proposed architecture configured with two different load factor parameters. Thus, as mentioned earlier, unlike the CAM unit, the proposed architecture can keep inserting new entries with a significant probability of success, even if the application goes beyond the target load factor. Note that a ratio of 2.8 in CAM to RAM capacities corresponds to the respective cost listed in Table 1.

314

P. Mahoney et al.

The insertion failure rate for a given application is defined as being the number of failed insertion attempts encountered while the application was executing, over the number of distinct entries for which an insertion was attempted. This characteristic depends on both the number of layers and the load factor. As illustrated in Fig. 4, the failure rate decreases in an exponential manner as the number of layers increases while keeping constant load factor values. Fig. 5 presents that same behaviour from a different angle, as the load factor varies for several values of number of layers. Fig. 6 combines L and α pairs yielding the same insertion failure rates. It allows rapidly identifying all architecture configurations yielding some predefined behavior. Fig. 6 also suggests that failure rates of any magnitude can be 0.25 L=2 L=4 L=8 L=16 L=32

Failure Rate (τ)

0.2

0.15

0.1

0.05

0 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Load Factor (α)

Fig. 5. τ vs α for several L values with constant total architecture capacity 1 0.9 0.8

Load Factor (α)

0.7 0.6 0.5 0.4 0.3 Displayed values of τ: 0.2 τ=1e-2 -3 τ=1e τ=1e-4

0.1 0 2

4

6

8

10

12

14

16

Number of Layers (L)

Fig. 6. L,α pairs yielding to constant τ values

18

Performance Characterization for the Implementation of CAMs

315

obtained by properly configuring the architecture. All applications can hence take advantage of the architecture by first identifying an insertion failure rate which it expects to support, and then choosing the L, α pair having the lowest production cost among those yielding the target behavior. For example, a rate of 1 failure in every 10 years of execution time could be seen as being negligible for many applications. Such a value combined with a hypothetical 100 MHz operating frequency would translate into an overall failure rate of 3.171×10−16. Unfortunately, prohibitive simulation times makes it difficult to characterize configurations producing rates of this order of magnitude. An analytical model is therefore needed to fill the gap.

5

Analytical Modeling

The insertion operations applied on the proposed architecture can be modeled with the balls and urns problem if ideal hash functions are used. Let there be n balls and m urns. Let event E be the insertion of a ball into an urn chosen in a perfectly random manner. The probability p that a given urn is chosen for a given insertion is: p=

1 m

(1)

If E is repeated for all the n balls, the probability that a given urn contains k balls is given by the probability mass function (pmf) equation of the binomial random variable:    k  n−k 1 1 n 1− Pk = k m m

(2)

The pmf of the Poisson random variable is known to be a good approximation of the binomial’s pmf when n is large and p is small. If we let β = n/m, equation 2 can be rewritten as: Pk =

β k −β e k!

(3)

Let the balls be the elements an application tries to insert into a hash table using an ideal hashing function, and let the urns be the rows of that table. In this case, β represents the load factor, and previously defined variable α can be used instead. Let Olocal (α) be the local occupancy relation, allowing one to obtain the occupancy rate of the hash table of a given layer. Since a given row will be occupied if the hash function assigns at least one element to it, O(α) is given by: m

Olocal (α) =

1  1 − P0 = 1 − P0 = 1 − e−α m 1

(4)

316

P. Mahoney et al.

It is also worthwhile to note that the insertion of an element into a hash table possesses a probability of failure equal to the hash table’s occupancy rate. Therefore, equation 4 also defines the failure probability for a single layer configuration as shown in Fig 2 for L = 1. In order to analytically express probability of failure and failure rate behaviours for all configurations, it is necessary to model multi layer units as systems composed of several single layer units. For this purpose, let T = C/L be the number of rows in the RAM unit of each layer, Oi , αi , and ni respectively be the occupancy rate, the load factor and the number of insertion attempts of the ith layer. We therefore have: Oi = Olocal (αi )

(5)

αi = Ni /T

(6)

When an insertion attempt made on a given layer fails, the attempt percolates to the next one. Thus the number of insertion attempts made in a given layer is equal to the number of insertion attempts that failed on the previous one, as Fig. 7 shows. Conversely, the number of insertion attempts made in a given layer is equal to the total number of insertion attempts made in the previous layer minus the number that succeeded. ni = Ni−1 − Oi−1 × T

(7)

Relations 7, 5 and 6 allow to express the global occupancy rate of a multi layered configuration as the sum of the occupancies of all the layers. L

Oglobal (α) =

1 Oi L i=1

(8)

The failure rate can be obtained by normalizing the part of the load factor that corresponds to failed insertion attempts by the load factor itself. τ=

α − Oglobal (α) α

(9)

The failure probability depends on the state of the architecture at the moment when the insertion is made. In multi layered configurations, this value represents the probability that the inserted element collides in every single layer. It is thus given by the product of the individual occupancy rates of every layer. Pf ailure =



Oi

(10)

Relations 9 and 10 allow analytically defining the behaviour of the architecture. They not only validate our simulation results as we were able to analytically reproduce Figs. 4 and 5, but they also allow quantifying the behaviour in domains located beyond the practical limits imposed by simulation times that can become excessive.

Performance Characterization for the Implementation of CAMs

317

Fig. 7. If the insertion attempt of an element in a layer causes a collision, this element percolates to the next layer Table 2. Six layers are necessary to obtain a value of τ = 10−16 , N1 =4000 Layer 1 2 3 4 5 6

αi 1.500 0.7231 0.2083 0.0202 2.041E-4 2.083E-8

Oi 0.7768 0.5147 0.1880 0.0200 2.041E-4 2.083E-8

Oglobal τ ni+1 0.1294 0.4820 1928.3 0.2152 0.1389 555.6 0.2466 0.0135 54.066 0.2499 1.361E-4 0.5444 0.2499 1.389E-8 5.556E-5 0.2499 2.220E-16 5.921E-13

For example, an application might need to store data up to a soft limit of 4000 elements. Its designer might judge a rate of 1 failure in every 10 years of execution time as being negligible, thus yielding to a τ value of 3.171 × 10−16 , as previously mentioned. Table 2 shows that a α = 0.25, L = 6 configuration offers the targeted performance. The design would thus have a total capacity of C = 4000/0.25 = 16000 elements divided into 6 RAM units of 16000/6 rows each. Failure Rate (τ)

0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 01 0.8 0.6 Load Factor (α)

0.4 0.2 0 16

14

12

4 6 8 10 Number of Layers (L)

2

Fig. 8. Joined τ vs R and τ vs α relations

318

P. Mahoney et al.

The analytical model allows plotting the surface illustrating the behaviour of τ with respect to parameters L and α, as shown in Fig. 8. It also allows extending the characterized domain to failure rates beyond the limits feasible with affordable simulation times. Fig 9 shows L, α pairs yielding to same failure rates. It is worthwhile noting that these relations are in fact iso-performance curves extracted from the surface of Fig. 8.

6

Proposed Design Methodology

One problem that has yet to be approached resides in the choice of the L, α pair to use for a given application. One possible methodology consists in identifying the target behaviour and then choosing the L, α pair leading to the lowest production cost among those yielding the desired behaviour. One technique suitable for the graphically inclined designer consists in first elaborating a cost model through a surface identifying the die area or cost in dollars associated with every L, α pair. Next, the technique consists in finding the parametric relation of the 3D curve resulting from the projection of the α vs L with constant target τ relation as found in Fig. 9 on the cost surface. This parametric relation defines the cost through a single parameter t from which both L and α can be obtained. The last step is to find the t value for which the lowest cost is observed. The resulting L and α values can then be considered as the best possible solution. 1.2

1

Load Factor (α)

0.8

0.6

0.4 Displayed values of τ: τ=1e1 τ=1e-2 τ=1e-5 τ=1e-10 τ=1e-15

0.2

0 2

4

6

8 10 Number of Layers (L)

12

14

16

Fig. 9. α/L relations yielding to same τ

For example, let the cost model be L2 + (1/α)2 as shown in Fig. 10, and the target failure rate be 10−15 . This cost model is only proposed for illustrative purposes. In order to make the projection, the α vs L with constant target τ = 10−15 relation needs to be modeled. Fig. 11 shows that least squares fitting

Performance Characterization for the Implementation of CAMs

319

Cost

200 180 160 140 120 100 80 60 40 20 0 10 8 6 Number of Layers (L)

4 2 0 1

0.9

0.8

0.7

0.5

0.6

0.4

0.3

0.2

0.1

Load Factor (α)

Fig. 10. Cost Model

finds relation 11 with parameters a = 9.60, b = 10.21 and c = 0.051 to be the best fit. This relation projected on the cost surface model yields the relation shown in Fig. 12 and is minimal at integer value t = 6. Parameters L = 6 and α = 0.286 are obtained by applying relations 12 and 13. Based on the cost surface model of Fig. 10, these values yield a cost of 62 + (1/6)2 = 48.2. f (x) = a − (

b ) xc

L(t) = t α(t) = a − (

7

(11) (12)

b ) tc

(13)

Hardware Implementation and Metric Comparison

A VLSI implementation of the proposed architecture has been realized in order to obtain a more thorough understanding of how it could compare to commercial CAMs. The original implementation consists in synchronous single port RAM units glued together with combinational logic described with register transfer level VHDL. It is a multi-cycle design where read operations consume one cycle and where write operations consume two. This implementation is dominated by RAMs and requires minimal glue logic. Table 3 presents area consumption data for a 4-layer implementation with a 1024x52 RAM. It shows that whether the synthesis tries to minimize either clock period or die area, glue logic area remains small compared to the one consumed by the RAM cells. The proposed architecture hence greatly benefits from the high density of RAM technology while emulating a CAM behaviour.

320

P. Mahoney et al.

0.8

0.7

0.6

Load Factor (α)

0.5

0.4

0.3

0.2

0.1

0

-0.1 2

4

6

8 10 Number of Layers (L)

12

14

16

Fig. 11. Model of α vs L relation for τ = 10−15 140 130 120 110

Cost

100 90 80 70 60 50 40 4

5

6

7 parameter t

8

9

10

Fig. 12. Minimal cost configuration for τ = 10−15 Table 3. Bringing the clock period from over 25ns down to under 5ns does not affect significantly the portion of the total area occupied by the RAMs. 180nm technology. Clock Period (ns) 26.98 9.48 4.35 4.27 4.24

Total Combinational RAM Area Area Portion (sq µm) (sq µm) 1,699,135 22,955 98.65% 1,699,518 23,338 98.63% 1,704,266 28,086 98.35% 1,707,043 30,863 98.19% 1,708,629 32,449 98.10%

Performance Characterization for the Implementation of CAMs

321

Table 3 also shows that a clock frequency greater than 200 MHz can be reached. This value could be easily increased by using faster RAM cells, as 2.8 ns of the 4.24ns clock period is consumed by the RAM access time. It is worthwhile noting that in order to obtain easily reproducible results, the implementation uses several standard modules from Synopsys’ DesignWare library. One of them is the combinational CRC which is used for the hash function blocks. Synopsys’ Design Compiler was also used to obtain both latency and die area estimates. As for power consumption of the logic, a tool developed by our industrial partner was used to produce the listed estimates. These estimates are based on the number of gates and on technology specific constraints. Table 4 presents a comparison of the design with a commercial CAM. It shows that for same total capacity, the parallel hashing memories consume 2.4 times less die area and 4.7 times less power than commercial CAM. As expected, these values are slightly lower than the ones presented in table 1, yet they still offer an appealing alternative to CAM technology free of any royalty fee. Table 4. Comparisons of proposed Parallel Hashing (PH) design with commercial CAMs. All values are for 180nm implementations running at 200MHz with a RAM access time of 2.8 ns. Parallel Portion CAM CAM/PH Hashing for RAM Factor Cost (mm2 /M bit) Power Consumption (mW/M bit)

8

8.02

98.10%

19.2

2.4

309

68.8%

1454

4.7

Proposed Optimisations

This section proposes three possible variations which can be brought to the design in order to optimize either power consumption, throughput or latency. The first of them consists in using dual port RAM cells in order to support both lookup and insertion operations in a single cycle. Eliminating multi-cycle operations greatly simplifies interactions with the design, but on the other hand, it also considerably increases the die area consumed by the cell. In fact, dual port RAM cells can occupy close to twice the area of their single port counterpart of equal size. With dual port RAM cells, one port can be dedicated solely to read operations, while the other takes care of write operations. It is hence possible to have the writing portion of insertion operations and the read portion of the following operation overlap each other, whether it is a lookup or an insertion. The writing portion of insertion operations is completely hidden to the user. The cell thus appears as completing every operation within a single cycle, as shown in Fig. 13. Since dual port RAM cells do not support simultaneous access to a same memory location, special wrapping logic is necessary to ensure correct behaviour

322

P. Mahoney et al.

Fig. 13. Dual Port RAM design optimisation. Operations 2 and 4 represent insertions.

when two or more consecutive operations are executed with the same RAM address. It basically consists in bypassing the read operation by presenting the data being written to the port wrapper where the read operation occurs. The first consequence of the described design variation would be to bring the latency down to a single cycle. As for die area, since RAM cells represent around 98% of the initial implementation’s consumption, the area could double. Finally, based on the 15% increase in power consumption of dual port RAMs over their single port counterparts and on the addition of the bypass logic, the increase in power consumption can be estimated as being close to 15%. The second design variation offers an increase in throughput. It consists in pipelining the architecture so as to allow a greater number of operations to be completed in a given time unit. The pipeline could possess 3 stages. The first stage would essentially allow the hashing function unit to translate the search keys into RAM addresses. The second would consist in a RAM access. The third and last one would allow post-processing, such as gating the payload field to the output pins, to take place. Since the longest atomic action of the initial design is the RAM memory access, the clock period of the pipelined design would be defined by the RAM latency. Available RAM specification sheets define this value to be 2.8 ns for a 180nm CMOS technology, hence the pipelined design could run at a 350MHz clock frequency. This design variation, combined with the dual-port RAM presented earlier, could allow the presented architecture to fully complete 350 × 106 operations per second, whether they are lookup or insertion operations. The third and last design variation offers a decrease in power consumption at the expense of a greater average latency. Because of the percolation process described in section III, the highest indexed layers fill up faster than the lower indexed ones. In fact, relation 7 shows that a given layer cannot possess a number of elements greater than any higher indexed layer. In other words, Ni ≤ Nj if i < j. Based on this rule, it is safe to assume that read operations executed on a given layer possess a greater probability of success than the ones executed on any of the lower indexed layers. This holds true particularly if the load factor is small, as illustrated in Fig. 14. This design variation aims at taking advantage of this characteristic by sequentially accessing groups of layers until a match is found. It offers a decrease in power consumption as lookup operations will not require accessing every RAM cell at all times. As for the increase of latency, its average value depends on both the group size and the load factor, while its worst case value equals the number of groups in the design.

Performance Characterization for the Implementation of CAMs

323

1 Displayed valued of α: α = 0.25 α = 0.40 α = 0.75

0.9 0.8

Local Occupancy Rate

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 1

2

3

4

5

6

Layer Index

Fig. 14. The occupancy rate of the individual layers decreases as the layer index increases

For example, let us reuse the 6 layer, 0.25 load factor configuration of Table 2 and apply the current design variation by grouping the layers in groups of 2. Based on the local occupancy rate of each layer, the first group can be shown as possessing 86.1% of all stored elements, while the last two would possess respectively 13.9% and 0.082%. A simple weighted sum leads one to identify the average latency of lookup operations executed on stored elements as being of 1.385 cycles. In a similar way, the decrease in RAM cell power consumption can be evaluated as being close to 62%. Even if neglecting power consumption of the additional logic required for this design variation, it is clear that many applications could benefit from this kind of tradeoff.

9

Conclusions

An architecture able to emulate CAM technology units was presented. It offers notable improvements on power consumption and die area metrics in addition to avoiding the high licensing fees charged by some CAM vendors. Because of its gracefully degradable performances, it can also best fit the needs of applications that do not possess hard upper limits on the number of items to store and whose designers currently see CAM technology as being their only choice. The performances of the presented architecture obtained in simulations were backed by an analytical model which allows to identify the configurations with low failure rates. A method to identify optimal configuration parameters was also presented, hence handing to the reader the tools he needs in order to start designing efficiently. Several design variations were also presented. The first consists in using dual port RAMs so as to have both lookup and insertion operations completed in a

324

P. Mahoney et al.

single cycle. The second pipelines the design in three stages in order to boost the throughput, and to bring the clock period down to the value of the RAM access time. The third and last one proposes to sequentially compare groups of layers until the sought after item is found. Unaccessed RAM cells being disabled, power consumption gets reduced at the expense of an increased average latency. Both simulation results and analytically obtained performance metrics assume perfect hashing functions. Even though CRC algorithms are considered as being the best available hashing functions [15], not much work has been done on the topic. Several papers offer analysis on the behaviour of CRC algorithms with respect to their ability to detect and repair corrupted data, yet none of them discuss their effectiveness to offer well dispersed hash keys. This kind of analysis might be necessary before bringing the proposed architecture in the field. Because of its simple design and of the widely spread use of the required tools, the presented architecture offers a very appealing alternative to CAM technology. Its performance metrics are also bound to improve over time, as CAM technology is not subject to the very intense commercial competition found in the RAM market and thus does not evolve as fast. Many applications like address lookups, cache tags and pattern recognition could greatly benefit from using it.

Acknowledgments The authors of this paper would like to acknowledge financial contributions from Micronet R&D, the Canadian Microelectronics Corporation and PMC-Sierra, Inc. They would also like to thank Benoit Cˆ ot´e, Martin Bisson and Normand B´elanger for their respective contributions to the project.

References 1. Peng, M., Azgomi, S.: Content-addressable memory (cam) and its network applications. In: International IC–Taipei Conference proceedings, pp. 1–3 (2001), http://www.eetasia.com/ 2. Mahoney, P., Savaria, Y., Bois, G., Plante, P.: Parallel hashing memories: an alternative to content addressable memories, pp. 223–226 (2005) 3. Natarajan, A., Jasinski, D., Burleson, W., Tessier, R.: A hybrid adiabatic content addressable memory for ultra low-power applications. In: ACM Great Lakes Symposium on VLSI, pp. 72–75 (2003) 4. Efthymiou, A., Garside, J.D.: An adaptive serial-parallel cam architecture for lowpower cache blocks. In: Proceedings of the 2002 international symposium on Low power electronics and design, pp. 136–141 (2002) 5. Pagiamtzis, K., Sheikholeslami, A.: A low-power content-addressable memory (CAM) using pipelined hierarchical search scheme. IEEE Journal of Solid-State Circuits 39(9), 1512–1519 (2004) 6. Delgado-Frias, J.G., Nyathi, J., Tatapudi, S.B.: Decoupled dynamic ternary content addressable memories. IEEE Transactions on Circuits and Systems 52(10), 2139– 2147 (2005)

Performance Characterization for the Implementation of CAMs

325

7. Liu, S.C., Wu, F.A., Kuo, J.B.: A novel low-voltage content-addressable-memory (cam) cell with a fast tag-compare capability using partially depleted (pd) soi cmos dynamic-threshold (dtmos) techniques. IEEE Journal of Solid-State Circuits 36(4), 712–716 (2001) 8. Lin, K., Wu, C.: A low-power cam design for lz data compression. IEEE Transactions on Computers 49(10), 1139–1145 (2000) 9. Zukowski, C.A., Wang, S.Y.: Use of selective precharge for low-power on the match lines of content-addressable memories. In: MTDT 1997: Proceedings of the 1997 IEEE International Workshop on Memory Technology, Design and Testing, Washington, DC, USA, pp. 64–68. IEEE Computer Society, Los Alamitos (1997) 10. Lin, C., Chan, J.: A low-power precomputation-based fully parallel contentaddressable memory. IEEE Journal of Solid-State Circuits 38(4), 654–662 (2003) 11. Broder, A., Karlin, A.: Multilevel adaptive hashing. In: Proceedings of the first annual ACM-SIAM symposium on Discrete algorithms, San Francisco, California, United States, pp. 43–53. ACM, New York (1990) 12. Lim, H., Seo, J., Jung, Y.: High speed ip address lookup architecture using hashing. IEEE Communications Letters 7(10), 502–504 (2003) 13. Lim, H., Jung, Y.: A parallel multiple hashing architecture for ip address lookup. In: Workshop on High Performance Switching and Routing, HPSR 2004, pp. 91–95 (2004) 14. Seznec, A.: A case for two-way skewed-associative caches. In: International Symposium on Computer Architecture, pp. 169–178 (1993) 15. Jain, R.: A comparison of hashing schemes for address lookup in computer networks. IEEE Transactions on Communications 40(10), 1570–1573 (1992)

Author Index

Abderazek, Ben 269 Aggarwal, Aneesh 201 Ahn, Minwook 149

Mahoney, Patrick 307 McKee, Sally A. 65 Morancho, Enric 173

Bhadauria, Major Bois, Guy 307

Nagarajan, Vijay 23 Navarro, Nacho 173 Niar, Smail 286

65

Cabezas, Javier 173 Canedo, Arquimedes 269 Chanet, Dominique 173 Choi, Woojin 107 De Bosschere, Koen 173 Dubois, Michel 107 Eeckhout, Lieven

45

Golander, Amit 242 Gupta, Rajiv 23 Hu, Chunling

85

Ibrahim, Khaled Z.

286

Jim´enez, Daniel A.

85

Kaxiras, Stefanos 4 Keramidas, Georgios 4 Kluyskens, Simon 45 Kremer, Ulrich 85 Krishnaswamy, Arvind 23

Paek, Yunheung 149 Park, Seok-Jun 107 Plante, Patrice 307 Rochange, Christine

222

Sainrat, Pascal 222 Savaria, Yvon 307 Seznec, Andr´e 128 Singh, Karan 65 Sowa, Masahiro 269 Stenstr¨ om, Per 3 Tyson, Gary S.

65

Vandierendonck, Hans

128

Weiss, Shlomo 242 Whalley, David 3 Xekalakis, Polychronis

4