Storage Networking Protocol Fundamentals 2003108300, 1587051605, 9781587051609

Citation preview

Storage Networking Protocol Fundamentals James Long

Cisco Press 800 East 96th Street Indianapolis, Indiana 46240 USA

ii

Storage Networking Protocol Fundamentals James Long Copyright © 2006 Cisco Systems, Inc. Published by: Cisco Press 800 East 96th Street Indianapolis, IN 46240 USA All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the publisher, except for the inclusion of brief quotations in a review. Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 First Printing

May 2006

Library of Congress Cataloging-in-Publication Number: 2003108300 ISBN: 1-58705-160-5

Trademark Acknowledgments All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Cisco Press or Cisco Systems, Inc. cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.

Warning and Disclaimer This book is designed to provide information about storage networking protocols. Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information is provided on an “as is” basis. The authors, Cisco Press, and Cisco Systems, Inc. shall have neither liability nor responsibility to any person or entity with respect to any loss or damages arising from the information contained in this book or from the use of the discs or programs that may accompany it. The opinions expressed in this book belong to the author and are not necessarily those of Cisco Systems, Inc.

Corporate and Government Sales Cisco Press offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales. For more information please contact: U.S. Corporate and Government Sales 1-800-382-3419 [email protected] For sales outside the U.S. please contact: International Sales [email protected]

Feedback Information At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with care and precision, undergoing rigorous development that involves the unique expertise of members from the professional technical community. Readers’ feedback is a natural continuation of this process. If you have any comments regarding how we could improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us through email at [email protected]. Please make sure to include the book title and ISBN in your message. We greatly appreciate your assistance.

iii

Publisher Cisco Representative Cisco Press Program Manager Executive Editor Production Manager Development Editor Project Editor Copy Editor Technical Editors Book and Cover Designer Composition Indexer

Paul Boger Anthony Wolfenden Jeff Brady Mary Beth Ray Patrick Kanouse Andrew Cupp Interactive Composition Corporation Interactive Composition Corporation Philip Lowden, Thomas Nosella, Rob Peglar Louisa Adair Interactive Composition Corporation Tim Wright

iv

About the Author James Long is a storage networking systems engineer who works for Cisco Systems, Inc., in the Field Sales Organization. James previously held the position of Global Storage Networking Solutions Architect within the Cisco IT Infrastructure Architecture team. During his tenure in the Cisco IT organization, James authored design guides, technology evaluations, and strategic planning documents. Before joining Cisco in 1999, James contracted with AT&T/TCI, Nextel International, and GTE following five years of employment in the open systems VAR community. James has more than 16 years of IT experience spanning server administration, database administration, software development, multiprotocol network design and administration, remote access solutions design and administration, IP telephony and IP contact center design, content distribution design, storage network design, and advanced technology evaluation. James holds numerous technical certifications from Cisco, Microsoft, Novell, SNIA, and CompTIA.

About the Technical Reviewers Philip Lowden currently works as a storage manager at Cisco Systems, Inc. Prior to this role, he worked for four years at Cisco and six years at Electronic Data Systems as a senior UNIX systems administrator performing production systems architecture and support duties on a variety of host and storage platforms. He was also an officer in the U.S. Air Force for six years. He holds a Masters of Science degree in computer engineering from North Carolina State University, a Bachelor of Science degree in computer science from the University of Nebraska, and a Bachelor of Art degree in English from Saint Meinrad College. He is an SNIA-certified FC-SAN Specialist. Philip is married and has two children. Thomas Nosella, CCIE No. 1395, is director of engineering within the Cisco Systems Data Center Switching Business Unit, an organization responsible for LAN, server fabric, and storage switching products and solutions. Thomas and his team of technical marketing engineers are responsible for the creation, validation, and promotion of intelligent and scalable designs and solutions for the Ciscos enterprise and service provider customer base. Thomas was one of the initial members of Andiamo Systems, Inc., and helped bring the Cisco MDS 9000 family of SAN switching products to market. Prior to working on storage, Thomas managed enterprise design teams focused on large-scale Ethernet design, server farm design, and content delivery networking. Thomas received his Bachelor of Engineering and Management from McMaster University in Ontario. Thomas received his CCIE certification in 1995. Rob Peglar is vice president of Technology, Marketing for Xiotech Corporation. A 28-year industry veteran and published author, he has global responsibility for the shaping and delivery of strategic marketing, emerging technologies, and defining Xiotech’s product and solution portfolio, including business and technology requirements, marketing direction, planning, execution, technology futures, strategic direction, and industry/customer liaison. Rob serves on the Board of Directors for the Blade Systems Alliance and is the co-author and track chair of the SNIA Virtualization Tutorial. He has extensive experience in the architecture, design, implementation, and operation of large heterogeneous SANs, distributed clustered virtual storage architectures, data management, disaster avoidance, and compliance, and is a sought-after speaker and panelist at leading storage and networking-related seminars and conferences worldwide. He holds the B.S. degree in computer science from Washington University, St. Louis, Missouri and performed graduate work at Washington University’s Sever Institute of Engineering. His research background includes I/O performance analysis, queuing theory, parallel systems architecture and OS design, storage networking protocols, and virtual systems optimization.

v

Dedication This book is posthumously dedicated to Don Jones. Don was a good man, a good friend, and a good mentor.

vi

Acknowledgments The quality of this book is directly attributable to the many people that assisted during the writing process. In particular, I would like to thank Mike Blair for his contribution to the SBCCS/ESCON/FICON section, Tom Burgee for his contribution to the optical section, Joel Christner for his contribution to the file-level protocols section, and Alan Conley for his contribution to the management protocols chapter. Additionally, I would like to thank Tuqiang Cao, Mike Frase, and Mark Bakke for their support. A special thank you goes to Tom Nosella, Phil Lowden, and Robert Peglar for serving as technical reviewers. Finally, I am very grateful to Henry White for hiring me at Cisco. Without Henry’s confidence in my potential, this book would not have been possible.

vii

This Book Is Safari Enabled The Safari® Enabled icon on the cover of your favorite technology book means the book is available through Safari Bookshelf. When you buy this book, you get free access to the online edition for 45 days. Safari Bookshelf is an electronic reference library that lets you easily search thousands of technical books, find code samples, download chapters, and access technical information whenever and wherever you need it. To gain 45-day Safari Enabled access to this book: • Go to http://www.ciscopress.com/safarienabled • Complete the brief registration form • Enter the coupon code YNQU-PVJQ-4NCS-65PJ-4EA9 If you have difficulty registering on Safari Bookshelf or accessing the online edition, please e-mail [email protected].

viii

Contents at a Glance Foreword

xxi

Introduction

xxii

Part I

Storage Networking Landscape

Chapter 1

Overview of Storage Networking

Chapter 2

OSI Reference Model Versus Other Network Models

Chapter 3

Overview of Network Operating Principles

Chapter 4

Overview of Modern SCSI Networking Protocols

Part II

OSI Layers

Chapter 5

OSI Physical and Data-Link Layers

Chapter 6

OSI Network Layer

Chapter 7

OSI Transport Layer

Chapter 8

OSI Session, Presentation, and Application Layers

Part III

Advanced Network Functionality

349

Chapter 9

Flow Control and Quality of Service

351

Chapter 10

Routing and Switching Protocols

Chapter 11

Load Balancing

Chapter 12

Storage Network Security

Chapter 13

Storage Management Protocols

Chapter 14

Protocol Decoding and Traffic Analysis

Part IV

3 5

367

377 385 395

419

Appendix B

Acronyms and Abbreviations

441

Appendix C

Answers to Review Questions

499

109

217

Standards and Specifications

Index

93

193

Appendix A

479

55

107

417

Glossary

39

463

405

245

ix

Table of Contents Foreword

xxi

Introduction Part I

xxii

Storage Networking Landscape

3

Chapter 1 Overview of Storage Networking

5

Brief History of Storage 5 Intelligent Interface Models 6 Serial Transmission Techniques 6 Modern Storage Networks 7 Drivers for Change

7

What Is a Storage Network?

9

Block Storage Protocol Review: ATA, SCSI, and SBCCS 11 ATA 12 SCSI 13 SBCCS 16 Mainframe Storage Networking: ESCON and FICON 17 ESCON 17 FICON 18 File Server Protocol Review: CIFS, NFS, and DAFS 20 CIFS 20 NFS 22 DAFS 22 Backup Protocols: NDMP and EXTENDED COPY 23 NDMP 23 SCSI-3 EXTENDED COPY 24 Optical Technologies: SONET/SDH, DWDM/CWDM, and RPR/802.17 SONET/SDH 25 DWDM/CWDM 27 RPR/802.17 31

25

Virtualization Implementations: Host, Storage Subsystem, and Network Host Implementations 34 Storage Subsystem Implementations 35 Network Implementations 35

32

Summary

36

Review Questions 37

x

Chapter 2 OSI Reference Model Versus Other Network Models

39

OSI Reference Model 39 Network Models, Specifications, and Implementations 39 Seven-Layer Model 40 Layer 1—Physical Layer 41 Layer 2—Data-Link Layer 42 Layer 3—Network Layer 42 Layer 4—Transport Layer 43 Layer 5—Session Layer 44 Layer 6—Presentation Layer 44 Layer 7—Application Layer 44 Implementation Considerations 44 SCSI Bus Interface and the ANSI T10 SCSI-3 Architecture Model Ethernet and the IEEE 802 Reference Model TCP/IP Suite and the ARPANET Model

47

48

Fibre Channel Architecture and ANSI T11 Model 49 Fibre Channel Model and the OSI Reference Model Fibre Channel Specifications 50 Summary

52

Review Questions 53 Chapter 3 Overview of Network Operating Principles Conceptual Underpinnings 55 Throughput 55 Topologies 56 Service and Device Discovery

55

61

SCSI Parallel Interface 62 The Right Tool for the Job 63 SPI Throughput 63 SPI Topologies 65 SPI Service and Device Discovery

66

Ethernet 66 Low Overhead Paradigm 67 Ethernet Throughput 70 Ethernet Topologies 72 Ethernet Service and Device Discovery TCP/IP Suite 73 Value of Ubiquitous Connectivity TCP/IP Throughput 75

74

73

50

45

xi

TCP/IP Topologies 77 TCP/IP Service and Device Discovery Discovery Contexts 78

78

Fibre Channel 84 Merger of Channels and Packet Switching FC Throughput 85 FC Topologies 87 FC Service and Device Discovery 89 Summary

84

90

Review Questions 90 Chapter 4 Overview of Modern SCSI Networking Protocols iSCSI 93 iSCSI Functional Overview 93 iSCSI Procedural Overview 96 FCP 97 FCP Functional Overview 97 FCP Procedural Overview 99 FCIP 100 FCIP Functional Overview 100 FCIP Procedural Overview 101 iFCP 102 iFCP Functional Overview 102 iFCP Procedural Overview 104 Summary

105

Review Questions 105 Part II

OSI Layers

107

Chapter 5 OSI Physical and Data-Link Layers

109

Conceptual Underpinnings 109 Addressing Schemes 109 Address Formats 114 Delivery Mechanisms 116 Dropped Frames or Packets 118 Duplicate Frames or Packets 118 Corrupt Frames or Packets 119 Acknowledgement 119 Guaranteed Delivery 120

93

xii

Flow Control and QoS 120 Guaranteed Bandwidth 121 Guaranteed Latency 121 Fragmentation, Reassembly, and PMTU Discovery In-order Delivery 122 Link Aggregation 125 Transceivers 125

121

SCSI Parallel Interface 126 SPI Media, Connectors, Transceivers, and Operating Ranges 126 SPI Encoding and Signaling 128 SPI Addressing Scheme 128 SPI Name Assignment and Resolution 128 SPI Address Assignment and Resolution 128 SPI Media Access 129 SPI Network Boundaries 130 SPI Frame Formats 130 SPI Delivery Mechanisms 131 SPI Link Aggregation 131 SPI Link Initialization 132 Ethernet 132 Ethernet Media, Connectors, Transceivers, and Operating Ranges 132 Ethernet Encoding and Signaling 135 Ethernet Addressing Scheme 137 Ethernet Name Assignment and Resolution 137 Ethernet Address Assignment and Resolution 138 Ethernet Media Access 138 Ethernet Network Boundaries 139 Ethernet Frame Formats 140 Ethernet Delivery Mechanisms 144 Ethernet Link Aggregation 145 Ethernet Link Initialization 146 Fibre Channel 150 FC Media, Connectors, Transceivers, and Operating Ranges 150 FC Encoding and Signaling 154 FC Addressing Scheme 156 FC Name Assignment and Resolution 161 FC Address Assignment and Resolution 162 FC Media Access 167 FC Network Boundaries 167 FC Frame Formats 168 FC Delivery Mechanisms 173 FC Link Aggregation 177 FC Link Initialization 177

xiii

Summary

189

Review Questions 190 Chapter 6 OSI Network Layer

193

Internet Protocol 193 IPv4 Overview 193 Data-Link Support 195 Ethernet 195 PPP 196 Addressing Scheme 197 Name Assignment and Resolution 202 Address Assignment and Resolution 204 Network Boundaries 205 Packet Formats 207 Delivery Mechanisms 209 ICMP 211 Interface and Port Initialization 214 Fibre Channel Summary

214

214

Review Questions 214 Chapter 7 OSI Transport Layer

217

TCP/IP Suite 217 UDP 217 UDP Operational Overview 217 UDP Packet Formats 219 UDP Delivery Mechanisms 220 UDP Connection Initialization 221 TCP 221 TCP Operational Overview 221 TCP Packet Formats 223 TCP Options 226 TCP Delivery Mechanisms 232 TCP Connection Initialization 234 Fibre Channel 236 FC Operational Overview 236 FC Frame Formats 238 FC Delivery Mechanisms 240 FC Connection Initialization 241 Summary

242

Review Questions 242

xiv

Chapter 8 OSI Session, Presentation, and Application Layers iSCSI Operational Details 245 iSCSI Addressing Scheme 245 iSCSI Name Assignment and Resolution 250 iSCSI Address Assignment and Resolution 251 iSCSI Session Types, Phases, and Stages 251 iSCSI Data Transfer Optimizations 252 iSCSI PDU Formats 254 iSCSI Login Parameters 282 iSCSI Delivery Mechanisms 287 Error Recovery Classes 287 Error Recovery Hierarchy 288 PDU Boundary Detection 289 PDU Retransmission 289 iSCSI In-Order Command Delivery 291 iSCSI Connection and Session Recovery 292 FCP Operational Details 293 FCP Addressing Scheme 293 FCP Name Assignment and Resolution 293 FCP Address Assignment and Resolution 294 FCP Session Establishment 294 FCP Data Transfer Optimizations 295 FCP IU Formats 295 FCP Additional Login Parameters 319 FCP Delivery Mechanisms 320 Error Detection 320 Exchange Level Error Recovery 321 Sequence Level Error Recovery 321 FCP IU Boundary Detection 322 FCP In-Order Command Delivery 322 FCP Connection and Session Recovery 323 FCIP Operational Details 324 Architecture and Functional Models 324 VE_Port Functional Model 325 B_Access Functional Model 329 FCIP Addressing Scheme 331 FCIP Packet and Frame Formats 333 FCIP Session Establishment 342 FCIP Delivery Mechanisms 343 FCIP Data Transfer Optimizations 345 Summary

346

Review Questions 346

245

xv

Part III

Advanced Network Functionality

349

Chapter 9 Flow Control and Quality of Service

351

Conceptual Underpinnings of Flow Control and Quality of Service

351

Ethernet Flow Control and QoS 355 Ethernet Flow Control 355 Ethernet QoS 355 IP Flow Control and QoS 356 IP Flow Control 356 IP QoS 358 TCP Flow Control and QoS 359 TCP Flow Control 359 TCP QoS 360 iSCSI Flow Control and QoS 360 iSCSI Flow Control 360 iSCSI QoS 361 FC Flow Control and QoS 361 FC Flow Control 361 FC QoS 362 FCP Flow Control and QoS 362 FCP Flow Control 363 FCP QoS 363 FCIP Flow Control and QoS 363 FCIP Flow Control 363 FCIP QoS 364 Summary

364

Review Questions 364 Chapter 10 Routing and Switching Protocols

367

Conceptual Underpinnings of Routing and Switching Protocols 367 Ethernet Switching Protocols 370 IP Routing Protocols 371 FC Switching Protocols 374 Summary

374

Review Questions 374

xvi

Chapter 11 Load Balancing

377

Conceptual Underpinnings of Load Balancing

377

Load Balancing with Networking Technologies 377 Ethernet Load Balancing 378 IP Load Balancing 378 FC Load Balancing 380 Load Balancing with Session-Oriented Technologies 380 iSCSI Load Balancing 381 FCP Load Balancing 382 FCIP Load Balancing 382 End Node Load-Balancing Techniques 382 Summary

383

Review Questions 383 Chapter 12 Storage Network Security

385

Conceptual Underpinnings of Storage Network Security AAA Protocols 386 Management Protocols 388 Ethernet Security IP Security

389

389

TCP Security

390

iSCSI Security

391

Fibre Channel Security FCP Security

392

FCIP Security Summary

391

392

393

Review Questions 393 Chapter 13 Storage Management Protocols

395

Conceptual Underpinnings of Storage Management Protocols 395 TCP/IP Management FC Management

400

397

385

xvii

SCSI Management Summary

402

403

Review Questions 403 Chapter 14 Protocol Decoding and Traffic Analysis The Purpose of Protocol Decoding Methods of Traffic Capture

405

405

Types of Protocol Decoders 407 Protocol Decoder Screenshots 408 Purpose of Traffic Analysis 414 Types of Traffic Analyzers 414 Summary

415

Review Questions 415 Part IV

Appendixes

417

Appendix A

Standards and Specifications

419

Appendix B

Acronyms and Abbreviations

441

Appendix C

Answers to Review Questions

Glossary Index

499

479

463

405

xviii

Icons Used in This Book FC

IP

FC

IP

FC

iSCSI

HBA

FC Storage Array

FC Tape Library

TOE

FC-Attached Host

iSCSI Storage Array

iSCSI Tape Library

iSCSI-Attached Host

SPI iFCP

FICON

iFCP Gateway

Cisco MDS 9500, FICON

FC

iSCSI

Cisco MDS 9500, FC

FCIP

Cisco MDS 9500, iSCSI

Cisco MDS 9500, FCIP

FC-BB Bridge

FC-HBA

Backplane

Ethernet Hub, Internal

Token Ring MAU, Internal

SPI Tape Library

FC-HBA

FC-attached ATA Storage Array

FC-attached SCSI Storage Array

FC-HBA

NAS+ iSCSI+FC

NAS iSNS

Network Attached Storage

Network Attached Storage with iSCSI and FC

Mainframe

Disk Control Unit

Terminator

iSNS Server

Terminator

Magnetic Disk

Rx

Tx

Tx

Rx

NIC

Common Tx/Rx Drivers

NIC

Dedicated Tx/Rx Drivers

xix

Host Bus #0

SAM First Level LUN: Logical Entity for LUN Discovery and SES

SAM device containing one SAM port

FC-HBA

Device Name 20:00:00:00:c9:2e:70:2f Port Name 10:00:00:00:c9:2e:70:2f Port ID ad 01 1f

FC Adapter

Device Name 50:06:01:60:90:20:1e:7a Port Name 50:06:01:68:10:20:1e:7a Port ID ad b4 c8

LUN 0 Cache

Fan

Port ID 7 Bus #1 Vendor-Specific Backplane

Device Name 50:06:01:60:90:20:1e:7a Port Name 50:06:01:61:10:20:1e:7a Port ID ad 21 7b

LUN 0

LUN 0

Magnetic Disk Drive

Magnetic Disk Drive

Port ID 7 Bus #2

Port ID 2

Port ID 1 LUN 0 Tape Drive

SAM Port SAM Second Level LUN

Port ID 2

Port ID 1

LUN 0 FC Adapter

Fibre Channel Switch

Power Supply

CPU

Storage Array

Tape Drive

xx

Medium Transport Element

LUN Robotic Arm

LUN Tape Drive 1

Port

Import/Export Element

Data Transfer Elements

Backplane

LUN Tape Drive 2 LUN Tape Drive 3

Storage Elements

Port

xxi

Foreword It is a great pleasure to write the foreword for this book. Storage networking technologies have been used for computer data storage in almost every major corporation worldwide since the late 90s. Banks, hospitals, credit card companies, airlines, and universities are just a few examples of the organizations that use these technologies. Storage networking gave birth to the concept of the Storage Area Network (SAN), a concept based fundamentally on separating storage and processing resources, with the ability to provision and manage storage as a separate service to the processing resources. A SAN consists of not only the storage subsystems, but also of the interconnection infrastructure, the data protection subsystems, migration and virtualization technologies, and more. A SAN can have very different characteristics; for example, it can be all contained in a single rack or it might span an entire continent in order to create the most dependable disaster-tolerant configuration. In all cases, a SAN is meant to be more flexible than direct attached storage in accommodating the growth and the changes in an organization. Several products, based on diverse technologies, are available today as building blocks to design Storage Area Networks. These technologies are quite different and each has its own distinctive properties, advantages, and disadvantages. It is important for an information technology professional to be able to evaluate them and pick the most appropriate for the various customer or organization needs. This is not an easy task, because each technology has a different history, sees the problems from its own point of view, and uses a unique terminology. The associated complex standards documentation, designed for products developers, usually produces more confusion than illumination for the casual reader. In this book, James takes on the challenge of comparing today’s most deployed storage networking architectures. To perform his analysis, he uses a powerful tool, the OSI reference model. By comparing each architecture with the OSI model, James conducts the reader through the nuances of each technology, layer by layer. An appropriate set of parameters is introduced for each layer, and used across the presented technologies to analyze and compare them. In this way, readers familiar with networking have a way to understand the world of storage controllers, while people familiar with controllers will find a different perspective on what they already know. The first part of the book introduces the world of storage and storage networking. The basics of storage technology, including block storage protocols and file access protocols, are presented, followed by a historical evolution of how they evolved to their current status. The seven OSI layers are also introduced, giving to the reader the tools for the subsequent analysis. The second part of the book is a deep comparative analysis of today’s technologies for networked storage, including iSCSI and Fibre Channel. Each protocol suite is analyzed at the physical and data-link layers; at the network layer; at the transport layer; and finally at the session, presentation, and application layers. The third and final part of the book relates to advanced functionalities of these technologies, such as quality of service, load-balancing functions, security, and management. In particular, security is an element of continuously increasing importance for storage networking. Because more and more digital data are vital to businesses, keeping these data secure from unauthorized access is crucial. At the same time, this growing mass of data needs to be properly managed, but managing a heterogeneous set of devices is not an easy task. Several underlying protocols for storage management have been defined or are being defined. Storage networking is a critical concept for today’s businesses, and this book provides a unique and helpful way to better understand it. Storage networking is also continuously evolving, and as such this book may be seen as an introduction to the information technology infrastructures of the future. Claudio DeSanti Technical Leader Data Center BU, Cisco Systems Vice-Chairman of the ANSI INCITS T11 Technical Committee

xxii

Introduction The modern business environment is characterized by pervasive use of computer and communication technologies. Corporations increasingly depend on such technologies to remain competitive in the global economy. Customer relationship management, enterprise resource planning, and electronic mail are just a few of the many applications that generate new data every day. All that data must be stored, managed, and accessed effectively if a business is to survive. This is one of the primary business challenges in the information age, and storage networking is a crucial component of the solution.

Objectives This book has four objectives: document details, explain concepts, dispel misconceptions, and compare protocols. The details of the major protocol suites are documented for reference. To that end, this book aims primarily to disclose rather than assess. In that respect, we give extra effort to objectivity. Additionally, I attempt to explain how each of the major protocol suites operates, and to identify common understandings. Discussions of how the protocols work are included, but you are encouraged to reference the original standards and specifications for a complete understanding of each protocol. This recommendation also ensures you have the latest information. Since many of the standards and specifications referenced in this book are draft versions, they are subject to change. Thus, it is reasonable to expect some of the content in this book will become inaccurate as in-progress specifications are finalized. Finally, comparisons are drawn between the major protocol suites to help you understand the implications of your network design choices. To achieve these objectives, a large amount of reference data must be included. However, this book is written so that you can read it from cover to cover. We have tried to integrate reference data so it is easily and quickly accessible. For this reason, I use tables and bulleted lists extensively. In support of the stated objectives, we have made every effort to improve clarity. Colloquialisms are avoided throughout the book. Moreover, special attention is paid to the use of the words may, might, must, and should. The word may implies permissibility. The word might implies possibility. The word must imposes a requirement. The word should implies a desirable behavior but does not impose a requirement.

Intended Audiences This book has two primary audiences. The first audience includes storage administrators who need to learn more about networking. We have included much networking history, and have explained many networking concepts to help acclimate storage administrators to the world of networking. The second audience includes network administrators who need to learn more about storage. This book examines networking technologies in the context of SCSI so that network administrators can fully understand the network requirements imposed by open systems storage applications. Many storage concepts, terms, and technologies exist that network administrators need to know and understand. Although this book provides some storage knowledge for network administrators, other resources should be consulted for a full understanding of storage. One such resource is Storage Networking Fundamentals: An Introduction to Storage Devices, Subsystems, Applications, Management, and File Systems (by Marc Farley, ISBN: 1-58705-162-1, Cisco Press).

xxiii

Organization This book discusses and compares the networking protocols that underlie modern open systems, block-oriented storage networks. To facilitate a methodical analysis, the book is divided into three parts. The first part introduces readers to the field of storage networking and the Open Systems Interconnection (OSI) reference model. The second part examines in detail each of the major protocol suites layerby-layer beginning with the lowest layer of the OSI reference model. The third part introduces readers to several advanced networking topics. As the book progresses, each chapter builds upon the previous chapters. Thus, you will benefit most by reading this book from front to back. However, all chapters can be leveraged in any order for reference material. Some of the content in this book is based upon emerging standards and in-progress specifications. Thus, you are encouraged to consult the latest version of in-progress specifications for recent updates and changes.

PART

I

Storage Networking Landscape Chapter 1

Overview of Storage Networking

Chapter 2

OSI Reference Model Versus Other Network Models

Chapter 3

Overview of Network Operating Principles

Chapter 4

Overview of Modern SCSI Networking Protocols

Upon completing this chapter, you will be able to:

•

Recount some of the key historical events and current business drivers in the storage market

• • •

Recognize the key characteristics and major benefits of each type of storage network

•

Describe the basic differences between Synchronous Optical Network (SONET)/ Synchronous Digital Hierarchy (SDH), coarse wavelength division multiplexing (CWDM)/dense wavelength division multiplexing (DWDM), and resilient packet ring (RPR)/802.17

• • •

Recognize each storage virtualization model

Distinguish between file-level and block-level protocols Explain the principal differences between mainframe and open systems storage networking

Describe how the technologies discussed in this chapter relate to one another Locate additional information on the major topics of this chapter

CHAPTER

1

Overview of Storage Networking This chapter provides an overview of the storage networking landscape. Although this chapter discusses many of the technologies and devices that compose storage networks, many others are omitted for the sake of brevity. The discussion of each topic enlightens rather than educates, because you should have some experience with storage and networks. The goal of this chapter is to provide a context for the discussions that follow in subsequent chapters. Pervasive use of computer and communication technologies characterizes the modern business environment. Corporations increasingly depend on such technologies to remain competitive in the global economy. Customer relationship management (CRM), enterprise resource planning (ERP), and electronic mail are just a few of the many applications that generate new data every day. If a business is to survive, it must effectively store, manage, and access all data. This is one of the primary business challenges in the information age, and storage networking is a crucial component of the solution.

Brief History of Storage Early computer systems used various storage media such as punch cards, magnetic tapes, magnetic drums, and floppy magnetic disks for primary storage of data and programs. When IBM invented the hard disk drive in the early 1950s, a new era in storage began. The key benefits introduced include the following:

• • • • • • •

Random access capability Fast access time High throughput High capacity High reliability Media reusability Media portability

Some of these features were supported by other storage media, but the hard disk was the first medium to support all of these features. Consequently, the hard disk helped make many

6

Chapter 1: Overview of Storage Networking

advances possible in computing. As evolutionary advances in hard disk technology came to market, they enabled further advances in computing.

Intelligent Interface Models One of the most important developments in storage technology was the advent of the intelligent interface model. Early storage devices bundled the physical interface, electrical interface, and storage protocol together. Each new storage technology introduced a new physical interface, electrical interface, and storage protocol. The intelligent interface model abstracts device-specific details from the storage protocol and decouples the storage protocol from the physical and electrical interfaces. This allows multiple storage technologies such as magnetic disk, optical disk, and magnetic tape to use a common storage protocol. It also allows the storage protocol to operate over different physical and electrical interfaces.

Serial Transmission Techniques Another key development in storage was the adoption of serial transmission techniques. Storage interfaces traditionally employ parallel electrical transmission techniques. Parallel electrical transmission has two significant limitations: bandwidth and distance. A common characteristic of electrical signals transmitted in parallel is electromagnetic interference among the wires in a sheath (called crosstalk). Crosstalk increases as the transmission rate increases. The ever-increasing capacities of modern disk drives necessitate ever-increasing transmission rates in their interfaces. Crosstalk makes it difficult to meet this challenge with parallel electrical transmission techniques. One countermeasure for crosstalk is to shorten the link distance, but distance constraints limit the usefulness of the interface. Cost control is also challenging. The economics of very long-distance cable infrastructure prohibit adoption of parallel transmission techniques. The serial nature of the global voice communications infrastructure provides evidence of this. Even for relatively short distances, such as data center requirements, parallel cabling costs can be adverse. Moreover, parallel cabling presents a logistical challenge. It can be difficult to fit through tight spaces or to bend around corners. As a result, the storage industry has embraced several serial transmission techniques in recent years. Serial electrical transmission techniques reduce the number of required conductors to one or two per direction. This does not eliminate crosstalk but significantly reduces it. Some of the adopted serial transmission techniques employ optical signaling, which completely eliminates crosstalk. Serial cabling is also less expensive and much easier to install and maintain. These benefits notwithstanding, some standards bodies have chosen to continue work on parallel electrical transmission techniques. The debate about the future of parallel technologies is still active, though serial technologies now clearly have the momentum.

Drivers for Change

7

Modern Storage Networks The adoption of the intelligent interface model and serial transmission techniques made it possible to leverage standard networking technologies for storage connectivity. The loop (also known as ring) topology was common in the early years of open systems storage networking but has since been broadly supplanted by switch-based solutions. Solutions based on Fibre Channel (FC) and Ethernet are most common today.

Drivers for Change Having the ability to network storage resources does not necessarily make it a good idea. Deploying technology for the sake of technology is not a sound business practice. So why have so many large- and medium-sized enterprises embraced storage networking? There are many reasons, but the primary drivers are the need for efficient management of storage resources and the need for competitive advantages that innovative storage solutions enable. The volume of data generated each day by the typical modern corporation is greater than ever before. The growth of the Internet and e-commerce have increased the value of data, prompting businesses to store raw data longer, manipulate raw data in new ways, and store more versions of manipulated data. Recently enacted U.S. legislation mandated changes in the retention policies and procedures for financial records of public corporations listed on U.S. exchanges. U.S. companies that have operations in other countries also might be subject to various international regulations in those countries regarding data security and data privacy. Additional U.S. legislation mandated improved patient record-keeping for medical institutions. Recent world events have increased awareness of the need for all corporations, government agencies, and research and educational institutions to replicate data to remote sites for disaster protection. Many of the older enterprise-level applications that were written for optimal functionality at the expense of data life-cycle management cannot be rewritten cost effectively. These and other factors have resulted in storage capacities and growth rates that challenge the efficacy of direct attached storage (DAS) management. Many corporations report that the cost of storage management is now greater than the cost of storage acquisition. Networking storage resources enables a new management paradigm. New protocols, products, and frameworks for the management of storage promise to reduce the cost of managing large-scale environments. The most prominent industry activity in this area is the Storage Management Initiative Specification (SMI-S) published by the Storage Networking Industry Association (SNIA). Vendor adoption of SMI-S is broad, and consumer interest is high. Competition is fierce in global markets. The era of U.S. corporate downsizing epitomized the effect of that competition. Any competitive edge is welcome in such a business climate.

8

Chapter 1: Overview of Storage Networking

Storage networks provide the potential to realize numerous competitive advantages such as the following:

•

Increased throughput—This allows input/output (I/O) sensitive applications to perform better.

•

Improved flexibility—Adds, moves, and changes are a fact of life in the operation of any computing or communication infrastructure. Storage networks allow storage administrators to make adds, moves, and changes more easily and with less downtime than otherwise possible in the DAS model.

•

Higher scalability—Storage networks allow greater numbers of servers and storage subsystems to be interconnected. This allows larger server clusters and web server farms by enabling horizontal expansion, and potentially reduces storage costs by enabling more servers to access shared storage hardware.

•

Data mobility—Storage networks increase the mobility of stored data for various purposes, such as migration to new hardware and replication to a secondary data center.

•

LAN-free backups—By deploying a storage area network (SAN), businesses can move network-based backup operations from the local-area network (LAN) to the SAN. This allows nightly operations such as batch processing, log transfers, and device discovery to run normally without competing against backup operations for LAN bandwidth. LAN-free backup was one of the original drivers for enterprises to adopt SAN technology.

•

Server-free backups—This is made possible by building intelligence into SAN switches and other SAN infrastructure devices to provide Small Computer System Interface (SCSI) EXTENDED COPY (XCOPY) functionality.

•

Advanced volume management—As virtualization technology migrates into SAN switches, it will enable robust heterogeneous volume management. Host and storage subsystem-based solutions exist today, but each has its limitations. Network-based virtualization promises to eliminate some of those limitations.

Another driver for change is the evolution of relevant standards. An understanding of the standards bodies that represent all aspects of storage functionality enables better understanding of historical storage designs and limitations, and of future directions. In addition to the aforementioned SNIA, other standards bodies are important to storage networking. The American National Standards Institute (ANSI) oversees the activities of the InterNational Committee for Information Technology Standards (INCITS). Two subcommittees of the INCITS are responsible for standards that are central to understanding storage networks. The INCITS T10 subcommittee owns SCSI standards, and the INCITS T11 subcommittee owns Fibre Channel standards. The Internet Engineering Task Force (IETF) has recently emerged as another driving force in storage. The IETF created the IP storage working Group (IPS-WG), which owns all IP-based storage standards. Standards bo